Task 15942066

Name	hadcm3n_4g79_1940_40_008307120_3
Workunit	8458255
Created	28 Aug 2013, 9:59:06 UTC
Sent	28 Aug 2013, 9:59:10 UTC
Report deadline	27 Nov 2013, 17:26:21 UTC
Received	16 Sep 2013, 9:57:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1289911
Run time	10 days 13 hours 58 min 16 sec
CPU time	9 days 20 hours 4 min 39 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0x1227c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0x1227c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0xa32e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0xa32e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0xa32e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0xa32e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0xa32e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0xa32e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(268,0xacc8ca28) malloc: * error for object 0xa32e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x116a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x116a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x2962804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x2962800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8197,0xacc8ca28) malloc: * error for object 0x295e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid 00:04:35 (28106): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... dyld: DYLD_ environment variables being ignored because main executable (/Library/Application Support/BOINC Data/slots/5/../../switcher/switcher) is setuid or setgid Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Sep 2013 13:40:30	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	907,200	840,962	0.9270
15 Sep 2013 05:58:11	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	881,280	816,849	0.9269
14 Sep 2013 11:28:45	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	855,360	792,773	0.9268
14 Sep 2013 03:50:53	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	829,440	768,830	0.9269
13 Sep 2013 09:36:15	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	803,520	745,007	0.9272
12 Sep 2013 22:53:45	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	777,600	720,977	0.9272
12 Sep 2013 05:58:16	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	751,680	697,011	0.9273
11 Sep 2013 12:06:56	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	725,760	672,707	0.9269
11 Sep 2013 04:25:01	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	699,840	648,625	0.9268
10 Sep 2013 12:45:35	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	673,920	624,538	0.9267
10 Sep 2013 05:03:03	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	648,000	600,338	0.9264
09 Sep 2013 13:15:52	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	622,080	575,998	0.9259
09 Sep 2013 05:37:21	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	596,160	551,790	0.9256
08 Sep 2013 11:53:03	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	570,240	527,469	0.9250
08 Sep 2013 04:15:24	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	544,320	503,401	0.9248
07 Sep 2013 08:29:01	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	518,400	479,331	0.9246
06 Sep 2013 10:47:44	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	492,480	455,313	0.9245
06 Sep 2013 03:10:01	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	466,560	431,300	0.9244
05 Sep 2013 11:28:48	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	440,640	407,093	0.9239
05 Sep 2013 03:41:57	1289911	15942066	hadcm3n_4g79_1940_40_008307120_3	414,720	382,783	0.9230