Task 14843956

Name	hadcm3n_o0ea_2100_40_008000727_2
Workunit	8155841
Created	26 Jun 2012, 18:28:59 UTC
Sent	26 Jun 2012, 18:29:04 UTC
Report deadline	26 Sep 2012, 1:56:15 UTC
Received	23 Jul 2012, 16:20:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1150085
Run time	8 days 1 hours 19 min 19 sec
CPU time	7 days 15 hours 40 min 14 sec
Validate state	Invalid
Credit	4,976.64
Device peak FLOPS	3.15 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.26</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:23:23 (34866): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x183c004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x183c000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x85fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x183c004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x183c000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x183c004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x183c000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x2820604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x283bc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x283bc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x2800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x2800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7111,0xa0a6e540) malloc: * error for object 0x815200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 21 - Return code = 1 Model crashed: READDUMP: BAD BUFFIN OF DATA tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2012 10:23:08	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	414,720	642,335	1.5488
20 Jul 2012 01:40:41	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	388,800	602,739	1.5503
19 Jul 2012 12:57:04	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	362,880	563,498	1.5528
19 Jul 2012 01:04:38	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	336,960	523,716	1.5542
18 Jul 2012 13:04:57	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	311,040	482,912	1.5526
18 Jul 2012 01:07:32	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	285,120	442,094	1.5506
17 Jul 2012 13:26:16	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	259,200	401,246	1.5480
17 Jul 2012 01:43:53	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	233,280	360,411	1.5450
16 Jul 2012 14:02:28	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	207,360	319,637	1.5415
13 Jul 2012 11:40:05	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	181,440	278,960	1.5375
02 Jul 2012 16:59:07	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	155,520	238,931	1.5363
02 Jul 2012 11:12:50	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	129,600	198,278	1.5299
02 Jul 2012 11:12:50	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	103,680	158,394	1.5277
02 Jul 2012 11:12:49	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	77,760	118,645	1.5258
30 Jun 2012 09:32:30	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	51,840	78,955	1.5231
29 Jun 2012 21:47:36	1150085	14843956	hadcm3n_o0ea_2100_40_008000727_2	25,920	39,476	1.5230