Task 15276565

Name	hadcm3n_zlk5_1880_40_008199425_0
Workunit	8354549
Created	13 Sep 2012, 0:49:15 UTC
Sent	14 Sep 2012, 18:38:47 UTC
Report deadline	15 Dec 2012, 2:05:58 UTC
Received	25 Nov 2012, 19:10:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	305509
Run time	13 days 15 hours 19 min 37 sec
CPU time	11 days 14 hours 13 min 43 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.57 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:09:31 (447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:09:32 (447): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 15:24:26 (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:24:27 (2184): No heartbeat from core client for 30 sec - exiting 15:24:28 (2184): No heartbeat from core client for 30 sec - exiting 15:24:29 (2184): No heartbeat from core client for 30 sec - exiting 15:24:30 (2184): No heartbeat from core client for 30 sec - exiting 15:24:31 (2184): No heartbeat from core client for 30 sec - exiting 15:24:32 (2184): No heartbeat from core client for 30 sec - exiting 15:24:33 (2184): No heartbeat from core client for 30 sec - exiting 15:24:34 (2184): No heartbeat from core client for 30 sec - exiting 15:24:35 (2184): No heartbeat from core client for 30 sec - exiting 15:24:36 (2184): No heartbeat from core client for 30 sec - exiting 15:24:37 (2184): No heartbeat from core client for 30 sec - exiting 15:24:38 (2184): No heartbeat from core client for 30 sec - exiting 15:24:39 (2184): No heartbeat from core client for 30 sec - exiting 15:24:40 (2184): No heartbeat from core client for 30 sec - exiting 15:24:41 (2184): No heartbeat from core client for 30 sec - exiting 15:24:43 (2184): No heartbeat from core client for 30 sec - exiting 15:24:44 (2184): No heartbeat from core client for 30 sec - exiting 15:24:45 (2184): No heartbeat from core client for 30 sec - exiting 15:24:46 (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:09:41 (401): No heartbeat from core client for 30 sec - exiting 13:09:42 (401): No heartbeat from core client for 30 sec - exiting 13:09:43 (401): No heartbeat from core client for 30 sec - exiting 13:09:44 (401): No heartbeat from core client for 30 sec - exiting 13:09:45 (401): No heartbeat from core client for 30 sec - exiting 13:09:46 (401): No heartbeat from core client for 30 sec - exiting 13:09:48 (401): No heartbeat from core client for 30 sec - exiting 13:09:49 (401): No heartbeat from core client for 30 sec - exiting 13:09:50 (401): No heartbeat from core client for 30 sec - exiting 13:09:51 (401): No heartbeat from core client for 30 sec - exiting 13:09:52 (401): No heartbeat from core client for 30 sec - exiting 13:09:53 (401): No heartbeat from core client for 30 sec - exiting 13:09:54 (401): No heartbeat from core client for 30 sec - exiting 13:09:55 (401): No heartbeat from core client for 30 sec - exiting 13:09:56 (401): No heartbeat from core client for 30 sec - exiting 13:09:57 (401): No heartbeat from core client for 30 sec - exiting 13:09:58 (401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa39404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(2501,0xac45a2c0) malloc: * error for object 0xa54a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(3773,0xac45a2c0) malloc: * error for object 0x143d004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3773,0xac45a2c0) malloc: * error for object 0xc13204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3773,0xac45a2c0) malloc: * error for object 0xc13200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3773,0xac45a2c0) malloc: * error for object 0xc13204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(400,0xac45a2c0) malloc: * error for object 0x9ce404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(400,0xac45a2c0) malloc: * error for object 0x9b0004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(400,0xac45a2c0) malloc: * error for object 0x9b0000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(400,0xac45a2c0) malloc: * error for object 0x9b0004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(400,0xac45a2c0) malloc: * error for object 0x9b0000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(400,0xac45a2c0) malloc: * error for object 0x9b0000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0xb7cc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0xb7cc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b41404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b41400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(988,0xac45a2c0) malloc: * error for object 0x1b40a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:35:15 (1648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:19:52 (540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb22c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0x1362804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0x1362800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0x1362804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0x1362800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb22c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0xb3e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1570,0xac45a2c0) malloc: * error for object 0x1347204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x117d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x96ce04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x96ce04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x116b400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x1139a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(938,0xac45a2c0) malloc: * error for object 0x1139a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /Library/Application Support/BOINC Data/projects/climateprediction.net/hadcm3n_zlk5_1880_40_008199425/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Nov 2012 17:50:04	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	777,600	984,201	1.2657
21 Nov 2012 16:19:32	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	751,680	953,897	1.2690
19 Nov 2012 18:05:37	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	725,760	923,660	1.2727
18 Nov 2012 16:10:48	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	699,840	893,472	1.2767
17 Nov 2012 14:36:08	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	673,920	861,719	1.2787
14 Nov 2012 16:19:50	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	648,000	826,774	1.2759
11 Nov 2012 22:00:57	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	622,080	793,704	1.2759
10 Nov 2012 23:46:29	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	596,160	763,900	1.2814
10 Nov 2012 10:21:09	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	570,240	733,966	1.2871
07 Nov 2012 19:19:44	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	544,320	701,494	1.2888
05 Nov 2012 21:38:50	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	518,400	667,202	1.2870
04 Nov 2012 18:53:47	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	492,480	637,119	1.2937
02 Nov 2012 18:22:00	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	466,560	602,405	1.2912
01 Nov 2012 16:22:44	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	440,640	568,038	1.2891
28 Oct 2012 20:00:12	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	414,720	535,703	1.2917
26 Oct 2012 19:34:47	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	388,800	502,749	1.2931
24 Oct 2012 13:02:19	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	362,880	470,707	1.2971
22 Oct 2012 17:31:22	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	336,960	438,614	1.3017
18 Oct 2012 15:27:59	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	311,040	405,462	1.3036
17 Oct 2012 16:16:47	305509	15276565	hadcm3n_zlk5_1880_40_008199425_0	285,120	371,801	1.3040