Task 15937770

Name	hadcm3n_38q6_2020_40_008410805_1
Workunit	8561661
Created	24 Aug 2013, 0:30:49 UTC
Sent	24 Aug 2013, 0:38:42 UTC
Report deadline	23 Nov 2013, 8:05:53 UTC
Received	18 Sep 2013, 6:59:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	853527
Run time	21 days 20 hours 24 min 38 sec
CPU time	19 days 22 hours 19 min 13 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	2.75 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:22:30 (46197): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(50104,0xa0a31540) malloc: * error for object 0x300e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:04:40 (52256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:19:39 (32612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:30:32 (46260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:26:49 (48983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:26:50 (48983): No heartbeat from core client for 30 sec - exiting 15:26:51 (48983): No heartbeat from core client for 30 sec - exiting 15:26:52 (48983): No heartbeat from core client for 30 sec - exiting 15:26:53 (48983): No heartbeat from core client for 30 sec - exiting 15:26:54 (48983): No heartbeat from core client for 30 sec - exiting 15:28:00 (49061): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:01 (49061): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:03:28 (49790): No heartbeat from core client for 30 sec - exiting 16:03:29 (49790): No heartbeat from core client for 30 sec - exiting 16:03:30 (49790): No heartbeat from core client for 30 sec - exiting 16:03:31 (49790): No heartbeat from core client for 30 sec - exiting 16:03:32 (49790): No heartbeat from core client for 30 sec - exiting 16:03:33 (49790): No heartbeat from core client for 30 sec - exiting 16:03:34 (49790): No heartbeat from core client for 30 sec - exiting 16:03:35 (49790): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:44:02 (50488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:38:15 (25096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:43:38 (56983): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1803404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(365,0xa0a31540) malloc: * error for object 0x1804200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:26:23 (58293): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:24:20 (59262): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:05:01 (59939): No heartbeat from core client for 30 sec - exiting 16:05:02 (59939): No heartbeat from core client for 30 sec - exiting 16:05:03 (59939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:18:46 (31320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:36:13 (31598): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:41:47 (31618): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:41:48 (31618): No heartbeat from core client for 30 sec - exiting 10:52:45 (31722): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:52:46 (31722): No heartbeat from core client for 30 sec - exiting 10:52:47 (31722): No heartbeat from core client for 30 sec - exiting 10:52:48 (31722): No heartbeat from core client for 30 sec - exiting 10:52:49 (31722): No heartbeat from core client for 30 sec - exiting 10:52:50 (31722): No heartbeat from core client for 30 sec - exiting 10:52:51 (31722): No heartbeat from core client for 30 sec - exiting 10:52:52 (31722): No heartbeat from core client for 30 sec - exiting 10:52:53 (31722): No heartbeat from core client for 30 sec - exiting 10:52:54 (31722): No heartbeat from core client for 30 sec - exiting 10:52:55 (31722): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:43:12 (32622): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:47:13 (32647): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:47:15 (32647): No heartbeat from core client for 30 sec - exiting 11:48:30 (32718): No heartbeat from core client for 30 sec - exiting 11:48:31 (32718): No heartbeat from core client for 30 sec - exiting 11:48:32 (32718): No heartbeat from core client for 30 sec - exiting 11:48:33 (32718): No heartbeat from core client for 30 sec - exiting 11:48:34 (32718): No heartbeat from core client for 30 sec - exiting 11:48:35 (32718): No heartbeat from core client for 30 sec - exiting 11:48:36 (32718): No heartbeat from core client for 30 sec - exiting 11:48:37 (32718): No heartbeat from core client for 30 sec - exiting 11:48:38 (32718): No heartbeat from core client for 30 sec - exiting 11:48:39 (32718): No heartbeat from core client for 30 sec - exiting 11:48:40 (32718): No heartbeat from core client for 30 sec - exiting 11:48:41 (32718): No heartbeat from core client for 30 sec - exiting 11:48:42 (32718): No heartbeat from core client for 30 sec - exiting 11:48:43 (32718): No heartbeat from core client for 30 sec - exiting 11:48:44 (32718): No heartbeat from core client for 30 sec - exiting 11:48:45 (32718): No heartbeat from core client for 30 sec - exiting 11:48:46 (32718): No heartbeat from core client for 30 sec - exiting 11:48:47 (32718): No heartbeat from core client for 30 sec - exiting 11:48:48 (32718): No heartbeat from core client for 30 sec - exiting 11:48:49 (32718): No heartbeat from core client for 30 sec - exiting 11:48:50 (32718): No heartbeat from core client for 30 sec - exiting 11:48:51 (32718): No heartbeat from core client for 30 sec - exiting 11:48:52 (32718): No heartbeat from core client for 30 sec - exiting 11:48:53 (32718): No heartbeat from core client for 30 sec - exiting 11:48:54 (32718): No heartbeat from core client for 30 sec - exiting 11:48:55 (32718): No heartbeat from core client for 30 sec - exiting 11:48:56 (32718): No heartbeat from core client for 30 sec - exiting 11:48:57 (32718): No heartbeat from core client for 30 sec - exiting 11:48:58 (32718): No heartbeat from core client for 30 sec - exiting 11:48:59 (32718): No heartbeat from core client for 30 sec - exiting 11:49:00 (32718): No heartbeat from core client for 30 sec - exiting 11:49:01 (32718): No heartbeat from core client for 30 sec - exiting 11:49:02 (32718): No heartbeat from core client for 30 sec - exiting 11:49:03 (32718): No heartbeat from core client for 30 sec - exiting 11:49:04 (32718): No heartbeat from core client for 30 sec - exiting 11:49:05 (32718): No heartbeat from core client for 30 sec - exiting 11:49:06 (32718): No heartbeat from core client for 30 sec - exiting 11:49:07 (32718): No heartbeat from core client for 30 sec - exiting 11:49:08 (32718): No heartbeat from core client for 30 sec - exiting 11:49:09 (32718): No heartbeat from core client for 30 sec - exiting 11:49:10 (32718): No heartbeat from core client for 30 sec - exiting 11:49:11 (32718): No heartbeat from core client for 30 sec - exiting 11:49:12 (32718): No heartbeat from core client for 30 sec - exiting 11:49:13 (32718): No heartbeat from core client for 30 sec - exiting 11:49:14 (32718): No heartbeat from core client for 30 sec - exiting 11:49:15 (32718): No heartbeat from core client for 30 sec - exiting 11:49:16 (32718): No heartbeat from core client for 30 sec - exiting 11:49:17 (32718): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:01 (32748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:53:01 (32760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:02 (32760): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:58:02 (32828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:13:10 (34145): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 14:03:09 (35115): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Sep 2013 03:51:30	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	984,960	1,715,896	1.7421
16 Sep 2013 08:51:52	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	959,040	1,673,446	1.7449
15 Sep 2013 20:27:26	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	933,120	1,630,494	1.7474
15 Sep 2013 07:28:30	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	907,200	1,586,547	1.7488
14 Sep 2013 18:30:23	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	881,280	1,542,626	1.7504
14 Sep 2013 05:21:41	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	855,360	1,498,283	1.7516
13 Sep 2013 16:02:42	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	829,440	1,453,312	1.7522
13 Sep 2013 02:39:39	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	803,520	1,408,111	1.7524
12 Sep 2013 09:14:35	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	777,600	1,363,223	1.7531
11 Sep 2013 15:12:46	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	751,680	1,318,350	1.7539
10 Sep 2013 17:11:45	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	725,760	1,273,108	1.7542
10 Sep 2013 02:11:34	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	699,840	1,228,386	1.7552
09 Sep 2013 07:57:52	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	673,920	1,183,868	1.7567
08 Sep 2013 09:32:18	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	648,000	1,138,903	1.7576
07 Sep 2013 16:56:17	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	622,080	1,092,764	1.7566
06 Sep 2013 17:30:12	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	596,160	1,047,462	1.7570
06 Sep 2013 03:45:13	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	570,240	1,002,921	1.7588
05 Sep 2013 09:43:20	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	544,320	957,806	1.7596
04 Sep 2013 19:19:53	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	518,400	912,808	1.7608
04 Sep 2013 04:04:37	853527	15937770	hadcm3n_38q6_2020_40_008410805_1	492,480	867,651	1.7618