Task 15773894

Name	hadcm3n_39py_1980_40_008364415_0
Workunit	8515274
Created	10 May 2013, 20:19:36 UTC
Sent	10 May 2013, 20:24:15 UTC
Report deadline	10 Aug 2013, 3:51:26 UTC
Received	18 Jul 2013, 13:31:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1203068
Run time	22 days 20 hours 8 min 19 sec
CPU time	14 days 18 hours 27 min 8 sec
Validate state	Invalid
Credit	8,398.08
Device peak FLOPS	2.63 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:40:55 (26677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:40:57 (26677): No heartbeat from core client for 30 sec - exiting 00:40:58 (26677): No heartbeat from core client for 30 sec - exiting 00:40:59 (26677): No heartbeat from core client for 30 sec - exiting 00:41:00 (26677): No heartbeat from core client for 30 sec - exiting 00:41:01 (26677): No heartbeat from core client for 30 sec - exiting 00:41:02 (26677): No heartbeat from core client for 30 sec - exiting 00:41:03 (26677): No heartbeat from core client for 30 sec - exiting 00:41:04 (26677): No heartbeat from core client for 30 sec - exiting 00:41:05 (26677): No heartbeat from core client for 30 sec - exiting 00:41:06 (26677): No heartbeat from core client for 30 sec - exiting 00:41:08 (26677): No heartbeat from core client for 30 sec - exiting 00:41:09 (26677): No heartbeat from core client for 30 sec - exiting 00:41:10 (26677): No heartbeat from core client for 30 sec - exiting 00:41:11 (26677): No heartbeat from core client for 30 sec - exiting 00:41:12 (26677): No heartbeat from core client for 30 sec - exiting 00:41:13 (26677): No heartbeat from core client for 30 sec - exiting 00:41:14 (26677): No heartbeat from core client for 30 sec - exiting 00:41:15 (26677): No heartbeat from core client for 30 sec - exiting 00:41:16 (26677): No heartbeat from core client for 30 sec - exiting 00:41:17 (26677): No heartbeat from core client for 30 sec - exiting 00:41:18 (26677): No heartbeat from core client for 30 sec - exiting 00:41:19 (26677): No heartbeat from core client for 30 sec - exiting 00:41:20 (26677): No heartbeat from core client for 30 sec - exiting 00:41:21 (26677): No heartbeat from core client for 30 sec - exiting 00:41:22 (26677): No heartbeat from core client for 30 sec - exiting 00:41:23 (26677): No heartbeat from core client for 30 sec - exiting 00:41:24 (26677): No heartbeat from core client for 30 sec - exiting 00:41:25 (26677): No heartbeat from core client for 30 sec - exiting 00:41:26 (26677): No heartbeat from core client for 30 sec - exiting 00:41:27 (26677): No heartbeat from core client for 30 sec - exiting 00:41:28 (26677): No heartbeat from core client for 30 sec - exiting 00:41:29 (26677): No heartbeat from core client for 30 sec - exiting 00:41:30 (26677): No heartbeat from core client for 30 sec - exiting 00:41:31 (26677): No heartbeat from core client for 30 sec - exiting 00:41:32 (26677): No heartbeat from core client for 30 sec - exiting 00:41:33 (26677): No heartbeat from core client for 30 sec - exiting 00:41:34 (26677): No heartbeat from core client for 30 sec - exiting 00:41:35 (26677): No heartbeat from core client for 30 sec - exiting 00:41:36 (26677): No heartbeat from core client for 30 sec - exiting 00:41:37 (26677): No heartbeat from core client for 30 sec - exiting 00:41:38 (26677): No heartbeat from core client for 30 sec - exiting 00:41:39 (26677): No heartbeat from core client for 30 sec - exiting 00:41:40 (26677): No heartbeat from core client for 30 sec - exiting 00:41:41 (26677): No heartbeat from core client for 30 sec - exiting 00:41:42 (26677): No heartbeat from core client for 30 sec - exiting 00:41:43 (26677): No heartbeat from core client for 30 sec - exiting 00:41:44 (26677): No heartbeat from core client for 30 sec - exiting 00:41:45 (26677): No heartbeat from core client for 30 sec - exiting 00:41:46 (26677): No heartbeat from core client for 30 sec - exiting 00:41:47 (26677): No heartbeat from core client for 30 sec - exiting 00:41:48 (26677): No heartbeat from core client for 30 sec - exiting 00:41:49 (26677): No heartbeat from core client for 30 sec - exiting 00:41:50 (26677): No heartbeat from core client for 30 sec - exiting 00:41:51 (26677): No heartbeat from core client for 30 sec - exiting 00:41:52 (26677): No heartbeat from core client for 30 sec - exiting 00:41:53 (26677): No heartbeat from core client for 30 sec - exiting 00:41:54 (26677): No heartbeat from core client for 30 sec - exiting 00:41:55 (26677): No heartbeat from core client for 30 sec - exiting 00:41:56 (26677): No heartbeat from core client for 30 sec - exiting 00:41:57 (26677): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:07:24 (87863): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x4b7b404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x12dfa94: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x12dfa90: pointer being freed was not allocated * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x1389804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x136f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x136f000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b65200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b4c204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(40838,0xac11e2c0) malloc: * error for object 0x2b4c200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug 05:27:39 (40838): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:27:40 (40838): No heartbeat from core client for 30 sec - exiting 05:27:41 (40838): No heartbeat from core client for 30 sec - exiting 05:27:42 (40838): No heartbeat from core client for 30 sec - exiting 05:27:43 (40838): No heartbeat from core client for 30 sec - exiting 05:27:44 (40838): No heartbeat from core client for 30 sec - exiting 05:27:45 (40838): No heartbeat from core client for 30 sec - exiting 05:27:46 (40838): No heartbeat from core client for 30 sec - exiting 05:27:47 (40838): No heartbeat from core client for 30 sec - exiting 05:27:48 (40838): No heartbeat from core client for 30 sec - exiting 05:27:49 (40838): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:27:14 (73010): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Jul 2013 16:53:21	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	699,840	1,275,669	1.8228
23 Jul 2013 16:53:21	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	673,920	1,229,927	1.8250
11 Jul 2013 21:30:27	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	648,000	1,177,003	1.8164
08 Jul 2013 21:09:48	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	622,080	1,139,754	1.8322
06 Jul 2013 04:32:29	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	596,160	1,098,975	1.8434
02 Jul 2013 12:03:53	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	570,240	1,053,491	1.8475
02 Jul 2013 09:56:50	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	544,320	1,011,541	1.8584
26 Jun 2013 03:27:15	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	518,400	975,651	1.8820
23 Jun 2013 02:18:58	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	492,480	941,090	1.9109
20 Jun 2013 03:10:36	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	466,560	888,731	1.9049
17 Jun 2013 01:22:53	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	440,640	849,433	1.9277
14 Jun 2013 02:49:51	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	414,720	801,389	1.9324
11 Jun 2013 04:01:15	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	388,800	756,492	1.9457
08 Jun 2013 22:05:21	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	362,880	711,451	1.9606
06 Jun 2013 02:23:24	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	336,960	663,913	1.9703
03 Jun 2013 21:43:07	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	311,040	612,596	1.9695
01 Jun 2013 01:27:47	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	285,120	565,047	1.9818
29 May 2013 22:21:42	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	259,200	508,108	1.9603
27 May 2013 03:37:06	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	233,280	450,879	1.9328
25 May 2013 22:01:49	1203068	15773894	hadcm3n_39py_1980_40_008364415_0	207,360	405,597	1.9560