Task 15640817

Name	hadcm3n_zkxn_1920_40_008254378_3
Workunit	8409502
Created	26 Feb 2013, 2:57:50 UTC
Sent	26 Feb 2013, 3:00:46 UTC
Report deadline	28 May 2013, 10:27:57 UTC
Received	18 Mar 2013, 10:52:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1231328
Run time	11 days 12 hours 16 min 24 sec
CPU time	6 days 21 hours 15 min 41 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	3.04 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 19:54:08 (7755): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:58 (8103): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:01 (8103): No heartbeat from core client for 30 sec - exiting 22:35:02 (8103): No heartbeat from core client for 30 sec - exiting 22:35:03 (8103): No heartbeat from core client for 30 sec - exiting 22:35:04 (8103): No heartbeat from core client for 30 sec - exiting 22:35:05 (8103): No heartbeat from core client for 30 sec - exiting 22:35:06 (8103): No heartbeat from core client for 30 sec - exiting 22:35:07 (8103): No heartbeat from core client for 30 sec - exiting 01:50:34 (9176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:06:47 (19399): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:06:54 (19399): No heartbeat from core client for 30 sec - exiting 18:06:55 (19399): No heartbeat from core client for 30 sec - exiting 18:06:56 (19399): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:56:27 (19922): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:07:00 (19946): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:07:02 (19946): No heartbeat from core client for 30 sec - exiting 19:07:03 (19946): No heartbeat from core client for 30 sec - exiting 19:07:04 (19946): No heartbeat from core client for 30 sec - exiting 19:07:05 (19946): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:39:52 (30354): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:12 (30622): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:07:47 (40419): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:53 (40419): No heartbeat from core client for 30 sec - exiting 18:07:54 (40419): No heartbeat from core client for 30 sec - exiting 18:07:55 (40419): No heartbeat from core client for 30 sec - exiting 18:07:56 (40419): No heartbeat from core client for 30 sec - exiting 18:07:57 (40419): No heartbeat from core client for 30 sec - exiting 18:07:58 (40419): No heartbeat from core client for 30 sec - exiting 18:07:59 (40419): No heartbeat from core client for 30 sec - exiting 18:08:00 (40419): No heartbeat from core client for 30 sec - exiting 18:08:01 (40419): No heartbeat from core client for 30 sec - exiting 18:08:02 (40419): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 20:58:09 (40699): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:27:34 (50321): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:27:36 (50321): No heartbeat from core client for 30 sec - exiting 06:29:44 (55358): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:29:45 (55358): No heartbeat from core client for 30 sec - exiting 06:29:46 (55358): No heartbeat from core client for 30 sec - exiting 06:29:47 (55358): No heartbeat from core client for 30 sec - exiting 06:29:48 (55358): No heartbeat from core client for 30 sec - exiting 06:29:49 (55358): No heartbeat from core client for 30 sec - exiting 06:29:50 (55358): No heartbeat from core client for 30 sec - exiting 06:29:51 (55358): No heartbeat from core client for 30 sec - exiting 06:29:52 (55358): No heartbeat from core client for 30 sec - exiting 06:29:53 (55358): No heartbeat from core client for 30 sec - exiting 06:29:54 (55358): No heartbeat from core client for 30 sec - exiting 06:29:55 (55358): No heartbeat from core client for 30 sec - exiting 06:29:56 (55358): No heartbeat from core client for 30 sec - exiting 06:29:57 (55358): No heartbeat from core client for 30 sec - exiting 06:29:58 (55358): No heartbeat from core client for 30 sec - exiting 06:29:59 (55358): No heartbeat from core client for 30 sec - exiting 06:30:00 (55358): No heartbeat from core client for 30 sec - exiting 06:30:01 (55358): No heartbeat from core client for 30 sec - exiting 06:30:02 (55358): No heartbeat from core client for 30 sec - exiting 06:30:03 (55358): No heartbeat from core client for 30 sec - exiting 06:30:04 (55358): No heartbeat from core client for 30 sec - exiting 06:30:05 (55358): No heartbeat from core client for 30 sec - exiting 06:30:06 (55358): No heartbeat from core client for 30 sec - exiting 06:30:07 (55358): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 19:01:16 (59948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:33 (60357): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:22:39 (69464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:47 (71569): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:05:20 (79029): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:19 (79857): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:20 (79857): No heartbeat from core client for 30 sec - exiting 05:06:21 (79857): No heartbeat from core client for 30 sec - exiting 05:06:22 (79857): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:06:46 (32878): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:46:56 (35285): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:46:58 (35285): No heartbeat from core client for 30 sec - exiting 01:46:59 (35285): No heartbeat from core client for 30 sec - exiting 01:47:00 (35285): No heartbeat from core client for 30 sec - exiting 01:47:01 (35285): No heartbeat from core client for 30 sec - exiting 01:47:02 (35285): No heartbeat from core client for 30 sec - exiting 01:47:03 (35285): No heartbeat from core client for 30 sec - exiting 01:47:04 (35285): No heartbeat from core client for 30 sec - exiting 01:47:05 (35285): No heartbeat from core client for 30 sec - exiting 01:47:06 (35285): No heartbeat from core client for 30 sec - exiting 01:47:07 (35285): No heartbeat from core client for 30 sec - exiting 01:47:08 (35285): No heartbeat from core client for 30 sec - exiting 01:47:09 (35285): No heartbeat from core client for 30 sec - exiting 01:47:10 (35285): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:12:31 (35968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:14:55 (42027): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:14:56 (42027): No heartbeat from core client for 30 sec - exiting 05:14:57 (42027): No heartbeat from core client for 30 sec - exiting 05:14:58 (42027): No heartbeat from core client for 30 sec - exiting 05:14:59 (42027): No heartbeat from core client for 30 sec - exiting 05:15:00 (42027): No heartbeat from core client for 30 sec - exiting 05:15:01 (42027): No heartbeat from core client for 30 sec - exiting 05:15:02 (42027): No heartbeat from core client for 30 sec - exiting 05:15:03 (42027): No heartbeat from core client for 30 sec - exiting 05:15:04 (42027): No heartbeat from core client for 30 sec - exiting 05:15:05 (42027): No heartbeat from core client for 30 sec - exiting 05:15:06 (42027): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:50:59 (63980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:51:01 (63980): No heartbeat from core client for 30 sec - exiting 00:51:02 (63980): No heartbeat from core client for 30 sec - exiting 00:51:03 (63980): No heartbeat from core client for 30 sec - exiting 00:51:04 (63980): No heartbeat from core client for 30 sec - exiting 00:51:05 (63980): No heartbeat from core client for 30 sec - exiting 00:51:06 (63980): No heartbeat from core client for 30 sec - exiting 00:51:07 (63980): No heartbeat from core client for 30 sec - exiting 00:51:08 (63980): No heartbeat from core client for 30 sec - exiting 00:51:09 (63980): No heartbeat from core client for 30 sec - exiting 00:51:10 (63980): No heartbeat from core client for 30 sec - exiting 00:51:11 (63980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:53:12 (73759): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x2802404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x2802400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x2802404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x2802400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x2802404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x2802400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x806604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x806600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x1000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(74133,0xa0767540) malloc: * error for object 0x1000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:22:20 (84936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:24 (84936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 01:17:00 (94659): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:40 (97506): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:41 (97506): No heartbeat from core client for 30 sec - exiting 03:27:42 (97506): No heartbeat from core client for 30 sec - exiting 03:27:43 (97506): No heartbeat from core client for 30 sec - exiting 03:27:44 (97506): No heartbeat from core client for 30 sec - exiting 03:27:45 (97506): No heartbeat from core client for 30 sec - exiting 03:27:46 (97506): No heartbeat from core client for 30 sec - exiting 03:27:47 (97506): No heartbeat from core client for 30 sec - exiting 03:27:48 (97506): No heartbeat from core client for 30 sec - exiting 03:27:49 (97506): No heartbeat from core client for 30 sec - exiting 03:27:50 (97506): No heartbeat from core client for 30 sec - exiting 05:25:24 (98464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:29 (99241): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:35:32 (99241): No heartbeat from core client for 30 sec - exiting 06:35:33 (99241): No heartbeat from core client for 30 sec - exiting 06:35:34 (99241): No heartbeat from core client for 30 sec - exiting 06:35:35 (99241): No heartbeat from core client for 30 sec - exiting 06:35:36 (99241): No heartbeat from core client for 30 sec - exiting 06:35:37 (99241): No heartbeat from core client for 30 sec - exiting 06:35:38 (99241): No heartbeat from core client for 30 sec - exiting 06:35:39 (99241): No heartbeat from core client for 30 sec - exiting 06:35:40 (99241): No heartbeat from core client for 30 sec - exiting 06:35:41 (99241): No heartbeat from core client for 30 sec - exiting 06:35:42 (99241): No heartbeat from core client for 30 sec - exiting 06:35:43 (99241): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... 21:57:32 (4366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:57:33 (4366): No heartbeat from core client for 30 sec - exiting 21:57:34 (4366): No heartbeat from core client for 30 sec - exiting 21:57:35 (4366): No heartbeat from core client for 30 sec - exiting 21:57:36 (4366): No heartbeat from core client for 30 sec - exiting 21:57:37 (4366): No heartbeat from core client for 30 sec - exiting 21:57:38 (4366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 04:53:16 (6457): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:11:28 (13965): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:00:17 (14831): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:01 (15169): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:03 (15169): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 03:26:01 (17699): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Atmos Hold Restart file rename failed on atmos_restart.hold Model crashed: U_MODEL: Illegal combination of submodels tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Mar 2013 12:24:46	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	311,040	570,417	1.8339
16 Mar 2013 03:31:27	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	285,120	522,109	1.8312
14 Mar 2013 10:10:13	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	259,200	474,930	1.8323
12 Mar 2013 13:30:08	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	233,280	426,735	1.8293
11 Mar 2013 04:36:19	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	207,360	378,255	1.8241
09 Mar 2013 08:42:16	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	181,440	329,203	1.8144
07 Mar 2013 10:41:29	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	155,520	280,215	1.8018
05 Mar 2013 13:20:05	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	129,600	231,622	1.7872
04 Mar 2013 03:11:55	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	103,680	183,104	1.7660
02 Mar 2013 10:39:02	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	77,760	138,484	1.7809
01 Mar 2013 07:02:02	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	51,840	96,087	1.8535
27 Feb 2013 12:12:36	1231328	15640817	hadcm3n_zkxn_1920_40_008254378_3	25,920	48,170	1.8584