Task 15521366

Name	hadcm3n_zazy_1920_40_008254153_2
Workunit	8409277
Created	2 Jan 2013, 10:10:50 UTC
Sent	2 Jan 2013, 10:11:04 UTC
Report deadline	3 Apr 2013, 17:38:15 UTC
Received	11 Jan 2013, 13:35:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1228333
Run time	1 days 7 hours 35 min 25 sec
CPU time	1 days 4 hours 18 min 20 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	2.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 01:23:29 (7456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:31:56 (10600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:45:15 (8056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:49 (9504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:17:04 (9504): No heartbeat from core client for 30 sec - exiting 06:17:05 (9504): No heartbeat from core client for 30 sec - exiting 06:17:06 (9504): No heartbeat from core client for 30 sec - exiting 06:17:07 (9504): No heartbeat from core client for 30 sec - exiting 06:18:02 (14308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:04 (7972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:21:59 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:13:31 (13636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:39:36 (7420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:35:13 (16844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:48:58 (11352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:00 (11352): No heartbeat from core client for 30 sec - exiting 08:49:01 (11352): No heartbeat from core client for 30 sec - exiting 08:49:02 (11352): No heartbeat from core client for 30 sec - exiting 08:49:03 (11352): No heartbeat from core client for 30 sec - exiting 08:49:04 (11352): No heartbeat from core client for 30 sec - exiting 08:49:05 (11352): No heartbeat from core client for 30 sec - exiting 08:49:06 (11352): No heartbeat from core client for 30 sec - exiting 08:49:07 (11352): No heartbeat from core client for 30 sec - exiting 08:49:08 (11352): No heartbeat from core client for 30 sec - exiting 02:07:52 (12692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:54 (12692): No heartbeat from core client for 30 sec - exiting 02:07:55 (12692): No heartbeat from core client for 30 sec - exiting 02:07:56 (12692): No heartbeat from core client for 30 sec - exiting 02:07:57 (12692): No heartbeat from core client for 30 sec - exiting 02:07:58 (12692): No heartbeat from core client for 30 sec - exiting 02:07:59 (12692): No heartbeat from core client for 30 sec - exiting 02:08:00 (12692): No heartbeat from core client for 30 sec - exiting 02:08:01 (12692): No heartbeat from core client for 30 sec - exiting 02:31:48 (8892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:31:50 (8892): No heartbeat from core client for 30 sec - exiting 02:31:51 (8892): No heartbeat from core client for 30 sec - exiting 02:31:52 (8892): No heartbeat from core client for 30 sec - exiting 02:34:01 (14412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:34:05 (14412): No heartbeat from core client for 30 sec - exiting 02:45:19 (12472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:03:38 (10488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:05:12 (15964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:06:57 (13660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:07:55 (9640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:09:48 (14348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:08 (2140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:19 (13372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:08:00 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:09:06 (16256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:11:14 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:19:32 (16692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:15 (13492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:05:32 (9164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:45:24 (9960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:30:12 (16844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:35:20 (8664): No heartbeat from core client for 30 sec - exiting 07:35:59 (8664): No heartbeat from core client for 30 sec - exiting 07:36:00 (8664): No heartbeat from core client for 30 sec - exiting 07:36:01 (8664): No heartbeat from core client for 30 sec - exiting 07:36:02 (8664): No heartbeat from core client for 30 sec - exiting 07:36:03 (8664): No heartbeat from core client for 30 sec - exiting 07:36:04 (8664): No heartbeat from core client for 30 sec - exiting 07:36:05 (8664): No heartbeat from core client for 30 sec - exiting 07:36:06 (8664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:46:50 (7268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:47:21 (14092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:49:14 (2956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:50:11 (18580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:50:48 (17280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:44:19 (11916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:08:37 (3140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:13:01 (15388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:20 (14952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:33:13 (11260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:44:11 (8064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:01 (17600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:45:02 (9100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:27:25 (11468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:45 (14216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:30:16 (10072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:30:20 (10072): No heartbeat from core client for 30 sec - exiting 13:32:10 (5500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:33:59 (15664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:37:40 (13156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:40:34 (15524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:41:24 (10896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:58 (15240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:46:07 (13468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:47:29 (10944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:52:04 (6164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:41:01 (7428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:00:08 (18408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:00:32 (7876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:02:24 (19012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:08 (6692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:42:08 (12588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:02 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:12 (15156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:13 (15156): No heartbeat from core client for 30 sec - exiting 13:40:40 (1364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:41:48 (8656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:42:30 (16428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:43:10 (10956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:12 (14272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:19 (15180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:14 (14696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:17 (14696): No heartbeat from core client for 30 sec - exiting 17:04:19 (14696): No heartbeat from core client for 30 sec - exiting 17:04:20 (14696): No heartbeat from core client for 30 sec - exiting 17:04:21 (14696): No heartbeat from core client for 30 sec - exiting 17:04:22 (14696): No heartbeat from core client for 30 sec - exiting 17:04:23 (14696): No heartbeat from core client for 30 sec - exiting 17:05:42 (12480): No heartbeat from core client for 30 sec - exiting 17:05:44 (12480): No heartbeat from core client for 30 sec - exiting 17:05:45 (12480): No heartbeat from core client for 30 sec - exiting 17:05:46 (12480): No heartbeat from core client for 30 sec - exiting 17:05:47 (12480): No heartbeat from core client for 30 sec - exiting 17:05:48 (12480): No heartbeat from core client for 30 sec - exiting 17:05:49 (12480): No heartbeat from core client for 30 sec - exiting 17:05:50 (12480): No heartbeat from core client for 30 sec - exiting 17:05:51 (12480): No heartbeat from core client for 30 sec - exiting 17:05:52 (12480): No heartbeat from core client for 30 sec - exiting 17:05:53 (12480): No heartbeat from core client for 30 sec - exiting 17:05:54 (12480): No heartbeat from core client for 30 sec - exiting 17:05:55 (12480): No heartbeat from core client for 30 sec - exiting 17:05:56 (12480): No heartbeat from core client for 30 sec - exiting 17:05:57 (12480): No heartbeat from core client for 30 sec - exiting 17:05:58 (12480): No heartbeat from core client for 30 sec - exiting 17:05:59 (12480): No heartbeat from core client for 30 sec - exiting 17:06:00 (12480): No heartbeat from core client for 30 sec - exiting 17:06:01 (12480): No heartbeat from core client for 30 sec - exiting 17:06:02 (12480): No heartbeat from core client for 30 sec - exiting 17:06:03 (12480): No heartbeat from core client for 30 sec - exiting 17:06:04 (12480): No heartbeat from core client for 30 sec - exiting 17:06:05 (12480): No heartbeat from core client for 30 sec - exiting 17:06:06 (12480): No heartbeat from core client for 30 sec - exiting 17:06:07 (12480): No heartbeat from core client for 30 sec - exiting 17:06:08 (12480): No heartbeat from core client for 30 sec - exiting 17:06:09 (12480): No heartbeat from core client for 30 sec - exiting 17:06:10 (12480): No heartbeat from core client for 30 sec - exiting 17:06:11 (12480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11192, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11192, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11192, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11192, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11192, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11192, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Jan 2013 13:05:17	1228333	15521366	hadcm3n_zazy_1920_40_008254153_2	51,840	80,662	1.5560
03 Jan 2013 00:49:11	1228333	15521366	hadcm3n_zazy_1920_40_008254153_2	25,920	39,919	1.5401