Task 16142977

Name	hadcm3n_z9hs_1960_40_008366480_3
Workunit	8517339
Created	13 Dec 2013, 5:20:30 UTC
Sent	13 Dec 2013, 5:20:43 UTC
Report deadline	14 Mar 2014, 12:47:54 UTC
Received	23 Jan 2014, 4:10:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1302832
Run time	31 days 7 hours 21 min 20 sec
CPU time	30 days 4 hours 17 min 9 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	1.46 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.31</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:20:27 (13512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:18:05 (89280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:18:08 (89280): No heartbeat from core client for 30 sec - exiting 00:18:09 (89280): No heartbeat from core client for 30 sec - exiting 00:18:10 (89280): No heartbeat from core client for 30 sec - exiting 00:18:12 (89280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:46:27 (93888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:29 (93888): No heartbeat from core client for 30 sec - exiting 18:47:42 (92324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 23:26:47 (93892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:26:49 (93892): No heartbeat from core client for 30 sec - exiting 23:26:50 (93892): No heartbeat from core client for 30 sec - exiting 23:28:06 (92592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:31:47 (90800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:31:48 (90800): No heartbeat from core client for 30 sec - exiting 23:31:50 (90800): No heartbeat from core client for 30 sec - exiting 23:32:54 (92484): No heartbeat from core client for 30 sec - exiting 23:32:55 (92484): No heartbeat from core client for 30 sec - exiting 23:32:56 (92484): No heartbeat from core client for 30 sec - exiting 23:32:57 (92484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:32:59 (92484): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 05:27:11 (92376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:27:13 (92376): No heartbeat from core client for 30 sec - exiting 05:27:14 (92376): No heartbeat from core client for 30 sec - exiting 05:27:15 (92376): No heartbeat from core client for 30 sec - exiting 05:28:32 (90240): No heartbeat from core client for 30 sec - exiting 05:28:33 (90240): No heartbeat from core client for 30 sec - exiting 05:28:34 (90240): No heartbeat from core client for 30 sec - exiting 05:28:35 (90240): No heartbeat from core client for 30 sec - exiting 05:28:37 (90240): No heartbeat from core client for 30 sec - exiting 05:28:38 (90240): No heartbeat from core client for 30 sec - exiting 05:28:39 (90240): No heartbeat from core client for 30 sec - exiting 05:28:40 (90240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:41 (90240): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=93096, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=93096, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=93096, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=93096, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=93096, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=93096, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jan 2014 12:51:10	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	829,440	2,574,881	3.1044
19 Jan 2014 13:01:02	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	803,520	2,492,261	3.1017
18 Jan 2014 13:13:25	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	777,600	2,409,401	3.0985
14 Jan 2014 14:26:46	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	751,680	2,325,193	3.0933
14 Jan 2014 00:40:58	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	725,760	2,244,413	3.0925
12 Jan 2014 19:24:37	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	699,840	2,164,277	3.0925
12 Jan 2014 19:24:37	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	673,920	2,084,007	3.0924
10 Jan 2014 08:52:24	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	648,000	2,004,110	3.0928
10 Jan 2014 08:52:24	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	622,080	1,924,318	3.0934
10 Jan 2014 08:52:24	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	596,160	1,844,391	3.0938
10 Jan 2014 08:52:24	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	570,240	1,764,576	3.0944
10 Jan 2014 08:52:24	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	544,320	1,684,714	3.0951
10 Jan 2014 08:52:24	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	518,400	1,604,861	3.0958
10 Jan 2014 08:52:24	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	492,480	1,525,003	3.0966
03 Jan 2014 19:05:52	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	466,560	1,445,113	3.0974
03 Jan 2014 19:05:52	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	440,640	1,365,374	3.0986
02 Jan 2014 13:06:44	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	414,720	1,285,667	3.1001
02 Jan 2014 13:06:44	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	388,800	1,205,903	3.1016
02 Jan 2014 13:06:44	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	362,880	1,126,269	3.1037
02 Jan 2014 13:06:44	1302832	16142977	hadcm3n_z9hs_1960_40_008366480_3	336,960	1,046,633	3.1061