Task 16594213

Name	hadcm3n_8fbm_1980_40_008729277_0
Workunit	8875255
Created	23 Apr 2014, 14:17:38 UTC
Sent	24 Apr 2014, 4:48:54 UTC
Report deadline	24 Jul 2014, 12:16:05 UTC
Received	4 Jun 2014, 13:31:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1125052
Run time	21 days 21 hours 10 min 46 sec
CPU time	16 days 18 hours 59 min 8 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	2.79 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:22:20 (3808): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:29:18 (9128): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:51:47 (3260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:51:48 (3260): No heartbeat from core client for 30 sec - exiting 14:51:49 (3260): No heartbeat from core client for 30 sec - exiting 14:51:50 (3260): No heartbeat from core client for 30 sec - exiting 14:51:51 (3260): No heartbeat from core client for 30 sec - exiting 14:51:53 (3260): No heartbeat from core client for 30 sec - exiting 14:51:54 (3260): No heartbeat from core client for 30 sec - exiting 14:51:55 (3260): No heartbeat from core client for 30 sec - exiting 14:51:56 (3260): No heartbeat from core client for 30 sec - exiting 14:51:57 (3260): No heartbeat from core client for 30 sec - exiting 14:51:58 (3260): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:12:53 (9700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:11:38 (6064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:11:39 (6064): No heartbeat from core client for 30 sec - exiting 14:11:40 (6064): No heartbeat from core client for 30 sec - exiting 14:11:42 (6064): No heartbeat from core client for 30 sec - exiting 14:11:43 (6064): No heartbeat from core client for 30 sec - exiting 14:11:44 (6064): No heartbeat from core client for 30 sec - exiting 14:11:45 (6064): No heartbeat from core client for 30 sec - exiting 14:11:46 (6064): No heartbeat from core client for 30 sec - exiting 14:11:47 (6064): No heartbeat from core client for 30 sec - exiting 14:11:48 (6064): No heartbeat from core client for 30 sec - exiting 14:11:49 (6064): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:02:40 (6948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:41 (6948): No heartbeat from core client for 30 sec - exiting 18:02:42 (6948): No heartbeat from core client for 30 sec - exiting 18:02:43 (6948): No heartbeat from core client for 30 sec - exiting 18:02:44 (6948): No heartbeat from core client for 30 sec - exiting 18:02:45 (6948): No heartbeat from core client for 30 sec - exiting 18:02:46 (6948): No heartbeat from core client for 30 sec - exiting 18:02:47 (6948): No heartbeat from core client for 30 sec - exiting 18:02:48 (6948): No heartbeat from core client for 30 sec - exiting 18:02:49 (6948): No heartbeat from core client for 30 sec - exiting 18:02:51 (6948): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:10:03 (8676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8528, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8528, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8528, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8528, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8528, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8528, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 May 2014 14:29:40	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	907,200	1,448,802	1.5970
26 May 2014 01:16:40	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	881,280	1,404,545	1.5938
25 May 2014 12:32:14	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	855,360	1,360,957	1.5911
24 May 2014 23:44:47	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	829,440	1,317,077	1.5879
24 May 2014 10:26:51	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	803,520	1,272,340	1.5835
23 May 2014 21:08:58	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	777,600	1,227,051	1.5780
23 May 2014 07:10:11	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	751,680	1,180,904	1.5710
22 May 2014 18:47:30	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	725,760	1,138,947	1.5693
22 May 2014 04:54:18	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	699,840	1,093,899	1.5631
21 May 2014 15:21:36	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	673,920	1,047,925	1.5550
21 May 2014 01:39:20	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	648,000	1,002,801	1.5475
20 May 2014 12:30:23	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	622,080	958,867	1.5414
19 May 2014 23:24:37	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	596,160	915,321	1.5354
19 May 2014 10:18:51	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	570,240	872,417	1.5299
18 May 2014 21:19:38	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	544,320	831,181	1.5270
18 May 2014 08:31:30	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	518,400	788,008	1.5201
17 May 2014 19:30:56	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	492,480	744,184	1.5111
17 May 2014 05:50:56	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	466,560	699,418	1.4991
16 May 2014 16:52:48	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	440,640	656,039	1.4888
15 May 2014 10:20:27	1125052	16594213	hadcm3n_8fbm_1980_40_008729277_0	414,720	614,912	1.4827