Task 16220306

Name	hadcm3n_7cjn_1980_40_008427542_1
Workunit	8578398
Created	14 Jan 2014, 0:39:14 UTC
Sent	14 Jan 2014, 0:39:31 UTC
Report deadline	15 Apr 2014, 8:06:42 UTC
Received	28 Jan 2014, 0:00:55 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1242385
Run time	11 days 6 hours 34 min 8 sec
CPU time	11 days 0 hours 51 min 53 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:30:06 (6292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:35:52 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:53 (2936): No heartbeat from core client for 30 sec - exiting 09:35:54 (2936): No heartbeat from core client for 30 sec - exiting 09:35:55 (2936): No heartbeat from core client for 30 sec - exiting 09:35:56 (2936): No heartbeat from core client for 30 sec - exiting 09:35:57 (2936): No heartbeat from core client for 30 sec - exiting 09:35:58 (2936): No heartbeat from core client for 30 sec - exiting 09:35:59 (2936): No heartbeat from core client for 30 sec - exiting 09:36:00 (2936): No heartbeat from core client for 30 sec - exiting 09:36:01 (2936): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6192, iMonCtr=1 Model crash detected, will try to restart... 08:31:07 (7080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:32:46 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:35:49 (6192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:29:10 (6296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:49 (3116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6008, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6008, iMonCtr=1 Model crash detected, will try to restart... 10:02:41 (6988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:28:57 (6368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:38:24 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7048, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7048, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Jan 2014 14:03:41	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	466,560	951,737	2.0399
26 Jan 2014 23:20:52	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	440,640	902,013	2.0471
26 Jan 2014 08:26:58	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	414,720	848,873	2.0469
25 Jan 2014 16:52:59	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	388,800	793,660	2.0413
25 Jan 2014 01:00:44	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	362,880	737,557	2.0325
24 Jan 2014 12:32:23	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	336,960	682,143	2.0244
24 Jan 2014 12:32:23	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	311,040	633,499	2.0367
22 Jan 2014 21:38:02	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	285,120	582,466	2.0429
22 Jan 2014 05:47:51	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	259,200	529,105	2.0413
21 Jan 2014 11:05:23	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	233,280	476,417	2.0423
20 Jan 2014 12:11:02	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	207,360	424,058	2.0450
19 Jan 2014 20:42:39	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	181,440	369,317	2.0355
19 Jan 2014 06:29:37	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	155,520	318,836	2.0501
18 Jan 2014 15:34:42	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	129,600	266,117	2.0534
18 Jan 2014 00:00:00	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	103,680	214,709	2.0709
17 Jan 2014 05:31:23	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	77,760	160,908	2.0693
16 Jan 2014 01:06:40	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	51,840	104,471	2.0153
15 Jan 2014 00:31:53	1242385	16220306	hadcm3n_7cjn_1980_40_008427542_1	25,920	50,638	1.9536