Task 15989916

Name	hadcm3n_7yl3_1980_40_008456106_0
Workunit	8606962
Created	30 Aug 2013, 18:52:13 UTC
Sent	12 Sep 2013, 13:15:13 UTC
Report deadline	12 Dec 2013, 20:42:24 UTC
Received	5 Oct 2013, 0:50:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1163325
Run time	5 days 2 hours 1 min 15 sec
CPU time	4 days 23 hours 43 min 51 sec
Validate state	Invalid
Credit	5,598.72
Device peak FLOPS	3.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:01:45 (7548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:14 (4836): No heartbeat from core client for 30 sec - exiting 07:05:26 (4836): No heartbeat from core client for 30 sec - exiting 07:05:27 (4836): No heartbeat from core client for 30 sec - exiting 07:05:28 (4836): No heartbeat from core client for 30 sec - exiting 07:05:29 (4836): No heartbeat from core client for 30 sec - exiting 07:05:30 (4836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:20:54 (6548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:21:01 (6548): No heartbeat from core client for 30 sec - exiting 09:21:02 (6548): No heartbeat from core client for 30 sec - exiting 09:21:03 (6548): No heartbeat from core client for 30 sec - exiting 09:21:04 (6548): No heartbeat from core client for 30 sec - exiting 09:21:05 (6548): No heartbeat from core client for 30 sec - exiting 09:21:06 (6548): No heartbeat from core client for 30 sec - exiting 09:21:07 (6548): No heartbeat from core client for 30 sec - exiting 09:22:36 (9100): No heartbeat from core client for 30 sec - exiting 09:23:10 (9100): No heartbeat from core client for 30 sec - exiting 09:23:11 (9100): No heartbeat from core client for 30 sec - exiting 09:23:12 (9100): No heartbeat from core client for 30 sec - exiting 09:23:13 (9100): No heartbeat from core client for 30 sec - exiting 09:23:14 (9100): No heartbeat from core client for 30 sec - exiting 09:23:15 (9100): No heartbeat from core client for 30 sec - exiting 09:23:16 (9100): No heartbeat from core client for 30 sec - exiting 09:23:17 (9100): No heartbeat from core client for 30 sec - exiting 09:23:18 (9100): No heartbeat from core client for 30 sec - exiting 09:23:19 (9100): No heartbeat from core client for 30 sec - exiting 09:23:20 (9100): No heartbeat from core client for 30 sec - exiting 09:23:21 (9100): No heartbeat from core client for 30 sec - exiting 09:23:22 (9100): No heartbeat from core client for 30 sec - exiting 09:23:23 (9100): No heartbeat from core client for 30 sec - exiting 09:23:24 (9100): No heartbeat from core client for 30 sec - exiting 09:23:25 (9100): No heartbeat from core client for 30 sec - exiting 09:23:26 (9100): No heartbeat from core client for 30 sec - exiting 09:23:27 (9100): No heartbeat from core client for 30 sec - exiting 09:23:28 (9100): No heartbeat from core client for 30 sec - exiting 09:23:29 (9100): No heartbeat from core client for 30 sec - exiting 09:23:30 (9100): No heartbeat from core client for 30 sec - exiting 09:23:31 (9100): No heartbeat from core client for 30 sec - exiting 09:23:32 (9100): No heartbeat from core client for 30 sec - exiting 09:23:33 (9100): No heartbeat from core client for 30 sec - exiting 09:23:34 (9100): No heartbeat from core client for 30 sec - exiting 09:23:35 (9100): No heartbeat from core client for 30 sec - exiting 09:23:36 (9100): No heartbeat from core client for 30 sec - exiting 09:23:37 (9100): No heartbeat from core client for 30 sec - exiting 09:23:41 (9100): No heartbeat from core client for 30 sec - exiting 09:23:42 (9100): No heartbeat from core client for 30 sec - exiting 09:23:43 (9100): No heartbeat from core client for 30 sec - exiting 09:23:44 (9100): No heartbeat from core client for 30 sec - exiting 09:23:45 (9100): No heartbeat from core client for 30 sec - exiting 09:23:46 (9100): No heartbeat from core client for 30 sec - exiting 09:23:47 (9100): No heartbeat from core client for 30 sec - exiting 09:23:48 (9100): No heartbeat from core client for 30 sec - exiting 09:23:49 (9100): No heartbeat from core client for 30 sec - exiting 09:23:50 (9100): No heartbeat from core client for 30 sec - exiting 09:23:51 (9100): No heartbeat from core client for 30 sec - exiting 09:23:52 (9100): No heartbeat from core client for 30 sec - exiting 09:23:53 (9100): No heartbeat from core client for 30 sec - exiting 09:23:54 (9100): No heartbeat from core client for 30 sec - exiting 09:23:55 (9100): No heartbeat from core client for 30 sec - exiting 09:23:56 (9100): No heartbeat from core client for 30 sec - exiting 09:23:57 (9100): No heartbeat from core client for 30 sec - exiting 09:23:58 (9100): No heartbeat from core client for 30 sec - exiting 09:23:59 (9100): No heartbeat from core client for 30 sec - exiting 09:24:00 (9100): No heartbeat from core client for 30 sec - exiting 09:24:01 (9100): No heartbeat from core client for 30 sec - exiting 09:24:02 (9100): No heartbeat from core client for 30 sec - exiting 09:24:03 (9100): No heartbeat from core client for 30 sec - exiting 09:24:04 (9100): No heartbeat from core client for 30 sec - exiting 09:24:05 (9100): No heartbeat from core client for 30 sec - exiting 09:24:06 (9100): No heartbeat from core client for 30 sec - exiting 09:24:07 (9100): No heartbeat from core client for 30 sec - exiting 09:24:08 (9100): No heartbeat from core client for 30 sec - exiting 09:24:09 (9100): No heartbeat from core client for 30 sec - exiting 09:24:10 (9100): No heartbeat from core client for 30 sec - exiting 09:24:11 (9100): No heartbeat from core client for 30 sec - exiting 09:24:12 (9100): No heartbeat from core client for 30 sec - exiting 09:24:13 (9100): No heartbeat from core client for 30 sec - exiting 09:24:14 (9100): No heartbeat from core client for 30 sec - exiting 09:24:15 (9100): No heartbeat from core client for 30 sec - exiting 09:24:16 (9100): No heartbeat from core client for 30 sec - exiting 09:24:17 (9100): No heartbeat from core client for 30 sec - exiting 09:24:18 (9100): No heartbeat from core client for 30 sec - exiting 09:24:19 (9100): No heartbeat from core client for 30 sec - exiting 09:24:50 (9100): No heartbeat from core client for 30 sec - exiting 09:24:51 (9100): No heartbeat from core client for 30 sec - exiting 09:24:55 (9100): No heartbeat from core client for 30 sec - exiting 09:24:56 (9100): No heartbeat from core client for 30 sec - exiting 09:24:57 (9100): No heartbeat from core client for 30 sec - exiting 09:24:58 (9100): No heartbeat from core client for 30 sec - exiting 09:24:59 (9100): No heartbeat from core client for 30 sec - exiting 09:25:00 (9100): No heartbeat from core client for 30 sec - exiting 09:25:06 (9100): No heartbeat from core client for 30 sec - exiting 09:25:09 (9100): No heartbeat from core client for 30 sec - exiting 09:25:13 (9100): No heartbeat from core client for 30 sec - exiting 09:25:14 (9100): No heartbeat from core client for 30 sec - exiting 09:25:15 (9100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:59:31 (7836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:29 (5696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:00:31 (5696): No heartbeat from core client for 30 sec - exiting 11:00:33 (5696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:54:00 (7740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:12 (800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:35 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:36 (6476): No heartbeat from core client for 30 sec - exiting 16:58:37 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:09:28 (11248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:09:51 (11248): No heartbeat from core client for 30 sec - exiting 22:11:51 (9744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:11:52 (9744): No heartbeat from core client for 30 sec - exiting 22:11:53 (9744): No heartbeat from core client for 30 sec - exiting 22:11:54 (9744): No heartbeat from core client for 30 sec - exiting 22:11:55 (9744): No heartbeat from core client for 30 sec - exiting 22:11:56 (9744): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 22:13:52 (7024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9224, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6624, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7160, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7160, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7160, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7160, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Oct 2013 15:59:11	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	466,560	417,005	0.8938
01 Oct 2013 06:34:41	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	440,640	395,302	0.8971
30 Sep 2013 03:53:01	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	414,720	371,353	0.8954
29 Sep 2013 02:01:36	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	388,800	347,498	0.8938
28 Sep 2013 01:46:41	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	362,880	324,010	0.8929
27 Sep 2013 00:04:33	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	336,960	302,304	0.8972
23 Sep 2013 09:31:01	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	311,040	279,317	0.8980
23 Sep 2013 09:31:01	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	285,120	256,202	0.8986
22 Sep 2013 01:18:13	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	259,200	230,021	0.8874
19 Sep 2013 22:35:27	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	233,280	205,508	0.8809
19 Sep 2013 01:05:27	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	207,360	182,035	0.8779
17 Sep 2013 21:29:54	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	181,440	158,896	0.8757
17 Sep 2013 00:10:46	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	155,520	135,912	0.8739
16 Sep 2013 08:56:55	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	129,600	113,718	0.8775
16 Sep 2013 00:48:45	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	103,680	91,445	0.8820
15 Sep 2013 01:52:12	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	77,760	68,280	0.8781
14 Sep 2013 06:17:00	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	51,840	45,710	0.8818
13 Sep 2013 10:51:32	1163325	15989916	hadcm3n_7yl3_1980_40_008456106_0	25,920	22,309	0.8607