Task 16022115

Name	hadcm3n_7wsj_1980_40_008453782_1
Workunit	8604638
Created	18 Sep 2013, 2:47:32 UTC
Sent	18 Sep 2013, 3:26:37 UTC
Report deadline	18 Dec 2013, 10:53:48 UTC
Received	2 Oct 2013, 12:26:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1194844
Run time	10 days 5 hours 5 min 53 sec
CPU time	9 days 15 hours 10 min 5 sec
Validate state	Invalid
Credit	9,642.24
Device peak FLOPS	3.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 07:47:06 (42112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:45:22 (35592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:40:48 (45780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:13:14 (44972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:59 (47964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:44:27 (49060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:39:32 (51588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:41:36 (55848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:18 (53260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 09:44:44 (53932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:44:45 (53932): No heartbeat from core client for 30 sec - exiting 09:44:46 (53932): No heartbeat from core client for 30 sec - exiting 09:44:47 (53932): No heartbeat from core client for 30 sec - exiting 09:44:48 (53932): No heartbeat from core client for 30 sec - exiting 09:44:49 (53932): No heartbeat from core client for 30 sec - exiting 09:44:50 (53932): No heartbeat from core client for 30 sec - exiting 09:44:51 (53932): No heartbeat from core client for 30 sec - exiting 09:44:52 (53932): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:54:17 (55140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:50:33 (70560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:11 (70404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:37 (73220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:34:45 (74720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:27:47 (76176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:22:17 (77304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:31 (79568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:15:19 (85900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:43:03 (94156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:53:34 (99164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:43:11 (100144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:32:47 (95908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 02:30:36 (103912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:08:22 (117368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:11:54 (117880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:37 (118120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:40:58 (117904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:47:12 (119748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:00:19 (124964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:41:19 (126052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:41:20 (126052): No heartbeat from core client for 30 sec - exiting 18:41:21 (126052): No heartbeat from core client for 30 sec - exiting 18:41:22 (126052): No heartbeat from core client for 30 sec - exiting 18:41:23 (126052): No heartbeat from core client for 30 sec - exiting 18:41:24 (126052): No heartbeat from core client for 30 sec - exiting 18:41:25 (126052): No heartbeat from core client for 30 sec - exiting 18:41:26 (126052): No heartbeat from core client for 30 sec - exiting 18:41:27 (126052): No heartbeat from core client for 30 sec - exiting 18:41:28 (126052): No heartbeat from core client for 30 sec - exiting 18:41:29 (126052): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 19:04:40 (133288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:33 (131312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:06:48 (131312): No heartbeat from core client for 30 sec - exiting 19:07:45 (131080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:01 (133576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:12:42 (140780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:16:07 (142412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:17:38 (141464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:18:03 (141464): No heartbeat from core client for 30 sec - exiting 05:18:04 (141464): No heartbeat from core client for 30 sec - exiting 05:18:05 (141464): No heartbeat from core client for 30 sec - exiting 05:19:14 (140056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:05 (140056): No heartbeat from core client for 30 sec - exiting 05:20:06 (140056): No heartbeat from core client for 30 sec - exiting 05:20:07 (140056): No heartbeat from core client for 30 sec - exiting 05:20:08 (140056): No heartbeat from core client for 30 sec - exiting 05:20:09 (140056): No heartbeat from core client for 30 sec - exiting 05:20:43 (141256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:21:21 (134532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:22:38 (139992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:23:02 (139992): No heartbeat from core client for 30 sec - exiting 05:23:03 (139992): No heartbeat from core client for 30 sec - exiting 05:23:04 (139992): No heartbeat from core client for 30 sec - exiting 05:23:05 (139992): No heartbeat from core client for 30 sec - exiting 05:23:06 (139992): No heartbeat from core client for 30 sec - exiting 05:23:07 (139992): No heartbeat from core client for 30 sec - exiting 05:23:08 (139992): No heartbeat from core client for 30 sec - exiting 05:23:09 (139992): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 05:25:44 (141976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:48 (140080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:29:48 (143624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:33:19 (142540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:06 (143672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:37:09 (143968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:05 (145300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:38:41 (144772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:39 (143512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:32 (145008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:42:19 (145172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:43:33 (144584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:44:45 (145168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:45:21 (144480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:24 (144168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:14 (144168): No heartbeat from core client for 30 sec - exiting 05:48:32 (143392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:49:46 (145100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:50:40 (145576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:51:43 (143884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:52:59 (145416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:54:53 (146268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:56:19 (145792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:57:57 (145796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:59:56 (144336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:00:44 (145468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:01:46 (146316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:02:20 (147168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:35 (146872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:04:55 (146472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:21 (145968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:10:43 (146040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:12:54 (145176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:16:32 (118720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:15 (147792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:28:38 (146544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:32:55 (144228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=147916, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=147916, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=147916, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=147916, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=147916, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=147916, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Oct 2013 22:26:43	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	803,520	812,862	1.0116
01 Oct 2013 07:50:18	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	777,600	786,058	1.0109
01 Oct 2013 00:11:50	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	751,680	759,656	1.0106
30 Sep 2013 16:26:32	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	725,760	733,233	1.0103
30 Sep 2013 08:55:22	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	699,840	706,899	1.0101
30 Sep 2013 01:21:54	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	673,920	680,498	1.0098
29 Sep 2013 16:06:07	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	648,000	654,312	1.0097
29 Sep 2013 07:38:06	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	622,080	627,817	1.0092
28 Sep 2013 23:55:57	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	596,160	601,316	1.0086
28 Sep 2013 16:01:21	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	570,240	574,959	1.0083
28 Sep 2013 00:00:53	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	544,320	548,622	1.0079
27 Sep 2013 04:50:57	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	518,400	522,243	1.0074
26 Sep 2013 13:54:47	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	492,480	495,794	1.0067
26 Sep 2013 05:45:55	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	466,560	469,050	1.0053
25 Sep 2013 22:07:56	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	440,640	442,557	1.0044
25 Sep 2013 12:37:12	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	414,720	416,320	1.0039
25 Sep 2013 09:28:14	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	388,800	389,949	1.0030
25 Sep 2013 09:24:26	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	362,880	363,318	1.0012
25 Sep 2013 09:20:14	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	336,960	336,841	0.9996
24 Sep 2013 06:02:51	1194844	16022115	hadcm3n_7wsj_1980_40_008453782_1	311,040	310,320	0.9977