Task 13395890

Name	hadcm3n_o3ny_1980_40_007451376_4
Workunit	7648879
Created	18 Sep 2011, 10:53:56 UTC
Sent	18 Sep 2011, 11:01:23 UTC
Report deadline	18 Dec 2011, 18:28:34 UTC
Received	12 Oct 2011, 0:20:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	23 days 2 hours 24 min 37 sec
CPU time	22 days 20 hours 11 min 56 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	1.70 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 16:59:21 (6848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:00:21 (4516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:26 (4516): No heartbeat from core client for 30 sec - exiting 17:00:27 (4516): No heartbeat from core client for 30 sec - exiting 17:00:28 (4516): No heartbeat from core client for 30 sec - exiting 17:00:29 (4516): No heartbeat from core client for 30 sec - exiting 17:00:30 (4516): No heartbeat from core client for 30 sec - exiting 17:00:31 (4516): No heartbeat from core client for 30 sec - exiting 17:00:32 (4516): No heartbeat from core client for 30 sec - exiting 17:00:34 (4516): No heartbeat from core client for 30 sec - exiting 17:00:35 (4516): No heartbeat from core client for 30 sec - exiting 17:00:36 (4516): No heartbeat from core client for 30 sec - exiting 17:09:18 (6620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:00:34 (1824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:34:45 (968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:34:46 (968): No heartbeat from core client for 30 sec - exiting 21:34:47 (968): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:24:06 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:29 (4796): No heartbeat from core client for 30 sec - exiting 02:24:30 (4796): No heartbeat from core client for 30 sec - exiting 02:24:31 (4796): No heartbeat from core client for 30 sec - exiting 02:24:32 (4796): No heartbeat from core client for 30 sec - exiting 02:24:33 (4796): No heartbeat from core client for 30 sec - exiting 02:24:34 (4796): No heartbeat from core client for 30 sec - exiting 02:24:35 (4796): No heartbeat from core client for 30 sec - exiting 02:24:36 (4796): No heartbeat from core client for 30 sec - exiting 02:24:37 (4796): No heartbeat from core client for 30 sec - exiting 02:24:38 (4796): No heartbeat from core client for 30 sec - exiting 02:24:40 (4796): No heartbeat from core client for 30 sec - exiting 02:24:41 (4796): No heartbeat from core client for 30 sec - exiting 02:24:42 (4796): No heartbeat from core client for 30 sec - exiting 02:24:43 (4796): No heartbeat from core client for 30 sec - exiting 02:24:44 (4796): No heartbeat from core client for 30 sec - exiting 02:24:45 (4796): No heartbeat from core client for 30 sec - exiting 02:24:46 (4796): No heartbeat from core client for 30 sec - exiting 02:24:47 (4796): No heartbeat from core client for 30 sec - exiting 02:24:48 (4796): No heartbeat from core client for 30 sec - exiting 02:24:49 (4796): No heartbeat from core client for 30 sec - exiting 02:24:50 (4796): No heartbeat from core client for 30 sec - exiting 02:24:52 (4796): No heartbeat from core client for 30 sec - exiting 02:24:53 (4796): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:38:40 (4996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:38:49 (4996): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 02:23:55 (3528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:24:44 (3528): No heartbeat from core client for 30 sec - exiting 02:24:45 (3528): No heartbeat from core client for 30 sec - exiting 02:24:46 (3528): No heartbeat from core client for 30 sec - exiting 02:24:48 (3528): No heartbeat from core client for 30 sec - exiting 02:24:49 (3528): No heartbeat from core client for 30 sec - exiting 02:24:50 (3528): No heartbeat from core client for 30 sec - exiting 02:24:51 (3528): No heartbeat from core client for 30 sec - exiting 02:24:52 (3528): No heartbeat from core client for 30 sec - exiting 02:24:53 (3528): No heartbeat from core client for 30 sec - exiting 02:24:54 (3528): No heartbeat from core client for 30 sec - exiting 02:24:55 (3528): No heartbeat from core client for 30 sec - exiting 02:24:56 (3528): No heartbeat from core client for 30 sec - exiting 02:24:57 (3528): No heartbeat from core client for 30 sec - exiting 02:24:58 (3528): No heartbeat from core client for 30 sec - exiting 02:25:00 (3528): No heartbeat from core client for 30 sec - exiting 02:25:01 (3528): No heartbeat from core client for 30 sec - exiting 02:25:02 (3528): No heartbeat from core client for 30 sec - exiting 02:25:03 (3528): No heartbeat from core client for 30 sec - exiting 02:25:04 (3528): No heartbeat from core client for 30 sec - exiting 02:25:05 (3528): No heartbeat from core client for 30 sec - exiting 02:25:06 (3528): No heartbeat from core client for 30 sec - exiting 02:25:07 (3528): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:11:48 (2624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:12:07 (2624): No heartbeat from core client for 30 sec - exiting 17:12:08 (2624): No heartbeat from core client for 30 sec - exiting 17:12:09 (2624): No heartbeat from core client for 30 sec - exiting 17:12:10 (2624): No heartbeat from core client for 30 sec - exiting 17:12:12 (2624): No heartbeat from core client for 30 sec - exiting 17:12:13 (2624): No heartbeat from core client for 30 sec - exiting 17:12:14 (2624): No heartbeat from core client for 30 sec - exiting 17:12:15 (2624): No heartbeat from core client for 30 sec - exiting 17:12:16 (2624): No heartbeat from core client for 30 sec - exiting 17:12:17 (2624): No heartbeat from core client for 30 sec - exiting 17:12:18 (2624): No heartbeat from core client for 30 sec - exiting 17:12:19 (2624): No heartbeat from core client for 30 sec - exiting 17:12:20 (2624): No heartbeat from core client for 30 sec - exiting 17:12:21 (2624): No heartbeat from core client for 30 sec - exiting 17:12:22 (2624): No heartbeat from core client for 30 sec - exiting 17:12:24 (2624): No heartbeat from core client for 30 sec - exiting 17:12:25 (2624): No heartbeat from core client for 30 sec - exiting 17:12:26 (2624): No heartbeat from core client for 30 sec - exiting 17:12:27 (2624): No heartbeat from core client for 30 sec - exiting 17:12:28 (2624): No heartbeat from core client for 30 sec - exiting 17:12:29 (2624): No heartbeat from core client for 30 sec - exiting 17:12:30 (2624): No heartbeat from core client for 30 sec - exiting 17:12:31 (2624): No heartbeat from core client for 30 sec - exiting 17:12:32 (2624): No heartbeat from core client for 30 sec - exiting 17:12:33 (2624): No heartbeat from core client for 30 sec - exiting 17:12:34 (2624): No heartbeat from core client for 30 sec - exiting 17:12:36 (2624): No heartbeat from core client for 30 sec - exiting 17:12:37 (2624): No heartbeat from core client for 30 sec - exiting 17:12:38 (2624): No heartbeat from core client for 30 sec - exiting 17:12:39 (2624): No heartbeat from core client for 30 sec - exiting 17:12:40 (2624): No heartbeat from core client for 30 sec - exiting 17:12:41 (2624): No heartbeat from core client for 30 sec - exiting 17:12:42 (2624): No heartbeat from core client for 30 sec - exiting 17:12:43 (2624): No heartbeat from core client for 30 sec - exiting 17:12:44 (2624): No heartbeat from core client for 30 sec - exiting 17:12:45 (2624): No heartbeat from core client for 30 sec - exiting 17:12:46 (2624): No heartbeat from core client for 30 sec - exiting 17:12:48 (2624): No heartbeat from core client for 30 sec - exiting 17:12:49 (2624): No heartbeat from core client for 30 sec - exiting 17:12:50 (2624): No heartbeat from core client for 30 sec - exiting 17:12:51 (2624): No heartbeat from core client for 30 sec - exiting 17:12:52 (2624): No heartbeat from core client for 30 sec - exiting 17:12:53 (2624): No heartbeat from core client for 30 sec - exiting 17:12:54 (2624): No heartbeat from core client for 30 sec - exiting 17:12:55 (2624): No heartbeat from core client for 30 sec - exiting 17:12:56 (2624): No heartbeat from core client for 30 sec - exiting 17:12:57 (2624): No heartbeat from core client for 30 sec - exiting 17:12:58 (2624): No heartbeat from core client for 30 sec - exiting 17:13:00 (2624): No heartbeat from core client for 30 sec - exiting 17:13:01 (2624): No heartbeat from core client for 30 sec - exiting 17:13:02 (2624): No heartbeat from core client for 30 sec - exiting 17:13:03 (2624): No heartbeat from core client for 30 sec - exiting 17:13:04 (2624): No heartbeat from core client for 30 sec - exiting 17:13:05 (2624): No heartbeat from core client for 30 sec - exiting 17:13:06 (2624): No heartbeat from core client for 30 sec - exiting 17:13:07 (2624): No heartbeat from core client for 30 sec - exiting 17:13:08 (2624): No heartbeat from core client for 30 sec - exiting 17:13:09 (2624): No heartbeat from core client for 30 sec - exiting 17:13:10 (2624): No heartbeat from core client for 30 sec - exiting 17:13:12 (2624): No heartbeat from core client for 30 sec - exiting 17:13:13 (2624): No heartbeat from core client for 30 sec - exiting 17:13:14 (2624): No heartbeat from core client for 30 sec - exiting 17:13:15 (2624): No heartbeat from core client for 30 sec - exiting 17:13:16 (2624): No heartbeat from core client for 30 sec - exiting 17:13:17 (2624): No heartbeat from core client for 30 sec - exiting 17:13:18 (2624): No heartbeat from core client for 30 sec - exiting 17:13:19 (2624): No heartbeat from core client for 30 sec - exiting 17:13:20 (2624): No heartbeat from core client for 30 sec - exiting 17:13:21 (2624): No heartbeat from core client for 30 sec - exiting 17:13:22 (2624): No heartbeat from core client for 30 sec - exiting 17:13:24 (2624): No heartbeat from core client for 30 sec - exiting 17:13:25 (2624): No heartbeat from core client for 30 sec - exiting 17:13:26 (2624): No heartbeat from core client for 30 sec - exiting 17:13:27 (2624): No heartbeat from core client for 30 sec - exiting 17:13:28 (2624): No heartbeat from core client for 30 sec - exiting 17:13:29 (2624): No heartbeat from core client for 30 sec - exiting 17:13:30 (2624): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5904, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Oct 2011 19:32:00	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	648,000	1,956,892	3.0199
10 Oct 2011 21:17:19	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	622,080	1,877,279	3.0177
09 Oct 2011 23:05:03	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	596,160	1,798,213	3.0163
09 Oct 2011 00:44:33	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	570,240	1,719,493	3.0154
08 Oct 2011 02:46:27	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	544,320	1,640,953	3.0147
07 Oct 2011 05:27:03	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	518,400	1,562,520	3.0141
06 Oct 2011 06:35:53	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	492,480	1,483,709	3.0127
05 Oct 2011 08:19:21	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	466,560	1,404,472	3.0103
04 Oct 2011 08:26:17	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	440,640	1,325,305	3.0077
03 Oct 2011 09:40:12	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	414,720	1,246,204	3.0049
02 Oct 2011 11:33:02	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	388,800	1,167,227	3.0021
01 Oct 2011 12:32:05	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	362,880	1,088,693	3.0001
30 Sep 2011 14:00:53	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	336,960	1,010,003	2.9974
29 Sep 2011 15:31:59	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	311,040	930,485	2.9915
28 Sep 2011 15:41:11	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	285,120	849,323	2.9788
27 Sep 2011 17:22:23	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	259,200	770,026	2.9708
26 Sep 2011 19:11:53	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	233,280	691,197	2.9630
25 Sep 2011 21:01:42	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	207,360	612,039	2.9516
24 Sep 2011 22:42:26	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	181,440	533,456	2.9401
24 Sep 2011 00:51:12	1122757	13395890	hadcm3n_o3ny_1980_40_007451376_4	155,520	455,180	2.9268