Task 16048122

Name	hadcm3n_of61_1900_40_008474780_1
Workunit	8625619
Created	28 Sep 2013, 11:07:19 UTC
Sent	28 Sep 2013, 11:27:38 UTC
Report deadline	28 Dec 2013, 18:54:49 UTC
Received	17 Oct 2013, 22:26:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1294904
Run time	18 days 4 hours 27 min 33 sec
CPU time	15 days 16 hours 5 min 10 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	2.13 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.18</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 14:31:02 (25712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:31:04 (25712): No heartbeat from core client for 30 sec - exiting 14:31:05 (25712): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:06:33 (9664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:07:11 (9664): No heartbeat from core client for 30 sec - exiting 05:08:49 (9664): No heartbeat from core client for 30 sec - exiting 05:09:06 (9664): No heartbeat from core client for 30 sec - exiting 05:09:11 (9664): No heartbeat from core client for 30 sec - exiting 05:09:12 (9664): No heartbeat from core client for 30 sec - exiting 05:09:13 (9664): No heartbeat from core client for 30 sec - exiting 05:09:14 (9664): No heartbeat from core client for 30 sec - exiting 05:09:15 (9664): No heartbeat from core client for 30 sec - exiting 05:09:16 (9664): No heartbeat from core client for 30 sec - exiting 05:09:19 (9664): No heartbeat from core client for 30 sec - exiting 05:09:20 (9664): No heartbeat from core client for 30 sec - exiting 05:09:22 (9664): No heartbeat from core client for 30 sec - exiting 05:09:23 (9664): No heartbeat from core client for 30 sec - exiting 05:09:24 (9664): No heartbeat from core client for 30 sec - exiting 05:09:25 (9664): No heartbeat from core client for 30 sec - exiting 05:09:27 (9664): No heartbeat from core client for 30 sec - exiting 05:09:28 (9664): No heartbeat from core client for 30 sec - exiting 05:09:30 (9664): No heartbeat from core client for 30 sec - exiting 05:09:32 (9664): No heartbeat from core client for 30 sec - exiting 05:09:33 (9664): No heartbeat from core client for 30 sec - exiting 05:09:35 (9664): No heartbeat from core client for 30 sec - exiting 05:09:36 (9664): No heartbeat from core client for 30 sec - exiting 05:09:37 (9664): No heartbeat from core client for 30 sec - exiting 05:09:38 (9664): No heartbeat from core client for 30 sec - exiting 05:09:39 (9664): No heartbeat from core client for 30 sec - exiting 05:09:40 (9664): No heartbeat from core client for 30 sec - exiting 05:09:41 (9664): No heartbeat from core client for 30 sec - exiting 05:09:42 (9664): No heartbeat from core client for 30 sec - exiting 05:09:43 (9664): No heartbeat from core client for 30 sec - exiting 05:09:45 (9664): No heartbeat from core client for 30 sec - exiting 05:13:09 (20456): No heartbeat from core client for 30 sec - exiting 05:13:10 (20456): No heartbeat from core client for 30 sec - exiting 05:13:11 (20456): No heartbeat from core client for 30 sec - exiting 05:13:12 (20456): No heartbeat from core client for 30 sec - exiting 05:13:13 (20456): No heartbeat from core client for 30 sec - exiting 05:13:14 (20456): No heartbeat from core client for 30 sec - exiting 05:13:15 (20456): No heartbeat from core client for 30 sec - exiting 05:13:16 (20456): No heartbeat from core client for 30 sec - exiting 05:13:17 (20456): No heartbeat from core client for 30 sec - exiting 05:13:18 (20456): No heartbeat from core client for 30 sec - exiting 05:13:19 (20456): No heartbeat from core client for 30 sec - exiting 05:13:20 (20456): No heartbeat from core client for 30 sec - exiting 05:13:21 (20456): No heartbeat from core client for 30 sec - exiting 05:13:22 (20456): No heartbeat from core client for 30 sec - exiting 05:13:23 (20456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:18:32 (25712): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 16:18:46 (25712): No heartbeat from core client for 30 sec - exiting 16:18:47 (25712): No heartbeat from core client for 30 sec - exiting 16:18:48 (25712): No heartbeat from core client for 30 sec - exiting 16:18:49 (25712): No heartbeat from core client for 30 sec - exiting 16:18:50 (25712): No heartbeat from core client for 30 sec - exiting 16:18:51 (25712): No heartbeat from core client for 30 sec - exiting 16:18:52 (25712): No heartbeat from core client for 30 sec - exiting 16:18:54 (25712): No heartbeat from core client for 30 sec - exiting 16:18:55 (25712): No heartbeat from core client for 30 sec - exiting 16:18:56 (25712): No heartbeat from core client for 30 sec - exiting 16:18:57 (25712): No heartbeat from core client for 30 sec - exiting 16:18:59 (25712): No heartbeat from core client for 30 sec - exiting 16:19:00 (25712): No heartbeat from core client for 30 sec - exiting 16:19:01 (25712): No heartbeat from core client for 30 sec - exiting 16:19:02 (25712): No heartbeat from core client for 30 sec - exiting 16:19:03 (25712): No heartbeat from core client for 30 sec - exiting 16:19:04 (25712): No heartbeat from core client for 30 sec - exiting 16:19:05 (25712): No heartbeat from core client for 30 sec - exiting 16:19:06 (25712): No heartbeat from core client for 30 sec - exiting 16:19:07 (25712): No heartbeat from core client for 30 sec - exiting 16:19:08 (25712): No heartbeat from core client for 30 sec - exiting 16:19:09 (25712): No heartbeat from core client for 30 sec - exiting 16:19:10 (25712): No heartbeat from core client for 30 sec - exiting 16:19:11 (25712): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:58:22 (5732): No heartbeat from core client for 30 sec - exiting 17:58:23 (5732): No heartbeat from core client for 30 sec - exiting 17:58:24 (5732): No heartbeat from core client for 30 sec - exiting 17:58:25 (5732): No heartbeat from core client for 30 sec - exiting 17:58:26 (5732): No heartbeat from core client for 30 sec - exiting 17:58:27 (5732): No heartbeat from core client for 30 sec - exiting 17:58:28 (5732): No heartbeat from core client for 30 sec - exiting 17:58:29 (5732): No heartbeat from core client for 30 sec - exiting 17:58:30 (5732): No heartbeat from core client for 30 sec - exiting 17:58:31 (5732): No heartbeat from core client for 30 sec - exiting 17:58:32 (5732): No heartbeat from core client for 30 sec - exiting 17:58:33 (5732): No heartbeat from core client for 30 sec - exiting 17:58:34 (5732): No heartbeat from core client for 30 sec - exiting 17:58:35 (5732): No heartbeat from core client for 30 sec - exiting 17:58:36 (5732): No heartbeat from core client for 30 sec - exiting 17:58:37 (5732): No heartbeat from core client for 30 sec - exiting 17:58:38 (5732): No heartbeat from core client for 30 sec - exiting 17:58:39 (5732): No heartbeat from core client for 30 sec - exiting 17:58:40 (5732): No heartbeat from core client for 30 sec - exiting 17:58:41 (5732): No heartbeat from core client for 30 sec - exiting 17:58:42 (5732): No heartbeat from core client for 30 sec - exiting 17:58:43 (5732): No heartbeat from core client for 30 sec - exiting 17:58:45 (5732): No heartbeat from core client for 30 sec - exiting 17:58:46 (5732): No heartbeat from core client for 30 sec - exiting 17:58:47 (5732): No heartbeat from core client for 30 sec - exiting 17:58:48 (5732): No heartbeat from core client for 30 sec - exiting 17:58:49 (5732): No heartbeat from core client for 30 sec - exiting 17:58:50 (5732): No heartbeat from core client for 30 sec - exiting 17:58:51 (5732): No heartbeat from core client for 30 sec - exiting 17:58:52 (5732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:24:08 (6788): No heartbeat from core client for 30 sec - exiting 03:24:09 (6788): No heartbeat from core client for 30 sec - exiting 03:24:11 (6788): No heartbeat from core client for 30 sec - exiting 03:24:12 (6788): No heartbeat from core client for 30 sec - exiting 03:24:13 (6788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:24 (732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:50:43 (2708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:27:38 (3860): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 05:27:40 (3860): No heartbeat from core client for 30 sec - exiting 05:27:41 (3860): No heartbeat from core client for 30 sec - exiting 05:27:42 (3860): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:16:10 (8092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:11 (8092): No heartbeat from core client for 30 sec - exiting 02:16:12 (8092): No heartbeat from core client for 30 sec - exiting 02:16:13 (8092): No heartbeat from core client for 30 sec - exiting 02:16:14 (8092): No heartbeat from core client for 30 sec - exiting 02:16:15 (8092): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:13:54 (7552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:55 (7552): No heartbeat from core client for 30 sec - exiting 08:13:56 (7552): No heartbeat from core client for 30 sec - exiting 11:07:04 (2760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:07:05 (2760): No heartbeat from core client for 30 sec - exiting 11:08:26 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:06:09 (8468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:24:19 (9200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:24:21 (9200): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8700, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8700, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Oct 2013 10:34:20	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	596,160	1,329,134	2.2295
16 Oct 2013 12:31:21	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	570,240	1,274,180	2.2345
15 Oct 2013 17:13:25	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	544,320	1,221,401	2.2439
14 Oct 2013 21:08:43	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	518,400	1,164,295	2.2459
14 Oct 2013 03:09:30	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	492,480	1,108,022	2.2499
13 Oct 2013 05:03:00	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	466,560	1,051,683	2.2541
12 Oct 2013 10:50:49	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	440,640	993,279	2.2542
11 Oct 2013 13:04:14	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	414,720	935,265	2.2552
10 Oct 2013 16:15:26	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	388,800	874,537	2.2493
09 Oct 2013 18:27:26	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	362,880	816,882	2.2511
08 Oct 2013 20:51:57	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	336,960	758,938	2.2523
08 Oct 2013 02:30:23	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	311,040	700,973	2.2536
07 Oct 2013 08:40:10	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	285,120	644,720	2.2612
06 Oct 2013 14:07:52	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	259,200	589,057	2.2726
05 Oct 2013 18:32:28	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	233,280	531,466	2.2782
05 Oct 2013 00:10:08	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	207,360	471,678	2.2747
04 Oct 2013 06:09:57	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	181,440	411,806	2.2697
03 Oct 2013 12:22:55	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	155,520	350,878	2.2562
02 Oct 2013 16:11:33	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	129,600	290,661	2.2428
01 Oct 2013 19:22:43	1294904	16048122	hadcm3n_of61_1900_40_008474780_1	103,680	231,405	2.2319