Task 15548874

Name	hadcm3n_n7n9_1880_40_008286096_0
Workunit	8437231
Created	17 Jan 2013, 17:41:04 UTC
Sent	17 Jan 2013, 17:41:11 UTC
Report deadline	19 Apr 2013, 1:08:22 UTC
Received	15 Mar 2013, 13:47:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	968211
Run time	11 days 9 hours 30 min 52 sec
CPU time	10 days 20 hours 3 min 32 sec
Validate state	Invalid
Credit	8,398.08
Device peak FLOPS	2.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.60</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:37:07 (6768): No heartbeat from core client for 30 sec - exiting 18:37:08 (6768): No heartbeat from core client for 30 sec - exiting 18:37:09 (6768): No heartbeat from core client for 30 sec - exiting 18:37:10 (6768): No heartbeat from core client for 30 sec - exiting 18:37:11 (6768): No heartbeat from core client for 30 sec - exiting 18:37:12 (6768): No heartbeat from core client for 30 sec - exiting 18:37:13 (6768): No heartbeat from core client for 30 sec - exiting 18:37:14 (6768): No heartbeat from core client for 30 sec - exiting 18:37:15 (6768): No heartbeat from core client for 30 sec - exiting 18:37:16 (6768): No heartbeat from core client for 30 sec - exiting 18:37:17 (6768): No heartbeat from core client for 30 sec - exiting 18:37:18 (6768): No heartbeat from core client for 30 sec - exiting 18:37:19 (6768): No heartbeat from core client for 30 sec - exiting 18:37:20 (6768): No heartbeat from core client for 30 sec - exiting 18:37:21 (6768): No heartbeat from core client for 30 sec - exiting 18:37:22 (6768): No heartbeat from core client for 30 sec - exiting 18:37:23 (6768): No heartbeat from core client for 30 sec - exiting 18:37:24 (6768): No heartbeat from core client for 30 sec - exiting 18:37:25 (6768): No heartbeat from core client for 30 sec - exiting 18:37:26 (6768): No heartbeat from core client for 30 sec - exiting 18:37:27 (6768): No heartbeat from core client for 30 sec - exiting 18:37:28 (6768): No heartbeat from core client for 30 sec - exiting 18:37:29 (6768): No heartbeat from core client for 30 sec - exiting 18:37:30 (6768): No heartbeat from core client for 30 sec - exiting 18:37:31 (6768): No heartbeat from core client for 30 sec - exiting 18:37:32 (6768): No heartbeat from core client for 30 sec - exiting 18:37:33 (6768): No heartbeat from core client for 30 sec - exiting 18:37:34 (6768): No heartbeat from core client for 30 sec - exiting 18:37:35 (6768): No heartbeat from core client for 30 sec - exiting 18:37:36 (6768): No heartbeat from core client for 30 sec - exiting 18:37:37 (6768): No heartbeat from core client for 30 sec - exiting 18:37:38 (6768): No heartbeat from core client for 30 sec - exiting 18:37:39 (6768): No heartbeat from core client for 30 sec - exiting 18:37:40 (6768): No heartbeat from core client for 30 sec - exiting 18:37:41 (6768): No heartbeat from core client for 30 sec - exiting 18:37:42 (6768): No heartbeat from core client for 30 sec - exiting 18:37:43 (6768): No heartbeat from core client for 30 sec - exiting 18:37:44 (6768): No heartbeat from core client for 30 sec - exiting 18:37:45 (6768): No heartbeat from core client for 30 sec - exiting 18:37:46 (6768): No heartbeat from core client for 30 sec - exiting 18:37:47 (6768): No heartbeat from core client for 30 sec - exiting 18:37:48 (6768): No heartbeat from core client for 30 sec - exiting 18:37:49 (6768): No heartbeat from core client for 30 sec - exiting 18:37:50 (6768): No heartbeat from core client for 30 sec - exiting 18:37:51 (6768): No heartbeat from core client for 30 sec - exiting 18:37:52 (6768): No heartbeat from core client for 30 sec - exiting 18:37:53 (6768): No heartbeat from core client for 30 sec - exiting 18:37:54 (6768): No heartbeat from core client for 30 sec - exiting 18:37:55 (6768): No heartbeat from core client for 30 sec - exiting 18:37:56 (6768): No heartbeat from core client for 30 sec - exiting 18:37:57 (6768): No heartbeat from core client for 30 sec - exiting 18:37:58 (6768): No heartbeat from core client for 30 sec - exiting 18:37:59 (6768): No heartbeat from core client for 30 sec - exiting 18:38:00 (6768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8632, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7656, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7656, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 07:44:50 (9776): No heartbeat from core client for 30 sec - exiting 07:44:51 (9776): No heartbeat from core client for 30 sec - exiting 07:44:52 (9776): No heartbeat from core client for 30 sec - exiting 07:44:53 (9776): No heartbeat from core client for 30 sec - exiting 07:44:54 (9776): No heartbeat from core client for 30 sec - exiting 07:44:55 (9776): No heartbeat from core client for 30 sec - exiting 07:44:56 (9776): No heartbeat from core client for 30 sec - exiting 07:44:57 (9776): No heartbeat from core client for 30 sec - exiting 07:44:58 (9776): No heartbeat from core client for 30 sec - exiting 07:44:59 (9776): No heartbeat from core client for 30 sec - exiting 07:45:00 (9776): No heartbeat from core client for 30 sec - exiting 07:45:01 (9776): No heartbeat from core client for 30 sec - exiting 07:45:02 (9776): No heartbeat from core client for 30 sec - exiting 07:45:03 (9776): No heartbeat from core client for 30 sec - exiting 07:45:04 (9776): No heartbeat from core client for 30 sec - exiting 07:45:05 (9776): No heartbeat from core client for 30 sec - exiting 07:45:06 (9776): No heartbeat from core client for 30 sec - exiting 07:45:07 (9776): No heartbeat from core client for 30 sec - exiting 07:45:08 (9776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10200, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10200, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Mar 2013 14:35:42	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	699,840	915,479	1.3081
01 Mar 2013 02:25:58	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	673,920	881,771	1.3084
28 Feb 2013 12:18:34	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	648,000	847,225	1.3074
28 Feb 2013 00:40:35	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	622,080	813,904	1.3084
27 Feb 2013 08:36:47	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	596,160	780,742	1.3096
26 Feb 2013 21:07:58	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	570,240	746,361	1.3089
26 Feb 2013 09:25:32	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	544,320	712,027	1.3081
25 Feb 2013 22:16:33	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	518,400	678,115	1.3081
25 Feb 2013 10:20:17	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	492,480	644,379	1.3084
22 Feb 2013 21:18:53	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	466,560	610,405	1.3083
22 Feb 2013 08:52:01	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	440,640	575,800	1.3067
21 Feb 2013 21:23:46	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	414,720	541,964	1.3068
21 Feb 2013 08:54:41	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	388,800	507,913	1.3064
20 Feb 2013 21:21:29	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	362,880	474,002	1.3062
20 Feb 2013 08:12:59	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	336,960	439,765	1.3051
19 Feb 2013 19:57:53	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	311,040	405,341	1.3032
19 Feb 2013 07:53:05	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	285,120	371,462	1.3028
18 Feb 2013 19:42:34	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	259,200	337,519	1.3022
18 Feb 2013 07:07:57	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	233,280	303,859	1.3026
17 Feb 2013 18:31:41	968211	15548874	hadcm3n_n7n9_1880_40_008286096_0	207,360	269,954	1.3019