Task 12747710

Name	hadcm3n_o63x_1900_40_007203248_0
Workunit	7401528
Created	28 Mar 2011, 14:15:57 UTC
Sent	29 Mar 2011, 9:54:42 UTC
Report deadline	28 Jun 2011, 17:21:53 UTC
Received	4 Apr 2011, 11:01:13 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1098635
Run time	5 days 8 hours 12 min 46 sec
CPU time	4 days 17 hours 54 min 1 sec
Validate state	Invalid
Credit	2,488.32
Device peak FLOPS	2.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:47:14 (768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:16 (768): No heartbeat from core client for 30 sec - exiting 17:47:17 (768): No heartbeat from core client for 30 sec - exiting 17:47:18 (768): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:50:04 (3432): No heartbeat from core client for 30 sec - exiting 21:50:05 (3432): No heartbeat from core client for 30 sec - exiting 21:50:07 (3432): No heartbeat from core client for 30 sec - exiting 21:50:08 (3432): No heartbeat from core client for 30 sec - exiting 21:50:09 (3432): No heartbeat from core client for 30 sec - exiting 21:50:10 (3432): No heartbeat from core client for 30 sec - exiting 21:50:11 (3432): No heartbeat from core client for 30 sec - exiting 21:50:12 (3432): No heartbeat from core client for 30 sec - exiting 21:50:13 (3432): No heartbeat from core client for 30 sec - exiting 21:50:14 (3432): No heartbeat from core client for 30 sec - exiting 21:50:15 (3432): No heartbeat from core client for 30 sec - exiting 21:50:16 (3432): No heartbeat from core client for 30 sec - exiting 21:50:18 (3432): No heartbeat from core client for 30 sec - exiting 21:50:19 (3432): No heartbeat from core client for 30 sec - exiting 21:50:20 (3432): No heartbeat from core client for 30 sec - exiting 21:50:21 (3432): No heartbeat from core client for 30 sec - exiting 21:50:22 (3432): No heartbeat from core client for 30 sec - exiting 21:50:23 (3432): No heartbeat from core client for 30 sec - exiting 21:50:24 (3432): No heartbeat from core client for 30 sec - exiting 21:50:25 (3432): No heartbeat from core client for 30 sec - exiting 21:50:26 (3432): No heartbeat from core client for 30 sec - exiting 21:50:27 (3432): No heartbeat from core client for 30 sec - exiting 21:50:28 (3432): No heartbeat from core client for 30 sec - exiting 21:50:30 (3432): No heartbeat from core client for 30 sec - exiting 21:50:31 (3432): No heartbeat from core client for 30 sec - exiting 21:50:32 (3432): No heartbeat from core client for 30 sec - exiting 21:50:33 (3432): No heartbeat from core client for 30 sec - exiting 21:50:34 (3432): No heartbeat from core client for 30 sec - exiting 21:50:35 (3432): No heartbeat from core client for 30 sec - exiting 21:50:36 (3432): No heartbeat from core client for 30 sec - exiting 21:50:37 (3432): No heartbeat from core client for 30 sec - exiting 21:50:38 (3432): No heartbeat from core client for 30 sec - exiting 21:50:39 (3432): No heartbeat from core client for 30 sec - exiting 21:50:40 (3432): No heartbeat from core client for 30 sec - exiting 21:50:42 (3432): No heartbeat from core client for 30 sec - exiting 21:50:43 (3432): No heartbeat from core client for 30 sec - exiting 21:50:44 (3432): No heartbeat from core client for 30 sec - exiting 21:50:45 (3432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:50:46 (3432): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:09:36 (3412): No heartbeat from core client for 30 sec - exiting 02:09:37 (3412): No heartbeat from core client for 30 sec - exiting 02:09:38 (3412): No heartbeat from core client for 30 sec - exiting 02:09:39 (3412): No heartbeat from core client for 30 sec - exiting 02:09:40 (3412): No heartbeat from core client for 30 sec - exiting 02:09:41 (3412): No heartbeat from core client for 30 sec - exiting 02:09:42 (3412): No heartbeat from core client for 30 sec - exiting 02:09:43 (3412): No heartbeat from core client for 30 sec - exiting 02:09:44 (3412): No heartbeat from core client for 30 sec - exiting 02:09:46 (3412): No heartbeat from core client for 30 sec - exiting 02:09:47 (3412): No heartbeat from core client for 30 sec - exiting 02:09:48 (3412): No heartbeat from core client for 30 sec - exiting 02:09:49 (3412): No heartbeat from core client for 30 sec - exiting 02:09:50 (3412): No heartbeat from core client for 30 sec - exiting 02:09:51 (3412): No heartbeat from core client for 30 sec - exiting 02:09:52 (3412): No heartbeat from core client for 30 sec - exiting 02:09:53 (3412): No heartbeat from core client for 30 sec - exiting 02:09:54 (3412): No heartbeat from core client for 30 sec - exiting 02:09:55 (3412): No heartbeat from core client for 30 sec - exiting 02:09:56 (3412): No heartbeat from core client for 30 sec - exiting 02:09:58 (3412): No heartbeat from core client for 30 sec - exiting 02:09:59 (3412): No heartbeat from core client for 30 sec - exiting 02:10:00 (3412): No heartbeat from core client for 30 sec - exiting 02:10:01 (3412): No heartbeat from core client for 30 sec - exiting 02:10:02 (3412): No heartbeat from core client for 30 sec - exiting 02:10:03 (3412): No heartbeat from core client for 30 sec - exiting 02:10:04 (3412): No heartbeat from core client for 30 sec - exiting 02:10:05 (3412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:10:06 (3412): No heartbeat from core client for 30 sec - exiting 02:10:07 (3412): No heartbeat from core client for 30 sec - exiting 02:10:09 (3412): No heartbeat from core client for 30 sec - exiting 02:10:10 (3412): No heartbeat from core client for 30 sec - exiting 02:10:11 (3412): No heartbeat from core client for 30 sec - exiting 02:10:12 (3412): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:50:09 (3476): No heartbeat from core client for 30 sec - exiting 10:50:11 (3476): No heartbeat from core client for 30 sec - exiting 10:50:12 (3476): No heartbeat from core client for 30 sec - exiting 10:50:13 (3476): No heartbeat from core client for 30 sec - exiting 10:50:14 (3476): No heartbeat from core client for 30 sec - exiting 10:50:15 (3476): No heartbeat from core client for 30 sec - exiting 10:50:16 (3476): No heartbeat from core client for 30 sec - exiting 10:50:17 (3476): No heartbeat from core client for 30 sec - exiting 10:50:18 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4020, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Apr 2011 20:07:23	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	207,360	371,065	1.7895
03 Apr 2011 05:13:59	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	181,440	323,034	1.7804
02 Apr 2011 14:05:11	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	155,520	277,592	1.7849
01 Apr 2011 21:41:00	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	129,600	231,367	1.7852
01 Apr 2011 07:24:01	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	103,680	185,615	1.7903
31 Mar 2011 14:39:36	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	77,760	139,525	1.7943
31 Mar 2011 00:35:16	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	51,840	93,751	1.8085
30 Mar 2011 03:06:06	1098635	12747710	hadcm3n_o63x_1900_40_007203248_0	25,920	47,249	1.8229