Task 12757713

Name	hadcm3n_o4ng_1900_40_007201359_2
Workunit	7399639
Created	30 Mar 2011, 21:58:32 UTC
Sent	30 Mar 2011, 22:01:42 UTC
Report deadline	30 Jun 2011, 5:28:53 UTC
Received	3 May 2011, 20:28:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1114586
Run time	17 days 22 hours 45 min 10 sec
CPU time	16 days 3 hours 53 min 39 sec
Validate state	Invalid
Credit	9,953.28
Device peak FLOPS	2.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:37:27 (5144): No heartbeat from core client for 30 sec - exiting 20:37:28 (5144): No heartbeat from core client for 30 sec - exiting 20:37:29 (5144): No heartbeat from core client for 30 sec - exiting 20:37:30 (5144): No heartbeat from core client for 30 sec - exiting 20:37:31 (5144): No heartbeat from core client for 30 sec - exiting 20:37:32 (5144): No heartbeat from core client for 30 sec - exiting 20:37:33 (5144): No heartbeat from core client for 30 sec - exiting 20:37:34 (5144): No heartbeat from core client for 30 sec - exiting 20:37:35 (5144): No heartbeat from core client for 30 sec - exiting 20:37:36 (5144): No heartbeat from core client for 30 sec - exiting 20:37:38 (5144): No heartbeat from core client for 30 sec - exiting 20:37:39 (5144): No heartbeat from core client for 30 sec - exiting 20:37:40 (5144): No heartbeat from core client for 30 sec - exiting 20:37:41 (5144): No heartbeat from core client for 30 sec - exiting 20:37:42 (5144): No heartbeat from core client for 30 sec - exiting 20:37:43 (5144): No heartbeat from core client for 30 sec - exiting 20:37:44 (5144): No heartbeat from core client for 30 sec - exiting 20:37:45 (5144): No heartbeat from core client for 30 sec - exiting 20:37:46 (5144): No heartbeat from core client for 30 sec - exiting 20:37:47 (5144): No heartbeat from core client for 30 sec - exiting 20:37:48 (5144): No heartbeat from core client for 30 sec - exiting 20:37:50 (5144): No heartbeat from core client for 30 sec - exiting 20:37:51 (5144): No heartbeat from core client for 30 sec - exiting 20:37:52 (5144): No heartbeat from core client for 30 sec - exiting 20:37:53 (5144): No heartbeat from core client for 30 sec - exiting 20:37:54 (5144): No heartbeat from core client for 30 sec - exiting 20:37:55 (5144): No heartbeat from core client for 30 sec - exiting 20:37:56 (5144): No heartbeat from core client for 30 sec - exiting 20:37:57 (5144): No heartbeat from core client for 30 sec - exiting 20:37:58 (5144): No heartbeat from core client for 30 sec - exiting 20:37:59 (5144): No heartbeat from core client for 30 sec - exiting 20:38:00 (5144): No heartbeat from core client for 30 sec - exiting 20:38:02 (5144): No heartbeat from core client for 30 sec - exiting 20:38:03 (5144): No heartbeat from core client for 30 sec - exiting 20:38:04 (5144): No heartbeat from core client for 30 sec - exiting 20:38:05 (5144): No heartbeat from core client for 30 sec - exiting 20:38:06 (5144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:05:51 (3964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:05:52 (3964): No heartbeat from core client for 30 sec - exiting 14:06:44 (4356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:46:28 (6052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:46:29 (6052): No heartbeat from core client for 30 sec - exiting 03:46:30 (6052): No heartbeat from core client for 30 sec - exiting 10:10:32 (5416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4752, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 16:45:06 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:45:08 (5060): No heartbeat from core client for 30 sec - exiting 16:45:09 (5060): No heartbeat from core client for 30 sec - exiting 16:45:10 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:29:30 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:29:32 (2936): No heartbeat from core client for 30 sec - exiting 06:29:33 (2936): No heartbeat from core client for 30 sec - exiting 06:29:34 (2936): No heartbeat from core client for 30 sec - exiting 06:29:35 (2936): No heartbeat from core client for 30 sec - exiting 06:29:36 (2936): No heartbeat from core client for 30 sec - exiting 06:29:37 (2936): No heartbeat from core client for 30 sec - exiting 06:29:38 (2936): No heartbeat from core client for 30 sec - exiting 06:29:39 (2936): No heartbeat from core client for 30 sec - exiting 06:29:40 (2936): No heartbeat from core client for 30 sec - exiting 06:29:41 (2936): No heartbeat from core client for 30 sec - exiting 06:29:42 (2936): No heartbeat from core client for 30 sec - exiting 06:31:45 (2980): No heartbeat from core client for 30 sec - exiting 06:31:46 (2980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:31:47 (2980): No heartbeat from core client for 30 sec - exiting 06:31:49 (2980): No heartbeat from core client for 30 sec - exiting 06:31:50 (2980): No heartbeat from core client for 30 sec - exiting 06:31:51 (2980): No heartbeat from core client for 30 sec - exiting 08:14:10 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:14:11 (5916): No heartbeat from core client for 30 sec - exiting 08:14:12 (5916): No heartbeat from core client for 30 sec - exiting 09:35:39 (5312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:40 (5312): No heartbeat from core client for 30 sec - exiting 09:35:41 (5312): No heartbeat from core client for 30 sec - exiting 09:36:35 (5568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:21:07 (4212): No heartbeat from core client for 30 sec - exiting 15:21:09 (4212): No heartbeat from core client for 30 sec - exiting 15:21:10 (4212): No heartbeat from core client for 30 sec - exiting 15:21:11 (4212): No heartbeat from core client for 30 sec - exiting 15:21:12 (4212): No heartbeat from core client for 30 sec - exiting 15:21:13 (4212): No heartbeat from core client for 30 sec - exiting 15:21:14 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5548, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5548, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5548, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5548, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5548, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5548, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 May 2011 17:41:52	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	829,440	1,364,904	1.6456
02 May 2011 03:20:22	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	803,520	1,323,951	1.6477
01 May 2011 06:32:35	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	777,600	1,282,295	1.6490
30 Apr 2011 10:05:29	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	751,680	1,240,518	1.6503
29 Apr 2011 06:27:41	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	725,760	1,198,733	1.6517
28 Apr 2011 11:37:54	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	699,840	1,156,835	1.6530
27 Apr 2011 18:22:37	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	673,920	1,115,036	1.6546
27 Apr 2011 00:39:21	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	648,000	1,072,772	1.6555
26 Apr 2011 01:13:36	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	622,080	1,031,486	1.6581
25 Apr 2011 10:41:12	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	596,160	987,125	1.6558
24 Apr 2011 21:16:12	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	570,240	942,889	1.6535
24 Apr 2011 09:23:50	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	544,320	900,559	1.6545
23 Apr 2011 20:58:34	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	518,400	858,592	1.6562
23 Apr 2011 04:55:32	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	492,480	817,510	1.6600
22 Apr 2011 09:20:28	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	466,560	776,033	1.6633
21 Apr 2011 21:55:28	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	440,640	735,304	1.6687
21 Apr 2011 09:43:00	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	414,720	693,832	1.6730
20 Apr 2011 20:19:01	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	388,800	650,298	1.6726
20 Apr 2011 19:06:33	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	362,880	605,905	1.6697
20 Apr 2011 19:06:33	1114586	12757713	hadcm3n_o4ng_1900_40_007201359_2	336,960	563,490	1.6723