Task 13018604

Name	hadcm3n_t5ts_1940_40_007313404_1
Workunit	7510834
Created	28 Jun 2011, 7:20:29 UTC
Sent	28 Jun 2011, 7:28:52 UTC
Report deadline	27 Sep 2011, 14:56:03 UTC
Received	29 Aug 2011, 19:46:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1089890
Run time	8 days 23 hours 31 min 36 sec
CPU time	8 days 17 hours 47 min 4 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:45:01 (8328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:03 (8328): No heartbeat from core client for 30 sec - exiting 08:45:04 (8328): No heartbeat from core client for 30 sec - exiting 08:45:05 (8328): No heartbeat from core client for 30 sec - exiting 08:45:06 (8328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 08:54:46 (2344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:16:21 (2352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:12:47 (2324): No heartbeat from core client for 30 sec - exiting 14:12:49 (2324): No heartbeat from core client for 30 sec - exiting 14:12:50 (2324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:40:56 (1824): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 14:40:59 (1824): No heartbeat from core client for 30 sec - exiting 14:41:00 (1824): No heartbeat from core client for 30 sec - exiting 14:41:01 (1824): No heartbeat from core client for 30 sec - exiting 14:41:02 (1824): No heartbeat from core client for 30 sec - exiting 14:41:03 (1824): No heartbeat from core client for 30 sec - exiting 14:41:04 (1824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 04:04:34 (2448): No heartbeat from core client for 30 sec - exiting 04:04:35 (2448): No heartbeat from core client for 30 sec - exiting 04:04:36 (2448): No heartbeat from core client for 30 sec - exiting 04:04:37 (2448): No heartbeat from core client for 30 sec - exiting 04:04:38 (2448): No heartbeat from core client for 30 sec - exiting 04:04:39 (2448): No heartbeat from core client for 30 sec - exiting 04:04:40 (2448): No heartbeat from core client for 30 sec - exiting 04:04:41 (2448): No heartbeat from core client for 30 sec - exiting 04:04:42 (2448): No heartbeat from core client for 30 sec - exiting 04:04:43 (2448): No heartbeat from core client for 30 sec - exiting 04:04:44 (2448): No heartbeat from core client for 30 sec - exiting 04:04:45 (2448): No heartbeat from core client for 30 sec - exiting 04:04:46 (2448): No heartbeat from core client for 30 sec - exiting 04:04:47 (2448): No heartbeat from core client for 30 sec - exiting 04:04:48 (2448): No heartbeat from core client for 30 sec - exiting 04:04:49 (2448): No heartbeat from core client for 30 sec - exiting 04:04:50 (2448): No heartbeat from core client for 30 sec - exiting 04:04:51 (2448): No heartbeat from core client for 30 sec - exiting 04:04:52 (2448): No heartbeat from core client for 30 sec - exiting 04:04:53 (2448): No heartbeat from core client for 30 sec - exiting 04:04:54 (2448): No heartbeat from core client for 30 sec - exiting 04:04:55 (2448): No heartbeat from core client for 30 sec - exiting 04:04:56 (2448): No heartbeat from core client for 30 sec - exiting 04:04:57 (2448): No heartbeat from core client for 30 sec - exiting 04:04:58 (2448): No heartbeat from core client for 30 sec - exiting 04:04:59 (2448): No heartbeat from core client for 30 sec - exiting 04:05:00 (2448): No heartbeat from core client for 30 sec - exiting 04:05:01 (2448): No heartbeat from core client for 30 sec - exiting 04:05:02 (2448): No heartbeat from core client for 30 sec - exiting 04:05:03 (2448): No heartbeat from core client for 30 sec - exiting 04:05:04 (2448): No heartbeat from core client for 30 sec - exiting 04:05:05 (2448): No heartbeat from core client for 30 sec - exiting 04:05:06 (2448): No heartbeat from core client for 30 sec - exiting 04:05:07 (2448): No heartbeat from core client for 30 sec - exiting 04:05:08 (2448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:09 (2448): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6500, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Aug 2011 23:53:13	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	492,480	721,687	1.4654
20 Aug 2011 05:24:10	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	466,560	685,050	1.4683
19 Aug 2011 19:24:31	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	440,640	649,181	1.4733
19 Aug 2011 09:09:34	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	414,720	613,342	1.4789
17 Aug 2011 18:25:13	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	388,800	575,643	1.4806
16 Aug 2011 11:50:03	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	362,880	538,297	1.4834
15 Aug 2011 00:33:53	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	336,960	501,059	1.4870
12 Aug 2011 16:56:29	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	311,040	463,288	1.4895
11 Aug 2011 18:06:29	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	285,120	426,822	1.4970
10 Aug 2011 11:30:44	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	259,200	390,853	1.5079
09 Aug 2011 05:23:15	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	233,280	354,386	1.5191
08 Aug 2011 02:44:07	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	207,360	318,023	1.5337
02 Jul 2011 13:43:46	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	181,440	281,560	1.5518
02 Jul 2011 02:01:35	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	155,520	240,464	1.5462
01 Jul 2011 14:11:40	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	129,600	199,213	1.5371
30 Jun 2011 18:56:46	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	103,680	158,788	1.5315
30 Jun 2011 07:39:07	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	77,760	118,637	1.5257
29 Jun 2011 20:09:27	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	51,840	78,692	1.5180
29 Jun 2011 08:28:34	1089890	13018604	hadcm3n_t5ts_1940_40_007313404_1	25,920	38,840	1.4985