Task 13302169

Name	hadcm3n_o05g_1980_40_007425757_1
Workunit	7623260
Created	26 Aug 2011, 18:53:15 UTC
Sent	26 Aug 2011, 18:53:18 UTC
Report deadline	26 Nov 2011, 2:20:29 UTC
Received	26 Oct 2011, 1:00:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1157770
Run time	24 days 4 hours 46 min 12 sec
CPU time	22 days 18 hours 58 min 9 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	1.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:10:27 (5528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3272, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:12:02 (4796): No heartbeat from core client for 30 sec - exiting 21:12:03 (4796): No heartbeat from core client for 30 sec - exiting 21:12:05 (4796): No heartbeat from core client for 30 sec - exiting 21:12:06 (4796): No heartbeat from core client for 30 sec - exiting 21:12:07 (4796): No heartbeat from core client for 30 sec - exiting 21:12:08 (4796): No heartbeat from core client for 30 sec - exiting 21:12:10 (4796): No heartbeat from core client for 30 sec - exiting 21:12:11 (4796): No heartbeat from core client for 30 sec - exiting 21:12:12 (4796): No heartbeat from core client for 30 sec - exiting 21:12:13 (4796): No heartbeat from core client for 30 sec - exiting 21:12:15 (4796): No heartbeat from core client for 30 sec - exiting 21:12:16 (4796): No heartbeat from core client for 30 sec - exiting 21:12:17 (4796): No heartbeat from core client for 30 sec - exiting 21:12:19 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:12:20 (4796): No heartbeat from core client for 30 sec - exiting 21:13:10 (5452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:13:49 (5452): No heartbeat from core client for 30 sec - exiting 21:19:41 (5696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:19:42 (5696): No heartbeat from core client for 30 sec - exiting 21:20:28 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:20:34 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5336, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5336, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5336, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4100, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4100, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4100, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Oct 2011 04:17:29	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	673,920	1,912,843	2.8384
04 Oct 2011 04:17:29	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	648,000	1,841,619	2.8420
04 Oct 2011 04:17:29	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	622,080	1,770,397	2.8459
04 Oct 2011 04:17:29	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	596,160	1,698,361	2.8488
30 Sep 2011 01:44:32	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	570,240	1,623,691	2.8474
28 Sep 2011 19:36:28	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	544,320	1,546,534	2.8412
28 Sep 2011 04:00:42	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	518,400	1,472,638	2.8407
28 Sep 2011 04:00:42	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	492,480	1,400,163	2.8431
28 Sep 2011 04:00:42	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	466,560	1,327,353	2.8450
25 Sep 2011 04:02:30	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	440,640	1,253,851	2.8455
23 Sep 2011 05:29:10	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	414,720	1,179,080	2.8431
21 Sep 2011 19:43:11	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	388,800	1,105,459	2.8433
20 Sep 2011 17:50:38	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	362,880	1,032,097	2.8442
18 Sep 2011 21:18:49	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	336,960	958,635	2.8450
18 Sep 2011 04:22:18	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	311,040	885,067	2.8455
16 Sep 2011 23:00:21	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	285,120	811,774	2.8471
16 Sep 2011 08:10:35	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	259,200	736,900	2.8430
15 Sep 2011 11:55:46	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	233,280	663,688	2.8450
13 Sep 2011 20:11:02	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	207,360	589,122	2.8411
11 Sep 2011 17:08:33	1157770	13302169	hadcm3n_o05g_1980_40_007425757_1	181,440	515,670	2.8421