Task 13324964

Name	hadcm3n_p15k_1940_40_007419647_2
Workunit	7617282
Created	1 Sep 2011, 18:11:09 UTC
Sent	2 Sep 2011, 5:57:54 UTC
Report deadline	2 Dec 2011, 13:25:05 UTC
Received	7 Oct 2011, 6:01:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1110645
Run time	11 days 22 hours 50 min 23 sec
CPU time	10 days 23 hours 35 min 13 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.8.23</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:51:15 (1396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7056, selfPID=7056, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:30:54 (8716): No heartbeat from core client for 30 sec - exiting 19:30:55 (8716): No heartbeat from core client for 30 sec - exiting 19:30:56 (8716): No heartbeat from core client for 30 sec - exiting 19:30:57 (8716): No heartbeat from core client for 30 sec - exiting 19:30:58 (8716): No heartbeat from core client for 30 sec - exiting 19:30:59 (8716): No heartbeat from core client for 30 sec - exiting 19:31:00 (8716): No heartbeat from core client for 30 sec - exiting 19:31:01 (8716): No heartbeat from core client for 30 sec - exiting 19:31:02 (8716): No heartbeat from core client for 30 sec - exiting 19:31:03 (8716): No heartbeat from core client for 30 sec - exiting 19:31:04 (8716): No heartbeat from core client for 30 sec - exiting 19:31:05 (8716): No heartbeat from core client for 30 sec - exiting 19:31:06 (8716): No heartbeat from core client for 30 sec - exiting 19:31:07 (8716): No heartbeat from core client for 30 sec - exiting 19:31:08 (8716): No heartbeat from core client for 30 sec - exiting 19:31:09 (8716): No heartbeat from core client for 30 sec - exiting 19:31:10 (8716): No heartbeat from core client for 30 sec - exiting 19:31:11 (8716): No heartbeat from core client for 30 sec - exiting 19:31:12 (8716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:42:27 (5088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:43:09 (12896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:44:03 (11752): No heartbeat from core client for 30 sec - exiting 20:44:04 (11752): No heartbeat from core client for 30 sec - exiting 20:44:05 (11752): No heartbeat from core client for 30 sec - exiting 20:44:06 (11752): No heartbeat from core client for 30 sec - exiting 20:44:07 (11752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:44:08 (11752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:32:53 (8024): No heartbeat from core client for 30 sec - exiting 23:32:55 (8024): No heartbeat from core client for 30 sec - exiting 23:32:56 (8024): No heartbeat from core client for 30 sec - exiting 23:32:57 (8024): No heartbeat from core client for 30 sec - exiting 23:32:58 (8024): No heartbeat from core client for 30 sec - exiting 23:32:59 (8024): No heartbeat from core client for 30 sec - exiting 23:33:00 (8024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:30:20 (9384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:31:05 (7372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9732, selfPID=9732, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:18:30 (10812): No heartbeat from core client for 30 sec - exiting 10:18:31 (10812): No heartbeat from core client for 30 sec - exiting 10:18:32 (10812): No heartbeat from core client for 30 sec - exiting 10:18:33 (10812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=868, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:57:21 (32796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38524, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38524, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38524, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38524, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38524, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=38524, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Oct 2011 01:42:39	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	492,480	941,694	1.9121
06 Oct 2011 10:40:15	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	466,560	891,346	1.9105
05 Oct 2011 20:15:38	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	440,640	841,181	1.9090
05 Oct 2011 03:34:47	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	414,720	791,754	1.9091
04 Oct 2011 14:22:03	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	388,800	745,509	1.9175
04 Oct 2011 00:13:40	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	362,880	696,632	1.9197
03 Oct 2011 09:45:17	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	336,960	647,166	1.9206
02 Oct 2011 16:22:46	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	311,040	597,795	1.9219
02 Oct 2011 01:30:02	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	285,120	548,407	1.9234
01 Oct 2011 11:00:23	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	259,200	500,040	1.9292
30 Sep 2011 21:22:28	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	233,280	452,731	1.9407
30 Sep 2011 05:38:21	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	207,360	403,843	1.9475
29 Sep 2011 12:59:27	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	181,440	354,498	1.9538
28 Sep 2011 01:28:54	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	155,520	304,360	1.9570
26 Sep 2011 13:38:00	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	129,600	254,841	1.9664
22 Sep 2011 02:08:27	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	103,680	204,650	1.9739
20 Sep 2011 04:34:51	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	77,760	153,789	1.9777
17 Sep 2011 05:08:35	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	51,840	102,549	1.9782
15 Sep 2011 07:17:59	1110645	13324964	hadcm3n_p15k_1940_40_007419647_2	25,920	51,729	1.9957