Task 14848046

Name	hadcm3n_zk9i_1880_40_008026395_1
Workunit	8181509
Created	29 Jun 2012, 16:41:52 UTC
Sent	29 Jun 2012, 16:42:02 UTC
Report deadline	29 Sep 2012, 0:09:13 UTC
Received	19 Aug 2012, 8:59:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1118216
Run time	21 days 18 hours 1 min 58 sec
CPU time	20 days 9 hours 29 min 58 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	2.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5628, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:37:11 (2428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:37:13 (2428): No heartbeat from core client for 30 sec - exiting 08:37:14 (2428): No heartbeat from core client for 30 sec - exiting 08:37:15 (2428): No heartbeat from core client for 30 sec - exiting 08:37:16 (2428): No heartbeat from core client for 30 sec - exiting 08:37:17 (2428): No heartbeat from core client for 30 sec - exiting 08:37:18 (2428): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1680, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:48:12 (5688): No heartbeat from core client for 30 sec - exiting 09:48:13 (5688): No heartbeat from core client for 30 sec - exiting 09:48:14 (5688): No heartbeat from core client for 30 sec - exiting 09:48:15 (5688): No heartbeat from core client for 30 sec - exiting 09:48:16 (5688): No heartbeat from core client for 30 sec - exiting 09:48:17 (5688): No heartbeat from core client for 30 sec - exiting 09:48:18 (5688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:01:24 (5704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:01:26 (5704): No heartbeat from core client for 30 sec - exiting 11:01:27 (5704): No heartbeat from core client for 30 sec - exiting 11:01:28 (5704): No heartbeat from core client for 30 sec - exiting 11:01:29 (5704): No heartbeat from core client for 30 sec - exiting 11:01:30 (5704): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... C07:13:02 (3856): No heartbeat from core client for 30 sec - exiting 07:13:03 (3856): No heartbeat from core client for 30 sec - exiting 07:13:04 (3856): No heartbeat from core client for 30 sec - exiting 07:13:05 (3856): No heartbeat from core client for 30 sec - exiting 07:13:06 (3856): No heartbeat from core client for 30 sec - exiting 07:13:07 (3856): No heartbeat from core client for 30 sec - exiting 07:13:08 (3856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8124, iMonCtr=1 Model crash detected, will try to restart... Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Aug 2012 17:19:47	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	1,010,880	1,796,208	1.7769
17 Aug 2012 18:38:50	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	984,960	1,749,318	1.7760
16 Aug 2012 13:26:52	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	959,040	1,702,008	1.7747
15 Aug 2012 10:45:24	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	933,120	1,654,264	1.7728
14 Aug 2012 10:56:53	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	907,200	1,606,845	1.7712
12 Aug 2012 17:53:48	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	881,280	1,559,315	1.7694
11 Aug 2012 21:32:44	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	855,360	1,511,892	1.7676
11 Aug 2012 07:46:43	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	829,440	1,465,152	1.7664
10 Aug 2012 09:16:51	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	803,520	1,419,297	1.7663
09 Aug 2012 13:52:16	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	777,600	1,374,593	1.7677
07 Aug 2012 09:49:31	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	751,680	1,330,292	1.7698
05 Aug 2012 09:41:43	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	725,760	1,287,554	1.7741
04 Aug 2012 08:40:31	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	699,840	1,244,330	1.7780
01 Aug 2012 02:14:13	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	673,920	1,200,572	1.7815
31 Jul 2012 12:23:19	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	648,000	1,153,141	1.7795
28 Jul 2012 14:35:12	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	622,080	1,105,212	1.7766
27 Jul 2012 16:30:20	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	596,160	1,057,405	1.7737
26 Jul 2012 16:22:23	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	570,240	1,010,333	1.7718
25 Jul 2012 16:45:14	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	544,320	962,609	1.7685
24 Jul 2012 16:21:14	1118216	14848046	hadcm3n_zk9i_1880_40_008026395_1	518,400	914,532	1.7641