Task 15700478

Name	hadcm3n_3hez_2020_40_008335337_3
Workunit	8486198
Created	3 Apr 2013, 5:16:22 UTC
Sent	3 Apr 2013, 5:16:29 UTC
Report deadline	3 Jul 2013, 12:43:40 UTC
Received	31 May 2013, 2:50:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1099480
Run time	20 days 5 hours 41 min 15 sec
CPU time	18 days 4 hours 5 min 30 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	2.73 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4412, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4412, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4248, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4248, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5616, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5000, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6020, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:18:00 (5972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:02 (5972): No heartbeat from core client for 30 sec - exiting 23:18:03 (5972): No heartbeat from core client for 30 sec - exiting 23:18:04 (5972): No heartbeat from core client for 30 sec - exiting 23:18:05 (5972): No heartbeat from core client for 30 sec - exiting 23:18:06 (5972): No heartbeat from core client for 30 sec - exiting 23:18:07 (5972): No heartbeat from core client for 30 sec - exiting 23:18:08 (5972): No heartbeat from core client for 30 sec - exiting 23:18:09 (5972): No heartbeat from core client for 30 sec - exiting 23:18:10 (5972): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5980, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5704, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6068, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6068, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6068, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5508, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 May 2013 01:35:26	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	907,200	1,544,598	1.7026
28 May 2013 11:29:41	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	881,280	1,496,809	1.6984
27 May 2013 20:37:52	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	855,360	1,448,326	1.6932
27 May 2013 05:32:50	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	829,440	1,400,098	1.6880
26 May 2013 14:01:42	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	803,520	1,351,446	1.6819
25 May 2013 23:42:17	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	777,600	1,303,196	1.6759
25 May 2013 09:22:20	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	751,680	1,255,239	1.6699
24 May 2013 19:04:20	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	725,760	1,207,374	1.6636
24 May 2013 04:06:19	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	699,840	1,159,129	1.6563
23 May 2013 13:24:40	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	673,920	1,110,980	1.6485
22 May 2013 06:52:34	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	648,000	1,062,788	1.6401
21 May 2013 04:01:14	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	622,080	1,015,353	1.6322
18 May 2013 14:25:52	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	596,160	967,933	1.6236
15 May 2013 06:09:35	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	570,240	921,335	1.6157
12 May 2013 07:00:28	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	544,320	875,827	1.6090
10 May 2013 03:04:36	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	518,400	829,273	1.5997
07 May 2013 05:00:37	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	492,480	789,845	1.6038
05 May 2013 08:04:46	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	466,560	751,068	1.6098
04 May 2013 21:11:30	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	440,640	712,192	1.6163
01 May 2013 09:34:08	1099480	15700478	hadcm3n_3hez_2020_40_008335337_3	414,720	672,062	1.6205