Task 12737674

Name	hadcm3n_o28x_1900_40_007198244_0
Workunit	7396524
Created	28 Mar 2011, 14:02:57 UTC
Sent	1 Apr 2011, 5:03:14 UTC
Report deadline	1 Jul 2011, 12:30:25 UTC
Received	20 Apr 2011, 15:34:07 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1131576
Run time	6 days 22 hours 30 min 22 sec
CPU time	6 days 13 hours 10 min 48 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4832, selfPID=4832, iMonCtr=1 CPDN Monitor - Quit request from BOINC... 01:00:34 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:35 (6592): No heartbeat from core client for 30 sec - exiting 01:00:36 (6592): No heartbeat from core client for 30 sec - exiting 01:00:37 (6592): No heartbeat from core client for 30 sec - exiting 01:00:38 (6592): No heartbeat from core client for 30 sec - exiting 01:00:39 (6592): No heartbeat from core client for 30 sec - exiting 01:00:40 (6592): No heartbeat from core client for 30 sec - exiting 01:00:41 (6592): No heartbeat from core client for 30 sec - exiting 01:00:43 (6592): No heartbeat from core client for 30 sec - exiting 01:00:44 (6592): No heartbeat from core client for 30 sec - exiting 01:00:45 (6592): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5656, selfPID=5656, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4556, selfPID=4556, iMonCtr=1 01:00:35 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:36 (4768): No heartbeat from core client for 30 sec - exiting 01:00:37 (4768): No heartbeat from core client for 30 sec - exiting 01:00:38 (4768): No heartbeat from core client for 30 sec - exiting 01:00:39 (4768): No heartbeat from core client for 30 sec - exiting 01:00:40 (4768): No heartbeat from core client for 30 sec - exiting 01:00:41 (4768): No heartbeat from core client for 30 sec - exiting 01:00:42 (4768): No heartbeat from core client for 30 sec - exiting 01:00:44 (4768): No heartbeat from core client for 30 sec - exiting 01:00:45 (4768): No heartbeat from core client for 30 sec - exiting 01:00:46 (4768): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2148, selfPID=2148, iMonCtr=1 CCPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4104, selfPID=4104, iMonCtr=1 01:00:32 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:33 (5652): No heartbeat from core client for 30 sec - exiting 01:00:34 (5652): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3308, selfPID=3308, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5148, selfPID=5148, iMonCtr=1 01:00:34 (3748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:35 (3748): No heartbeat from core client for 30 sec - exiting 01:00:36 (3748): No heartbeat from core client for 30 sec - exiting 01:00:37 (3748): No heartbeat from core client for 30 sec - exiting 01:00:39 (3748): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4440, selfPID=4440, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2920, selfPID=2920, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5496, selfPID=5496, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2376, selfPID=2376, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5732, selfPID=5732, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3280, selfPID=3280, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6804, selfPID=6804, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3280, selfPID=3280, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4360, selfPID=4360, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3088, selfPID=3088, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6968, selfPID=6968, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8188, selfPID=8188, iMonCtr=1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8132, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5300, selfPID=5300, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5808, selfPID=5808, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3960, selfPID=3960, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5680, selfPID=5680, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4580, selfPID=4580, iMonCtr=1 01:00:27 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4188, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Atmos Hold Restart file rename failed on atmos_restart.hold Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Apr 2011 15:40:13	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	311,040	571,971	1.8389
20 Apr 2011 15:40:13	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	285,120	524,240	1.8387
13 Apr 2011 02:02:52	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	259,200	476,721	1.8392
11 Apr 2011 11:17:47	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	233,280	428,805	1.8382
10 Apr 2011 07:41:46	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	207,360	380,772	1.8363
09 Apr 2011 04:34:49	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	181,440	331,055	1.8246
08 Apr 2011 02:51:32	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	155,520	282,767	1.8182
07 Apr 2011 01:59:54	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	129,600	236,818	1.8273
06 Apr 2011 00:46:27	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	103,680	189,541	1.8281
04 Apr 2011 11:43:47	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	77,760	141,327	1.8175
03 Apr 2011 08:50:07	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	51,840	93,597	1.8055
02 Apr 2011 07:31:22	1131576	12737674	hadcm3n_o28x_1900_40_007198244_0	25,920	46,160	1.7809