Task 13373498

Name	hadcm3n_ymq1_1940_40_007453048_2
Workunit	7650551
Created	10 Sep 2011, 20:21:52 UTC
Sent	10 Sep 2011, 20:31:08 UTC
Report deadline	11 Dec 2011, 3:58:19 UTC
Received	1 Oct 2011, 10:55:10 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1167875
Run time	1 days 7 hours 24 min 13 sec
CPU time	1 days 5 hours 5 min 14 sec
Validate state	Invalid
Credit	1,244.16
Device peak FLOPS	4.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4856, iMonCtr=1 Model crash detected, will try to restart... 11:49:38 (4632): No heartbeat from core client for 30 sec - exiting 11:49:39 (4632): No heartbeat from core client for 30 sec - exiting 11:49:40 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:24:06 (4700): No heartbeat from core client for 30 sec - exiting 12:24:08 (4700): No heartbeat from core client for 30 sec - exiting 12:24:09 (4700): No heartbeat from core client for 30 sec - exiting 12:24:10 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4656, iMonCtr=1 Model crash detected, will try to restart... 14:49:34 (4620): No heartbeat from core client for 30 sec - exiting 14:49:35 (4620): No heartbeat from core client for 30 sec - exiting 14:49:36 (4620): No heartbeat from core client for 30 sec - exiting 14:49:37 (4620): No heartbeat from core client for 30 sec - exiting 14:49:38 (4620): No heartbeat from core client for 30 sec - exiting 14:49:39 (4620): No heartbeat from core client for 30 sec - exiting 14:49:40 (4620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4896, iMonCtr=1 Model crash detected, will try to restart... 18:42:20 (4800): No heartbeat from core client for 30 sec - exiting 18:42:21 (4800): No heartbeat from core client for 30 sec - exiting 18:42:22 (4800): No heartbeat from core client for 30 sec - exiting 18:42:23 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5756, iMonCtr=1 Model crash detected, will try to restart... 21:33:25 (4828): No heartbeat from core client for 30 sec - exiting 21:33:26 (4828): No heartbeat from core client for 30 sec - exiting 21:33:27 (4828): No heartbeat from core client for 30 sec - exiting 21:33:28 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3172, iMonCtr=1 Model crash detected, will try to restart... 20:39:43 (4864): No heartbeat from core client for 30 sec - exiting 20:39:44 (4864): No heartbeat from core client for 30 sec - exiting 20:39:45 (4864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:51 (4824): No heartbeat from core client for 30 sec - exiting 21:33:52 (4824): No heartbeat from core client for 30 sec - exiting 21:33:53 (4824): No heartbeat from core client for 30 sec - exiting 21:33:54 (4824): No heartbeat from core client for 30 sec - exiting 21:33:55 (4824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:17:07 (4548): No heartbeat from core client for 30 sec - exiting 19:17:08 (4548): No heartbeat from core client for 30 sec - exiting 19:17:09 (4548): No heartbeat from core client for 30 sec - exiting 19:17:10 (4548): No heartbeat from core client for 30 sec - exiting 19:17:11 (4548): No heartbeat from core client for 30 sec - exiting 19:17:12 (4548): No heartbeat from core client for 30 sec - exiting 19:17:13 (4548): No heartbeat from core client for 30 sec - exiting 19:17:14 (4548): No heartbeat from core client for 30 sec - exiting 19:17:15 (4548): No heartbeat from core client for 30 sec - exiting 19:17:16 (4548): No heartbeat from core client for 30 sec - exiting 19:17:17 (4548): No heartbeat from core client for 30 sec - exiting 19:17:18 (4548): No heartbeat from core client for 30 sec - exiting 19:17:19 (4548): No heartbeat from core client for 30 sec - exiting 19:17:20 (4548): No heartbeat from core client for 30 sec - exiting 19:17:21 (4548): No heartbeat from core client for 30 sec - exiting 19:17:22 (4548): No heartbeat from core client for 30 sec - exiting 19:17:23 (4548): No heartbeat from core client for 30 sec - exiting 19:17:24 (4548): No heartbeat from core client for 30 sec - exiting 19:17:25 (4548): No heartbeat from core client for 30 sec - exiting 19:17:26 (4548): No heartbeat from core client for 30 sec - exiting 19:17:27 (4548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:23:06 (4712): No heartbeat from core client for 30 sec - exiting 22:23:07 (4712): No heartbeat from core client for 30 sec - exiting 22:23:08 (4712): No heartbeat from core client for 30 sec - exiting 22:23:09 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:13 (4464): No heartbeat from core client for 30 sec - exiting 19:08:14 (4464): No heartbeat from core client for 30 sec - exiting 19:08:15 (4464): No heartbeat from core client for 30 sec - exiting 19:08:16 (4464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:59:39 (4240): No heartbeat from core client for 30 sec - exiting 10:59:40 (4240): No heartbeat from core client for 30 sec - exiting 10:59:41 (4240): No heartbeat from core client for 30 sec - exiting 10:59:42 (4240): No heartbeat from core client for 30 sec - exiting 10:59:43 (4240): No heartbeat from core client for 30 sec - exiting 10:59:44 (4240): No heartbeat from core client for 30 sec - exiting 10:59:45 (4240): No heartbeat from core client for 30 sec - exiting 10:59:46 (4240): No heartbeat from core client for 30 sec - exiting 10:59:47 (4240): No heartbeat from core client for 30 sec - exiting 10:59:48 (4240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3904, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4888, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3872, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3860, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1976, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4836, iMonCtr=1 Model crash detected, will try to restart... 21:45:47 (4652): No heartbeat from core client for 30 sec - exiting 21:45:48 (4652): No heartbeat from core client for 30 sec - exiting 21:45:49 (4652): No heartbeat from core client for 30 sec - exiting 21:45:50 (4652): No heartbeat from core client for 30 sec - exiting 21:45:51 (4652): No heartbeat from core client for 30 sec - exiting 21:45:52 (4652): No heartbeat from core client for 30 sec - exiting 21:45:53 (4652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:46 (4568): No heartbeat from core client for 30 sec - exiting 19:00:47 (4568): No heartbeat from core client for 30 sec - exiting 19:00:48 (4568): No heartbeat from core client for 30 sec - exiting 19:00:49 (4568): No heartbeat from core client for 30 sec - exiting 19:00:50 (4568): No heartbeat from core client for 30 sec - exiting 19:00:51 (4568): No heartbeat from core client for 30 sec - exiting 19:00:52 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:30 (4756): No heartbeat from core client for 30 sec - exiting 21:18:31 (4756): No heartbeat from core client for 30 sec - exiting 21:18:32 (4756): No heartbeat from core client for 30 sec - exiting 21:18:33 (4756): No heartbeat from core client for 30 sec - exiting 21:18:34 (4756): No heartbeat from core client for 30 sec - exiting 21:18:35 (4756): No heartbeat from core client for 30 sec - exiting 21:18:36 (4756): No heartbeat from core client for 30 sec - exiting 21:18:37 (4756): No heartbeat from core client for 30 sec - exiting 21:18:38 (4756): No heartbeat from core client for 30 sec - exiting 21:18:39 (4756): No heartbeat from core client for 30 sec - exiting 21:18:40 (4756): No heartbeat from core client for 30 sec - exiting 21:18:41 (4756): No heartbeat from core client for 30 sec - exiting 21:18:42 (4756): No heartbeat from core client for 30 sec - exiting 21:18:43 (4756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4788, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Sep 2011 21:37:43	1167875	13373498	hadcm3n_ymq1_1940_40_007453048_2	103,680	98,380	0.9489
25 Sep 2011 16:45:45	1167875	13373498	hadcm3n_ymq1_1940_40_007453048_2	77,760	74,383	0.9566
24 Sep 2011 18:13:11	1167875	13373498	hadcm3n_ymq1_1940_40_007453048_2	51,840	50,454	0.9733
19 Sep 2011 21:34:48	1167875	13373498	hadcm3n_ymq1_1940_40_007453048_2	25,920	24,419	0.9421