Task 16004736

Name	hadcm3n_o4ld_2060_40_008396012_4
Workunit	8546871
Created	5 Sep 2013, 17:00:17 UTC
Sent	5 Sep 2013, 17:20:03 UTC
Report deadline	6 Dec 2013, 0:47:14 UTC
Received	7 Oct 2013, 7:59:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1280504
Run time	19 hours 27 min 4 sec
CPU time	18 hours 5 min 47 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	2.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:42:33 (8208): No heartbeat from core client for 30 sec - exiting 23:42:34 (8208): No heartbeat from core client for 30 sec - exiting 23:42:35 (8208): No heartbeat from core client for 30 sec - exiting 23:42:36 (8208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:01:19 (9988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:20 (9988): No heartbeat from core client for 30 sec - exiting 03:01:21 (9988): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 03:38:39 (1084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:40 (1084): No heartbeat from core client for 30 sec - exiting 03:43:39 (3744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:02:01 (6812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:11:01 (13156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:02 (13156): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:37:42 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:05:08 (10536): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:15:14 (13880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:15:15 (13880): No heartbeat from core client for 30 sec - exiting 16:15:16 (13880): No heartbeat from core client for 30 sec - exiting 16:15:17 (13880): No heartbeat from core client for 30 sec - exiting 16:15:18 (13880): No heartbeat from core client for 30 sec - exiting 16:15:19 (13880): No heartbeat from core client for 30 sec - exiting 16:15:20 (13880): No heartbeat from core client for 30 sec - exiting 16:15:21 (13880): No heartbeat from core client for 30 sec - exiting 16:15:22 (13880): No heartbeat from core client for 30 sec - exiting 16:15:23 (13880): No heartbeat from core client for 30 sec - exiting 16:15:24 (13880): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:40:16 (13772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:40:17 (13772): No heartbeat from core client for 30 sec - exiting 16:50:18 (16784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:55:18 (14748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:01 (16784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:22 (16372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:24:03 (18316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7468, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... 13:14:13 (5144): No heartbeat from core client for 30 sec - exiting 13:14:14 (5144): No heartbeat from core client for 30 sec - exiting 13:14:15 (5144): No heartbeat from core client for 30 sec - exiting 13:14:16 (5144): No heartbeat from core client for 30 sec - exiting 13:14:17 (5144): No heartbeat from core client for 30 sec - exiting 13:14:18 (5144): No heartbeat from core client for 30 sec - exiting 13:14:19 (5144): No heartbeat from core client for 30 sec - exiting 13:14:20 (5144): No heartbeat from core client for 30 sec - exiting 13:14:21 (5144): No heartbeat from core client for 30 sec - exiting 13:14:22 (5144): No heartbeat from core client for 30 sec - exiting 13:14:23 (5144): No heartbeat from core client for 30 sec - exiting 13:14:24 (5144): No heartbeat from core client for 30 sec - exiting 13:14:25 (5144): No heartbeat from core client for 30 sec - exiting 13:14:26 (5144): No heartbeat from core client for 30 sec - exiting 13:14:27 (5144): No heartbeat from core client for 30 sec - exiting 13:14:28 (5144): No heartbeat from core client for 30 sec - exiting 13:14:29 (5144): No heartbeat from core client for 30 sec - exiting 13:14:30 (5144): No heartbeat from core client for 30 sec - exiting 13:14:31 (5144): No heartbeat from core client for 30 sec - exiting 13:14:32 (5144): No heartbeat from core client for 30 sec - exiting 13:14:33 (5144): No heartbeat from core client for 30 sec - exiting 13:14:34 (5144): No heartbeat from core client for 30 sec - exiting 13:14:35 (5144): No heartbeat from core client for 30 sec - exiting 13:14:36 (5144): No heartbeat from core client for 30 sec - exiting 13:14:37 (5144): No heartbeat from core client for 30 sec - exiting 13:14:38 (5144): No heartbeat from core client for 30 sec - exiting 13:14:39 (5144): No heartbeat from core client for 30 sec - exiting 13:14:40 (5144): No heartbeat from core client for 30 sec - exiting 13:14:41 (5144): No heartbeat from core client for 30 sec - exiting 13:14:42 (5144): No heartbeat from core client for 30 sec - exiting 13:14:43 (5144): No heartbeat from core client for 30 sec - exiting 13:14:44 (5144): No heartbeat from core client for 30 sec - exiting 13:14:45 (5144): No heartbeat from core client for 30 sec - exiting 13:14:46 (5144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:14:47 (5144): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6596, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6596, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22392, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=22392, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Sep 2013 10:11:30	1280504	16004736	hadcm3n_o4ld_2060_40_008396012_4	51,840	58,947	1.1371
07 Sep 2013 05:33:14	1280504	16004736	hadcm3n_o4ld_2060_40_008396012_4	25,920	22,376	0.8633