Task 13668511

Name	hadcm3n_yf3z_1980_40_007545767_4
Workunit	7742999
Created	29 Nov 2011, 2:09:03 UTC
Sent	29 Nov 2011, 2:12:04 UTC
Report deadline	28 Feb 2012, 9:39:15 UTC
Received	30 Nov 2011, 12:32:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	1 days 3 hours 41 min 3 sec
CPU time	1 days 3 hours 2 min 4 sec
Validate state	Invalid
Credit	311.04
Device peak FLOPS	1.69 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 22:50:48 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:05 (2812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:18:06 (2812): No heartbeat from core client for 30 sec - exiting 23:18:07 (2812): No heartbeat from core client for 30 sec - exiting 23:18:09 (2812): No heartbeat from core client for 30 sec - exiting 23:18:10 (2812): No heartbeat from core client for 30 sec - exiting 23:18:11 (2812): No heartbeat from core client for 30 sec - exiting 23:18:12 (2812): No heartbeat from core client for 30 sec - exiting 23:18:13 (2812): No heartbeat from core client for 30 sec - exiting 23:18:14 (2812): No heartbeat from core client for 30 sec - exiting 23:18:15 (2812): No heartbeat from core client for 30 sec - exiting 23:18:16 (2812): No heartbeat from core client for 30 sec - exiting 23:19:56 (1700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:20:46 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:21:21 (960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:35:57 (3580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:36:41 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:37:21 (4584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:38 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:26 (3124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:10 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:16 (4212): No heartbeat from core client for 30 sec - exiting 23:44:17 (4212): No heartbeat from core client for 30 sec - exiting 23:44:18 (4212): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 04:31:44 (4660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:31:57 (4660): No heartbeat from core client for 30 sec - exiting 04:31:58 (4660): No heartbeat from core client for 30 sec - exiting 04:31:59 (4660): No heartbeat from core client for 30 sec - exiting 04:32:00 (4660): No heartbeat from core client for 30 sec - exiting 04:32:01 (4660): No heartbeat from core client for 30 sec - exiting 04:32:03 (4660): No heartbeat from core client for 30 sec - exiting 04:32:04 (4660): No heartbeat from core client for 30 sec - exiting 04:32:05 (4660): No heartbeat from core client for 30 sec - exiting 04:32:06 (4660): No heartbeat from core client for 30 sec - exiting 04:32:07 (4660): No heartbeat from core client for 30 sec - exiting 04:32:08 (4660): No heartbeat from core client for 30 sec - exiting 04:32:09 (4660): No heartbeat from core client for 30 sec - exiting 04:32:10 (4660): No heartbeat from core client for 30 sec - exiting 04:32:11 (4660): No heartbeat from core client for 30 sec - exiting 04:32:12 (4660): No heartbeat from core client for 30 sec - exiting 04:35:26 (1688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:35:31 (1688): No heartbeat from core client for 30 sec - exiting 04:35:32 (1688): No heartbeat from core client for 30 sec - exiting 04:35:33 (1688): No heartbeat from core client for 30 sec - exiting 04:36:37 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:42 (5000): No heartbeat from core client for 30 sec - exiting 04:36:43 (5000): No heartbeat from core client for 30 sec - exiting 04:36:45 (5000): No heartbeat from core client for 30 sec - exiting 04:37:48 (1112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:37:49 (1112): No heartbeat from core client for 30 sec - exiting 04:37:51 (1112): No heartbeat from core client for 30 sec - exiting 04:37:52 (1112): No heartbeat from core client for 30 sec - exiting 04:37:53 (1112): No heartbeat from core client for 30 sec - exiting 04:37:54 (1112): No heartbeat from core client for 30 sec - exiting 04:37:55 (1112): No heartbeat from core client for 30 sec - exiting 04:39:29 (4492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:40:52 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:42:10 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:43:33 (4184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:44:51 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:19:44 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:22 (4696): No heartbeat from core client for 30 sec - exiting 15:20:23 (4696): No heartbeat from core client for 30 sec - exiting 15:20:24 (4696): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:08:50 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:01 (5040): No heartbeat from core client for 30 sec - exiting 00:09:02 (5040): No heartbeat from core client for 30 sec - exiting 00:09:03 (5040): No heartbeat from core client for 30 sec - exiting 00:09:04 (5040): No heartbeat from core client for 30 sec - exiting 00:09:05 (5040): No heartbeat from core client for 30 sec - exiting 00:09:06 (5040): No heartbeat from core client for 30 sec - exiting 00:09:07 (5040): No heartbeat from core client for 30 sec - exiting 00:09:08 (5040): No heartbeat from core client for 30 sec - exiting 00:09:10 (5040): No heartbeat from core client for 30 sec - exiting 00:09:11 (5040): No heartbeat from core client for 30 sec - exiting 00:12:45 (4548): No heartbeat from core client for 30 sec - exiting 00:12:54 (4548): No heartbeat from core client for 30 sec - exiting 00:12:55 (4548): No heartbeat from core client for 30 sec - exiting 00:12:56 (4548): No heartbeat from core client for 30 sec - exiting 00:12:57 (4548): No heartbeat from core client for 30 sec - exiting 00:12:58 (4548): No heartbeat from core client for 30 sec - exiting 00:12:59 (4548): No heartbeat from core client for 30 sec - exiting 00:13:01 (4548): No heartbeat from core client for 30 sec - exiting 00:13:02 (4548): No heartbeat from core client for 30 sec - exiting 00:13:03 (4548): No heartbeat from core client for 30 sec - exiting 00:13:04 (4548): No heartbeat from core client for 30 sec - exiting 00:13:05 (4548): No heartbeat from core client for 30 sec - exiting 00:13:06 (4548): No heartbeat from core client for 30 sec - exiting 00:13:07 (4548): No heartbeat from core client for 30 sec - exiting 00:13:08 (4548): No heartbeat from core client for 30 sec - exiting 00:13:09 (4548): No heartbeat from core client for 30 sec - exiting 00:13:10 (4548): No heartbeat from core client for 30 sec - exiting 00:13:11 (4548): No heartbeat from core client for 30 sec - exiting 00:13:13 (4548): No heartbeat from core client for 30 sec - exiting 00:13:14 (4548): No heartbeat from core client for 30 sec - exiting 00:13:15 (4548): No heartbeat from core client for 30 sec - exiting 00:13:16 (4548): No heartbeat from core client for 30 sec - exiting 00:13:17 (4548): No heartbeat from core client for 30 sec - exiting 00:13:55 (4548): No heartbeat from core client for 30 sec - exiting 00:13:56 (4548): No heartbeat from core client for 30 sec - exiting 00:13:57 (4548): No heartbeat from core client for 30 sec - exiting 00:13:58 (4548): No heartbeat from core client for 30 sec - exiting 00:14:00 (4548): No heartbeat from core client for 30 sec - exiting 00:14:01 (4548): No heartbeat from core client for 30 sec - exiting 00:14:02 (4548): No heartbeat from core client for 30 sec - exiting 00:14:03 (4548): No heartbeat from core client for 30 sec - exiting 00:14:04 (4548): No heartbeat from core client for 30 sec - exiting 00:14:05 (4548): No heartbeat from core client for 30 sec - exiting 00:14:06 (4548): No heartbeat from core client for 30 sec - exiting 00:14:07 (4548): No heartbeat from core client for 30 sec - exiting 00:14:08 (4548): No heartbeat from core client for 30 sec - exiting 00:14:09 (4548): No heartbeat from core client for 30 sec - exiting 00:14:10 (4548): No heartbeat from core client for 30 sec - exiting 00:14:12 (4548): No heartbeat from core client for 30 sec - exiting 00:14:13 (4548): No heartbeat from core client for 30 sec - exiting 00:14:14 (4548): No heartbeat from core client for 30 sec - exiting 00:14:15 (4548): No heartbeat from core client for 30 sec - exiting 00:14:16 (4548): No heartbeat from core client for 30 sec - exiting 00:14:17 (4548): No heartbeat from core client for 30 sec - exiting 00:14:18 (4548): No heartbeat from core client for 30 sec - exiting 00:14:19 (4548): No heartbeat from core client for 30 sec - exiting 00:14:20 (4548): No heartbeat from core client for 30 sec - exiting 00:14:21 (4548): No heartbeat from core client for 30 sec - exiting 00:14:22 (4548): No heartbeat from core client for 30 sec - exiting 00:14:24 (4548): No heartbeat from core client for 30 sec - exiting 00:14:25 (4548): No heartbeat from core client for 30 sec - exiting 00:14:26 (4548): No heartbeat from core client for 30 sec - exiting 00:14:27 (4548): No heartbeat from core client for 30 sec - exiting 00:14:28 (4548): No heartbeat from core client for 30 sec - exiting 00:14:29 (4548): No heartbeat from core client for 30 sec - exiting 00:14:30 (4548): No heartbeat from core client for 30 sec - exiting 00:14:31 (4548): No heartbeat from core client for 30 sec - exiting 00:14:32 (4548): No heartbeat from core client for 30 sec - exiting 00:14:33 (4548): No heartbeat from core client for 30 sec - exiting 00:14:34 (4548): No heartbeat from core client for 30 sec - exiting 00:14:36 (4548): No heartbeat from core client for 30 sec - exiting 00:14:37 (4548): No heartbeat from core client for 30 sec - exiting 00:14:38 (4548): No heartbeat from core client for 30 sec - exiting 00:14:39 (4548): No heartbeat from core client for 30 sec - exiting 00:14:40 (4548): No heartbeat from core client for 30 sec - exiting 00:14:41 (4548): No heartbeat from core client for 30 sec - exiting 00:14:42 (4548): No heartbeat from core client for 30 sec - exiting 00:14:43 (4548): No heartbeat from core client for 30 sec - exiting 00:14:44 (4548): No heartbeat from core client for 30 sec - exiting 00:14:45 (4548): No heartbeat from core client for 30 sec - exiting 00:14:46 (4548): No heartbeat from core client for 30 sec - exiting 00:14:48 (4548): No heartbeat from core client for 30 sec - exiting 00:14:49 (4548): No heartbeat from core client for 30 sec - exiting 00:14:50 (4548): No heartbeat from core client for 30 sec - exiting 00:14:51 (4548): No heartbeat from core client for 30 sec - exiting 00:14:52 (4548): No heartbeat from core client for 30 sec - exiting 00:14:53 (4548): No heartbeat from core client for 30 sec - exiting 00:14:54 (4548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:55 (4548): No heartbeat from core client for 30 sec - exiting 00:14:56 (4548): No heartbeat from core client for 30 sec - exiting 00:14:57 (4548): No heartbeat from core client for 30 sec - exiting 00:14:58 (4548): No heartbeat from core client for 30 sec - exiting 00:15:00 (4548): No heartbeat from core client for 30 sec - exiting 00:15:01 (4548): No heartbeat from core client for 30 sec - exiting 00:15:02 (4548): No heartbeat from core client for 30 sec - exiting 00:15:03 (4548): No heartbeat from core client for 30 sec - exiting 00:15:04 (4548): No heartbeat from core client for 30 sec - exiting 00:15:05 (4548): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 03:16:48 (5596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:53 (5596): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2844, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2844, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2844, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2844, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2844, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2844, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Nov 2011 07:41:18	1122757	13668511	hadcm3n_yf3z_1980_40_007545767_4	25,920	82,271	3.1740