Task 14070956

Name	hadcm3n_o4a6_1980_40_007537928_3
Workunit	7735160
Created	6 Feb 2012, 15:53:14 UTC
Sent	6 Feb 2012, 15:53:19 UTC
Report deadline	7 May 2012, 23:20:30 UTC
Received	17 Feb 2012, 16:26:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1167875
Run time	1 days 1 hours 40 min 53 sec
CPU time	1 days 0 hours 21 min 3 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	4.16 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 22:13:40 (5308): No heartbeat from core client for 30 sec - exiting 22:13:41 (5308): No heartbeat from core client for 30 sec - exiting 22:13:42 (5308): No heartbeat from core client for 30 sec - exiting 22:13:43 (5308): No heartbeat from core client for 30 sec - exiting 22:13:44 (5308): No heartbeat from core client for 30 sec - exiting 22:13:45 (5308): No heartbeat from core client for 30 sec - exiting 22:13:46 (5308): No heartbeat from core client for 30 sec - exiting 22:13:47 (5308): No heartbeat from core client for 30 sec - exiting 22:13:48 (5308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:30:08 (5384): No heartbeat from core client for 30 sec - exiting 07:30:10 (5384): No heartbeat from core client for 30 sec - exiting 07:30:11 (5384): No heartbeat from core client for 30 sec - exiting 07:30:12 (5384): No heartbeat from core client for 30 sec - exiting 07:30:13 (5384): No heartbeat from core client for 30 sec - exiting 07:30:14 (5384): No heartbeat from core client for 30 sec - exiting 07:30:15 (5384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:48:53 (5852): No heartbeat from core client for 30 sec - exiting 18:48:54 (5852): No heartbeat from core client for 30 sec - exiting 18:48:55 (5852): No heartbeat from core client for 30 sec - exiting 18:48:56 (5852): No heartbeat from core client for 30 sec - exiting 18:48:57 (5852): No heartbeat from core client for 30 sec - exiting 18:48:58 (5852): No heartbeat from core client for 30 sec - exiting 18:48:59 (5852): No heartbeat from core client for 30 sec - exiting 18:49:00 (5852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:43:13 (5356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:29:55 (4480): No heartbeat from core client for 30 sec - exiting 07:29:57 (4480): No heartbeat from core client for 30 sec - exiting 07:29:58 (4480): No heartbeat from core client for 30 sec - exiting 07:29:59 (4480): No heartbeat from core client for 30 sec - exiting 07:30:00 (4480): No heartbeat from core client for 30 sec - exiting 07:30:01 (4480): No heartbeat from core client for 30 sec - exiting 07:30:02 (4480): No heartbeat from core client for 30 sec - exiting 07:30:03 (4480): No heartbeat from core client for 30 sec - exiting 07:30:04 (4480): No heartbeat from core client for 30 sec - exiting 07:30:05 (4480): No heartbeat from core client for 30 sec - exiting 07:30:06 (4480): No heartbeat from core client for 30 sec - exiting 07:30:07 (4480): No heartbeat from core client for 30 sec - exiting 07:30:08 (4480): No heartbeat from core client for 30 sec - exiting 07:30:09 (4480): No heartbeat from core client for 30 sec - exiting 07:30:10 (4480): No heartbeat from core client for 30 sec - exiting 07:30:11 (4480): No heartbeat from core client for 30 sec - exiting 07:30:12 (4480): No heartbeat from core client for 30 sec - exiting 07:30:13 (4480): No heartbeat from core client for 30 sec - exiting 07:30:14 (4480): No heartbeat from core client for 30 sec - exiting 07:30:15 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:44:02 (6084): No heartbeat from core client for 30 sec - exiting 18:44:03 (6084): No heartbeat from core client for 30 sec - exiting 18:44:04 (6084): No heartbeat from core client for 30 sec - exiting 18:44:05 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:44:06 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6212, iMonCtr=1 Model crash detected, will try to restart... 19:29:54 (6044): No heartbeat from core client for 30 sec - exiting 19:29:55 (6044): No heartbeat from core client for 30 sec - exiting 19:29:56 (6044): No heartbeat from core client for 30 sec - exiting 19:29:57 (6044): No heartbeat from core client for 30 sec - exiting 19:29:58 (6044): No heartbeat from core client for 30 sec - exiting 19:29:59 (6044): No heartbeat from core client for 30 sec - exiting 19:30:00 (6044): No heartbeat from core client for 30 sec - exiting 19:30:01 (6044): No heartbeat from core client for 30 sec - exiting 19:30:02 (6044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:34:00 (5880): No heartbeat from core client for 30 sec - exiting 16:34:01 (5880): No heartbeat from core client for 30 sec - exiting 16:34:02 (5880): No heartbeat from core client for 30 sec - exiting 16:34:03 (5880): No heartbeat from core client for 30 sec - exiting 16:34:04 (5880): No heartbeat from core client for 30 sec - exiting 16:34:05 (5880): No heartbeat from core client for 30 sec - exiting 16:34:06 (5880): No heartbeat from core client for 30 sec - exiting 16:34:07 (5880): No heartbeat from core client for 30 sec - exiting 16:34:08 (5880): No heartbeat from core client for 30 sec - exiting 16:34:09 (5880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:35:46 (5976): No heartbeat from core client for 30 sec - exiting 23:35:47 (5976): No heartbeat from core client for 30 sec - exiting 23:35:48 (5976): No heartbeat from core client for 30 sec - exiting 23:35:49 (5976): No heartbeat from core client for 30 sec - exiting 23:35:50 (5976): No heartbeat from core client for 30 sec - exiting 23:35:51 (5976): No heartbeat from core client for 30 sec - exiting 23:35:52 (5976): No heartbeat from core client for 30 sec - exiting 23:35:53 (5976): No heartbeat from core client for 30 sec - exiting 23:35:54 (5976): No heartbeat from core client for 30 sec - exiting 23:35:55 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:48:42 (5876): No heartbeat from core client for 30 sec - exiting 10:48:44 (5876): No heartbeat from core client for 30 sec - exiting 10:48:45 (5876): No heartbeat from core client for 30 sec - exiting 10:48:46 (5876): No heartbeat from core client for 30 sec - exiting 10:48:47 (5876): No heartbeat from core client for 30 sec - exiting 10:48:48 (5876): No heartbeat from core client for 30 sec - exiting 10:48:49 (5876): No heartbeat from core client for 30 sec - exiting 10:48:50 (5876): No heartbeat from core client for 30 sec - exiting 10:48:51 (5876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:50:07 (5396): No heartbeat from core client for 30 sec - exiting 19:50:08 (5396): No heartbeat from core client for 30 sec - exiting 19:50:09 (5396): No heartbeat from core client for 30 sec - exiting 19:50:10 (5396): No heartbeat from core client for 30 sec - exiting 19:50:11 (5396): No heartbeat from core client for 30 sec - exiting 19:50:12 (5396): No heartbeat from core client for 30 sec - exiting 19:50:13 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:53:00 (5700): No heartbeat from core client for 30 sec - exiting 18:53:01 (5700): No heartbeat from core client for 30 sec - exiting 18:53:02 (5700): No heartbeat from core client for 30 sec - exiting 18:53:03 (5700): No heartbeat from core client for 30 sec - exiting 18:53:04 (5700): No heartbeat from core client for 30 sec - exiting 18:53:05 (5700): No heartbeat from core client for 30 sec - exiting 18:53:06 (5700): No heartbeat from core client for 30 sec - exiting 18:53:07 (5700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:36:36 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6588, iMonCtr=1 Model crash detected, will try to restart... 23:06:35 (5944): No heartbeat from core client for 30 sec - exiting 23:06:37 (5944): No heartbeat from core client for 30 sec - exiting 23:06:38 (5944): No heartbeat from core client for 30 sec - exiting 23:06:39 (5944): No heartbeat from core client for 30 sec - exiting 23:06:40 (5944): No heartbeat from core client for 30 sec - exiting 23:06:41 (5944): No heartbeat from core client for 30 sec - exiting 23:06:42 (5944): No heartbeat from core client for 30 sec - exiting 23:06:43 (5944): No heartbeat from core client for 30 sec - exiting 23:06:44 (5944): No heartbeat from core client for 30 sec - exiting 23:06:45 (5944): No heartbeat from core client for 30 sec - exiting 23:06:46 (5944): No heartbeat from core client for 30 sec - exiting 23:06:47 (5944): No heartbeat from core client for 30 sec - exiting 23:06:48 (5944): No heartbeat from core client for 30 sec - exiting 23:06:49 (5944): No heartbeat from core client for 30 sec - exiting 23:06:50 (5944): No heartbeat from core client for 30 sec - exiting 23:06:51 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:07:39 (4480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:52:44 (1904): No heartbeat from core client for 30 sec - exiting 06:52:46 (1904): No heartbeat from core client for 30 sec - exiting 06:52:47 (1904): No heartbeat from core client for 30 sec - exiting 06:52:48 (1904): No heartbeat from core client for 30 sec - exiting 06:52:49 (1904): No heartbeat from core client for 30 sec - exiting 06:52:50 (1904): No heartbeat from core client for 30 sec - exiting 06:52:51 (1904): No heartbeat from core client for 30 sec - exiting 06:52:52 (1904): No heartbeat from core client for 30 sec - exiting 06:52:53 (1904): No heartbeat from core client for 30 sec - exiting 06:52:54 (1904): No heartbeat from core client for 30 sec - exiting 06:52:55 (1904): No heartbeat from core client for 30 sec - exiting 06:52:56 (1904): No heartbeat from core client for 30 sec - exiting 06:52:57 (1904): No heartbeat from core client for 30 sec - exiting 06:52:58 (1904): No heartbeat from core client for 30 sec - exiting 06:52:59 (1904): No heartbeat from core client for 30 sec - exiting 06:53:00 (1904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:53:52 (5820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:50:40 (6748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:26:07 (5604): No heartbeat from core client for 30 sec - exiting 10:26:09 (5604): No heartbeat from core client for 30 sec - exiting 10:26:10 (5604): No heartbeat from core client for 30 sec - exiting 10:26:11 (5604): No heartbeat from core client for 30 sec - exiting 10:26:12 (5604): No heartbeat from core client for 30 sec - exiting 10:26:13 (5604): No heartbeat from core client for 30 sec - exiting 10:26:14 (5604): No heartbeat from core client for 30 sec - exiting 10:26:15 (5604): No heartbeat from core client for 30 sec - exiting 10:26:16 (5604): No heartbeat from core client for 30 sec - exiting 10:26:17 (5604): No heartbeat from core client for 30 sec - exiting 10:26:18 (5604): No heartbeat from core client for 30 sec - exiting 10:26:19 (5604): No heartbeat from core client for 30 sec - exiting 10:26:20 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:21 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:16:10 (3584): No heartbeat from core client for 30 sec - exiting 13:16:11 (3584): No heartbeat from core client for 30 sec - exiting 13:16:12 (3584): No heartbeat from core client for 30 sec - exiting 13:16:13 (3584): No heartbeat from core client for 30 sec - exiting 13:16:14 (3584): No heartbeat from core client for 30 sec - exiting 13:16:15 (3584): No heartbeat from core client for 30 sec - exiting 13:16:16 (3584): No heartbeat from core client for 30 sec - exiting 13:16:17 (3584): No heartbeat from core client for 30 sec - exiting 13:16:18 (3584): No heartbeat from core client for 30 sec - exiting 13:16:19 (3584): No heartbeat from core client for 30 sec - exiting 13:16:20 (3584): No heartbeat from core client for 30 sec - exiting 13:16:21 (3584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Feb 2012 23:13:40	1167875	14070956	hadcm3n_o4a6_1980_40_007537928_3	77,760	74,921	0.9635
12 Feb 2012 14:19:29	1167875	14070956	hadcm3n_o4a6_1980_40_007537928_3	51,840	50,156	0.9675
07 Feb 2012 21:54:51	1167875	14070956	hadcm3n_o4a6_1980_40_007537928_3	25,920	24,986	0.9640