Task 15797265

Name	hadcm3n_3hbl_1980_40_008319935_1
Workunit	8471070
Created	26 May 2013, 19:22:28 UTC
Sent	26 May 2013, 19:22:40 UTC
Report deadline	26 Aug 2013, 2:49:51 UTC
Received	28 May 2013, 16:58:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1230144
Run time	1 days 9 hours 51 min 18 sec
CPU time	1 days 6 hours 29 min 19 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	2.21 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 08:14:49 (3132): No heartbeat from core client for 30 sec - exiting 08:14:50 (3132): No heartbeat from core client for 30 sec - exiting 08:14:51 (3132): No heartbeat from core client for 30 sec - exiting 08:14:52 (3132): No heartbeat from core client for 30 sec - exiting 08:14:53 (3132): No heartbeat from core client for 30 sec - exiting 08:14:54 (3132): No heartbeat from core client for 30 sec - exiting 08:14:55 (3132): No heartbeat from core client for 30 sec - exiting 08:14:56 (3132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:12 (2864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:48:32 (4132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:48:34 (4132): No heartbeat from core client for 30 sec - exiting 23:15:18 (4212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:58:27 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:26 (5476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:44:29 (5060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:45:58 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:14:39 (4768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:20:48 (3840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:27:46 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:30:11 (3540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:53 (5452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:18:29 (5916): No heartbeat from core client for 30 sec - exiting 02:18:30 (5916): No heartbeat from core client for 30 sec - exiting 02:18:31 (5916): No heartbeat from core client for 30 sec - exiting 02:18:32 (5916): No heartbeat from core client for 30 sec - exiting 02:18:33 (5916): No heartbeat from core client for 30 sec - exiting 02:18:34 (5916): No heartbeat from core client for 30 sec - exiting 02:18:35 (5916): No heartbeat from core client for 30 sec - exiting 02:18:37 (5916): No heartbeat from core client for 30 sec - exiting 02:18:38 (5916): No heartbeat from core client for 30 sec - exiting 02:18:39 (5916): No heartbeat from core client for 30 sec - exiting 02:18:40 (5916): No heartbeat from core client for 30 sec - exiting 02:18:41 (5916): No heartbeat from core client for 30 sec - exiting 02:18:42 (5916): No heartbeat from core client for 30 sec - exiting 02:18:43 (5916): No heartbeat from core client for 30 sec - exiting 02:18:44 (5916): No heartbeat from core client for 30 sec - exiting 02:18:45 (5916): No heartbeat from core client for 30 sec - exiting 02:18:46 (5916): No heartbeat from core client for 30 sec - exiting 02:18:48 (5916): No heartbeat from core client for 30 sec - exiting 02:18:49 (5916): No heartbeat from core client for 30 sec - exiting 02:18:50 (5916): No heartbeat from core client for 30 sec - exiting 02:18:51 (5916): No heartbeat from core client for 30 sec - exiting 02:18:52 (5916): No heartbeat from core client for 30 sec - exiting 02:18:53 (5916): No heartbeat from core client for 30 sec - exiting 02:18:54 (5916): No heartbeat from core client for 30 sec - exiting 02:18:55 (5916): No heartbeat from core client for 30 sec - exiting 02:18:56 (5916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:40:52 (4488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:43:46 (5252): No heartbeat from core client for 30 sec - exiting 02:43:47 (5252): No heartbeat from core client for 30 sec - exiting 02:43:48 (5252): No heartbeat from core client for 30 sec - exiting 02:43:50 (5252): No heartbeat from core client for 30 sec - exiting 02:43:51 (5252): No heartbeat from core client for 30 sec - exiting 02:43:52 (5252): No heartbeat from core client for 30 sec - exiting 02:43:53 (5252): No heartbeat from core client for 30 sec - exiting 02:43:54 (5252): No heartbeat from core client for 30 sec - exiting 02:43:55 (5252): No heartbeat from core client for 30 sec - exiting 02:43:56 (5252): No heartbeat from core client for 30 sec - exiting 02:43:57 (5252): No heartbeat from core client for 30 sec - exiting 02:43:58 (5252): No heartbeat from core client for 30 sec - exiting 02:43:59 (5252): No heartbeat from core client for 30 sec - exiting 02:44:00 (5252): No heartbeat from core client for 30 sec - exiting 02:44:02 (5252): No heartbeat from core client for 30 sec - exiting 02:44:03 (5252): No heartbeat from core client for 30 sec - exiting 02:44:04 (5252): No heartbeat from core client for 30 sec - exiting 02:44:05 (5252): No heartbeat from core client for 30 sec - exiting 02:44:06 (5252): No heartbeat from core client for 30 sec - exiting 02:44:07 (5252): No heartbeat from core client for 30 sec - exiting 02:44:09 (5252): No heartbeat from core client for 30 sec - exiting 02:44:10 (5252): No heartbeat from core client for 30 sec - exiting 02:44:11 (5252): No heartbeat from core client for 30 sec - exiting 02:44:12 (5252): No heartbeat from core client for 30 sec - exiting 02:44:13 (5252): No heartbeat from core client for 30 sec - exiting 02:44:14 (5252): No heartbeat from core client for 30 sec - exiting 02:44:15 (5252): No heartbeat from core client for 30 sec - exiting 02:44:16 (5252): No heartbeat from core client for 30 sec - exiting 02:44:17 (5252): No heartbeat from core client for 30 sec - exiting 02:44:18 (5252): No heartbeat from core client for 30 sec - exiting 02:44:19 (5252): No heartbeat from core client for 30 sec - exiting 02:44:21 (5252): No heartbeat from core client for 30 sec - exiting 02:44:22 (5252): No heartbeat from core client for 30 sec - exiting 02:44:23 (5252): No heartbeat from core client for 30 sec - exiting 02:44:24 (5252): No heartbeat from core client for 30 sec - exiting 02:44:25 (5252): No heartbeat from core client for 30 sec - exiting 02:44:26 (5252): No heartbeat from core client for 30 sec - exiting 02:44:27 (5252): No heartbeat from core client for 30 sec - exiting 02:44:28 (5252): No heartbeat from core client for 30 sec - exiting 02:44:29 (5252): No heartbeat from core client for 30 sec - exiting 02:44:30 (5252): No heartbeat from core client for 30 sec - exiting 02:44:32 (5252): No heartbeat from core client for 30 sec - exiting 02:44:33 (5252): No heartbeat from core client for 30 sec - exiting 02:44:34 (5252): No heartbeat from core client for 30 sec - exiting 02:44:35 (5252): No heartbeat from core client for 30 sec - exiting 02:44:36 (5252): No heartbeat from core client for 30 sec - exiting 02:44:37 (5252): No heartbeat from core client for 30 sec - exiting 02:44:38 (5252): No heartbeat from core client for 30 sec - exiting 02:44:39 (5252): No heartbeat from core client for 30 sec - exiting 02:44:41 (5252): No heartbeat from core client for 30 sec - exiting 02:44:42 (5252): No heartbeat from core client for 30 sec - exiting 02:44:43 (5252): No heartbeat from core client for 30 sec - exiting 02:44:44 (5252): No heartbeat from core client for 30 sec - exiting 02:44:45 (5252): No heartbeat from core client for 30 sec - exiting 02:44:46 (5252): No heartbeat from core client for 30 sec - exiting 02:44:47 (5252): No heartbeat from core client for 30 sec - exiting 02:44:48 (5252): No heartbeat from core client for 30 sec - exiting 02:44:49 (5252): No heartbeat from core client for 30 sec - exiting 02:44:50 (5252): No heartbeat from core client for 30 sec - exiting 02:44:51 (5252): No heartbeat from core client for 30 sec - exiting 02:44:53 (5252): No heartbeat from core client for 30 sec - exiting 02:44:54 (5252): No heartbeat from core client for 30 sec - exiting 02:44:55 (5252): No heartbeat from core client for 30 sec - exiting 02:44:56 (5252): No heartbeat from core client for 30 sec - exiting 02:44:57 (5252): No heartbeat from core client for 30 sec - exiting 02:44:58 (5252): No heartbeat from core client for 30 sec - exiting 02:44:59 (5252): No heartbeat from core client for 30 sec - exiting 02:45:00 (5252): No heartbeat from core client for 30 sec - exiting 02:45:02 (5252): No heartbeat from core client for 30 sec - exiting 02:45:03 (5252): No heartbeat from core client for 30 sec - exiting 02:45:04 (5252): No heartbeat from core client for 30 sec - exiting 02:45:05 (5252): No heartbeat from core client for 30 sec - exiting 02:45:06 (5252): No heartbeat from core client for 30 sec - exiting 02:45:07 (5252): No heartbeat from core client for 30 sec - exiting 02:45:08 (5252): No heartbeat from core client for 30 sec - exiting 02:45:09 (5252): No heartbeat from core client for 30 sec - exiting 02:45:11 (5252): No heartbeat from core client for 30 sec - exiting 02:45:12 (5252): No heartbeat from core client for 30 sec - exiting 02:45:13 (5252): No heartbeat from core client for 30 sec - exiting 02:45:14 (5252): No heartbeat from core client for 30 sec - exiting 02:45:15 (5252): No heartbeat from core client for 30 sec - exiting 02:45:16 (5252): No heartbeat from core client for 30 sec - exiting 02:45:17 (5252): No heartbeat from core client for 30 sec - exiting 02:45:18 (5252): No heartbeat from core client for 30 sec - exiting 02:45:19 (5252): No heartbeat from core client for 30 sec - exiting 02:45:21 (5252): No heartbeat from core client for 30 sec - exiting 02:45:22 (5252): No heartbeat from core client for 30 sec - exiting 02:45:23 (5252): No heartbeat from core client for 30 sec - exiting 02:45:24 (5252): No heartbeat from core client for 30 sec - exiting 02:45:25 (5252): No heartbeat from core client for 30 sec - exiting 02:45:26 (5252): No heartbeat from core client for 30 sec - exiting 02:45:27 (5252): No heartbeat from core client for 30 sec - exiting 02:45:28 (5252): No heartbeat from core client for 30 sec - exiting 02:45:29 (5252): No heartbeat from core client for 30 sec - exiting 02:45:30 (5252): No heartbeat from core client for 30 sec - exiting 02:45:32 (5252): No heartbeat from core client for 30 sec - exiting 02:45:33 (5252): No heartbeat from core client for 30 sec - exiting 02:45:34 (5252): No heartbeat from core client for 30 sec - exiting 02:45:35 (5252): No heartbeat from core client for 30 sec - exiting 02:45:36 (5252): No heartbeat from core client for 30 sec - exiting 02:45:37 (5252): No heartbeat from core client for 30 sec - exiting 02:45:38 (5252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:12:58 (5848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:15:19 (5176): No heartbeat from core client for 30 sec - exiting 04:15:20 (5176): No heartbeat from core client for 30 sec - exiting 04:15:21 (5176): No heartbeat from core client for 30 sec - exiting 04:15:23 (5176): No heartbeat from core client for 30 sec - exiting 04:15:24 (5176): No heartbeat from core client for 30 sec - exiting 04:15:25 (5176): No heartbeat from core client for 30 sec - exiting 04:15:26 (5176): No heartbeat from core client for 30 sec - exiting 04:15:27 (5176): No heartbeat from core client for 30 sec - exiting 04:15:28 (5176): No heartbeat from core client for 30 sec - exiting 04:15:29 (5176): No heartbeat from core client for 30 sec - exiting 04:15:30 (5176): No heartbeat from core client for 30 sec - exiting 04:15:31 (5176): No heartbeat from core client for 30 sec - exiting 04:15:32 (5176): No heartbeat from core client for 30 sec - exiting 04:15:34 (5176): No heartbeat from core client for 30 sec - exiting 04:15:35 (5176): No heartbeat from core client for 30 sec - exiting 04:15:36 (5176): No heartbeat from core client for 30 sec - exiting 04:15:37 (5176): No heartbeat from core client for 30 sec - exiting 04:15:38 (5176): No heartbeat from core client for 30 sec - exiting 04:15:39 (5176): No heartbeat from core client for 30 sec - exiting 04:15:40 (5176): No heartbeat from core client for 30 sec - exiting 04:15:41 (5176): No heartbeat from core client for 30 sec - exiting 04:15:42 (5176): No heartbeat from core client for 30 sec - exiting 04:15:44 (5176): No heartbeat from core client for 30 sec - exiting 04:15:46 (5176): No heartbeat from core client for 30 sec - exiting 04:15:47 (5176): No heartbeat from core client for 30 sec - exiting 04:15:48 (5176): No heartbeat from core client for 30 sec - exiting 04:15:49 (5176): No heartbeat from core client for 30 sec - exiting 04:15:50 (5176): No heartbeat from core client for 30 sec - exiting 04:15:51 (5176): No heartbeat from core client for 30 sec - exiting 04:15:52 (5176): No heartbeat from core client for 30 sec - exiting 04:15:53 (5176): No heartbeat from core client for 30 sec - exiting 04:15:55 (5176): No heartbeat from core client for 30 sec - exiting 04:15:56 (5176): No heartbeat from core client for 30 sec - exiting 04:15:57 (5176): No heartbeat from core client for 30 sec - exiting 04:15:58 (5176): No heartbeat from core client for 30 sec - exiting 04:15:59 (5176): No heartbeat from core client for 30 sec - exiting 04:16:00 (5176): No heartbeat from core client for 30 sec - exiting 04:16:01 (5176): No heartbeat from core client for 30 sec - exiting 04:16:02 (5176): No heartbeat from core client for 30 sec - exiting 04:16:03 (5176): No heartbeat from core client for 30 sec - exiting 04:16:04 (5176): No heartbeat from core client for 30 sec - exiting 04:16:05 (5176): No heartbeat from core client for 30 sec - exiting 04:16:07 (5176): No heartbeat from core client for 30 sec - exiting 04:16:08 (5176): No heartbeat from core client for 30 sec - exiting 04:16:09 (5176): No heartbeat from core client for 30 sec - exiting 04:16:10 (5176): No heartbeat from core client for 30 sec - exiting 04:16:11 (5176): No heartbeat from core client for 30 sec - exiting 04:16:12 (5176): No heartbeat from core client for 30 sec - exiting 04:16:13 (5176): No heartbeat from core client for 30 sec - exiting 04:16:14 (5176): No heartbeat from core client for 30 sec - exiting 04:16:15 (5176): No heartbeat from core client for 30 sec - exiting 04:16:16 (5176): No heartbeat from core client for 30 sec - exiting 04:16:17 (5176): No heartbeat from core client for 30 sec - exiting 04:16:18 (5176): No heartbeat from core client for 30 sec - exiting 04:16:19 (5176): No heartbeat from core client for 30 sec - exiting 04:16:21 (5176): No heartbeat from core client for 30 sec - exiting 04:16:22 (5176): No heartbeat from core client for 30 sec - exiting 04:16:23 (5176): No heartbeat from core client for 30 sec - exiting 04:16:24 (5176): No heartbeat from core client for 30 sec - exiting 04:16:25 (5176): No heartbeat from core client for 30 sec - exiting 04:16:26 (5176): No heartbeat from core client for 30 sec - exiting 04:16:27 (5176): No heartbeat from core client for 30 sec - exiting 04:16:29 (5176): No heartbeat from core client for 30 sec - exiting 04:16:30 (5176): No heartbeat from core client for 30 sec - exiting 04:16:31 (5176): No heartbeat from core client for 30 sec - exiting 04:16:32 (5176): No heartbeat from core client for 30 sec - exiting 04:16:33 (5176): No heartbeat from core client for 30 sec - exiting 04:16:34 (5176): No heartbeat from core client for 30 sec - exiting 04:16:35 (5176): No heartbeat from core client for 30 sec - exiting 04:16:36 (5176): No heartbeat from core client for 30 sec - exiting 04:16:37 (5176): No heartbeat from core client for 30 sec - exiting 04:16:38 (5176): No heartbeat from core client for 30 sec - exiting 04:16:40 (5176): No heartbeat from core client for 30 sec - exiting 04:16:41 (5176): No heartbeat from core client for 30 sec - exiting 04:16:42 (5176): No heartbeat from core client for 30 sec - exiting 04:16:43 (5176): No heartbeat from core client for 30 sec - exiting 04:16:44 (5176): No heartbeat from core client for 30 sec - exiting 04:16:45 (5176): No heartbeat from core client for 30 sec - exiting 04:16:46 (5176): No heartbeat from core client for 30 sec - exiting 04:16:47 (5176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7608, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7608, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7608, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7608, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7608, iMonCtr=1 Model crash detected, will try to restart... 05:04:06 (4260): No heartbeat from core client for 30 sec - exiting 05:04:08 (4260): No heartbeat from core client for 30 sec - exiting 05:04:09 (4260): No heartbeat from core client for 30 sec - exiting 05:04:10 (4260): No heartbeat from core client for 30 sec - exiting 05:04:11 (4260): No heartbeat from core client for 30 sec - exiting 05:04:12 (4260): No heartbeat from core client for 30 sec - exiting 05:04:13 (4260): No heartbeat from core client for 30 sec - exiting 05:04:14 (4260): No heartbeat from core client for 30 sec - exiting 05:04:15 (4260): No heartbeat from core client for 30 sec - exiting 05:04:17 (4260): No heartbeat from core client for 30 sec - exiting 05:04:18 (4260): No heartbeat from core client for 30 sec - exiting 05:04:19 (4260): No heartbeat from core client for 30 sec - exiting 05:04:20 (4260): No heartbeat from core client for 30 sec - exiting 05:04:21 (4260): No heartbeat from core client for 30 sec - exiting 05:04:22 (4260): No heartbeat from core client for 30 sec - exiting 05:04:23 (4260): No heartbeat from core client for 30 sec - exiting 05:04:24 (4260): No heartbeat from core client for 30 sec - exiting 05:04:25 (4260): No heartbeat from core client for 30 sec - exiting 05:04:26 (4260): No heartbeat from core client for 30 sec - exiting 05:04:27 (4260): No heartbeat from core client for 30 sec - exiting 05:04:29 (4260): No heartbeat from core client for 30 sec - exiting 05:04:30 (4260): No heartbeat from core client for 30 sec - exiting 05:04:31 (4260): No heartbeat from core client for 30 sec - exiting 05:04:32 (4260): No heartbeat from core client for 30 sec - exiting 05:04:33 (4260): No heartbeat from core client for 30 sec - exiting 05:04:34 (4260): No heartbeat from core client for 30 sec - exiting 05:04:35 (4260): No heartbeat from core client for 30 sec - exiting 05:04:36 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4528, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4528, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 May 2013 22:29:33	1230144	15797265	hadcm3n_3hbl_1980_40_008319935_1	51,840	84,376	1.6276
27 May 2013 07:48:37	1230144	15797265	hadcm3n_3hbl_1980_40_008319935_1	25,920	41,650	1.6069