Task 14842513

Name	hadcm3n_u6xu_1980_40_007998992_4
Workunit	8154106
Created	25 Jun 2012, 18:12:55 UTC
Sent	25 Jun 2012, 18:13:17 UTC
Report deadline	25 Sep 2012, 1:40:28 UTC
Received	2 Jul 2012, 12:26:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	6 days 2 hours 35 min 2 sec
CPU time	5 days 10 hours 52 min 18 sec
Validate state	Invalid
Credit	1,555.20
Device peak FLOPS	1.69 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 02:02:07 (6672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:02:52 (6672): No heartbeat from core client for 30 sec - exiting 02:02:53 (6672): No heartbeat from core client for 30 sec - exiting 02:02:54 (6672): No heartbeat from core client for 30 sec - exiting 02:02:56 (6672): No heartbeat from core client for 30 sec - exiting 02:02:57 (6672): No heartbeat from core client for 30 sec - exiting 02:02:58 (6672): No heartbeat from core client for 30 sec - exiting 02:02:59 (6672): No heartbeat from core client for 30 sec - exiting 02:03:00 (6672): No heartbeat from core client for 30 sec - exiting 02:03:01 (6672): No heartbeat from core client for 30 sec - exiting 02:03:02 (6672): No heartbeat from core client for 30 sec - exiting 02:03:03 (6672): No heartbeat from core client for 30 sec - exiting 02:12:44 (4220): No heartbeat from core client for 30 sec - exiting 02:12:45 (4220): No heartbeat from core client for 30 sec - exiting 02:12:46 (4220): No heartbeat from core client for 30 sec - exiting 02:12:47 (4220): No heartbeat from core client for 30 sec - exiting 02:12:48 (4220): No heartbeat from core client for 30 sec - exiting 02:12:53 (4220): No heartbeat from core client for 30 sec - exiting 02:12:54 (4220): No heartbeat from core client for 30 sec - exiting 02:12:55 (4220): No heartbeat from core client for 30 sec - exiting 02:12:56 (4220): No heartbeat from core client for 30 sec - exiting 02:12:57 (4220): No heartbeat from core client for 30 sec - exiting 02:12:58 (4220): No heartbeat from core client for 30 sec - exiting 02:13:00 (4220): No heartbeat from core client for 30 sec - exiting 02:13:01 (4220): No heartbeat from core client for 30 sec - exiting 02:13:05 (4220): No heartbeat from core client for 30 sec - exiting 02:13:07 (4220): No heartbeat from core client for 30 sec - exiting 02:13:08 (4220): No heartbeat from core client for 30 sec - exiting 02:13:09 (4220): No heartbeat from core client for 30 sec - exiting 02:13:14 (4220): No heartbeat from core client for 30 sec - exiting 02:13:15 (4220): No heartbeat from core client for 30 sec - exiting 02:13:16 (4220): No heartbeat from core client for 30 sec - exiting 02:13:17 (4220): No heartbeat from core client for 30 sec - exiting 02:13:22 (4220): No heartbeat from core client for 30 sec - exiting 02:13:23 (4220): No heartbeat from core client for 30 sec - exiting 02:13:26 (4220): No heartbeat from core client for 30 sec - exiting 02:13:30 (4220): No heartbeat from core client for 30 sec - exiting 02:13:31 (4220): No heartbeat from core client for 30 sec - exiting 02:13:34 (4220): No heartbeat from core client for 30 sec - exiting 02:13:35 (4220): No heartbeat from core client for 30 sec - exiting 02:13:37 (4220): No heartbeat from core client for 30 sec - exiting 02:13:38 (4220): No heartbeat from core client for 30 sec - exiting 02:13:39 (4220): No heartbeat from core client for 30 sec - exiting 02:13:40 (4220): No heartbeat from core client for 30 sec - exiting 02:13:41 (4220): No heartbeat from core client for 30 sec - exiting 02:13:42 (4220): No heartbeat from core client for 30 sec - exiting 02:13:47 (4220): No heartbeat from core client for 30 sec - exiting 02:13:48 (4220): No heartbeat from core client for 30 sec - exiting 02:13:49 (4220): No heartbeat from core client for 30 sec - exiting 02:13:50 (4220): No heartbeat from core client for 30 sec - exiting 02:13:55 (4220): No heartbeat from core client for 30 sec - exiting 02:13:56 (4220): No heartbeat from core client for 30 sec - exiting 02:13:59 (4220): No heartbeat from core client for 30 sec - exiting 02:14:00 (4220): No heartbeat from core client for 30 sec - exiting 02:14:03 (4220): No heartbeat from core client for 30 sec - exiting 02:14:04 (4220): No heartbeat from core client for 30 sec - exiting 02:14:05 (4220): No heartbeat from core client for 30 sec - exiting 02:14:39 (4220): No heartbeat from core client for 30 sec - exiting 02:14:40 (4220): No heartbeat from core client for 30 sec - exiting 02:14:41 (4220): No heartbeat from core client for 30 sec - exiting 02:14:43 (4220): No heartbeat from core client for 30 sec - exiting 02:14:44 (4220): No heartbeat from core client for 30 sec - exiting 02:14:45 (4220): No heartbeat from core client for 30 sec - exiting 02:14:46 (4220): No heartbeat from core client for 30 sec - exiting 02:14:47 (4220): No heartbeat from core client for 30 sec - exiting 02:14:48 (4220): No heartbeat from core client for 30 sec - exiting 02:14:49 (4220): No heartbeat from core client for 30 sec - exiting 02:14:50 (4220): No heartbeat from core client for 30 sec - exiting 02:14:51 (4220): No heartbeat from core client for 30 sec - exiting 02:14:52 (4220): No heartbeat from core client for 30 sec - exiting 02:14:57 (4220): No heartbeat from core client for 30 sec - exiting 02:14:58 (4220): No heartbeat from core client for 30 sec - exiting 02:15:01 (4220): No heartbeat from core client for 30 sec - exiting 02:15:02 (4220): No heartbeat from core client for 30 sec - exiting 02:15:03 (4220): No heartbeat from core client for 30 sec - exiting 02:15:04 (4220): No heartbeat from core client for 30 sec - exiting 02:15:06 (4220): No heartbeat from core client for 30 sec - exiting 02:15:07 (4220): No heartbeat from core client for 30 sec - exiting 02:15:08 (4220): No heartbeat from core client for 30 sec - exiting 02:15:09 (4220): No heartbeat from core client for 30 sec - exiting 02:15:10 (4220): No heartbeat from core client for 30 sec - exiting 02:15:11 (4220): No heartbeat from core client for 30 sec - exiting 02:15:12 (4220): No heartbeat from core client for 30 sec - exiting 02:15:13 (4220): No heartbeat from core client for 30 sec - exiting 02:15:14 (4220): No heartbeat from core client for 30 sec - exiting 02:15:15 (4220): No heartbeat from core client for 30 sec - exiting 02:15:16 (4220): No heartbeat from core client for 30 sec - exiting 02:15:18 (4220): No heartbeat from core client for 30 sec - exiting 02:15:22 (4220): No heartbeat from core client for 30 sec - exiting 02:15:23 (4220): No heartbeat from core client for 30 sec - exiting 02:15:26 (4220): No heartbeat from core client for 30 sec - exiting 02:15:27 (4220): No heartbeat from core client for 30 sec - exiting 02:15:30 (4220): No heartbeat from core client for 30 sec - exiting 02:15:31 (4220): No heartbeat from core client for 30 sec - exiting 02:15:34 (4220): No heartbeat from core client for 30 sec - exiting 02:15:36 (4220): No heartbeat from core client for 30 sec - exiting 02:15:37 (4220): No heartbeat from core client for 30 sec - exiting 02:15:38 (4220): No heartbeat from core client for 30 sec - exiting 02:15:43 (4220): No heartbeat from core client for 30 sec - exiting 02:15:44 (4220): No heartbeat from core client for 30 sec - exiting 02:15:45 (4220): No heartbeat from core client for 30 sec - exiting 02:15:46 (4220): No heartbeat from core client for 30 sec - exiting 02:15:47 (4220): No heartbeat from core client for 30 sec - exiting 02:15:48 (4220): No heartbeat from core client for 30 sec - exiting 02:15:49 (4220): No heartbeat from core client for 30 sec - exiting 02:15:50 (4220): No heartbeat from core client for 30 sec - exiting 02:15:51 (4220): No heartbeat from core client for 30 sec - exiting 02:15:53 (4220): No heartbeat from core client for 30 sec - exiting 02:15:54 (4220): No heartbeat from core client for 30 sec - exiting 02:15:55 (4220): No heartbeat from core client for 30 sec - exiting 02:15:59 (4220): No heartbeat from core client for 30 sec - exiting 02:16:00 (4220): No heartbeat from core client for 30 sec - exiting 02:16:02 (4220): No heartbeat from core client for 30 sec - exiting 02:16:03 (4220): No heartbeat from core client for 30 sec - exiting 02:16:04 (4220): No heartbeat from core client for 30 sec - exiting 02:16:05 (4220): No heartbeat from core client for 30 sec - exiting 02:16:06 (4220): No heartbeat from core client for 30 sec - exiting 02:16:07 (4220): No heartbeat from core client for 30 sec - exiting 02:16:08 (4220): No heartbeat from core client for 30 sec - exiting 02:16:24 (4220): No heartbeat from core client for 30 sec - exiting 02:16:29 (4220): No heartbeat from core client for 30 sec - exiting 02:16:30 (4220): No heartbeat from core client for 30 sec - exiting 02:16:33 (4220): No heartbeat from core client for 30 sec - exiting 02:16:37 (4220): No heartbeat from core client for 30 sec - exiting 02:16:38 (4220): No heartbeat from core client for 30 sec - exiting 02:16:41 (4220): No heartbeat from core client for 30 sec - exiting 02:16:42 (4220): No heartbeat from core client for 30 sec - exiting 02:16:45 (4220): No heartbeat from core client for 30 sec - exiting 02:16:46 (4220): No heartbeat from core client for 30 sec - exiting 02:16:49 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:16:50 (4220): No heartbeat from core client for 30 sec - exiting 02:16:53 (4220): No heartbeat from core client for 30 sec - exiting 02:16:55 (4220): No heartbeat from core client for 30 sec - exiting 02:16:56 (4220): No heartbeat from core client for 30 sec - exiting 03:39:45 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:51:47 (5216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:56 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:56:07 (5684): No heartbeat from core client for 30 sec - exiting 01:56:08 (5684): No heartbeat from core client for 30 sec - exiting 01:56:09 (5684): No heartbeat from core client for 30 sec - exiting 01:56:10 (5684): No heartbeat from core client for 30 sec - exiting 01:56:11 (5684): No heartbeat from core client for 30 sec - exiting 01:56:12 (5684): No heartbeat from core client for 30 sec - exiting 01:56:13 (5684): No heartbeat from core client for 30 sec - exiting 01:56:14 (5684): No heartbeat from core client for 30 sec - exiting 01:56:16 (5684): No heartbeat from core client for 30 sec - exiting 01:56:17 (5684): No heartbeat from core client for 30 sec - exiting 01:56:18 (5684): No heartbeat from core client for 30 sec - exiting 09:21:46 (3884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:21:50 (3884): No heartbeat from core client for 30 sec - exiting 01:55:39 (3720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:59 (3720): No heartbeat from core client for 30 sec - exiting 01:56:01 (3720): No heartbeat from core client for 30 sec - exiting 01:56:02 (3720): No heartbeat from core client for 30 sec - exiting 01:56:03 (3720): No heartbeat from core client for 30 sec - exiting 01:56:04 (3720): No heartbeat from core client for 30 sec - exiting 01:56:05 (3720): No heartbeat from core client for 30 sec - exiting 01:56:06 (3720): No heartbeat from core client for 30 sec - exiting 01:56:07 (3720): No heartbeat from core client for 30 sec - exiting 01:56:08 (3720): No heartbeat from core client for 30 sec - exiting 01:56:09 (3720): No heartbeat from core client for 30 sec - exiting 01:56:10 (3720): No heartbeat from core client for 30 sec - exiting 01:56:11 (3720): No heartbeat from core client for 30 sec - exiting 01:56:13 (3720): No heartbeat from core client for 30 sec - exiting 01:56:14 (3720): No heartbeat from core client for 30 sec - exiting 01:56:15 (3720): No heartbeat from core client for 30 sec - exiting 01:56:16 (3720): No heartbeat from core client for 30 sec - exiting 01:56:17 (3720): No heartbeat from core client for 30 sec - exiting 01:56:18 (3720): No heartbeat from core client for 30 sec - exiting 01:56:19 (3720): No heartbeat from core client for 30 sec - exiting 01:56:20 (3720): No heartbeat from core client for 30 sec - exiting 01:56:52 (1616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:52 (1616): No heartbeat from core client for 30 sec - exiting 01:57:53 (1616): No heartbeat from core client for 30 sec - exiting 01:57:54 (1616): No heartbeat from core client for 30 sec - exiting 01:57:55 (1616): No heartbeat from core client for 30 sec - exiting 01:57:56 (1616): No heartbeat from core client for 30 sec - exiting 01:57:57 (1616): No heartbeat from core client for 30 sec - exiting 01:57:58 (1616): No heartbeat from core client for 30 sec - exiting 01:57:59 (1616): No heartbeat from core client for 30 sec - exiting 01:58:01 (1616): No heartbeat from core client for 30 sec - exiting 01:58:02 (1616): No heartbeat from core client for 30 sec - exiting 01:58:03 (1616): No heartbeat from core client for 30 sec - exiting 01:58:04 (1616): No heartbeat from core client for 30 sec - exiting 01:58:05 (1616): No heartbeat from core client for 30 sec - exiting 01:58:06 (1616): No heartbeat from core client for 30 sec - exiting 01:58:07 (1616): No heartbeat from core client for 30 sec - exiting 01:58:08 (1616): No heartbeat from core client for 30 sec - exiting 01:58:09 (1616): No heartbeat from core client for 30 sec - exiting 02:23:06 (6876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3672, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3672, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2012 12:31:55	1122757	14842513	hadcm3n_u6xu_1980_40_007998992_4	129,600	418,232	3.2271
29 Jun 2012 16:40:34	1122757	14842513	hadcm3n_u6xu_1980_40_007998992_4	103,680	328,590	3.1693
28 Jun 2012 15:35:20	1122757	14842513	hadcm3n_u6xu_1980_40_007998992_4	77,760	240,950	3.0986
27 Jun 2012 15:50:50	1122757	14842513	hadcm3n_u6xu_1980_40_007998992_4	51,840	158,047	3.0487
26 Jun 2012 16:07:58	1122757	14842513	hadcm3n_u6xu_1980_40_007998992_4	25,920	76,143	2.9376