Task 13668513

Name	hadcm3n_u41e_1980_40_007546104_1
Workunit	7743336
Created	29 Nov 2011, 2:09:13 UTC
Sent	29 Nov 2011, 2:12:04 UTC
Report deadline	28 Feb 2012, 9:39:15 UTC
Received	1 Dec 2011, 17:03:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122757
Run time	2 days 2 hours 53 min 56 sec
CPU time	2 days 2 hours 1 min 40 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	1.69 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 22:50:47 (1700): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 15:19:43 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:01 (3456): No heartbeat from core client for 30 sec - exiting 15:20:02 (3456): No heartbeat from core client for 30 sec - exiting 15:20:03 (3456): No heartbeat from core client for 30 sec - exiting 15:20:04 (3456): No heartbeat from core client for 30 sec - exiting 15:20:05 (3456): No heartbeat from core client for 30 sec - exiting 15:20:06 (3456): No heartbeat from core client for 30 sec - exiting 15:20:07 (3456): No heartbeat from core client for 30 sec - exiting 15:20:08 (3456): No heartbeat from core client for 30 sec - exiting 15:20:10 (3456): No heartbeat from core client for 30 sec - exiting 15:20:11 (3456): No heartbeat from core client for 30 sec - exiting 15:20:12 (3456): No heartbeat from core client for 30 sec - exiting 15:20:13 (3456): No heartbeat from core client for 30 sec - exiting 15:20:14 (3456): No heartbeat from core client for 30 sec - exiting 15:20:15 (3456): No heartbeat from core client for 30 sec - exiting 15:20:16 (3456): No heartbeat from core client for 30 sec - exiting 15:20:17 (3456): No heartbeat from core client for 30 sec - exiting 15:20:18 (3456): No heartbeat from core client for 30 sec - exiting 15:20:19 (3456): No heartbeat from core client for 30 sec - exiting 15:20:20 (3456): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:08:50 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:09:01 (5080): No heartbeat from core client for 30 sec - exiting 00:09:02 (5080): No heartbeat from core client for 30 sec - exiting 00:09:03 (5080): No heartbeat from core client for 30 sec - exiting 00:09:04 (5080): No heartbeat from core client for 30 sec - exiting 00:09:05 (5080): No heartbeat from core client for 30 sec - exiting 00:09:06 (5080): No heartbeat from core client for 30 sec - exiting 00:09:07 (5080): No heartbeat from core client for 30 sec - exiting 00:09:08 (5080): No heartbeat from core client for 30 sec - exiting 00:09:10 (5080): No heartbeat from core client for 30 sec - exiting 00:09:11 (5080): No heartbeat from core client for 30 sec - exiting 00:12:45 (4712): No heartbeat from core client for 30 sec - exiting 00:12:54 (4712): No heartbeat from core client for 30 sec - exiting 00:12:55 (4712): No heartbeat from core client for 30 sec - exiting 00:12:56 (4712): No heartbeat from core client for 30 sec - exiting 00:12:57 (4712): No heartbeat from core client for 30 sec - exiting 00:12:58 (4712): No heartbeat from core client for 30 sec - exiting 00:12:59 (4712): No heartbeat from core client for 30 sec - exiting 00:13:01 (4712): No heartbeat from core client for 30 sec - exiting 00:13:02 (4712): No heartbeat from core client for 30 sec - exiting 00:13:03 (4712): No heartbeat from core client for 30 sec - exiting 00:13:04 (4712): No heartbeat from core client for 30 sec - exiting 00:13:05 (4712): No heartbeat from core client for 30 sec - exiting 00:13:06 (4712): No heartbeat from core client for 30 sec - exiting 00:13:07 (4712): No heartbeat from core client for 30 sec - exiting 00:13:08 (4712): No heartbeat from core client for 30 sec - exiting 00:13:09 (4712): No heartbeat from core client for 30 sec - exiting 00:13:10 (4712): No heartbeat from core client for 30 sec - exiting 00:13:11 (4712): No heartbeat from core client for 30 sec - exiting 00:13:13 (4712): No heartbeat from core client for 30 sec - exiting 00:13:14 (4712): No heartbeat from core client for 30 sec - exiting 00:13:15 (4712): No heartbeat from core client for 30 sec - exiting 00:13:16 (4712): No heartbeat from core client for 30 sec - exiting 00:13:17 (4712): No heartbeat from core client for 30 sec - exiting 00:13:55 (4712): No heartbeat from core client for 30 sec - exiting 00:13:56 (4712): No heartbeat from core client for 30 sec - exiting 00:13:57 (4712): No heartbeat from core client for 30 sec - exiting 00:13:58 (4712): No heartbeat from core client for 30 sec - exiting 00:14:00 (4712): No heartbeat from core client for 30 sec - exiting 00:14:01 (4712): No heartbeat from core client for 30 sec - exiting 00:14:02 (4712): No heartbeat from core client for 30 sec - exiting 00:14:03 (4712): No heartbeat from core client for 30 sec - exiting 00:14:04 (4712): No heartbeat from core client for 30 sec - exiting 00:14:05 (4712): No heartbeat from core client for 30 sec - exiting 00:14:06 (4712): No heartbeat from core client for 30 sec - exiting 00:14:07 (4712): No heartbeat from core client for 30 sec - exiting 00:14:08 (4712): No heartbeat from core client for 30 sec - exiting 00:14:09 (4712): No heartbeat from core client for 30 sec - exiting 00:14:10 (4712): No heartbeat from core client for 30 sec - exiting 00:14:12 (4712): No heartbeat from core client for 30 sec - exiting 00:14:13 (4712): No heartbeat from core client for 30 sec - exiting 00:14:14 (4712): No heartbeat from core client for 30 sec - exiting 00:14:15 (4712): No heartbeat from core client for 30 sec - exiting 00:14:16 (4712): No heartbeat from core client for 30 sec - exiting 00:14:17 (4712): No heartbeat from core client for 30 sec - exiting 00:14:18 (4712): No heartbeat from core client for 30 sec - exiting 00:14:19 (4712): No heartbeat from core client for 30 sec - exiting 00:14:20 (4712): No heartbeat from core client for 30 sec - exiting 00:14:21 (4712): No heartbeat from core client for 30 sec - exiting 00:14:22 (4712): No heartbeat from core client for 30 sec - exiting 00:14:24 (4712): No heartbeat from core client for 30 sec - exiting 00:14:25 (4712): No heartbeat from core client for 30 sec - exiting 00:14:26 (4712): No heartbeat from core client for 30 sec - exiting 00:14:27 (4712): No heartbeat from core client for 30 sec - exiting 00:14:28 (4712): No heartbeat from core client for 30 sec - exiting 00:14:29 (4712): No heartbeat from core client for 30 sec - exiting 00:14:30 (4712): No heartbeat from core client for 30 sec - exiting 00:14:31 (4712): No heartbeat from core client for 30 sec - exiting 00:14:32 (4712): No heartbeat from core client for 30 sec - exiting 00:14:33 (4712): No heartbeat from core client for 30 sec - exiting 00:14:34 (4712): No heartbeat from core client for 30 sec - exiting 00:14:36 (4712): No heartbeat from core client for 30 sec - exiting 00:14:37 (4712): No heartbeat from core client for 30 sec - exiting 00:14:38 (4712): No heartbeat from core client for 30 sec - exiting 00:14:39 (4712): No heartbeat from core client for 30 sec - exiting 00:14:40 (4712): No heartbeat from core client for 30 sec - exiting 00:14:41 (4712): No heartbeat from core client for 30 sec - exiting 00:14:42 (4712): No heartbeat from core client for 30 sec - exiting 00:14:43 (4712): No heartbeat from core client for 30 sec - exiting 00:14:44 (4712): No heartbeat from core client for 30 sec - exiting 00:14:45 (4712): No heartbeat from core client for 30 sec - exiting 00:14:46 (4712): No heartbeat from core client for 30 sec - exiting 00:14:48 (4712): No heartbeat from core client for 30 sec - exiting 00:14:49 (4712): No heartbeat from core client for 30 sec - exiting 00:14:50 (4712): No heartbeat from core client for 30 sec - exiting 00:14:51 (4712): No heartbeat from core client for 30 sec - exiting 00:14:52 (4712): No heartbeat from core client for 30 sec - exiting 00:14:53 (4712): No heartbeat from core client for 30 sec - exiting 00:14:54 (4712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:55 (4712): No heartbeat from core client for 30 sec - exiting 00:14:56 (4712): No heartbeat from core client for 30 sec - exiting 00:14:57 (4712): No heartbeat from core client for 30 sec - exiting 00:14:58 (4712): No heartbeat from core client for 30 sec - exiting 00:15:00 (4712): No heartbeat from core client for 30 sec - exiting 00:15:01 (4712): No heartbeat from core client for 30 sec - exiting 03:16:49 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:16:53 (5684): No heartbeat from core client for 30 sec - exiting 03:18:30 (4348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:12:56 (5592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:13:57 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:21:29 (5264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:21:33 (5264): No heartbeat from core client for 30 sec - exiting 04:21:35 (5264): No heartbeat from core client for 30 sec - exiting 04:21:36 (5264): No heartbeat from core client for 30 sec - exiting 04:21:37 (5264): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 22:55:47 (5756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:56:44 (1076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:21 (5944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:03:36 (776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5888, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5888, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5888, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5888, iMonCtr=1 Model crash detected, will try to restart... 08:04:53 (5888): No heartbeat from core client for 30 sec - exiting 08:04:54 (5888): No heartbeat from core client for 30 sec - exiting 08:04:55 (5888): No heartbeat from core client for 30 sec - exiting 08:04:56 (5888): No heartbeat from core client for 30 sec - exiting 08:04:57 (5888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5636, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5636, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Dec 2011 11:07:00	1122757	13668513	hadcm3n_u41e_1980_40_007546104_1	51,840	159,593	3.0786
30 Nov 2011 12:37:41	1122757	13668513	hadcm3n_u41e_1980_40_007546104_1	25,920	82,087	3.1669