Task 13416401

Name	hadcm3n_u6n0_1980_40_007460697_4
Workunit	7658200
Created	24 Sep 2011, 1:49:22 UTC
Sent	24 Sep 2011, 1:59:22 UTC
Report deadline	24 Dec 2011, 9:26:33 UTC
Received	14 Oct 2011, 19:26:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1020546
Run time	16 days 2 hours 55 min 27 sec
CPU time	15 days 16 hours 28 min 31 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	2.54 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:02:50 (4948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:03:06 (4948): No heartbeat from core client for 30 sec - exiting 14:03:07 (4948): No heartbeat from core client for 30 sec - exiting 14:03:08 (4948): No heartbeat from core client for 30 sec - exiting 14:07:39 (7028): No heartbeat from core client for 30 sec - exiting 14:07:40 (7028): No heartbeat from core client for 30 sec - exiting 14:07:41 (7028): No heartbeat from core client for 30 sec - exiting 14:07:42 (7028): No heartbeat from core client for 30 sec - exiting 14:07:43 (7028): No heartbeat from core client for 30 sec - exiting 14:07:44 (7028): No heartbeat from core client for 30 sec - exiting 14:07:45 (7028): No heartbeat from core client for 30 sec - exiting 14:07:46 (7028): No heartbeat from core client for 30 sec - exiting 14:07:47 (7028): No heartbeat from core client for 30 sec - exiting 14:07:48 (7028): No heartbeat from core client for 30 sec - exiting 14:07:49 (7028): No heartbeat from core client for 30 sec - exiting 14:07:50 (7028): No heartbeat from core client for 30 sec - exiting 14:07:51 (7028): No heartbeat from core client for 30 sec - exiting 14:07:52 (7028): No heartbeat from core client for 30 sec - exiting 14:07:53 (7028): No heartbeat from core client for 30 sec - exiting 14:07:54 (7028): No heartbeat from core client for 30 sec - exiting 14:07:55 (7028): No heartbeat from core client for 30 sec - exiting 14:07:56 (7028): No heartbeat from core client for 30 sec - exiting 14:07:57 (7028): No heartbeat from core client for 30 sec - exiting 14:07:58 (7028): No heartbeat from core client for 30 sec - exiting 14:07:59 (7028): No heartbeat from core client for 30 sec - exiting 14:08:00 (7028): No heartbeat from core client for 30 sec - exiting 14:08:01 (7028): No heartbeat from core client for 30 sec - exiting 14:08:02 (7028): No heartbeat from core client for 30 sec - exiting 14:08:03 (7028): No heartbeat from core client for 30 sec - exiting 14:08:04 (7028): No heartbeat from core client for 30 sec - exiting 14:08:05 (7028): No heartbeat from core client for 30 sec - exiting 14:08:06 (7028): No heartbeat from core client for 30 sec - exiting 14:08:07 (7028): No heartbeat from core client for 30 sec - exiting 14:08:08 (7028): No heartbeat from core client for 30 sec - exiting 14:08:09 (7028): No heartbeat from core client for 30 sec - exiting 14:08:10 (7028): No heartbeat from core client for 30 sec - exiting 14:08:11 (7028): No heartbeat from core client for 30 sec - exiting 14:08:12 (7028): No heartbeat from core client for 30 sec - exiting 14:08:13 (7028): No heartbeat from core client for 30 sec - exiting 14:08:14 (7028): No heartbeat from core client for 30 sec - exiting 14:08:15 (7028): No heartbeat from core client for 30 sec - exiting 14:08:16 (7028): No heartbeat from core client for 30 sec - exiting 14:08:17 (7028): No heartbeat from core client for 30 sec - exiting 14:08:18 (7028): No heartbeat from core client for 30 sec - exiting 14:08:19 (7028): No heartbeat from core client for 30 sec - exiting 14:08:20 (7028): No heartbeat from core client for 30 sec - exiting 14:08:21 (7028): No heartbeat from core client for 30 sec - exiting 14:08:22 (7028): No heartbeat from core client for 30 sec - exiting 14:08:23 (7028): No heartbeat from core client for 30 sec - exiting 14:08:24 (7028): No heartbeat from core client for 30 sec - exiting 14:08:25 (7028): No heartbeat from core client for 30 sec - exiting 14:08:26 (7028): No heartbeat from core client for 30 sec - exiting 14:08:27 (7028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:48:27 (2420): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 14:48:36 (2420): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 06:40:55 (6788): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 06:41:03 (6788): No heartbeat from core client for 30 sec - exiting 06:41:04 (6788): No heartbeat from core client for 30 sec - exiting 06:41:05 (6788): No heartbeat from core client for 30 sec - exiting 15:13:14 (5328): No heartbeat from core client for 30 sec - exiting 15:13:15 (5328): No heartbeat from core client for 30 sec - exiting 15:13:16 (5328): No heartbeat from core client for 30 sec - exiting 15:13:17 (5328): No heartbeat from core client for 30 sec - exiting 15:13:18 (5328): No heartbeat from core client for 30 sec - exiting 15:13:19 (5328): No heartbeat from core client for 30 sec - exiting 15:13:20 (5328): No heartbeat from core client for 30 sec - exiting 15:13:21 (5328): No heartbeat from core client for 30 sec - exiting 15:13:22 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:13:23 (5328): No heartbeat from core client for 30 sec - exiting 15:13:24 (5328): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:42:47 (6360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:25:53 (10668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:40:19 (15368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:51:42 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:31:58 (14600): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 20:20:11 (11336): No heartbeat from core client for 30 sec - exiting 20:20:12 (11336): No heartbeat from core client for 30 sec - exiting 20:20:13 (11336): No heartbeat from core client for 30 sec - exiting 20:20:14 (11336): No heartbeat from core client for 30 sec - exiting 20:20:15 (11336): No heartbeat from core client for 30 sec - exiting 20:20:16 (11336): No heartbeat from core client for 30 sec - exiting 20:20:17 (11336): No heartbeat from core client for 30 sec - exiting 20:20:18 (11336): No heartbeat from core client for 30 sec - exiting 20:20:19 (11336): No heartbeat from core client for 30 sec - exiting 20:20:20 (11336): No heartbeat from core client for 30 sec - exiting 20:20:21 (11336): No heartbeat from core client for 30 sec - exiting 20:20:22 (11336): No heartbeat from core client for 30 sec - exiting 20:20:23 (11336): No heartbeat from core client for 30 sec - exiting 20:20:24 (11336): No heartbeat from core client for 30 sec - exiting 20:20:25 (11336): No heartbeat from core client for 30 sec - exiting 20:20:26 (11336): No heartbeat from core client for 30 sec - exiting 20:20:27 (11336): No heartbeat from core client for 30 sec - exiting 20:20:28 (11336): No heartbeat from core client for 30 sec - exiting 20:20:29 (11336): No heartbeat from core client for 30 sec - exiting 20:20:30 (11336): No heartbeat from core client for 30 sec - exiting 20:20:31 (11336): No heartbeat from core client for 30 sec - exiting 20:20:32 (11336): No heartbeat from core client for 30 sec - exiting 20:20:33 (11336): No heartbeat from core client for 30 sec - exiting 20:20:34 (11336): No heartbeat from core client for 30 sec - exiting 20:20:35 (11336): No heartbeat from core client for 30 sec - exiting 20:20:36 (11336): No heartbeat from core client for 30 sec - exiting 20:20:37 (11336): No heartbeat from core client for 30 sec - exiting 20:20:38 (11336): No heartbeat from core client for 30 sec - exiting 20:20:39 (11336): No heartbeat from core client for 30 sec - exiting 20:20:40 (11336): No heartbeat from core client for 30 sec - exiting 20:20:41 (11336): No heartbeat from core client for 30 sec - exiting 20:20:42 (11336): No heartbeat from core client for 30 sec - exiting 20:20:43 (11336): No heartbeat from core client for 30 sec - exiting 20:20:44 (11336): No heartbeat from core client for 30 sec - exiting 20:20:45 (11336): No heartbeat from core client for 30 sec - exiting 20:20:46 (11336): No heartbeat from core client for 30 sec - exiting 20:20:47 (11336): No heartbeat from core client for 30 sec - exiting 20:20:48 (11336): No heartbeat from core client for 30 sec - exiting 20:20:49 (11336): No heartbeat from core client for 30 sec - exiting 20:20:50 (11336): No heartbeat from core client for 30 sec - exiting 20:20:51 (11336): No heartbeat from core client for 30 sec - exiting 20:20:52 (11336): No heartbeat from core client for 30 sec - exiting 20:20:53 (11336): No heartbeat from core client for 30 sec - exiting 20:20:54 (11336): No heartbeat from core client for 30 sec - exiting 20:20:55 (11336): No heartbeat from core client for 30 sec - exiting 20:20:56 (11336): No heartbeat from core client for 30 sec - exiting 20:20:57 (11336): No heartbeat from core client for 30 sec - exiting 20:20:58 (11336): No heartbeat from core client for 30 sec - exiting 20:20:59 (11336): No heartbeat from core client for 30 sec - exiting 20:21:00 (11336): No heartbeat from core client for 30 sec - exiting 20:21:01 (11336): No heartbeat from core client for 30 sec - exiting 20:21:02 (11336): No heartbeat from core client for 30 sec - exiting 20:21:03 (11336): No heartbeat from core client for 30 sec - exiting 20:21:04 (11336): No heartbeat from core client for 30 sec - exiting 20:21:05 (11336): No heartbeat from core client for 30 sec - exiting 20:21:06 (11336): No heartbeat from core client for 30 sec - exiting 20:21:07 (11336): No heartbeat from core client for 30 sec - exiting 20:21:08 (11336): No heartbeat from core client for 30 sec - exiting 20:21:09 (11336): No heartbeat from core client for 30 sec - exiting 20:21:10 (11336): No heartbeat from core client for 30 sec - exiting 20:21:11 (11336): No heartbeat from core client for 30 sec - exiting 20:21:12 (11336): No heartbeat from core client for 30 sec - exiting 20:21:13 (11336): No heartbeat from core client for 30 sec - exiting 20:21:14 (11336): No heartbeat from core client for 30 sec - exiting 20:21:15 (11336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:21:16 (11336): No heartbeat from core client for 30 sec - exiting 20:21:17 (11336): No heartbeat from core client for 30 sec - exiting 20:25:02 (12144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:08 (12144): No heartbeat from core client for 30 sec - exiting 20:26:42 (9296): No heartbeat from core client for 30 sec - exiting 20:26:55 (9296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:26:56 (9296): No heartbeat from core client for 30 sec - exiting 20:26:57 (9296): No heartbeat from core client for 30 sec - exiting 20:26:58 (9296): No heartbeat from core client for 30 sec - exiting 20:26:59 (9296): No heartbeat from core client for 30 sec - exiting 20:27:00 (9296): No heartbeat from core client for 30 sec - exiting 20:27:02 (9296): No heartbeat from core client for 30 sec - exiting 20:27:03 (9296): No heartbeat from core client for 30 sec - exiting 20:27:04 (9296): No heartbeat from core client for 30 sec - exiting 20:27:05 (9296): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:43:19 (8104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:34:37 (7576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:27:42 (7016): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 16:27:47 (7016): No heartbeat from core client for 30 sec - exiting 16:27:48 (7016): No heartbeat from core client for 30 sec - exiting 16:27:49 (7016): No heartbeat from core client for 30 sec - exiting 01:29:28 (15488): No heartbeat from core client for 30 sec - exiting 01:29:29 (15488): No heartbeat from core client for 30 sec - exiting 01:29:30 (15488): No heartbeat from core client for 30 sec - exiting 01:29:31 (15488): No heartbeat from core client for 30 sec - exiting 01:29:32 (15488): No heartbeat from core client for 30 sec - exiting 01:29:33 (15488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 21:53:56 (15476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:19:41 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 07:45:43 (5832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:07:20 (1140): No heartbeat from core client for 30 sec - exiting 20:07:21 (1140): No heartbeat from core client for 30 sec - exiting 20:07:22 (1140): No heartbeat from core client for 30 sec - exiting 20:07:23 (1140): No heartbeat from core client for 30 sec - exiting 20:07:24 (1140): No heartbeat from core client for 30 sec - exiting 20:07:25 (1140): No heartbeat from core client for 30 sec - exiting 20:07:26 (1140): No heartbeat from core client for 30 sec - exiting 20:07:27 (1140): No heartbeat from core client for 30 sec - exiting 20:07:28 (1140): No heartbeat from core client for 30 sec - exiting 20:07:29 (1140): No heartbeat from core client for 30 sec - exiting 20:07:30 (1140): No heartbeat from core client for 30 sec - exiting 20:07:31 (1140): No heartbeat from core client for 30 sec - exiting 20:07:32 (1140): No heartbeat from core client for 30 sec - exiting 20:07:33 (1140): No heartbeat from core client for 30 sec - exiting 20:07:34 (1140): No heartbeat from core client for 30 sec - exiting 20:07:35 (1140): No heartbeat from core client for 30 sec - exiting 20:07:36 (1140): No heartbeat from core client for 30 sec - exiting 20:07:37 (1140): No heartbeat from core client for 30 sec - exiting 20:07:38 (1140): No heartbeat from core client for 30 sec - exiting 20:07:39 (1140): No heartbeat from core client for 30 sec - exiting 20:07:40 (1140): No heartbeat from core client for 30 sec - exiting 20:07:41 (1140): No heartbeat from core client for 30 sec - exiting 20:07:42 (1140): No heartbeat from core client for 30 sec - exiting 20:07:43 (1140): No heartbeat from core client for 30 sec - exiting 20:07:44 (1140): No heartbeat from core client for 30 sec - exiting 20:07:45 (1140): No heartbeat from core client for 30 sec - exiting 20:07:46 (1140): No heartbeat from core client for 30 sec - exiting 20:07:47 (1140): No heartbeat from core client for 30 sec - exiting 20:07:48 (1140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:07:49 (1140): No heartbeat from core client for 30 sec - exiting 20:09:53 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:37 (6856): No heartbeat from core client for 30 sec - exiting 21:23:39 (6856): No heartbeat from core client for 30 sec - exiting 21:23:40 (6856): No heartbeat from core client for 30 sec - exiting 21:23:41 (6856): No heartbeat from core client for 30 sec - exiting 21:23:42 (6856): No heartbeat from core client for 30 sec - exiting 21:23:43 (6856): No heartbeat from core client for 30 sec - exiting 21:23:44 (6856): No heartbeat from core client for 30 sec - exiting 21:23:45 (6856): No heartbeat from core client for 30 sec - exiting 21:23:46 (6856): No heartbeat from core client for 30 sec - exiting 21:23:47 (6856): No heartbeat from core client for 30 sec - exiting 21:23:48 (6856): No heartbeat from core client for 30 sec - exiting 21:23:49 (6856): No heartbeat from core client for 30 sec - exiting 21:23:50 (6856): No heartbeat from core client for 30 sec - exiting 21:23:51 (6856): No heartbeat from core client for 30 sec - exiting 21:23:52 (6856): No heartbeat from core client for 30 sec - exiting 21:23:53 (6856): No heartbeat from core client for 30 sec - exiting 21:23:54 (6856): No heartbeat from core client for 30 sec - exiting 21:23:55 (6856): No heartbeat from core client for 30 sec - exiting 21:23:56 (6856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8688, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8688, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Oct 2011 07:02:42	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	673,920	1,328,354	1.9711
13 Oct 2011 14:36:34	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	648,000	1,277,200	1.9710
12 Oct 2011 23:45:18	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	622,080	1,225,641	1.9702
12 Oct 2011 08:04:40	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	596,160	1,174,069	1.9694
11 Oct 2011 07:55:27	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	570,240	1,123,848	1.9708
10 Oct 2011 17:47:27	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	544,320	1,072,767	1.9708
10 Oct 2011 03:13:28	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	518,400	1,021,527	1.9705
09 Oct 2011 11:29:01	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	492,480	969,127	1.9679
08 Oct 2011 20:55:36	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	466,560	917,784	1.9671
08 Oct 2011 07:16:23	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	440,640	866,963	1.9675
07 Oct 2011 09:51:46	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	414,720	815,846	1.9672
06 Oct 2011 19:14:00	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	388,800	764,723	1.9669
06 Oct 2011 04:18:46	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	362,880	714,222	1.9682
05 Oct 2011 12:38:56	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	336,960	663,302	1.9685
04 Oct 2011 20:42:13	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	311,040	612,003	1.9676
04 Oct 2011 03:26:35	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	285,120	561,311	1.9687
03 Oct 2011 13:07:51	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	259,200	511,075	1.9717
02 Oct 2011 21:01:42	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	233,280	459,696	1.9706
02 Oct 2011 04:32:26	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	207,360	407,544	1.9654
01 Oct 2011 12:47:18	1020546	13416401	hadcm3n_u6n0_1980_40_007460697_4	181,440	356,247	1.9634