Task 16010790

Name	hadcm3n_7vwg_1980_40_008452627_1
Workunit	8603483
Created	10 Sep 2013, 15:05:13 UTC
Sent	10 Sep 2013, 15:42:02 UTC
Report deadline	10 Dec 2013, 23:09:13 UTC
Received	19 Sep 2013, 23:53:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1230461
Run time	2 days 6 hours 12 min 9 sec
CPU time	1 days 20 hours 23 min 5 sec
Validate state	Invalid
Credit	933.12
Device peak FLOPS	2.77 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 16:13:22 (67356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:54:47 (63208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:33 (66208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:41:00 (67776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:49:26 (63732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:26:40 (68252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:09 (68428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:18 (68428): No heartbeat from core client for 30 sec - exiting 22:24:19 (68428): No heartbeat from core client for 30 sec - exiting 22:24:20 (68428): No heartbeat from core client for 30 sec - exiting 22:24:21 (68428): No heartbeat from core client for 30 sec - exiting 22:24:22 (68428): No heartbeat from core client for 30 sec - exiting 22:24:23 (68428): No heartbeat from core client for 30 sec - exiting 22:24:24 (68428): No heartbeat from core client for 30 sec - exiting 22:24:25 (68428): No heartbeat from core client for 30 sec - exiting 22:24:26 (68428): No heartbeat from core client for 30 sec - exiting 22:24:27 (68428): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 23:05:12 (68368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:05:13 (68368): No heartbeat from core client for 30 sec - exiting 23:05:14 (68368): No heartbeat from core client for 30 sec - exiting 23:05:15 (68368): No heartbeat from core client for 30 sec - exiting 23:05:16 (68368): No heartbeat from core client for 30 sec - exiting 23:05:17 (68368): No heartbeat from core client for 30 sec - exiting 23:05:18 (68368): No heartbeat from core client for 30 sec - exiting 23:05:19 (68368): No heartbeat from core client for 30 sec - exiting 23:05:20 (68368): No heartbeat from core client for 30 sec - exiting 23:05:21 (68368): No heartbeat from core client for 30 sec - exiting 23:05:22 (68368): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 23:27:16 (51432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:27:17 (51432): No heartbeat from core client for 30 sec - exiting 23:27:18 (51432): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 00:22:00 (63732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:29:46 (66108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:42 (67788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:53 (68124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:38:53 (48740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:50:04 (67176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:25:47 (67092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:48 (67092): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 03:34:31 (47892): No heartbeat from core client for 30 sec - exiting 03:34:32 (47892): No heartbeat from core client for 30 sec - exiting 03:34:33 (47892): No heartbeat from core client for 30 sec - exiting 03:34:34 (47892): No heartbeat from core client for 30 sec - exiting 03:34:35 (47892): No heartbeat from core client for 30 sec - exiting 03:34:36 (47892): No heartbeat from core client for 30 sec - exiting 03:34:37 (47892): No heartbeat from core client for 30 sec - exiting 03:34:38 (47892): No heartbeat from core client for 30 sec - exiting 03:34:39 (47892): No heartbeat from core client for 30 sec - exiting 03:34:40 (47892): No heartbeat from core client for 30 sec - exiting 03:34:41 (47892): No heartbeat from core client for 30 sec - exiting 03:34:42 (47892): No heartbeat from core client for 30 sec - exiting 03:34:43 (47892): No heartbeat from core client for 30 sec - exiting 03:34:44 (47892): No heartbeat from core client for 30 sec - exiting 03:34:45 (47892): No heartbeat from core client for 30 sec - exiting 03:34:46 (47892): No heartbeat from core client for 30 sec - exiting 03:34:47 (47892): No heartbeat from core client for 30 sec - exiting 03:34:48 (47892): No heartbeat from core client for 30 sec - exiting 03:34:49 (47892): No heartbeat from core client for 30 sec - exiting 03:34:50 (47892): No heartbeat from core client for 30 sec - exiting 03:34:51 (47892): No heartbeat from core client for 30 sec - exiting 03:34:52 (47892): No heartbeat from core client for 30 sec - exiting 03:34:53 (47892): No heartbeat from core client for 30 sec - exiting 03:34:54 (47892): No heartbeat from core client for 30 sec - exiting 03:34:55 (47892): No heartbeat from core client for 30 sec - exiting 03:34:56 (47892): No heartbeat from core client for 30 sec - exiting 03:34:57 (47892): No heartbeat from core client for 30 sec - exiting 03:34:58 (47892): No heartbeat from core client for 30 sec - exiting 03:34:59 (47892): No heartbeat from core client for 30 sec - exiting 03:35:00 (47892): No heartbeat from core client for 30 sec - exiting 03:35:01 (47892): No heartbeat from core client for 30 sec - exiting 03:35:02 (47892): No heartbeat from core client for 30 sec - exiting 03:35:03 (47892): No heartbeat from core client for 30 sec - exiting 03:35:04 (47892): No heartbeat from core client for 30 sec - exiting 03:35:05 (47892): No heartbeat from core client for 30 sec - exiting 03:35:06 (47892): No heartbeat from core client for 30 sec - exiting 03:35:07 (47892): No heartbeat from core client for 30 sec - exiting 03:35:08 (47892): No heartbeat from core client for 30 sec - exiting 03:35:09 (47892): No heartbeat from core client for 30 sec - exiting 03:35:10 (47892): No heartbeat from core client for 30 sec - exiting 03:35:11 (47892): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:11:42 (68300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:11:43 (68300): No heartbeat from core client for 30 sec - exiting 05:11:44 (68300): No heartbeat from core client for 30 sec - exiting 05:11:45 (68300): No heartbeat from core client for 30 sec - exiting 05:11:46 (68300): No heartbeat from core client for 30 sec - exiting 05:11:47 (68300): No heartbeat from core client for 30 sec - exiting 05:11:48 (68300): No heartbeat from core client for 30 sec - exiting 05:11:49 (68300): No heartbeat from core client for 30 sec - exiting 05:11:50 (68300): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... 05:25:23 (63752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:25:24 (63752): No heartbeat from core client for 30 sec - exiting 08:41:55 (67088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:57:10 (60232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:15 (63940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:16 (63940): No heartbeat from core client for 30 sec - exiting 08:58:17 (63940): No heartbeat from core client for 30 sec - exiting 08:58:18 (63940): No heartbeat from core client for 30 sec - exiting 08:58:19 (63940): No heartbeat from core client for 30 sec - exiting 08:58:20 (63940): No heartbeat from core client for 30 sec - exiting 08:58:21 (63940): No heartbeat from core client for 30 sec - exiting 08:58:22 (63940): No heartbeat from core client for 30 sec - exiting 08:58:23 (63940): No heartbeat from core client for 30 sec - exiting 08:58:24 (63940): No heartbeat from core client for 30 sec - exiting 08:59:03 (68696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:15:45 (69284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:15:46 (69284): No heartbeat from core client for 30 sec - exiting 09:15:47 (69284): No heartbeat from core client for 30 sec - exiting 09:15:48 (69284): No heartbeat from core client for 30 sec - exiting 09:15:49 (69284): No heartbeat from core client for 30 sec - exiting 09:15:50 (69284): No heartbeat from core client for 30 sec - exiting 09:15:51 (69284): No heartbeat from core client for 30 sec - exiting 09:15:52 (69284): No heartbeat from core client for 30 sec - exiting 09:15:53 (69284): No heartbeat from core client for 30 sec - exiting 09:15:54 (69284): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:55:57 (70900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:58 (70900): No heartbeat from core client for 30 sec - exiting 09:55:59 (70900): No heartbeat from core client for 30 sec - exiting 09:56:00 (70900): No heartbeat from core client for 30 sec - exiting 09:56:01 (70900): No heartbeat from core client for 30 sec - exiting 09:56:02 (70900): No heartbeat from core client for 30 sec - exiting 09:56:03 (70900): No heartbeat from core client for 30 sec - exiting 09:56:04 (70900): No heartbeat from core client for 30 sec - exiting 09:56:05 (70900): No heartbeat from core client for 30 sec - exiting 09:56:06 (70900): No heartbeat from core client for 30 sec - exiting 09:56:07 (70900): No heartbeat from core client for 30 sec - exiting 10:16:58 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:16:59 (5204): No heartbeat from core client for 30 sec - exiting 10:17:00 (5204): No heartbeat from core client for 30 sec - exiting 10:17:01 (5204): No heartbeat from core client for 30 sec - exiting 10:17:02 (5204): No heartbeat from core client for 30 sec - exiting 10:17:03 (5204): No heartbeat from core client for 30 sec - exiting 10:18:46 (74528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:42:31 (74540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:43:03 (74540): No heartbeat from core client for 30 sec - exiting 11:43:04 (74540): No heartbeat from core client for 30 sec - exiting 11:43:05 (74540): No heartbeat from core client for 30 sec - exiting 11:43:06 (74540): No heartbeat from core client for 30 sec - exiting 11:43:07 (74540): No heartbeat from core client for 30 sec - exiting 11:43:08 (74540): No heartbeat from core client for 30 sec - exiting 11:43:09 (74540): No heartbeat from core client for 30 sec - exiting 11:43:10 (74540): No heartbeat from core client for 30 sec - exiting 11:43:11 (74540): No heartbeat from core client for 30 sec - exiting 11:43:12 (74540): No heartbeat from core client for 30 sec - exiting 11:43:13 (74540): No heartbeat from core client for 30 sec - exiting 11:43:14 (74540): No heartbeat from core client for 30 sec - exiting 11:58:36 (77080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:18:58 (82616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:32:43 (80956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:46:26 (85840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:53:52 (84728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:53 (84728): No heartbeat from core client for 30 sec - exiting 13:36:56 (85564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:41:00 (84980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:45:29 (87564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:45:31 (87564): No heartbeat from core client for 30 sec - exiting 13:45:32 (87564): No heartbeat from core client for 30 sec - exiting 13:45:33 (87564): No heartbeat from core client for 30 sec - exiting 13:45:34 (87564): No heartbeat from core client for 30 sec - exiting 13:45:35 (87564): No heartbeat from core client for 30 sec - exiting 13:45:36 (87564): No heartbeat from core client for 30 sec - exiting 13:45:37 (87564): No heartbeat from core client for 30 sec - exiting 13:45:38 (87564): No heartbeat from core client for 30 sec - exiting 13:45:39 (87564): No heartbeat from core client for 30 sec - exiting 13:45:40 (87564): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 14:09:43 (88640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:04 (87628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:03:45 (87628): No heartbeat from core client for 30 sec - exiting 15:03:46 (87628): No heartbeat from core client for 30 sec - exiting 15:03:47 (87628): No heartbeat from core client for 30 sec - exiting 15:03:48 (87628): No heartbeat from core client for 30 sec - exiting 15:03:49 (87628): No heartbeat from core client for 30 sec - exiting 15:03:50 (87628): No heartbeat from core client for 30 sec - exiting 15:03:51 (87628): No heartbeat from core client for 30 sec - exiting 15:03:52 (87628): No heartbeat from core client for 30 sec - exiting 15:03:53 (87628): No heartbeat from core client for 30 sec - exiting 15:03:54 (87628): No heartbeat from core client for 30 sec - exiting 15:03:55 (87628): No heartbeat from core client for 30 sec - exiting 15:03:56 (87628): No heartbeat from core client for 30 sec - exiting 15:03:57 (87628): No heartbeat from core client for 30 sec - exiting 15:03:58 (87628): No heartbeat from core client for 30 sec - exiting 15:03:59 (87628): No heartbeat from core client for 30 sec - exiting 15:04:00 (87628): No heartbeat from core client for 30 sec - exiting 15:04:01 (87628): No heartbeat from core client for 30 sec - exiting 15:04:02 (87628): No heartbeat from core client for 30 sec - exiting 15:04:03 (87628): No heartbeat from core client for 30 sec - exiting 15:04:04 (87628): No heartbeat from core client for 30 sec - exiting 15:04:05 (87628): No heartbeat from core client for 30 sec - exiting 15:04:06 (87628): No heartbeat from core client for 30 sec - exiting 15:04:07 (87628): No heartbeat from core client for 30 sec - exiting 15:04:08 (87628): No heartbeat from core client for 30 sec - exiting 15:04:09 (87628): No heartbeat from core client for 30 sec - exiting 15:04:10 (87628): No heartbeat from core client for 30 sec - exiting 15:04:11 (87628): No heartbeat from core client for 30 sec - exiting 15:04:12 (87628): No heartbeat from core client for 30 sec - exiting 15:04:13 (87628): No heartbeat from core client for 30 sec - exiting 15:04:14 (87628): No heartbeat from core client for 30 sec - exiting 15:04:15 (87628): No heartbeat from core client for 30 sec - exiting 15:04:16 (87628): No heartbeat from core client for 30 sec - exiting 15:04:17 (87628): No heartbeat from core client for 30 sec - exiting 15:04:18 (87628): No heartbeat from core client for 30 sec - exiting 15:04:19 (87628): No heartbeat from core client for 30 sec - exiting 15:04:20 (87628): No heartbeat from core client for 30 sec - exiting 15:04:21 (87628): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 15:36:19 (91476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:09:31 (47988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:09:32 (47988): No heartbeat from core client for 30 sec - exiting 22:09:33 (47988): No heartbeat from core client for 30 sec - exiting 22:09:35 (47988): No heartbeat from core client for 30 sec - exiting 22:09:36 (47988): No heartbeat from core client for 30 sec - exiting 22:09:37 (47988): No heartbeat from core client for 30 sec - exiting 22:09:38 (47988): No heartbeat from core client for 30 sec - exiting 22:09:39 (47988): No heartbeat from core client for 30 sec - exiting 22:09:40 (47988): No heartbeat from core client for 30 sec - exiting 22:09:41 (47988): No heartbeat from core client for 30 sec - exiting 22:09:42 (47988): No heartbeat from core client for 30 sec - exiting 22:42:10 (87736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:42:15 (87736): No heartbeat from core client for 30 sec - exiting 22:42:16 (87736): No heartbeat from core client for 30 sec - exiting 22:42:17 (87736): No heartbeat from core client for 30 sec - exiting 22:42:18 (87736): No heartbeat from core client for 30 sec - exiting 22:42:19 (87736): No heartbeat from core client for 30 sec - exiting 22:42:21 (87736): No heartbeat from core client for 30 sec - exiting 22:42:22 (87736): No heartbeat from core client for 30 sec - exiting 22:42:23 (87736): No heartbeat from core client for 30 sec - exiting 22:42:24 (87736): No heartbeat from core client for 30 sec - exiting 22:42:25 (87736): No heartbeat from core client for 30 sec - exiting 23:04:17 (91212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:11:17 (87580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:28:57 (89248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:29:45 (89248): No heartbeat from core client for 30 sec - exiting 23:29:46 (89248): No heartbeat from core client for 30 sec - exiting 23:29:47 (89248): No heartbeat from core client for 30 sec - exiting 23:29:48 (89248): No heartbeat from core client for 30 sec - exiting 23:29:49 (89248): No heartbeat from core client for 30 sec - exiting 23:29:50 (89248): No heartbeat from core client for 30 sec - exiting 23:29:51 (89248): No heartbeat from core client for 30 sec - exiting 23:29:52 (89248): No heartbeat from core client for 30 sec - exiting 23:29:53 (89248): No heartbeat from core client for 30 sec - exiting 23:29:54 (89248): No heartbeat from core client for 30 sec - exiting 23:33:29 (90256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:12 (92192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:25 (91588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:15:30 (91588): No heartbeat from core client for 30 sec - exiting 00:15:31 (91588): No heartbeat from core client for 30 sec - exiting 00:15:32 (91588): No heartbeat from core client for 30 sec - exiting 00:15:33 (91588): No heartbeat from core client for 30 sec - exiting 00:15:35 (91588): No heartbeat from core client for 30 sec - exiting 00:15:36 (91588): No heartbeat from core client for 30 sec - exiting 00:15:37 (91588): No heartbeat from core client for 30 sec - exiting 00:15:38 (91588): No heartbeat from core client for 30 sec - exiting 00:15:39 (91588): No heartbeat from core client for 30 sec - exiting 00:15:40 (91588): No heartbeat from core client for 30 sec - exiting 00:37:54 (45008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:32 (92500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:55:03 (58260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:56:47 (89964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:45 (92352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:49:14 (6944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:18:27 (88840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:19:22 (77668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:15 (90136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:17:32 (92764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:18:05 (92764): No heartbeat from core client for 30 sec - exiting 05:18:06 (92764): No heartbeat from core client for 30 sec - exiting 05:18:07 (92764): No heartbeat from core client for 30 sec - exiting 05:18:09 (92764): No heartbeat from core client for 30 sec - exiting 05:18:10 (92764): No heartbeat from core client for 30 sec - exiting 05:31:11 (92668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:34 (92792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:35:35 (92792): No heartbeat from core client for 30 sec - exiting 05:35:36 (92792): No heartbeat from core client for 30 sec - exiting 05:35:37 (92792): No heartbeat from core client for 30 sec - exiting 05:35:38 (92792): No heartbeat from core client for 30 sec - exiting 05:35:39 (92792): No heartbeat from core client for 30 sec - exiting 05:35:40 (92792): No heartbeat from core client for 30 sec - exiting 05:35:41 (92792): No heartbeat from core client for 30 sec - exiting 05:35:43 (92792): No heartbeat from core client for 30 sec - exiting 05:35:44 (92792): No heartbeat from core client for 30 sec - exiting 05:35:45 (92792): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 06:28:34 (85592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:28:35 (85592): No heartbeat from core client for 30 sec - exiting 06:28:36 (85592): No heartbeat from core client for 30 sec - exiting 06:28:37 (85592): No heartbeat from core client for 30 sec - exiting 07:18:59 (92796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:46:42 (93088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:46:50 (93088): No heartbeat from core client for 30 sec - exiting 07:46:51 (93088): No heartbeat from core client for 30 sec - exiting 07:46:52 (93088): No heartbeat from core client for 30 sec - exiting 07:46:53 (93088): No heartbeat from core client for 30 sec - exiting 07:46:54 (93088): No heartbeat from core client for 30 sec - exiting 07:46:55 (93088): No heartbeat from core client for 30 sec - exiting 07:46:56 (93088): No heartbeat from core client for 30 sec - exiting 07:46:57 (93088): No heartbeat from core client for 30 sec - exiting 07:46:58 (93088): No heartbeat from core client for 30 sec - exiting 07:46:59 (93088): No heartbeat from core client for 30 sec - exiting 08:44:21 (88912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:22 (88912): No heartbeat from core client for 30 sec - exiting 08:44:23 (88912): No heartbeat from core client for 30 sec - exiting 08:44:24 (88912): No heartbeat from core client for 30 sec - exiting 08:44:25 (88912): No heartbeat from core client for 30 sec - exiting 08:44:26 (88912): No heartbeat from core client for 30 sec - exiting 08:44:27 (88912): No heartbeat from core client for 30 sec - exiting 08:44:28 (88912): No heartbeat from core client for 30 sec - exiting 08:44:29 (88912): No heartbeat from core client for 30 sec - exiting 08:44:30 (88912): No heartbeat from core client for 30 sec - exiting 08:44:32 (88912): No heartbeat from core client for 30 sec - exiting 10:03:55 (92132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:47:11 (91532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:44 (88144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:05:55 (88144): No heartbeat from core client for 30 sec - exiting 11:05:56 (88144): No heartbeat from core client for 30 sec - exiting 11:05:57 (88144): No heartbeat from core client for 30 sec - exiting 11:05:58 (88144): No heartbeat from core client for 30 sec - exiting 11:05:59 (88144): No heartbeat from core client for 30 sec - exiting 11:06:00 (88144): No heartbeat from core client for 30 sec - exiting 11:06:02 (88144): No heartbeat from core client for 30 sec - exiting 11:06:03 (88144): No heartbeat from core client for 30 sec - exiting 11:06:04 (88144): No heartbeat from core client for 30 sec - exiting 11:06:05 (88144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:29:21 (87408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:29:30 (87408): No heartbeat from core client for 30 sec - exiting 14:29:31 (87408): No heartbeat from core client for 30 sec - exiting 14:29:32 (87408): No heartbeat from core client for 30 sec - exiting 14:29:33 (87408): No heartbeat from core client for 30 sec - exiting 14:29:34 (87408): No heartbeat from core client for 30 sec - exiting 14:29:35 (87408): No heartbeat from core client for 30 sec - exiting 14:31:51 (91180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:58:48 (93112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:09:05 (93048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:22:57 (92636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:44:15 (47988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:44:18 (47988): No heartbeat from core client for 30 sec - exiting 16:44:19 (47988): No heartbeat from core client for 30 sec - exiting 16:44:20 (47988): No heartbeat from core client for 30 sec - exiting 16:44:21 (47988): No heartbeat from core client for 30 sec - exiting 16:44:22 (47988): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 17:24:13 (92376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:07 (90128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:39:12 (90128): No heartbeat from core client for 30 sec - exiting 17:39:13 (90128): No heartbeat from core client for 30 sec - exiting 17:39:14 (90128): No heartbeat from core client for 30 sec - exiting 17:39:15 (90128): No heartbeat from core client for 30 sec - exiting 17:39:16 (90128): No heartbeat from core client for 30 sec - exiting 17:45:45 (31520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:45:51 (31520): No heartbeat from core client for 30 sec - exiting 17:45:52 (31520): No heartbeat from core client for 30 sec - exiting 17:45:54 (31520): No heartbeat from core client for 30 sec - exiting 17:45:55 (31520): No heartbeat from core client for 30 sec - exiting 17:45:56 (31520): No heartbeat from core client for 30 sec - exiting 17:45:57 (31520): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:30:41 (87408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:42 (87408): No heartbeat from core client for 30 sec - exiting 18:30:43 (87408): No heartbeat from core client for 30 sec - exiting 18:30:44 (87408): No heartbeat from core client for 30 sec - exiting 18:30:45 (87408): No heartbeat from core client for 30 sec - exiting 18:30:46 (87408): No heartbeat from core client for 30 sec - exiting 19:37:18 (88728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:29 (88728): No heartbeat from core client for 30 sec - exiting 19:37:30 (88728): No heartbeat from core client for 30 sec - exiting 19:37:32 (88728): No heartbeat from core client for 30 sec - exiting 19:37:33 (88728): No heartbeat from core client for 30 sec - exiting 19:37:34 (88728): No heartbeat from core client for 30 sec - exiting 19:37:35 (88728): No heartbeat from core client for 30 sec - exiting 19:37:36 (88728): No heartbeat from core client for 30 sec - exiting 19:37:37 (88728): No heartbeat from core client for 30 sec - exiting 19:37:38 (88728): No heartbeat from core client for 30 sec - exiting 19:37:39 (88728): No heartbeat from core client for 30 sec - exiting 19:58:58 (92648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:04 (92648): No heartbeat from core client for 30 sec - exiting 19:59:05 (92648): No heartbeat from core client for 30 sec - exiting 19:59:07 (92648): No heartbeat from core client for 30 sec - exiting 19:59:08 (92648): No heartbeat from core client for 30 sec - exiting 19:59:09 (92648): No heartbeat from core client for 30 sec - exiting 19:59:10 (92648): No heartbeat from core client for 30 sec - exiting 19:59:11 (92648): No heartbeat from core client for 30 sec - exiting 19:59:12 (92648): No heartbeat from core client for 30 sec - exiting 19:59:13 (92648): No heartbeat from core client for 30 sec - exiting 19:59:14 (92648): No heartbeat from core client for 30 sec - exiting 20:18:04 (684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:44:56 (2872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:23:54 (47264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:26:43 (87216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:44 (87216): No heartbeat from core client for 30 sec - exiting 03:26:45 (87216): No heartbeat from core client for 30 sec - exiting 03:26:46 (87216): No heartbeat from core client for 30 sec - exiting 03:26:47 (87216): No heartbeat from core client for 30 sec - exiting 03:26:48 (87216): No heartbeat from core client for 30 sec - exiting 03:26:49 (87216): No heartbeat from core client for 30 sec - exiting 03:26:50 (87216): No heartbeat from core client for 30 sec - exiting 03:26:52 (87216): No heartbeat from core client for 30 sec - exiting 03:26:53 (87216): No heartbeat from core client for 30 sec - exiting 03:26:54 (87216): No heartbeat from core client for 30 sec - exiting 03:35:26 (92292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:27 (92292): No heartbeat from core client for 30 sec - exiting 03:35:29 (92292): No heartbeat from core client for 30 sec - exiting 03:35:30 (92292): No heartbeat from core client for 30 sec - exiting 03:35:31 (92292): No heartbeat from core client for 30 sec - exiting 03:35:32 (92292): No heartbeat from core client for 30 sec - exiting 03:35:33 (92292): No heartbeat from core client for 30 sec - exiting 03:35:34 (92292): No heartbeat from core client for 30 sec - exiting 03:35:35 (92292): No heartbeat from core client for 30 sec - exiting 03:35:36 (92292): No heartbeat from core client for 30 sec - exiting 03:50:57 (90368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:23:13 (63000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:53 (92232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:30:18 (92076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:48:45 (92164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:07 (91260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:37:08 (91260): No heartbeat from core client for 30 sec - exiting 07:37:09 (91260): No heartbeat from core client for 30 sec - exiting 07:37:11 (91260): No heartbeat from core client for 30 sec - exiting 07:37:12 (91260): No heartbeat from core client for 30 sec - exiting 07:37:13 (91260): No heartbeat from core client for 30 sec - exiting 07:37:14 (91260): No heartbeat from core client for 30 sec - exiting 07:37:15 (91260): No heartbeat from core client for 30 sec - exiting 07:37:16 (91260): No heartbeat from core client for 30 sec - exiting 07:37:17 (91260): No heartbeat from core client for 30 sec - exiting 07:37:18 (91260): No heartbeat from core client for 30 sec - exiting 07:37:59 (90940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:00 (90940): No heartbeat from core client for 30 sec - exiting 07:38:01 (90940): No heartbeat from core client for 30 sec - exiting 07:38:02 (90940): No heartbeat from core client for 30 sec - exiting 07:38:03 (90940): No heartbeat from core client for 30 sec - exiting 07:38:04 (90940): No heartbeat from core client for 30 sec - exiting 07:38:05 (90940): No heartbeat from core client for 30 sec - exiting 07:38:06 (90940): No heartbeat from core client for 30 sec - exiting 07:38:08 (90940): No heartbeat from core client for 30 sec - exiting 08:08:02 (92988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:07 (92988): No heartbeat from core client for 30 sec - exiting 08:08:08 (92988): No heartbeat from core client for 30 sec - exiting 08:08:09 (92988): No heartbeat from core client for 30 sec - exiting 08:08:10 (92988): No heartbeat from core client for 30 sec - exiting 08:08:11 (92988): No heartbeat from core client for 30 sec - exiting 08:08:12 (92988): No heartbeat from core client for 30 sec - exiting 08:08:13 (92988): No heartbeat from core client for 30 sec - exiting 08:08:14 (92988): No heartbeat from core client for 30 sec - exiting 08:08:15 (92988): No heartbeat from core client for 30 sec - exiting 08:08:16 (92988): No heartbeat from core client for 30 sec - exiting 08:27:39 (93168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:20:39 (6128): No heartbeat from core client for 30 sec - exiting 15:20:40 (6128): No heartbeat from core client for 30 sec - exiting 15:20:41 (6128): No heartbeat from core client for 30 sec - exiting 15:20:42 (6128): No heartbeat from core client for 30 sec - exiting 15:20:43 (6128): No heartbeat from core client for 30 sec - exiting 15:20:44 (6128): No heartbeat from core client for 30 sec - exiting 15:20:45 (6128): No heartbeat from core client for 30 sec - exiting 15:20:46 (6128): No heartbeat from core client for 30 sec - exiting 15:20:48 (6128): No heartbeat from core client for 30 sec - exiting 15:20:49 (6128): No heartbeat from core client for 30 sec - exiting 15:20:50 (6128): No heartbeat from core client for 30 sec - exiting 15:20:51 (6128): No heartbeat from core client for 30 sec - exiting 15:20:52 (6128): No heartbeat from core client for 30 sec - exiting 15:20:53 (6128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:52:00 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:43 (6260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:54 (6260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 06:05:50 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:33:49 (6692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:33:54 (6692): No heartbeat from core client for 30 sec - exiting 06:33:55 (6692): No heartbeat from core client for 30 sec - exiting 06:33:57 (6692): No heartbeat from core client for 30 sec - exiting 06:33:58 (6692): No heartbeat from core client for 30 sec - exiting 06:33:59 (6692): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:49:34 (6168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:28:06 (1580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:28:07 (1580): No heartbeat from core client for 30 sec - exiting 09:28:08 (1580): No heartbeat from core client for 30 sec - exiting 09:28:09 (1580): No heartbeat from core client for 30 sec - exiting 09:28:10 (1580): No heartbeat from core client for 30 sec - exiting 09:28:11 (1580): No heartbeat from core client for 30 sec - exiting 11:20:57 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:48:37 (6972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:21 (4692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:57 (4432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:52 (5604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:13:54 (5604): No heartbeat from core client for 30 sec - exiting 13:13:55 (5604): No heartbeat from core client for 30 sec - exiting 13:13:56 (5604): No heartbeat from core client for 30 sec - exiting 13:13:57 (5604): No heartbeat from core client for 30 sec - exiting 13:13:58 (5604): No heartbeat from core client for 30 sec - exiting 13:14:00 (5604): No heartbeat from core client for 30 sec - exiting 13:14:01 (5604): No heartbeat from core client for 30 sec - exiting 13:14:02 (5604): No heartbeat from core client for 30 sec - exiting 13:14:03 (5604): No heartbeat from core client for 30 sec - exiting 13:14:04 (5604): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 14:03:01 (1960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:35:03 (6796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:03:26 (3220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:46:56 (1552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:52 (8024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:41:26 (3788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:04:59 (7288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:54 (7444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:58 (7840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:22:07 (7644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8852, iMonCtr=1 Model crash detected, will try to restart... 08:56:03 (8852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8048, iMonCtr=1 Model crash detected, will try to restart... 14:01:02 (8048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8232, iMonCtr=1 Model crash detected, will try to restart... 21:14:56 (8232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:41:22 (13416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:12:36 (12776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:57:17 (31172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:58:49 (32052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:04:59 (32540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:25:16 (32260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:12 (30168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:40:51 (31932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:25:37 (31868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31548, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31548, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=31548, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Sep 2013 12:44:04	1230461	16010790	hadcm3n_7vwg_1980_40_008452627_1	77,760	142,701	1.8351
13 Sep 2013 00:29:13	1230461	16010790	hadcm3n_7vwg_1980_40_008452627_1	51,840	111,108	2.1433
12 Sep 2013 07:39:07	1230461	16010790	hadcm3n_7vwg_1980_40_008452627_1	25,920	73,348	2.8298