Task 15993153

Name	hadcm3n_8109_1980_40_008459244_0
Workunit	8610100
Created	30 Aug 2013, 20:45:15 UTC
Sent	6 Sep 2013, 9:29:40 UTC
Report deadline	6 Dec 2013, 16:56:51 UTC
Received	26 Sep 2013, 10:39:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1158872
Run time	12 days 7 hours 8 min 26 sec
CPU time	12 days 4 hours 20 min 30 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	1.39 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 07:47:44 (2328): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 07:47:46 (2328): No heartbeat from core client for 30 sec - exiting 07:47:47 (2328): No heartbeat from core client for 30 sec - exiting 23:40:33 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:55:47 (10308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:59:37 (4228): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 22:24:41 (8260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:18 (8260): No heartbeat from core client for 30 sec - exiting 22:28:19 (8260): No heartbeat from core client for 30 sec - exiting 22:28:20 (8260): No heartbeat from core client for 30 sec - exiting 22:28:21 (8260): No heartbeat from core client for 30 sec - exiting 22:33:10 (12972): No heartbeat from core client for 30 sec - exiting 22:33:11 (12972): No heartbeat from core client for 30 sec - exiting 22:33:12 (12972): No heartbeat from core client for 30 sec - exiting 22:33:13 (12972): No heartbeat from core client for 30 sec - exiting 22:33:14 (12972): No heartbeat from core client for 30 sec - exiting 22:33:15 (12972): No heartbeat from core client for 30 sec - exiting 22:33:16 (12972): No heartbeat from core client for 30 sec - exiting 22:33:17 (12972): No heartbeat from core client for 30 sec - exiting 22:33:18 (12972): No heartbeat from core client for 30 sec - exiting 22:33:19 (12972): No heartbeat from core client for 30 sec - exiting 22:33:20 (12972): No heartbeat from core client for 30 sec - exiting 22:33:21 (12972): No heartbeat from core client for 30 sec - exiting 22:33:22 (12972): No heartbeat from core client for 30 sec - exiting 22:33:23 (12972): No heartbeat from core client for 30 sec - exiting 22:33:24 (12972): No heartbeat from core client for 30 sec - exiting 22:33:25 (12972): No heartbeat from core client for 30 sec - exiting 22:33:26 (12972): No heartbeat from core client for 30 sec - exiting 22:33:27 (12972): No heartbeat from core client for 30 sec - exiting 22:33:28 (12972): No heartbeat from core client for 30 sec - exiting 22:33:29 (12972): No heartbeat from core client for 30 sec - exiting 22:33:30 (12972): No heartbeat from core client for 30 sec - exiting 22:33:31 (12972): No heartbeat from core client for 30 sec - exiting 22:33:32 (12972): No heartbeat from core client for 30 sec - exiting 22:33:33 (12972): No heartbeat from core client for 30 sec - exiting 22:33:34 (12972): No heartbeat from core client for 30 sec - exiting 22:33:35 (12972): No heartbeat from core client for 30 sec - exiting 22:33:36 (12972): No heartbeat from core client for 30 sec - exiting 22:33:37 (12972): No heartbeat from core client for 30 sec - exiting 22:33:38 (12972): No heartbeat from core client for 30 sec - exiting 22:33:39 (12972): No heartbeat from core client for 30 sec - exiting 22:33:40 (12972): No heartbeat from core client for 30 sec - exiting 22:33:41 (12972): No heartbeat from core client for 30 sec - exiting 22:33:42 (12972): No heartbeat from core client for 30 sec - exiting 22:33:43 (12972): No heartbeat from core client for 30 sec - exiting 22:33:44 (12972): No heartbeat from core client for 30 sec - exiting 22:33:45 (12972): No heartbeat from core client for 30 sec - exiting 22:33:46 (12972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:49 (1896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:07 (3040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:08 (3040): No heartbeat from core client for 30 sec - exiting 03:09:09 (3040): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9544, iMonCtr=1 Model crash detected, will try to restart... 04:14:00 (3424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:01 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:29 (4424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:31:12 (1480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:47 (6592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:48 (6592): No heartbeat from core client for 30 sec - exiting 21:04:50 (6592): No heartbeat from core client for 30 sec - exiting 21:04:51 (6592): No heartbeat from core client for 30 sec - exiting 21:04:52 (6592): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:41:08 (9828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:49 (10472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:50 (10472): No heartbeat from core client for 30 sec - exiting 20:23:09 (6608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:20 (6356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:05:57 (10260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:05:59 (10260): No heartbeat from core client for 30 sec - exiting 10:38:02 (7352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:44:12 (2736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:09:42 (8744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:40 (7704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:51:13 (8536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:11 (6296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:15 (6296): No heartbeat from core client for 30 sec - exiting 01:07:16 (6296): No heartbeat from core client for 30 sec - exiting 01:07:17 (6296): No heartbeat from core client for 30 sec - exiting 01:07:18 (6296): No heartbeat from core client for 30 sec - exiting 01:07:19 (6296): No heartbeat from core client for 30 sec - exiting 01:07:20 (6296): No heartbeat from core client for 30 sec - exiting 01:07:21 (6296): No heartbeat from core client for 30 sec - exiting 01:07:23 (6296): No heartbeat from core client for 30 sec - exiting 01:07:24 (6296): No heartbeat from core client for 30 sec - exiting 01:07:25 (6296): No heartbeat from core client for 30 sec - exiting 01:07:26 (6296): No heartbeat from core client for 30 sec - exiting 01:07:27 (6296): No heartbeat from core client for 30 sec - exiting 01:07:28 (6296): No heartbeat from core client for 30 sec - exiting 01:07:29 (6296): No heartbeat from core client for 30 sec - exiting 08:17:28 (4116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:58:50 (7308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:46 (10068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:27:46 (7724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:29 (5560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:38:33 (11520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:42 (10688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:46 (8644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:49:33 (11496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:49:34 (11496): No heartbeat from core client for 30 sec - exiting 04:39:47 (6520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:51:44 (6232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:20:02 (10828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:20:03 (10828): No heartbeat from core client for 30 sec - exiting 10:20:04 (10828): No heartbeat from core client for 30 sec - exiting 10:20:05 (10828): No heartbeat from core client for 30 sec - exiting 10:20:06 (10828): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:00:05 (5712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:54:02 (11868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:58:55 (8052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:58:56 (8052): No heartbeat from core client for 30 sec - exiting 00:58:57 (8052): No heartbeat from core client for 30 sec - exiting 00:58:58 (8052): No heartbeat from core client for 30 sec - exiting 00:58:59 (8052): No heartbeat from core client for 30 sec - exiting 00:59:00 (8052): No heartbeat from core client for 30 sec - exiting 00:59:01 (8052): No heartbeat from core client for 30 sec - exiting 00:59:02 (8052): No heartbeat from core client for 30 sec - exiting 00:59:03 (8052): No heartbeat from core client for 30 sec - exiting 00:59:04 (8052): No heartbeat from core client for 30 sec - exiting 00:59:05 (8052): No heartbeat from core client for 30 sec - exiting 01:10:27 (10116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:39:52 (3468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:01:41 (11252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:31:42 (7160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:01:44 (10736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:31:46 (6824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:47 (9984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:48 (1192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:14 (8968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:32:31 (12104): No heartbeat from core client for 30 sec - exiting 04:32:32 (12104): No heartbeat from core client for 30 sec - exiting 04:32:34 (12104): No heartbeat from core client for 30 sec - exiting 04:32:35 (12104): No heartbeat from core client for 30 sec - exiting 04:32:36 (12104): No heartbeat from core client for 30 sec - exiting 04:32:37 (12104): No heartbeat from core client for 30 sec - exiting 04:32:38 (12104): No heartbeat from core client for 30 sec - exiting 04:32:39 (12104): No heartbeat from core client for 30 sec - exiting 04:32:40 (12104): No heartbeat from core client for 30 sec - exiting 04:32:41 (12104): No heartbeat from core client for 30 sec - exiting 04:32:42 (12104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/8109ko.pji8c10 Error converting file to netcdf: dataout/8109ko.pii8c10 Error converting file to netcdf: dataout/8109ko.pfi8c10 Error converting file to netcdf: dataout/8109ka.phi8c10 Error converting file to netcdf: dataout/8109ka.pgi8c10 Error converting file to netcdf: dataout/8109ka.pei8c10 Error converting file to netcdf: dataout/8109ka.pdi8c10 04:38:24 (2792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/8109ko.pji8c10 Error converting file to netcdf: dataout/8109ko.pii8c10 Error converting file to netcdf: dataout/8109ko.pfi8c10 Error converting file to netcdf: dataout/8109ka.phi8c10 Error converting file to netcdf: dataout/8109ka.pgi8c10 Error converting file to netcdf: dataout/8109ka.pei8c10 Error converting file to netcdf: dataout/8109ka.pdi8c10 06:52:36 (1936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:31:51 (5296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:23:06 (8524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:13:10 (11916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:53:14 (6928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:33:33 (2704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:40 (1536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:41 (1536): No heartbeat from core client for 30 sec - exiting 04:33:42 (10116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:43:17 (5988): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 19:02:28 (7616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:52 (10824): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:53 (10824): No heartbeat from core client for 30 sec - exiting 19:27:54 (10824): No heartbeat from core client for 30 sec - exiting 19:27:55 (10824): No heartbeat from core client for 30 sec - exiting 19:27:57 (10824): No heartbeat from core client for 30 sec - exiting 19:27:58 (10824): No heartbeat from core client for 30 sec - exiting 19:27:59 (10824): No heartbeat from core client for 30 sec - exiting 19:28:00 (10824): No heartbeat from core client for 30 sec - exiting 19:28:01 (10824): No heartbeat from core client for 30 sec - exiting 19:28:02 (10824): No heartbeat from core client for 30 sec - exiting 19:28:03 (10824): No heartbeat from core client for 30 sec - exiting 19:58:16 (8552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:08:21 (8444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:13:05 (9788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:24:40 (10152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:31 (4660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:38 (10564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:46:20 (4924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:55:17 (9636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:06:43 (9292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:17:07 (3476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:49 (6224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:46 (5308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:41 (2756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:55:23 (7728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:47 (10204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:46 (6944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:37:31 (7392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:14:37 (11952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:45:38 (8596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:08:36 (560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:48:42 (6816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:30:42 (3112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:32 (8876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:54:29 (1104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:38:44 (11364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:38:45 (11364): No heartbeat from core client for 30 sec - exiting 07:40:05 (6476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:48:41 (6356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:42 (9124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:07:26 (8236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:19:45 (8136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:30:18 (6156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:54 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:10:25 (3552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:24 (11868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:51:38 (1344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:50:34 (8772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:34 (4956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:10:34 (6836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:20:34 (552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:00:35 (4596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:20:37 (308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:39 (8564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:20:40 (3076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:42 (10368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:00:43 (10368): No heartbeat from core client for 30 sec - exiting 18:10:43 (9672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:44 (8716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:31:56 (8488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:32:18 (8488): No heartbeat from core client for 30 sec - exiting 19:42:56 (1052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:11 (11664): No heartbeat from core client for 30 sec - exiting 19:48:12 (11664): No heartbeat from core client for 30 sec - exiting 19:48:13 (11664): No heartbeat from core client for 30 sec - exiting 19:48:14 (11664): No heartbeat from core client for 30 sec - exiting 19:48:15 (11664): No heartbeat from core client for 30 sec - exiting 19:48:16 (11664): No heartbeat from core client for 30 sec - exiting 19:48:17 (11664): No heartbeat from core client for 30 sec - exiting 19:48:19 (11664): No heartbeat from core client for 30 sec - exiting 19:48:20 (11664): No heartbeat from core client for 30 sec - exiting 19:48:21 (11664): No heartbeat from core client for 30 sec - exiting 19:48:22 (11664): No heartbeat from core client for 30 sec - exiting 19:48:23 (11664): No heartbeat from core client for 30 sec - exiting 19:48:24 (11664): No heartbeat from core client for 30 sec - exiting 19:48:25 (11664): No heartbeat from core client for 30 sec - exiting 19:48:26 (11664): No heartbeat from core client for 30 sec - exiting 19:48:27 (11664): No heartbeat from core client for 30 sec - exiting 19:48:28 (11664): No heartbeat from core client for 30 sec - exiting 19:48:29 (11664): No heartbeat from core client for 30 sec - exiting 19:48:31 (11664): No heartbeat from core client for 30 sec - exiting 19:48:32 (11664): No heartbeat from core client for 30 sec - exiting 19:48:33 (11664): No heartbeat from core client for 30 sec - exiting 19:48:34 (11664): No heartbeat from core client for 30 sec - exiting 19:48:35 (11664): No heartbeat from core client for 30 sec - exiting 19:48:36 (11664): No heartbeat from core client for 30 sec - exiting 19:48:37 (11664): No heartbeat from core client for 30 sec - exiting 19:48:38 (11664): No heartbeat from core client for 30 sec - exiting 19:48:39 (11664): No heartbeat from core client for 30 sec - exiting 19:48:40 (11664): No heartbeat from core client for 30 sec - exiting 19:48:42 (11664): No heartbeat from core client for 30 sec - exiting 19:48:43 (11664): No heartbeat from core client for 30 sec - exiting 19:48:44 (11664): No heartbeat from core client for 30 sec - exiting 19:48:45 (11664): No heartbeat from core client for 30 sec - exiting 19:48:46 (11664): No heartbeat from core client for 30 sec - exiting 19:48:47 (11664): No heartbeat from core client for 30 sec - exiting 19:48:48 (11664): No heartbeat from core client for 30 sec - exiting 19:48:49 (11664): No heartbeat from core client for 30 sec - exiting 19:48:50 (11664): No heartbeat from core client for 30 sec - exiting 19:48:51 (11664): No heartbeat from core client for 30 sec - exiting 19:48:52 (11664): No heartbeat from core client for 30 sec - exiting 19:48:54 (11664): No heartbeat from core client for 30 sec - exiting 19:48:55 (11664): No heartbeat from core client for 30 sec - exiting 19:48:56 (11664): No heartbeat from core client for 30 sec - exiting 19:48:57 (11664): No heartbeat from core client for 30 sec - exiting 19:48:58 (11664): No heartbeat from core client for 30 sec - exiting 19:48:59 (11664): No heartbeat from core client for 30 sec - exiting 19:49:00 (11664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:49:01 (11664): No heartbeat from core client for 30 sec - exiting 19:49:02 (11664): No heartbeat from core client for 30 sec - exiting 19:49:03 (11664): No heartbeat from core client for 30 sec - exiting 19:49:04 (11664): No heartbeat from core client for 30 sec - exiting 19:49:06 (11664): No heartbeat from core client for 30 sec - exiting 19:49:07 (11664): No heartbeat from core client for 30 sec - exiting 19:49:08 (11664): No heartbeat from core client for 30 sec - exiting 19:49:09 (11664): No heartbeat from core client for 30 sec - exiting 19:49:10 (11664): No heartbeat from core client for 30 sec - exiting 19:49:11 (11664): No heartbeat from core client for 30 sec - exiting 19:53:33 (8312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:53:34 (8312): No heartbeat from core client for 30 sec - exiting 19:53:35 (8312): No heartbeat from core client for 30 sec - exiting 19:53:36 (8312): No heartbeat from core client for 30 sec - exiting 19:53:37 (8312): No heartbeat from core client for 30 sec - exiting 19:53:38 (8312): No heartbeat from core client for 30 sec - exiting 19:53:40 (8312): No heartbeat from core client for 30 sec - exiting 19:53:41 (8312): No heartbeat from core client for 30 sec - exiting 19:53:42 (8312): No heartbeat from core client for 30 sec - exiting 19:53:43 (8312): No heartbeat from core client for 30 sec - exiting 19:53:44 (8312): No heartbeat from core client for 30 sec - exiting 19:56:50 (9984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:56:51 (9984): No heartbeat from core client for 30 sec - exiting 19:56:52 (9984): No heartbeat from core client for 30 sec - exiting 19:56:54 (9984): No heartbeat from core client for 30 sec - exiting 19:56:55 (9984): No heartbeat from core client for 30 sec - exiting 19:56:56 (9984): No heartbeat from core client for 30 sec - exiting 19:56:57 (9984): No heartbeat from core client for 30 sec - exiting 19:56:58 (9984): No heartbeat from core client for 30 sec - exiting 19:56:59 (9984): No heartbeat from core client for 30 sec - exiting 19:57:00 (9984): No heartbeat from core client for 30 sec - exiting 19:57:01 (9984): No heartbeat from core client for 30 sec - exiting 20:00:57 (6324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:00:58 (6324): No heartbeat from core client for 30 sec - exiting 20:00:59 (6324): No heartbeat from core client for 30 sec - exiting 20:05:17 (1556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:05:18 (1556): No heartbeat from core client for 30 sec - exiting 20:05:19 (1556): No heartbeat from core client for 30 sec - exiting 20:05:20 (1556): No heartbeat from core client for 30 sec - exiting 20:05:22 (1556): No heartbeat from core client for 30 sec - exiting 20:05:23 (1556): No heartbeat from core client for 30 sec - exiting 20:05:24 (1556): No heartbeat from core client for 30 sec - exiting 20:05:25 (1556): No heartbeat from core client for 30 sec - exiting 20:05:26 (1556): No heartbeat from core client for 30 sec - exiting 20:05:27 (1556): No heartbeat from core client for 30 sec - exiting 20:05:28 (1556): No heartbeat from core client for 30 sec - exiting 20:08:57 (9132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:09:02 (9132): No heartbeat from core client for 30 sec - exiting 20:09:03 (9132): No heartbeat from core client for 30 sec - exiting 20:09:04 (9132): No heartbeat from core client for 30 sec - exiting 20:09:05 (9132): No heartbeat from core client for 30 sec - exiting 20:09:06 (9132): No heartbeat from core client for 30 sec - exiting 20:09:07 (9132): No heartbeat from core client for 30 sec - exiting 20:09:08 (9132): No heartbeat from core client for 30 sec - exiting 20:09:09 (9132): No heartbeat from core client for 30 sec - exiting 20:09:10 (9132): No heartbeat from core client for 30 sec - exiting 20:09:11 (9132): No heartbeat from core client for 30 sec - exiting 20:10:22 (1536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:37 (1536): No heartbeat from core client for 30 sec - exiting 20:10:38 (1536): No heartbeat from core client for 30 sec - exiting 20:10:39 (1536): No heartbeat from core client for 30 sec - exiting 20:12:28 (7056): No heartbeat from core client for 30 sec - exiting 20:12:30 (7056): No heartbeat from core client for 30 sec - exiting 20:12:31 (7056): No heartbeat from core client for 30 sec - exiting 20:12:32 (7056): No heartbeat from core client for 30 sec - exiting 20:12:33 (7056): No heartbeat from core client for 30 sec - exiting 20:12:34 (7056): No heartbeat from core client for 30 sec - exiting 20:12:35 (7056): No heartbeat from core client for 30 sec - exiting 20:12:36 (7056): No heartbeat from core client for 30 sec - exiting 20:12:37 (7056): No heartbeat from core client for 30 sec - exiting 20:12:38 (7056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:12:39 (7056): No heartbeat from core client for 30 sec - exiting 20:12:41 (7056): No heartbeat from core client for 30 sec - exiting 20:12:42 (7056): No heartbeat from core client for 30 sec - exiting 20:12:43 (7056): No heartbeat from core client for 30 sec - exiting 20:12:44 (7056): No heartbeat from core client for 30 sec - exiting 20:12:45 (7056): No heartbeat from core client for 30 sec - exiting 20:12:46 (7056): No heartbeat from core client for 30 sec - exiting 20:12:47 (7056): No heartbeat from core client for 30 sec - exiting 20:12:48 (7056): No heartbeat from core client for 30 sec - exiting 20:12:49 (7056): No heartbeat from core client for 30 sec - exiting 20:15:22 (8336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:15:23 (8336): No heartbeat from core client for 30 sec - exiting 20:15:24 (8336): No heartbeat from core client for 30 sec - exiting 20:15:26 (8336): No heartbeat from core client for 30 sec - exiting 20:15:27 (8336): No heartbeat from core client for 30 sec - exiting 20:15:28 (8336): No heartbeat from core client for 30 sec - exiting 20:15:29 (8336): No heartbeat from core client for 30 sec - exiting 20:15:30 (8336): No heartbeat from core client for 30 sec - exiting 20:15:31 (8336): No heartbeat from core client for 30 sec - exiting 20:15:32 (8336): No heartbeat from core client for 30 sec - exiting 20:15:33 (8336): No heartbeat from core client for 30 sec - exiting 20:19:38 (7008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:19:39 (7008): No heartbeat from core client for 30 sec - exiting 20:19:40 (7008): No heartbeat from core client for 30 sec - exiting 20:19:42 (7008): No heartbeat from core client for 30 sec - exiting 20:19:43 (7008): No heartbeat from core client for 30 sec - exiting 20:19:44 (7008): No heartbeat from core client for 30 sec - exiting 20:19:45 (7008): No heartbeat from core client for 30 sec - exiting 20:19:46 (7008): No heartbeat from core client for 30 sec - exiting 20:19:47 (7008): No heartbeat from core client for 30 sec - exiting 20:19:48 (7008): No heartbeat from core client for 30 sec - exiting 20:19:49 (7008): No heartbeat from core client for 30 sec - exiting 20:25:06 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:25:07 (5332): No heartbeat from core client for 30 sec - exiting 20:25:08 (5332): No heartbeat from core client for 30 sec - exiting 20:25:10 (5332): No heartbeat from core client for 30 sec - exiting 20:25:11 (5332): No heartbeat from core client for 30 sec - exiting 20:25:12 (5332): No heartbeat from core client for 30 sec - exiting 20:25:13 (5332): No heartbeat from core client for 30 sec - exiting 20:25:14 (5332): No heartbeat from core client for 30 sec - exiting 20:25:15 (5332): No heartbeat from core client for 30 sec - exiting 20:25:16 (5332): No heartbeat from core client for 30 sec - exiting 20:25:17 (5332): No heartbeat from core client for 30 sec - exiting 20:28:12 (9284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:29:12 (9284): No heartbeat from core client for 30 sec - exiting 20:29:13 (9284): No heartbeat from core client for 30 sec - exiting 20:29:14 (9284): No heartbeat from core client for 30 sec - exiting 20:29:15 (9284): No heartbeat from core client for 30 sec - exiting 20:29:16 (9284): No heartbeat from core client for 30 sec - exiting 20:29:17 (9284): No heartbeat from core client for 30 sec - exiting 20:29:18 (9284): No heartbeat from core client for 30 sec - exiting 20:29:19 (9284): No heartbeat from core client for 30 sec - exiting 20:29:21 (9284): No heartbeat from core client for 30 sec - exiting 20:29:22 (9284): No heartbeat from core client for 30 sec - exiting 20:32:58 (8216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:33:00 (8216): No heartbeat from core client for 30 sec - exiting 20:33:01 (8216): No heartbeat from core client for 30 sec - exiting 20:33:02 (8216): No heartbeat from core client for 30 sec - exiting 20:33:03 (8216): No heartbeat from core client for 30 sec - exiting 20:33:04 (8216): No heartbeat from core client for 30 sec - exiting 20:33:05 (8216): No heartbeat from core client for 30 sec - exiting 20:33:06 (8216): No heartbeat from core client for 30 sec - exiting 20:33:07 (8216): No heartbeat from core client for 30 sec - exiting 20:33:08 (8216): No heartbeat from core client for 30 sec - exiting 20:33:09 (8216): No heartbeat from core client for 30 sec - exiting 20:35:49 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:35:51 (5692): No heartbeat from core client for 30 sec - exiting 20:35:52 (5692): No heartbeat from core client for 30 sec - exiting 20:35:53 (5692): No heartbeat from core client for 30 sec - exiting 20:35:54 (5692): No heartbeat from core client for 30 sec - exiting 20:35:55 (5692): No heartbeat from core client for 30 sec - exiting 20:35:56 (5692): No heartbeat from core client for 30 sec - exiting 20:35:57 (5692): No heartbeat from core client for 30 sec - exiting 20:35:58 (5692): No heartbeat from core client for 30 sec - exiting 20:35:59 (5692): No heartbeat from core client for 30 sec - exiting 20:36:00 (5692): No heartbeat from core client for 30 sec - exiting 20:38:46 (5200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:38:47 (5200): No heartbeat from core client for 30 sec - exiting 20:38:48 (5200): No heartbeat from core client for 30 sec - exiting 20:38:49 (5200): No heartbeat from core client for 30 sec - exiting 20:38:50 (5200): No heartbeat from core client for 30 sec - exiting 20:38:51 (5200): No heartbeat from core client for 30 sec - exiting 20:38:52 (5200): No heartbeat from core client for 30 sec - exiting 20:38:53 (5200): No heartbeat from core client for 30 sec - exiting 20:38:54 (5200): No heartbeat from core client for 30 sec - exiting 20:38:55 (5200): No heartbeat from core client for 30 sec - exiting 20:38:57 (5200): No heartbeat from core client for 30 sec - exiting 20:48:46 (1900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:20:58 (12256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:09 (12256): No heartbeat from core client for 30 sec - exiting 21:21:10 (12256): No heartbeat from core client for 30 sec - exiting 21:21:11 (12256): No heartbeat from core client for 30 sec - exiting 21:21:13 (12256): No heartbeat from core client for 30 sec - exiting 21:21:14 (12256): No heartbeat from core client for 30 sec - exiting 21:21:15 (12256): No heartbeat from core client for 30 sec - exiting 21:21:16 (12256): No heartbeat from core client for 30 sec - exiting 21:21:17 (12256): No heartbeat from core client for 30 sec - exiting 21:21:18 (12256): No heartbeat from core client for 30 sec - exiting 21:21:19 (12256): No heartbeat from core client for 30 sec - exiting 21:21:20 (12256): No heartbeat from core client for 30 sec - exiting 21:21:21 (12256): No heartbeat from core client for 30 sec - exiting 21:21:22 (12256): No heartbeat from core client for 30 sec - exiting 21:21:23 (12256): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:18:20 (8804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:58:19 (8692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:03:55 (9956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:08:21 (9408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:30 (10140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:28:30 (2600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:48:30 (9580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:08:32 (2028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:18:33 (5704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:28:34 (8296): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:27 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:45 (10212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:48:46 (5164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:28:47 (10884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:47:32 (8932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:21 (9612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:51:51 (9612): No heartbeat from core client for 30 sec - exiting 19:51:52 (9612): No heartbeat from core client for 30 sec - exiting 19:51:53 (9612): No heartbeat from core client for 30 sec - exiting 19:51:54 (9612): No heartbeat from core client for 30 sec - exiting 19:51:55 (9612): No heartbeat from core client for 30 sec - exiting 19:51:56 (9612): No heartbeat from core client for 30 sec - exiting 19:51:57 (9612): No heartbeat from core client for 30 sec - exiting 19:51:59 (9612): No heartbeat from core client for 30 sec - exiting 19:52:00 (9612): No heartbeat from core client for 30 sec - exiting 19:52:01 (9612): No heartbeat from core client for 30 sec - exiting 20:58:05 (5628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:58:55 (6792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:18:48 (7032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:18:57 (7032): No heartbeat from core client for 30 sec - exiting 22:18:58 (7032): No heartbeat from core client for 30 sec - exiting 22:18:59 (7032): No heartbeat from core client for 30 sec - exiting 22:19:01 (7032): No heartbeat from core client for 30 sec - exiting 22:19:02 (7032): No heartbeat from core client for 30 sec - exiting 22:19:03 (7032): No heartbeat from core client for 30 sec - exiting 22:19:04 (7032): No heartbeat from core client for 30 sec - exiting 22:19:05 (7032): No heartbeat from core client for 30 sec - exiting 23:47:57 (7636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:58:36 (7188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:11 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:11 (6952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:41:13 (4272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Sep 2013 03:04:52	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	336,960	1,031,206	3.0603
25 Sep 2013 09:20:21	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	311,040	953,689	3.0661
23 Sep 2013 11:11:39	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	285,120	873,943	3.0652
23 Sep 2013 10:09:04	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	259,200	793,413	3.0610
21 Sep 2013 02:03:16	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	233,280	714,100	3.0611
20 Sep 2013 03:42:22	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	207,360	635,409	3.0643
19 Sep 2013 05:26:31	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	181,440	559,586	3.0841
18 Sep 2013 04:56:03	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	155,520	480,753	3.0913
17 Sep 2013 04:56:56	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	129,600	400,676	3.0916
16 Sep 2013 05:50:35	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	103,680	319,665	3.0832
15 Sep 2013 06:58:24	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	77,760	239,730	3.0829
14 Sep 2013 08:17:50	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	51,840	159,873	3.0840
13 Sep 2013 09:41:16	1158872	15993153	hadcm3n_8109_1980_40_008459244_0	25,920	80,186	3.0936