Task 12746155

Name	hadcm3n_o5id_1900_40_007202472_1
Workunit	7400752
Created	28 Mar 2011, 14:13:54 UTC
Sent	29 Mar 2011, 19:52:49 UTC
Report deadline	29 Jun 2011, 3:20:00 UTC
Received	31 Mar 2011, 5:43:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1125716
Run time	19 hours 37 min 14 sec
CPU time	14 hours 55 min 57 sec
Validate state	Invalid
Credit	622.08
Device peak FLOPS	3.49 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 22:30:49 (4365): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:30:50 (4365): No heartbeat from core client for 30 sec - exiting 22:30:51 (4365): No heartbeat from core client for 30 sec - exiting 22:30:52 (4365): No heartbeat from core client for 30 sec - exiting 22:30:53 (4365): No heartbeat from core client for 30 sec - exiting 22:30:54 (4365): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 22:41:48 (4936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:41:49 (4936): No heartbeat from core client for 30 sec - exiting 22:41:50 (4936): No heartbeat from core client for 30 sec - exiting 22:41:51 (4936): No heartbeat from core client for 30 sec - exiting 22:41:52 (4936): No heartbeat from core client for 30 sec - exiting 23:04:02 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:04:03 (5672): No heartbeat from core client for 30 sec - exiting 23:04:04 (5672): No heartbeat from core client for 30 sec - exiting 23:04:05 (5672): No heartbeat from core client for 30 sec - exiting 23:04:06 (5672): No heartbeat from core client for 30 sec - exiting 23:04:07 (5672): No heartbeat from core client for 30 sec - exiting 23:04:08 (5672): No heartbeat from core client for 30 sec - exiting 23:04:09 (5672): No heartbeat from core client for 30 sec - exiting 23:04:10 (5672): No heartbeat from core client for 30 sec - exiting 23:04:11 (5672): No heartbeat from core client for 30 sec - exiting 23:04:12 (5672): No heartbeat from core client for 30 sec - exiting 23:04:13 (5672): No heartbeat from core client for 30 sec - exiting 23:04:14 (5672): No heartbeat from core client for 30 sec - exiting 23:04:15 (5672): No heartbeat from core client for 30 sec - exiting 23:04:16 (5672): No heartbeat from core client for 30 sec - exiting 23:04:17 (5672): No heartbeat from core client for 30 sec - exiting 23:04:18 (5672): No heartbeat from core client for 30 sec - exiting 23:04:19 (5672): No heartbeat from core client for 30 sec - exiting 23:04:20 (5672): No heartbeat from core client for 30 sec - exiting 23:04:21 (5672): No heartbeat from core client for 30 sec - exiting 23:04:22 (5672): No heartbeat from core client for 30 sec - exiting 23:04:23 (5672): No heartbeat from core client for 30 sec - exiting 23:04:24 (5672): No heartbeat from core client for 30 sec - exiting 23:04:25 (5672): No heartbeat from core client for 30 sec - exiting 23:04:26 (5672): No heartbeat from core client for 30 sec - exiting 23:04:27 (5672): No heartbeat from core client for 30 sec - exiting 23:04:28 (5672): No heartbeat from core client for 30 sec - exiting 23:04:29 (5672): No heartbeat from core client for 30 sec - exiting 23:04:30 (5672): No heartbeat from core client for 30 sec - exiting 23:04:31 (5672): No heartbeat from core client for 30 sec - exiting 23:04:32 (5672): No heartbeat from core client for 30 sec - exiting 23:04:33 (5672): No heartbeat from core client for 30 sec - exiting 23:04:34 (5672): No heartbeat from core client for 30 sec - exiting 23:04:35 (5672): No heartbeat from core client for 30 sec - exiting 23:04:36 (5672): No heartbeat from core client for 30 sec - exiting 23:04:37 (5672): No heartbeat from core client for 30 sec - exiting 23:04:38 (5672): No heartbeat from core client for 30 sec - exiting 23:04:39 (5672): No heartbeat from core client for 30 sec - exiting 23:04:40 (5672): No heartbeat from core client for 30 sec - exiting 23:04:41 (5672): No heartbeat from core client for 30 sec - exiting 23:04:42 (5672): No heartbeat from core client for 30 sec - exiting 23:04:43 (5672): No heartbeat from core client for 30 sec - exiting 23:04:44 (5672): No heartbeat from core client for 30 sec - exiting 23:04:45 (5672): No heartbeat from core client for 30 sec - exiting 23:04:46 (5672): No heartbeat from core client for 30 sec - exiting 23:04:47 (5672): No heartbeat from core client for 30 sec - exiting 23:04:48 (5672): No heartbeat from core client for 30 sec - exiting 23:04:49 (5672): No heartbeat from core client for 30 sec - exiting 23:04:50 (5672): No heartbeat from core client for 30 sec - exiting 23:04:51 (5672): No heartbeat from core client for 30 sec - exiting 23:04:52 (5672): No heartbeat from core client for 30 sec - exiting 23:04:53 (5672): No heartbeat from core client for 30 sec - exiting 23:04:54 (5672): No heartbeat from core client for 30 sec - exiting 23:04:55 (5672): No heartbeat from core client for 30 sec - exiting 23:04:56 (5672): No heartbeat from core client for 30 sec - exiting 23:04:57 (5672): No heartbeat from core client for 30 sec - exiting 23:04:58 (5672): No heartbeat from core client for 30 sec - exiting 23:04:59 (5672): No heartbeat from core client for 30 sec - exiting 23:05:00 (5672): No heartbeat from core client for 30 sec - exiting 23:05:01 (5672): No heartbeat from core client for 30 sec - exiting 23:05:02 (5672): No heartbeat from core client for 30 sec - exiting 23:05:03 (5672): No heartbeat from core client for 30 sec - exiting 23:05:04 (5672): No heartbeat from core client for 30 sec - exiting 23:08:09 (7462): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:36 (7687): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:37 (7687): No heartbeat from core client for 30 sec - exiting 23:12:38 (7687): No heartbeat from core client for 30 sec - exiting 23:12:39 (7687): No heartbeat from core client for 30 sec - exiting 23:12:40 (7687): No heartbeat from core client for 30 sec - exiting 23:12:41 (7687): No heartbeat from core client for 30 sec - exiting 23:34:45 (7876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:42:42 (9761): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:54:40 (10460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:17:18 (11538): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:19:24 (13518): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:20 (13731): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:09:10 (17869): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:33:37 (18095): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:55 (20215): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:56 (20215): No heartbeat from core client for 30 sec - exiting 01:37:58 (20215): No heartbeat from core client for 30 sec - exiting 01:48:33 (20630): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:44 (21562): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:12:00 (21767): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:22:48 (28860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:28:13 (29860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:32:58 (30340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:32:59 (30340): No heartbeat from core client for 30 sec - exiting 03:33:00 (30340): No heartbeat from core client for 30 sec - exiting 03:33:01 (30340): No heartbeat from core client for 30 sec - exiting 03:33:02 (30340): No heartbeat from core client for 30 sec - exiting 03:35:08 (30771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:09 (30771): No heartbeat from core client for 30 sec - exiting 03:35:10 (30771): No heartbeat from core client for 30 sec - exiting 03:35:11 (30771): No heartbeat from core client for 30 sec - exiting 03:35:12 (30771): No heartbeat from core client for 30 sec - exiting 03:35:13 (30771): No heartbeat from core client for 30 sec - exiting 03:35:14 (30771): No heartbeat from core client for 30 sec - exiting 03:35:15 (30771): No heartbeat from core client for 30 sec - exiting 03:35:16 (30771): No heartbeat from core client for 30 sec - exiting 03:35:17 (30771): No heartbeat from core client for 30 sec - exiting 03:38:15 (30973): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:38:16 (30973): No heartbeat from core client for 30 sec - exiting 03:38:17 (30973): No heartbeat from core client for 30 sec - exiting 03:38:18 (30973): No heartbeat from core client for 30 sec - exiting 03:38:19 (30973): No heartbeat from core client for 30 sec - exiting 03:38:20 (30973): No heartbeat from core client for 30 sec - exiting 03:38:21 (30973): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 03:44:05 (31244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:11 (31825): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:04:59 (792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:05:00 (792): No heartbeat from core client for 30 sec - exiting 04:05:01 (792): No heartbeat from core client for 30 sec - exiting 04:05:02 (792): No heartbeat from core client for 30 sec - exiting 04:05:03 (792): No heartbeat from core client for 30 sec - exiting 04:05:04 (792): No heartbeat from core client for 30 sec - exiting 04:05:05 (792): No heartbeat from core client for 30 sec - exiting 04:05:06 (792): No heartbeat from core client for 30 sec - exiting 04:05:07 (792): No heartbeat from core client for 30 sec - exiting 04:05:08 (792): No heartbeat from core client for 30 sec - exiting 04:05:09 (792): No heartbeat from core client for 30 sec - exiting 04:05:10 (792): No heartbeat from core client for 30 sec - exiting 04:05:11 (792): No heartbeat from core client for 30 sec - exiting 04:05:12 (792): No heartbeat from core client for 30 sec - exiting 04:05:13 (792): No heartbeat from core client for 30 sec - exiting 04:05:14 (792): No heartbeat from core client for 30 sec - exiting 04:05:15 (792): No heartbeat from core client for 30 sec - exiting 04:05:16 (792): No heartbeat from core client for 30 sec - exiting 04:05:17 (792): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 04:06:35 (1178): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 04:13:00 (1381): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:13:01 (1381): No heartbeat from core client for 30 sec - exiting 04:32:25 (2031): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 05:10:12 (3755): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:10:13 (3755): No heartbeat from core client for 30 sec - exiting 05:10:14 (3755): No heartbeat from core client for 30 sec - exiting 05:10:15 (3755): No heartbeat from core client for 30 sec - exiting 05:10:16 (3755): No heartbeat from core client for 30 sec - exiting 05:10:17 (3755): No heartbeat from core client for 30 sec - exiting 05:10:18 (3755): No heartbeat from core client for 30 sec - exiting 05:28:28 (7076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:20:01 (8694): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:42:15 (13178): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:50:39 (15152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:59:15 (15900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:23:51 (16651): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:27:53 (18808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:29:47 (19171): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:36:09 (19375): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:41:00 (20388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:01 (20788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:51:02 (20788): No heartbeat from core client for 30 sec - exiting 07:51:03 (20788): No heartbeat from core client for 30 sec - exiting 07:51:04 (20788): No heartbeat from core client for 30 sec - exiting 07:51:05 (20788): No heartbeat from core client for 30 sec - exiting 07:51:06 (20788): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 07:58:35 (21733): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:58:36 (21733): No heartbeat from core client for 30 sec - exiting 07:58:37 (21733): No heartbeat from core client for 30 sec - exiting 07:58:38 (21733): No heartbeat from core client for 30 sec - exiting 07:58:39 (21733): No heartbeat from core client for 30 sec - exiting 07:58:40 (21733): No heartbeat from core client for 30 sec - exiting 07:58:41 (21733): No heartbeat from core client for 30 sec - exiting 07:58:42 (21733): No heartbeat from core client for 30 sec - exiting 07:58:43 (21733): No heartbeat from core client for 30 sec - exiting 07:58:44 (21733): No heartbeat from core client for 30 sec - exiting 08:04:09 (22360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:12:13 (22883): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:18 (23561): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:18:34 (23951): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:57 (24159): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:30 (25084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:43 (26194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:04 (27474): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:05 (27474): No heartbeat from core client for 30 sec - exiting 09:04:06 (27474): No heartbeat from core client for 30 sec - exiting 09:04:07 (27474): No heartbeat from core client for 30 sec - exiting 09:04:08 (27474): No heartbeat from core client for 30 sec - exiting 09:04:09 (27474): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:07:02 (28187): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:17:18 (28457): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:32:54 (29377): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:32:55 (29377): No heartbeat from core client for 30 sec - exiting 09:32:56 (29377): No heartbeat from core client for 30 sec - exiting 09:32:57 (29377): No heartbeat from core client for 30 sec - exiting 09:32:58 (29377): No heartbeat from core client for 30 sec - exiting 09:32:59 (29377): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:52:24 (30749): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:37 (32453): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:39 (730): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:30:54 (2472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:03:45 (3517): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:03:46 (3517): No heartbeat from core client for 30 sec - exiting 11:03:47 (3517): No heartbeat from core client for 30 sec - exiting 11:03:48 (3517): No heartbeat from core client for 30 sec - exiting 11:03:49 (3517): No heartbeat from core client for 30 sec - exiting 11:03:50 (3517): No heartbeat from core client for 30 sec - exiting 11:32:05 (6395): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:20 (8889): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:50:24 (9272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:52:36 (10508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:03:02 (10702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:47:19 (23874): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:24:06 (31612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:30:29 (32533): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:42:26 (643): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:50:03 (1804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:56:04 (2489): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:08 (3020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:09 (3020): No heartbeat from core client for 30 sec - exiting 17:00:10 (3020): No heartbeat from core client for 30 sec - exiting 17:00:11 (3020): No heartbeat from core client for 30 sec - exiting 17:00:12 (3020): No heartbeat from core client for 30 sec - exiting 17:00:13 (3020): No heartbeat from core client for 30 sec - exiting 17:00:14 (3020): No heartbeat from core client for 30 sec - exiting 17:00:16 (3020): No heartbeat from core client for 30 sec - exiting 17:00:17 (3020): No heartbeat from core client for 30 sec - exiting 17:00:18 (3020): No heartbeat from core client for 30 sec - exiting 17:00:19 (3020): No heartbeat from core client for 30 sec - exiting 17:00:20 (3020): No heartbeat from core client for 30 sec - exiting 17:00:21 (3020): No heartbeat from core client for 30 sec - exiting 17:00:22 (3020): No heartbeat from core client for 30 sec - exiting 17:00:23 (3020): No heartbeat from core client for 30 sec - exiting 17:00:24 (3020): No heartbeat from core client for 30 sec - exiting 17:00:25 (3020): No heartbeat from core client for 30 sec - exiting 17:04:24 (3386): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:06:36 (3766): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:01 (3969): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:02 (3969): No heartbeat from core client for 30 sec - exiting 17:11:03 (3969): No heartbeat from core client for 30 sec - exiting 17:11:04 (3969): No heartbeat from core client for 30 sec - exiting 17:11:05 (3969): No heartbeat from core client for 30 sec - exiting 17:11:06 (3969): No heartbeat from core client for 30 sec - exiting 17:11:07 (3969): No heartbeat from core client for 30 sec - exiting 17:11:08 (3969): No heartbeat from core client for 30 sec - exiting 17:15:05 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:34 (4731): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:35 (4731): No heartbeat from core client for 30 sec - exiting 17:21:36 (4731): No heartbeat from core client for 30 sec - exiting 17:21:37 (4731): No heartbeat from core client for 30 sec - exiting 17:21:38 (4731): No heartbeat from core client for 30 sec - exiting 17:21:39 (4731): No heartbeat from core client for 30 sec - exiting 17:21:40 (4731): No heartbeat from core client for 30 sec - exiting 17:21:41 (4731): No heartbeat from core client for 30 sec - exiting 17:21:42 (4731): No heartbeat from core client for 30 sec - exiting 17:25:38 (5288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:29:52 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:41:20 (6041): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:43:29 (7067): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:55 (7270): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:47:56 (7270): No heartbeat from core client for 30 sec - exiting 17:47:57 (7270): No heartbeat from core client for 30 sec - exiting 17:47:58 (7270): No heartbeat from core client for 30 sec - exiting 17:47:59 (7270): No heartbeat from core client for 30 sec - exiting 17:48:00 (7270): No heartbeat from core client for 30 sec - exiting 17:48:01 (7270): No heartbeat from core client for 30 sec - exiting 17:48:02 (7270): No heartbeat from core client for 30 sec - exiting 17:48:03 (7270): No heartbeat from core client for 30 sec - exiting 18:00:19 (7646): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:02:31 (8747): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:08:49 (8939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:46 (9516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:05:51 (13084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:31:56 (14493): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:46:00 (16771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:39 (18036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:11:25 (19224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:49:39 (20275): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:01:52 (23601): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 21:16:27 (24653): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:35 (25961): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 01:14:32 (14343): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:26:38 (14771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:28:52 (15853): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:03:10 (16024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:07:02 (19033): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:23:04 (19400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:28:55 (20798): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 02:33:45 (21312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:35:57 (21771): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:35:58 (21771): No heartbeat from core client for 30 sec - exiting 02:48:00 (21952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:56:12 (23038): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:02:29 (23754): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:21:21 (24308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:31:19 (25956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:35:13 (26854): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:09 (27211): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:54:59 (28443): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:32 (28940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:24 (29423): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:08:35 (29627): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:20:30 (30168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:34 (31211): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:35 (31211): No heartbeat from core client for 30 sec - exiting 04:22:36 (31211): No heartbeat from core client for 30 sec - exiting 04:22:37 (31211): No heartbeat from core client for 30 sec - exiting 04:22:38 (31211): No heartbeat from core client for 30 sec - exiting 04:22:39 (31211): No heartbeat from core client for 30 sec - exiting 04:22:40 (31211): No heartbeat from core client for 30 sec - exiting 04:22:41 (31211): No heartbeat from core client for 30 sec - exiting 04:22:42 (31211): No heartbeat from core client for 30 sec - exiting 04:22:43 (31211): No heartbeat from core client for 30 sec - exiting 04:22:44 (31211): No heartbeat from core client for 30 sec - exiting 04:41:35 (31404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:47:48 (617): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:09:46 (1194): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:17 (3257): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:26:20 (4164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:31 (4727): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:28:32 (4727): No heartbeat from core client for 30 sec - exiting 05:28:33 (4727): No heartbeat from core client for 30 sec - exiting 05:28:34 (4727): No heartbeat from core client for 30 sec - exiting 05:28:35 (4727): No heartbeat from core client for 30 sec - exiting 05:28:36 (4727): No heartbeat from core client for 30 sec - exiting 05:28:37 (4727): No heartbeat from core client for 30 sec - exiting 05:28:38 (4727): No heartbeat from core client for 30 sec - exiting 05:28:39 (4727): No heartbeat from core client for 30 sec - exiting 05:28:40 (4727): No heartbeat from core client for 30 sec - exiting 05:28:41 (4727): No heartbeat from core client for 30 sec - exiting 05:28:42 (4727): No heartbeat from core client for 30 sec - exiting 05:28:43 (4727): No heartbeat from core client for 30 sec - exiting 05:28:44 (4727): No heartbeat from core client for 30 sec - exiting 05:28:45 (4727): No heartbeat from core client for 30 sec - exiting 05:28:46 (4727): No heartbeat from core client for 30 sec - exiting 05:28:47 (4727): No heartbeat from core client for 30 sec - exiting 05:28:48 (4727): No heartbeat from core client for 30 sec - exiting 05:28:49 (4727): No heartbeat from core client for 30 sec - exiting 05:41:11 (4932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:41:12 (4932): No heartbeat from core client for 30 sec - exiting 05:41:13 (4932): No heartbeat from core client for 30 sec - exiting 05:41:14 (4932): No heartbeat from core client for 30 sec - exiting 05:41:15 (4932): No heartbeat from core client for 30 sec - exiting 05:41:16 (4932): No heartbeat from core client for 30 sec - exiting 05:41:17 (4932): No heartbeat from core client for 30 sec - exiting 05:46:36 (6067): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:48:43 (6537): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:50:46 (6714): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:50:47 (6714): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 06:22:34 (6939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:36:52 (9739): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:07:20 (11000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 07:15:06 (13678): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 07:19:14 (14366): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 07:25:33 (14755): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 BUFFIN: Read Failed: Numerical result out of range BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Mar 2011 04:35:35	1125716	12746155	hadcm3n_o5id_1900_40_007202472_1	51,840	51,891	1.0010
30 Mar 2011 09:26:59	1125716	12746155	hadcm3n_o5id_1900_40_007202472_1	25,920	26,648	1.0281