Task 13348460

Name	hadcm3n_o35u_1940_40_007443255_1
Workunit	7640758
Created	8 Sep 2011, 23:27:07 UTC
Sent	19 Sep 2011, 17:12:13 UTC
Report deadline	20 Dec 2011, 0:39:24 UTC
Received	6 Dec 2011, 19:44:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1072815
Run time	21 days 17 hours 22 min 30 sec
CPU time	21 days 11 hours 22 min 10 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 20:20:15 (3320): No heartbeat from core client for 30 sec - exiting 20:20:16 (3320): No heartbeat from core client for 30 sec - exiting 20:20:17 (3320): No heartbeat from core client for 30 sec - exiting 20:20:18 (3320): No heartbeat from core client for 30 sec - exiting 20:20:19 (3320): No heartbeat from core client for 30 sec - exiting 20:20:20 (3320): No heartbeat from core client for 30 sec - exiting 20:20:21 (3320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:27:03 (5748): No heartbeat from core client for 30 sec - exiting 19:27:04 (5748): No heartbeat from core client for 30 sec - exiting 19:27:05 (5748): No heartbeat from core client for 30 sec - exiting 19:27:06 (5748): No heartbeat from core client for 30 sec - exiting 19:27:07 (5748): No heartbeat from core client for 30 sec - exiting 19:27:08 (5748): No heartbeat from core client for 30 sec - exiting 19:27:09 (5748): No heartbeat from core client for 30 sec - exiting 19:27:10 (5748): No heartbeat from core client for 30 sec - exiting 19:27:11 (5748): No heartbeat from core client for 30 sec - exiting 19:27:12 (5748): No heartbeat from core client for 30 sec - exiting 19:27:13 (5748): No heartbeat from core client for 30 sec - exiting 19:27:14 (5748): No heartbeat from core client for 30 sec - exiting 19:27:15 (5748): No heartbeat from core client for 30 sec - exiting 19:27:16 (5748): No heartbeat from core client for 30 sec - exiting 19:27:17 (5748): No heartbeat from core client for 30 sec - exiting 19:27:18 (5748): No heartbeat from core client for 30 sec - exiting 19:27:19 (5748): No heartbeat from core client for 30 sec - exiting 19:27:20 (5748): No heartbeat from core client for 30 sec - exiting 19:27:21 (5748): No heartbeat from core client for 30 sec - exiting 19:27:22 (5748): No heartbeat from core client for 30 sec - exiting 19:27:23 (5748): No heartbeat from core client for 30 sec - exiting 19:27:24 (5748): No heartbeat from core client for 30 sec - exiting 19:27:25 (5748): No heartbeat from core client for 30 sec - exiting 19:27:26 (5748): No heartbeat from core client for 30 sec - exiting 19:27:27 (5748): No heartbeat from core client for 30 sec - exiting 19:27:28 (5748): No heartbeat from core client for 30 sec - exiting 19:27:29 (5748): No heartbeat from core client for 30 sec - exiting 19:27:30 (5748): No heartbeat from core client for 30 sec - exiting 19:27:31 (5748): No heartbeat from core client for 30 sec - exiting 19:27:32 (5748): No heartbeat from core client for 30 sec - exiting 19:27:33 (5748): No heartbeat from core client for 30 sec - exiting 19:27:34 (5748): No heartbeat from core client for 30 sec - exiting 19:27:35 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:39:47 (6104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:45:59 (5588): No heartbeat from core client for 30 sec - exiting 19:46:00 (5588): No heartbeat from core client for 30 sec - exiting 19:46:01 (5588): No heartbeat from core client for 30 sec - exiting 19:46:02 (5588): No heartbeat from core client for 30 sec - exiting 19:46:03 (5588): No heartbeat from core client for 30 sec - exiting 19:46:04 (5588): No heartbeat from core client for 30 sec - exiting 19:46:05 (5588): No heartbeat from core client for 30 sec - exiting 19:46:07 (5588): No heartbeat from core client for 30 sec - exiting 19:46:08 (5588): No heartbeat from core client for 30 sec - exiting 19:46:09 (5588): No heartbeat from core client for 30 sec - exiting 19:46:10 (5588): No heartbeat from core client for 30 sec - exiting 19:46:11 (5588): No heartbeat from core client for 30 sec - exiting 19:46:12 (5588): No heartbeat from core client for 30 sec - exiting 19:46:13 (5588): No heartbeat from core client for 30 sec - exiting 19:46:14 (5588): No heartbeat from core client for 30 sec - exiting 19:46:15 (5588): No heartbeat from core client for 30 sec - exiting 19:46:16 (5588): No heartbeat from core client for 30 sec - exiting 19:46:17 (5588): No heartbeat from core client for 30 sec - exiting 19:46:19 (5588): No heartbeat from core client for 30 sec - exiting 19:46:20 (5588): No heartbeat from core client for 30 sec - exiting 19:46:21 (5588): No heartbeat from core client for 30 sec - exiting 19:46:22 (5588): No heartbeat from core client for 30 sec - exiting 19:46:23 (5588): No heartbeat from core client for 30 sec - exiting 19:46:24 (5588): No heartbeat from core client for 30 sec - exiting 19:46:25 (5588): No heartbeat from core client for 30 sec - exiting 19:46:26 (5588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:50 (5820): No heartbeat from core client for 30 sec - exiting 11:20:51 (5820): No heartbeat from core client for 30 sec - exiting 11:20:52 (5820): No heartbeat from core client for 30 sec - exiting 11:20:53 (5820): No heartbeat from core client for 30 sec - exiting 11:20:54 (5820): No heartbeat from core client for 30 sec - exiting 11:20:55 (5820): No heartbeat from core client for 30 sec - exiting 11:20:56 (5820): No heartbeat from core client for 30 sec - exiting 11:20:57 (5820): No heartbeat from core client for 30 sec - exiting 11:20:58 (5820): No heartbeat from core client for 30 sec - exiting 11:21:00 (5820): No heartbeat from core client for 30 sec - exiting 11:21:01 (5820): No heartbeat from core client for 30 sec - exiting 11:21:02 (5820): No heartbeat from core client for 30 sec - exiting 11:21:03 (5820): No heartbeat from core client for 30 sec - exiting 11:21:04 (5820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:22:15 (6004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2960, iMonCtr=1 Model crash detected, will try to restart... 22:15:50 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7052, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:26:34 (5576): No heartbeat from core client for 30 sec - exiting 19:26:35 (5576): No heartbeat from core client for 30 sec - exiting 19:26:36 (5576): No heartbeat from core client for 30 sec - exiting 19:26:37 (5576): No heartbeat from core client for 30 sec - exiting 19:26:38 (5576): No heartbeat from core client for 30 sec - exiting 19:26:39 (5576): No heartbeat from core client for 30 sec - exiting 19:26:40 (5576): No heartbeat from core client for 30 sec - exiting 19:26:41 (5576): No heartbeat from core client for 30 sec - exiting 19:26:43 (5576): No heartbeat from core client for 30 sec - exiting 19:26:44 (5576): No heartbeat from core client for 30 sec - exiting 19:26:45 (5576): No heartbeat from core client for 30 sec - exiting 19:26:46 (5576): No heartbeat from core client for 30 sec - exiting 19:26:47 (5576): No heartbeat from core client for 30 sec - exiting 19:26:48 (5576): No heartbeat from core client for 30 sec - exiting 19:26:49 (5576): No heartbeat from core client for 30 sec - exiting 19:26:50 (5576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:06:27 (616): No heartbeat from core client for 30 sec - exiting 19:06:28 (616): No heartbeat from core client for 30 sec - exiting 19:06:29 (616): No heartbeat from core client for 30 sec - exiting 19:06:30 (616): No heartbeat from core client for 30 sec - exiting 19:06:31 (616): No heartbeat from core client for 30 sec - exiting 19:06:32 (616): No heartbeat from core client for 30 sec - exiting 19:06:33 (616): No heartbeat from core client for 30 sec - exiting 19:06:34 (616): No heartbeat from core client for 30 sec - exiting 19:06:35 (616): No heartbeat from core client for 30 sec - exiting 19:06:36 (616): No heartbeat from core client for 30 sec - exiting 19:06:37 (616): No heartbeat from core client for 30 sec - exiting 19:06:38 (616): No heartbeat from core client for 30 sec - exiting 19:06:40 (616): No heartbeat from core client for 30 sec - exiting 19:06:41 (616): No heartbeat from core client for 30 sec - exiting 19:06:42 (616): No heartbeat from core client for 30 sec - exiting 19:06:43 (616): No heartbeat from core client for 30 sec - exiting 19:06:44 (616): No heartbeat from core client for 30 sec - exiting 19:06:45 (616): No heartbeat from core client for 30 sec - exiting 19:06:46 (616): No heartbeat from core client for 30 sec - exiting 19:06:47 (616): No heartbeat from core client for 30 sec - exiting 19:06:48 (616): No heartbeat from core client for 30 sec - exiting 19:06:49 (616): No heartbeat from core client for 30 sec - exiting 19:06:50 (616): No heartbeat from core client for 30 sec - exiting 19:06:51 (616): No heartbeat from core client for 30 sec - exiting 19:06:52 (616): No heartbeat from core client for 30 sec - exiting 19:06:53 (616): No heartbeat from core client for 30 sec - exiting 19:06:54 (616): No heartbeat from core client for 30 sec - exiting 19:06:55 (616): No heartbeat from core client for 30 sec - exiting 19:06:56 (616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:33:13 (5160): No heartbeat from core client for 30 sec - exiting 18:33:14 (5160): No heartbeat from core client for 30 sec - exiting 18:33:15 (5160): No heartbeat from core client for 30 sec - exiting 18:33:16 (5160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:04:09 (5984): No heartbeat from core client for 30 sec - exiting 00:04:10 (5984): No heartbeat from core client for 30 sec - exiting 00:04:11 (5984): No heartbeat from core client for 30 sec - exiting 00:04:13 (5984): No heartbeat from core client for 30 sec - exiting 00:04:14 (5984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:59:32 (5228): No heartbeat from core client for 30 sec - exiting 19:59:33 (5228): No heartbeat from core client for 30 sec - exiting 19:59:34 (5228): No heartbeat from core client for 30 sec - exiting 19:59:35 (5228): No heartbeat from core client for 30 sec - exiting 19:59:36 (5228): No heartbeat from core client for 30 sec - exiting 19:59:37 (5228): No heartbeat from core client for 30 sec - exiting 19:59:39 (5228): No heartbeat from core client for 30 sec - exiting 19:59:40 (5228): No heartbeat from core client for 30 sec - exiting 19:59:41 (5228): No heartbeat from core client for 30 sec - exiting 19:59:42 (5228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:54:57 (5096): No heartbeat from core client for 30 sec - exiting 22:54:58 (5096): No heartbeat from core client for 30 sec - exiting 22:54:59 (5096): No heartbeat from core client for 30 sec - exiting 22:55:00 (5096): No heartbeat from core client for 30 sec - exiting 22:55:01 (5096): No heartbeat from core client for 30 sec - exiting 22:55:02 (5096): No heartbeat from core client for 30 sec - exiting 22:55:03 (5096): No heartbeat from core client for 30 sec - exiting 22:55:04 (5096): No heartbeat from core client for 30 sec - exiting 22:55:05 (5096): No heartbeat from core client for 30 sec - exiting 22:55:06 (5096): No heartbeat from core client for 30 sec - exiting 22:55:07 (5096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:18:20 (4328): No heartbeat from core client for 30 sec - exiting 19:18:21 (4328): No heartbeat from core client for 30 sec - exiting 19:18:22 (4328): No heartbeat from core client for 30 sec - exiting 19:18:23 (4328): No heartbeat from core client for 30 sec - exiting 19:18:24 (4328): No heartbeat from core client for 30 sec - exiting 19:18:25 (4328): No heartbeat from core client for 30 sec - exiting 19:18:26 (4328): No heartbeat from core client for 30 sec - exiting 19:18:28 (4328): No heartbeat from core client for 30 sec - exiting 19:18:29 (4328): No heartbeat from core client for 30 sec - exiting 19:18:30 (4328): No heartbeat from core client for 30 sec - exiting 19:18:31 (4328): No heartbeat from core client for 30 sec - exiting 19:18:32 (4328): No heartbeat from core client for 30 sec - exiting 19:18:33 (4328): No heartbeat from core client for 30 sec - exiting 19:18:34 (4328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2768, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:55:34 (5688): No heartbeat from core client for 30 sec - exiting 18:55:35 (5688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:59:55 (5624): No heartbeat from core client for 30 sec - exiting 17:59:56 (5624): No heartbeat from core client for 30 sec - exiting 17:59:57 (5624): No heartbeat from core client for 30 sec - exiting 17:59:58 (5624): No heartbeat from core client for 30 sec - exiting 17:59:59 (5624): No heartbeat from core client for 30 sec - exiting 18:00:00 (5624): No heartbeat from core client for 30 sec - exiting 18:00:01 (5624): No heartbeat from core client for 30 sec - exiting 18:00:02 (5624): No heartbeat from core client for 30 sec - exiting 18:00:03 (5624): No heartbeat from core client for 30 sec - exiting 18:00:04 (5624): No heartbeat from core client for 30 sec - exiting 18:00:05 (5624): No heartbeat from core client for 30 sec - exiting 18:00:06 (5624): No heartbeat from core client for 30 sec - exiting 18:00:07 (5624): No heartbeat from core client for 30 sec - exiting 18:00:08 (5624): No heartbeat from core client for 30 sec - exiting 18:00:09 (5624): No heartbeat from core client for 30 sec - exiting 18:00:10 (5624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:38:46 (4832): No heartbeat from core client for 30 sec - exiting 19:38:47 (4832): No heartbeat from core client for 30 sec - exiting 19:38:48 (4832): No heartbeat from core client for 30 sec - exiting 19:38:49 (4832): No heartbeat from core client for 30 sec - exiting 19:38:50 (4832): No heartbeat from core client for 30 sec - exiting 19:38:51 (4832): No heartbeat from core client for 30 sec - exiting 19:38:52 (4832): No heartbeat from core client for 30 sec - exiting 19:38:53 (4832): No heartbeat from core client for 30 sec - exiting 19:38:54 (4832): No heartbeat from core client for 30 sec - exiting 19:38:55 (4832): No heartbeat from core client for 30 sec - exiting 19:38:56 (4832): No heartbeat from core client for 30 sec - exiting 19:38:57 (4832): No heartbeat from core client for 30 sec - exiting 19:38:58 (4832): No heartbeat from core client for 30 sec - exiting 19:38:59 (4832): No heartbeat from core client for 30 sec - exiting 19:39:00 (4832): No heartbeat from core client for 30 sec - exiting 19:39:01 (4832): No heartbeat from core client for 30 sec - exiting 19:39:02 (4832): No heartbeat from core client for 30 sec - exiting 19:39:03 (4832): No heartbeat from core client for 30 sec - exiting 19:39:04 (4832): No heartbeat from core client for 30 sec - exiting 19:39:05 (4832): No heartbeat from core client for 30 sec - exiting 19:39:06 (4832): No heartbeat from core client for 30 sec - exiting 19:39:07 (4832): No heartbeat from core client for 30 sec - exiting 19:39:08 (4832): No heartbeat from core client for 30 sec - exiting 19:39:09 (4832): No heartbeat from core client for 30 sec - exiting 19:39:10 (4832): No heartbeat from core client for 30 sec - exiting 19:39:11 (4832): No heartbeat from core client for 30 sec - exiting 19:39:12 (4832): No heartbeat from core client for 30 sec - exiting 19:39:13 (4832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5720, selfPID=5720, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:21:11 (4016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:40:01 (6032): No heartbeat from core client for 30 sec - exiting 12:40:02 (6032): No heartbeat from core client for 30 sec - exiting 12:40:03 (6032): No heartbeat from core client for 30 sec - exiting 12:40:05 (6032): No heartbeat from core client for 30 sec - exiting 12:40:06 (6032): No heartbeat from core client for 30 sec - exiting 12:40:07 (6032): No heartbeat from core client for 30 sec - exiting 12:40:08 (6032): No heartbeat from core client for 30 sec - exiting 12:40:09 (6032): No heartbeat from core client for 30 sec - exiting 12:40:10 (6032): No heartbeat from core client for 30 sec - exiting 12:40:11 (6032): No heartbeat from core client for 30 sec - exiting 12:40:12 (6032): No heartbeat from core client for 30 sec - exiting 12:40:13 (6032): No heartbeat from core client for 30 sec - exiting 12:40:14 (6032): No heartbeat from core client for 30 sec - exiting 12:40:15 (6032): No heartbeat from core client for 30 sec - exiting 12:40:16 (6032): No heartbeat from core client for 30 sec - exiting 12:40:17 (6032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:50:23 (5032): No heartbeat from core client for 30 sec - exiting 19:50:24 (5032): No heartbeat from core client for 30 sec - exiting 19:50:26 (5032): No heartbeat from core client for 30 sec - exiting 19:50:27 (5032): No heartbeat from core client for 30 sec - exiting 19:50:28 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:57:14 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:29:22 (624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:51:00 (5636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6312, iMonCtr=1 Model crash detected, will try to restart... 18:53:32 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:55:17 (5480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5348, iMonCtr=1 Model crash detected, will try to restart... 20:54:14 (4456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:57:43 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:32:36 (4756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:48:16 (3208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2260, iMonCtr=1 Model crash detected, will try to restart... 19:17:51 (6000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:45:36 (4488): No heartbeat from core client for 30 sec - exiting 21:45:37 (4488): No heartbeat from core client for 30 sec - exiting 21:45:38 (4488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:45:39 (4488): No heartbeat from core client for 30 sec - exiting 20:18:38 (5464): No heartbeat from core client for 30 sec - exiting 20:18:39 (5464): No heartbeat from core client for 30 sec - exiting 20:18:40 (5464): No heartbeat from core client for 30 sec - exiting 20:18:41 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:18:42 (5464): No heartbeat from core client for 30 sec - exiting 23:06:41 (5376): No heartbeat from core client for 30 sec - exiting 23:06:42 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2888, iMonCtr=1 Model crash detected, will try to restart... 17:59:59 (5304): No heartbeat from core client for 30 sec - exiting 18:00:00 (5304): No heartbeat from core client for 30 sec - exiting 18:00:01 (5304): No heartbeat from core client for 30 sec - exiting 18:00:02 (5304): No heartbeat from core client for 30 sec - exiting 18:00:03 (5304): No heartbeat from core client for 30 sec - exiting 18:00:04 (5304): No heartbeat from core client for 30 sec - exiting 18:00:05 (5304): No heartbeat from core client for 30 sec - exiting 18:00:06 (5304): No heartbeat from core client for 30 sec - exiting 18:00:08 (5304): No heartbeat from core client for 30 sec - exiting 18:00:09 (5304): No heartbeat from core client for 30 sec - exiting 18:00:10 (5304): No heartbeat from core client for 30 sec - exiting 18:00:11 (5304): No heartbeat from core client for 30 sec - exiting 18:00:12 (5304): No heartbeat from core client for 30 sec - exiting 18:00:13 (5304): No heartbeat from core client for 30 sec - exiting 18:00:14 (5304): No heartbeat from core client for 30 sec - exiting 18:00:15 (5304): No heartbeat from core client for 30 sec - exiting 18:00:16 (5304): No heartbeat from core client for 30 sec - exiting 18:00:17 (5304): No heartbeat from core client for 30 sec - exiting 18:00:18 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:38 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:38 (5344): No heartbeat from core client for 30 sec - exiting 22:25:39 (5344): No heartbeat from core client for 30 sec - exiting 22:25:40 (5344): No heartbeat from core client for 30 sec - exiting 22:25:42 (5344): No heartbeat from core client for 30 sec - exiting 22:25:43 (5344): No heartbeat from core client for 30 sec - exiting 22:25:44 (5344): No heartbeat from core client for 30 sec - exiting 22:25:45 (5344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:32:28 (5304): No heartbeat from core client for 30 sec - exiting 20:32:29 (5304): No heartbeat from core client for 30 sec - exiting 20:32:30 (5304): No heartbeat from core client for 30 sec - exiting 20:32:31 (5304): No heartbeat from core client for 30 sec - exiting 20:32:32 (5304): No heartbeat from core client for 30 sec - exiting 20:32:33 (5304): No heartbeat from core client for 30 sec - exiting 20:32:34 (5304): No heartbeat from core client for 30 sec - exiting 20:32:35 (5304): No heartbeat from core client for 30 sec - exiting 20:32:36 (5304): No heartbeat from core client for 30 sec - exiting 20:32:37 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:11 (6048): No heartbeat from core client for 30 sec - exiting 11:19:12 (6048): No heartbeat from core client for 30 sec - exiting 11:19:13 (6048): No heartbeat from core client for 30 sec - exiting 11:19:14 (6048): No heartbeat from core client for 30 sec - exiting 11:19:15 (6048): No heartbeat from core client for 30 sec - exiting 11:19:16 (6048): No heartbeat from core client for 30 sec - exiting 11:19:17 (6048): No heartbeat from core client for 30 sec - exiting 11:19:18 (6048): No heartbeat from core client for 30 sec - exiting 11:19:19 (6048): No heartbeat from core client for 30 sec - exiting 11:19:20 (6048): No heartbeat from core client for 30 sec - exiting 11:19:21 (6048): No heartbeat from core client for 30 sec - exiting 11:19:22 (6048): No heartbeat from core client for 30 sec - exiting 11:19:23 (6048): No heartbeat from core client for 30 sec - exiting 11:19:24 (6048): No heartbeat from core client for 30 sec - exiting 11:19:25 (6048): No heartbeat from core client for 30 sec - exiting 11:19:26 (6048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:19:27 (6048): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4208, iMonCtr=1 Model crash detected, will try to restart... 18:27:50 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5692, iMonCtr=1 Model crash detected, will try to restart... 09:36:46 (5768): No heartbeat from core client for 30 sec - exiting 09:36:48 (5768): No heartbeat from core client for 30 sec - exiting 09:36:49 (5768): No heartbeat from core client for 30 sec - exiting 09:36:50 (5768): No heartbeat from core client for 30 sec - exiting 09:36:51 (5768): No heartbeat from core client for 30 sec - exiting 09:36:52 (5768): No heartbeat from core client for 30 sec - exiting 09:36:53 (5768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4400, iMonCtr=1 Model crash detected, will try to restart... 01:02:55 (5468): No heartbeat from core client for 30 sec - exiting 01:02:56 (5468): No heartbeat from core client for 30 sec - exiting 01:02:57 (5468): No heartbeat from core client for 30 sec - exiting 01:02:58 (5468): No heartbeat from core client for 30 sec - exiting 01:02:59 (5468): No heartbeat from core client for 30 sec - exiting 01:03:00 (5468): No heartbeat from core client for 30 sec - exiting 01:03:01 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:03 (5752): No heartbeat from core client for 30 sec - exiting 18:40:04 (5752): No heartbeat from core client for 30 sec - exiting 18:40:05 (5752): No heartbeat from core client for 30 sec - exiting 18:40:06 (5752): No heartbeat from core client for 30 sec - exiting 18:40:07 (5752): No heartbeat from core client for 30 sec - exiting 18:40:08 (5752): No heartbeat from core client for 30 sec - exiting 18:40:09 (5752): No heartbeat from core client for 30 sec - exiting 18:40:10 (5752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:24:12 (5688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:29 (5636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:37:08 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:13:43 (5008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:43:39 (3176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:07:04 (4468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:51:47 (4448): No heartbeat from core client for 30 sec - exiting 21:51:48 (4448): No heartbeat from core client for 30 sec - exiting 21:51:49 (4448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:33:06 (6016): No heartbeat from core client for 30 sec - exiting 15:33:07 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:48:20 (5708): No heartbeat from core client for 30 sec - exiting 21:48:22 (5708): No heartbeat from core client for 30 sec - exiting 21:48:23 (5708): No heartbeat from core client for 30 sec - exiting 21:48:24 (5708): No heartbeat from core client for 30 sec - exiting 21:48:25 (5708): No heartbeat from core client for 30 sec - exiting 21:48:26 (5708): No heartbeat from core client for 30 sec - exiting 21:48:27 (5708): No heartbeat from core client for 30 sec - exiting 21:48:28 (5708): No heartbeat from core client for 30 sec - exiting 21:48:29 (5708): No heartbeat from core client for 30 sec - exiting 21:48:30 (5708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:11:14 (5932): No heartbeat from core client for 30 sec - exiting 12:11:15 (5932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:42:24 (1384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6104, iMonCtr=1 Model crash detected, will try to restart... 16:20:34 (5924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:45:40 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:44 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:08 (5764): No heartbeat from core client for 30 sec - exiting 19:59:09 (5764): No heartbeat from core client for 30 sec - exiting 19:59:10 (5764): No heartbeat from core client for 30 sec - exiting 19:59:11 (5764): No heartbeat from core client for 30 sec - exiting 19:59:12 (5764): No heartbeat from core client for 30 sec - exiting 19:59:14 (5764): No heartbeat from core client for 30 sec - exiting 19:59:15 (5764): No heartbeat from core client for 30 sec - exiting 19:59:16 (5764): No heartbeat from core client for 30 sec - exiting 19:59:17 (5764): No heartbeat from core client for 30 sec - exiting 19:59:18 (5764): No heartbeat from core client for 30 sec - exiting 19:59:19 (5764): No heartbeat from core client for 30 sec - exiting 19:59:20 (5764): No heartbeat from core client for 30 sec - exiting 19:59:21 (5764): No heartbeat from core client for 30 sec - exiting 19:59:22 (5764): No heartbeat from core client for 30 sec - exiting 19:59:23 (5764): No heartbeat from core client for 30 sec - exiting 19:59:25 (5764): No heartbeat from core client for 30 sec - exiting 19:59:26 (5764): No heartbeat from core client for 30 sec - exiting 19:59:27 (5764): No heartbeat from core client for 30 sec - exiting 19:59:28 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:34 (5852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:47 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:09:33 (4808): No heartbeat from core client for 30 sec - exiting 19:09:34 (4808): No heartbeat from core client for 30 sec - exiting 19:09:35 (4808): No heartbeat from core client for 30 sec - exiting 19:09:36 (4808): No heartbeat from core client for 30 sec - exiting 19:09:37 (4808): No heartbeat from core client for 30 sec - exiting 19:09:39 (4808): No heartbeat from core client for 30 sec - exiting 19:09:40 (4808): No heartbeat from core client for 30 sec - exiting 19:09:41 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:44 (5740): No heartbeat from core client for 30 sec - exiting 19:18:45 (5740): No heartbeat from core client for 30 sec - exiting 19:18:46 (5740): No heartbeat from core client for 30 sec - exiting 19:18:47 (5740): No heartbeat from core client for 30 sec - exiting 19:18:48 (5740): No heartbeat from core client for 30 sec - exiting 19:18:49 (5740): No heartbeat from core client for 30 sec - exiting 19:18:50 (5740): No heartbeat from core client for 30 sec - exiting 19:18:51 (5740): No heartbeat from core client for 30 sec - exiting 19:18:52 (5740): No heartbeat from core client for 30 sec - exiting 19:18:53 (5740): No heartbeat from core client for 30 sec - exiting 19:18:54 (5740): No heartbeat from core client for 30 sec - exiting 19:18:55 (5740): No heartbeat from core client for 30 sec - exiting 19:18:56 (5740): No heartbeat from core client for 30 sec - exiting 19:18:57 (5740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:19:48 (6548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:29 (5736): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:03:55 (5632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:20:21 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:09:29 (6140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5264, iMonCtr=1 Model crash detected, will try to restart... 19:57:14 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:59:03 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:55:08 (5780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:05 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:51:15 (6124): No heartbeat from core client for 30 sec - exiting 12:51:16 (6124): No heartbeat from core client for 30 sec - exiting 12:51:17 (6124): No heartbeat from core client for 30 sec - exiting 12:51:18 (6124): No heartbeat from core client for 30 sec - exiting 12:51:19 (6124): No heartbeat from core client for 30 sec - exiting 12:51:20 (6124): No heartbeat from core client for 30 sec - exiting 12:51:22 (6124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:08:40 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:06:57 (6188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:07:38 (5904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:20 (5788): No heartbeat from core client for 30 sec - exiting 19:52:21 (5788): No heartbeat from core client for 30 sec - exiting 19:52:22 (5788): No heartbeat from core client for 30 sec - exiting 19:52:23 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:25 (5788): No heartbeat from core client for 30 sec - exiting 21:30:01 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:38:32 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:53:24 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:40 (5548): No heartbeat from core client for 30 sec - exiting 19:00:41 (5548): No heartbeat from core client for 30 sec - exiting 19:00:42 (5548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:02:09 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6084, iMonCtr=1 Model crash detected, will try to restart... 19:11:32 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5916, iMonCtr=1 Model crash detected, will try to restart... 11:11:33 (4044): No heartbeat from core client for 30 sec - exiting 11:11:34 (4044): No heartbeat from core client for 30 sec - exiting 11:11:36 (4044): No heartbeat from core client for 30 sec - exiting 11:11:37 (4044): No heartbeat from core client for 30 sec - exiting 11:11:38 (4044): No heartbeat from core client for 30 sec - exiting 11:11:39 (4044): No heartbeat from core client for 30 sec - exiting 11:11:40 (4044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4952, iMonCtr=1 Model crash detected, will try to restart... 11:00:35 (5884): No heartbeat from core client for 30 sec - exiting 11:00:36 (5884): No heartbeat from core client for 30 sec - exiting 11:00:37 (5884): No heartbeat from core client for 30 sec - exiting 11:00:38 (5884): No heartbeat from core client for 30 sec - exiting 11:00:39 (5884): No heartbeat from core client for 30 sec - exiting 11:00:40 (5884): No heartbeat from core client for 30 sec - exiting 11:00:41 (5884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:04:27 (5456): No heartbeat from core client for 30 sec - exiting 13:04:29 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:36:32 (5948): No heartbeat from core client for 30 sec - exiting 19:36:34 (5948): No heartbeat from core client for 30 sec - exiting 19:36:35 (5948): No heartbeat from core client for 30 sec - exiting 19:36:36 (5948): No heartbeat from core client for 30 sec - exiting 19:36:37 (5948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x772C3A93 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o35u_1940_40_007443255/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Dec 2011 19:45:26	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	1,036,800	1,855,316	1.7895
05 Dec 2011 13:12:11	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	1,010,880	1,808,865	1.7894
04 Dec 2011 11:30:36	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	984,960	1,761,878	1.7888
03 Dec 2011 13:17:18	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	959,040	1,714,207	1.7874
01 Dec 2011 21:13:44	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	933,120	1,667,332	1.7868
29 Nov 2011 20:42:31	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	907,200	1,621,100	1.7869
27 Nov 2011 18:43:58	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	881,280	1,574,572	1.7867
26 Nov 2011 16:51:31	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	855,360	1,528,253	1.7867
23 Nov 2011 22:48:41	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	829,440	1,482,320	1.7871
20 Nov 2011 00:20:54	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	803,520	1,435,401	1.7864
18 Nov 2011 00:01:48	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	777,600	1,390,008	1.7876
15 Nov 2011 23:10:26	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	751,680	1,344,074	1.7881
15 Nov 2011 19:57:17	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	725,760	1,298,465	1.7891
15 Nov 2011 19:57:17	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	699,840	1,252,121	1.7892
15 Nov 2011 19:57:17	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	673,920	1,205,603	1.7889
09 Nov 2011 22:56:07	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	648,000	1,159,857	1.7899
08 Nov 2011 19:35:32	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	622,080	1,113,918	1.7906
07 Nov 2011 20:14:18	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	596,160	1,067,889	1.7913
06 Nov 2011 20:54:06	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	570,240	1,021,616	1.7916
05 Nov 2011 13:55:06	1072815	13348460	hadcm3n_o35u_1940_40_007443255_1	544,320	975,915	1.7929