Task 14358873

Name	hadcm3n_o1xv_2020_40_007856806_0
Workunit	8011918
Created	4 Apr 2012, 19:09:22 UTC
Sent	4 Apr 2012, 19:14:33 UTC
Report deadline	5 Jul 2012, 2:41:44 UTC
Received	17 May 2012, 16:48:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1135421
Run time	8 days 5 hours 25 min 41 sec
CPU time	7 days 9 hours 48 min 10 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.51 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4944, iMonCtr=1 Model crash detected, will try to restart... 18:45:08 (4108): No heartbeat from core client for 30 sec - exiting 18:45:09 (4108): No heartbeat from core client for 30 sec - exiting 18:45:10 (4108): No heartbeat from core client for 30 sec - exiting 18:45:11 (4108): No heartbeat from core client for 30 sec - exiting 18:45:12 (4108): No heartbeat from core client for 30 sec - exiting 18:45:13 (4108): No heartbeat from core client for 30 sec - exiting 18:45:14 (4108): No heartbeat from core client for 30 sec - exiting 18:45:15 (4108): No heartbeat from core client for 30 sec - exiting 18:45:16 (4108): No heartbeat from core client for 30 sec - exiting 18:45:17 (4108): No heartbeat from core client for 30 sec - exiting 18:45:18 (4108): No heartbeat from core client for 30 sec - exiting 18:45:19 (4108): No heartbeat from core client for 30 sec - exiting 18:45:20 (4108): No heartbeat from core client for 30 sec - exiting 18:45:21 (4108): No heartbeat from core client for 30 sec - exiting 18:45:22 (4108): No heartbeat from core client for 30 sec - exiting 18:45:23 (4108): No heartbeat from core client for 30 sec - exiting 18:45:24 (4108): No heartbeat from core client for 30 sec - exiting 18:45:25 (4108): No heartbeat from core client for 30 sec - exiting 18:45:26 (4108): No heartbeat from core client for 30 sec - exiting 18:45:27 (4108): No heartbeat from core client for 30 sec - exiting 18:45:28 (4108): No heartbeat from core client for 30 sec - exiting 18:45:29 (4108): No heartbeat from core client for 30 sec - exiting 18:45:30 (4108): No heartbeat from core client for 30 sec - exiting 18:45:31 (4108): No heartbeat from core client for 30 sec - exiting 18:45:32 (4108): No heartbeat from core client for 30 sec - exiting 18:45:33 (4108): No heartbeat from core client for 30 sec - exiting 18:45:34 (4108): No heartbeat from core client for 30 sec - exiting 18:45:35 (4108): No heartbeat from core client for 30 sec - exiting 18:45:36 (4108): No heartbeat from core client for 30 sec - exiting 18:45:37 (4108): No heartbeat from core client for 30 sec - exiting 18:45:38 (4108): No heartbeat from core client for 30 sec - exiting 18:45:39 (4108): No heartbeat from core client for 30 sec - exiting 18:45:40 (4108): No heartbeat from core client for 30 sec - exiting 18:45:41 (4108): No heartbeat from core client for 30 sec - exiting 18:45:42 (4108): No heartbeat from core client for 30 sec - exiting 18:45:43 (4108): No heartbeat from core client for 30 sec - exiting 18:45:44 (4108): No heartbeat from core client for 30 sec - exiting 18:45:45 (4108): No heartbeat from core client for 30 sec - exiting 18:45:46 (4108): No heartbeat from core client for 30 sec - exiting 18:45:47 (4108): No heartbeat from core client for 30 sec - exiting 18:45:48 (4108): No heartbeat from core client for 30 sec - exiting 18:45:49 (4108): No heartbeat from core client for 30 sec - exiting 18:45:50 (4108): No heartbeat from core client for 30 sec - exiting 18:45:51 (4108): No heartbeat from core client for 30 sec - exiting 18:45:52 (4108): No heartbeat from core client for 30 sec - exiting 18:45:53 (4108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:31:30 (4656): No heartbeat from core client for 30 sec - exiting 18:31:31 (4656): No heartbeat from core client for 30 sec - exiting 18:31:32 (4656): No heartbeat from core client for 30 sec - exiting 18:31:33 (4656): No heartbeat from core client for 30 sec - exiting 18:31:34 (4656): No heartbeat from core client for 30 sec - exiting 18:31:35 (4656): No heartbeat from core client for 30 sec - exiting 18:31:36 (4656): No heartbeat from core client for 30 sec - exiting 18:31:37 (4656): No heartbeat from core client for 30 sec - exiting 18:31:38 (4656): No heartbeat from core client for 30 sec - exiting 18:31:39 (4656): No heartbeat from core client for 30 sec - exiting 18:31:40 (4656): No heartbeat from core client for 30 sec - exiting 18:31:41 (4656): No heartbeat from core client for 30 sec - exiting 18:31:42 (4656): No heartbeat from core client for 30 sec - exiting 18:31:43 (4656): No heartbeat from core client for 30 sec - exiting 18:31:44 (4656): No heartbeat from core client for 30 sec - exiting 18:31:45 (4656): No heartbeat from core client for 30 sec - exiting 18:31:46 (4656): No heartbeat from core client for 30 sec - exiting 18:31:47 (4656): No heartbeat from core client for 30 sec - exiting 18:31:48 (4656): No heartbeat from core client for 30 sec - exiting 18:31:49 (4656): No heartbeat from core client for 30 sec - exiting 18:31:50 (4656): No heartbeat from core client for 30 sec - exiting 18:31:51 (4656): No heartbeat from core client for 30 sec - exiting 18:31:52 (4656): No heartbeat from core client for 30 sec - exiting 18:31:53 (4656): No heartbeat from core client for 30 sec - exiting 18:31:54 (4656): No heartbeat from core client for 30 sec - exiting 18:31:55 (4656): No heartbeat from core client for 30 sec - exiting 18:31:56 (4656): No heartbeat from core client for 30 sec - exiting 18:31:57 (4656): No heartbeat from core client for 30 sec - exiting 18:31:58 (4656): No heartbeat from core client for 30 sec - exiting 18:31:59 (4656): No heartbeat from core client for 30 sec - exiting 18:32:00 (4656): No heartbeat from core client for 30 sec - exiting 18:32:01 (4656): No heartbeat from core client for 30 sec - exiting 18:32:02 (4656): No heartbeat from core client for 30 sec - exiting 18:32:03 (4656): No heartbeat from core client for 30 sec - exiting 18:32:04 (4656): No heartbeat from core client for 30 sec - exiting 18:32:05 (4656): No heartbeat from core client for 30 sec - exiting 18:32:06 (4656): No heartbeat from core client for 30 sec - exiting 18:32:07 (4656): No heartbeat from core client for 30 sec - exiting 18:32:08 (4656): No heartbeat from core client for 30 sec - exiting 18:32:09 (4656): No heartbeat from core client for 30 sec - exiting 18:32:10 (4656): No heartbeat from core client for 30 sec - exiting 18:32:11 (4656): No heartbeat from core client for 30 sec - exiting 18:32:12 (4656): No heartbeat from core client for 30 sec - exiting 18:32:13 (4656): No heartbeat from core client for 30 sec - exiting 18:32:14 (4656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:05 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:36:15 (3332): No heartbeat from core client for 30 sec - exiting 18:36:16 (3332): No heartbeat from core client for 30 sec - exiting 18:36:17 (3332): No heartbeat from core client for 30 sec - exiting 18:36:18 (3332): No heartbeat from core client for 30 sec - exiting 18:36:19 (3332): No heartbeat from core client for 30 sec - exiting 18:36:20 (3332): No heartbeat from core client for 30 sec - exiting 18:36:21 (3332): No heartbeat from core client for 30 sec - exiting 18:36:22 (3332): No heartbeat from core client for 30 sec - exiting 18:36:23 (3332): No heartbeat from core client for 30 sec - exiting 18:36:24 (3332): No heartbeat from core client for 30 sec - exiting 18:36:25 (3332): No heartbeat from core client for 30 sec - exiting 18:36:26 (3332): No heartbeat from core client for 30 sec - exiting 18:36:27 (3332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:08 (4968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:29:10 (4968): No heartbeat from core client for 30 sec - exiting 21:30:22 (3040): No heartbeat from core client for 30 sec - exiting 21:30:23 (3040): No heartbeat from core client for 30 sec - exiting 21:30:24 (3040): No heartbeat from core client for 30 sec - exiting 21:30:25 (3040): No heartbeat from core client for 30 sec - exiting 21:30:26 (3040): No heartbeat from core client for 30 sec - exiting 21:30:27 (3040): No heartbeat from core client for 30 sec - exiting 21:30:28 (3040): No heartbeat from core client for 30 sec - exiting 21:30:29 (3040): No heartbeat from core client for 30 sec - exiting 21:30:30 (3040): No heartbeat from core client for 30 sec - exiting 21:30:31 (3040): No heartbeat from core client for 30 sec - exiting 21:30:32 (3040): No heartbeat from core client for 30 sec - exiting 21:30:33 (3040): No heartbeat from core client for 30 sec - exiting 21:30:34 (3040): No heartbeat from core client for 30 sec - exiting 21:30:35 (3040): No heartbeat from core client for 30 sec - exiting 21:30:36 (3040): No heartbeat from core client for 30 sec - exiting 21:30:37 (3040): No heartbeat from core client for 30 sec - exiting 21:30:38 (3040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:31:45 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2312, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2312, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2312, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2312, iMonCtr=1 Model crash detected, will try to restart... 06:37:07 (2312): No heartbeat from core client for 30 sec - exiting 06:37:08 (2312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:34:24 (4396): No heartbeat from core client for 30 sec - exiting 18:34:26 (4396): No heartbeat from core client for 30 sec - exiting 18:34:27 (4396): No heartbeat from core client for 30 sec - exiting 18:34:28 (4396): No heartbeat from core client for 30 sec - exiting 18:34:29 (4396): No heartbeat from core client for 30 sec - exiting 18:34:30 (4396): No heartbeat from core client for 30 sec - exiting 18:34:31 (4396): No heartbeat from core client for 30 sec - exiting 18:34:32 (4396): No heartbeat from core client for 30 sec - exiting 18:34:33 (4396): No heartbeat from core client for 30 sec - exiting 18:34:34 (4396): No heartbeat from core client for 30 sec - exiting 18:34:35 (4396): No heartbeat from core client for 30 sec - exiting 18:34:36 (4396): No heartbeat from core client for 30 sec - exiting 18:34:37 (4396): No heartbeat from core client for 30 sec - exiting 18:34:38 (4396): No heartbeat from core client for 30 sec - exiting 18:34:39 (4396): No heartbeat from core client for 30 sec - exiting 18:34:40 (4396): No heartbeat from core client for 30 sec - exiting 18:34:41 (4396): No heartbeat from core client for 30 sec - exiting 18:34:42 (4396): No heartbeat from core client for 30 sec - exiting 18:34:43 (4396): No heartbeat from core client for 30 sec - exiting 18:34:44 (4396): No heartbeat from core client for 30 sec - exiting 18:34:45 (4396): No heartbeat from core client for 30 sec - exiting 18:34:46 (4396): No heartbeat from core client for 30 sec - exiting 18:34:47 (4396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:10:21 (4580): No heartbeat from core client for 30 sec - exiting 18:10:22 (4580): No heartbeat from core client for 30 sec - exiting 18:10:23 (4580): No heartbeat from core client for 30 sec - exiting 18:10:24 (4580): No heartbeat from core client for 30 sec - exiting 18:10:25 (4580): No heartbeat from core client for 30 sec - exiting 18:10:26 (4580): No heartbeat from core client for 30 sec - exiting 18:10:27 (4580): No heartbeat from core client for 30 sec - exiting 18:10:28 (4580): No heartbeat from core client for 30 sec - exiting 18:10:29 (4580): No heartbeat from core client for 30 sec - exiting 18:10:30 (4580): No heartbeat from core client for 30 sec - exiting 18:10:31 (4580): No heartbeat from core client for 30 sec - exiting 18:10:32 (4580): No heartbeat from core client for 30 sec - exiting 18:10:33 (4580): No heartbeat from core client for 30 sec - exiting 18:10:34 (4580): No heartbeat from core client for 30 sec - exiting 18:10:35 (4580): No heartbeat from core client for 30 sec - exiting 18:10:36 (4580): No heartbeat from core client for 30 sec - exiting 18:10:37 (4580): No heartbeat from core client for 30 sec - exiting 18:10:38 (4580): No heartbeat from core client for 30 sec - exiting 18:10:39 (4580): No heartbeat from core client for 30 sec - exiting 18:10:40 (4580): No heartbeat from core client for 30 sec - exiting 18:10:41 (4580): No heartbeat from core client for 30 sec - exiting 18:10:42 (4580): No heartbeat from core client for 30 sec - exiting 18:10:43 (4580): No heartbeat from core client for 30 sec - exiting 18:10:44 (4580): No heartbeat from core client for 30 sec - exiting 18:10:45 (4580): No heartbeat from core client for 30 sec - exiting 18:10:46 (4580): No heartbeat from core client for 30 sec - exiting 18:10:47 (4580): No heartbeat from core client for 30 sec - exiting 18:10:48 (4580): No heartbeat from core client for 30 sec - exiting 18:10:49 (4580): No heartbeat from core client for 30 sec - exiting 18:10:50 (4580): No heartbeat from core client for 30 sec - exiting 18:10:51 (4580): No heartbeat from core client for 30 sec - exiting 18:10:52 (4580): No heartbeat from core client for 30 sec - exiting 18:10:53 (4580): No heartbeat from core client for 30 sec - exiting 18:10:54 (4580): No heartbeat from core client for 30 sec - exiting 18:10:55 (4580): No heartbeat from core client for 30 sec - exiting 18:10:56 (4580): No heartbeat from core client for 30 sec - exiting 18:10:57 (4580): No heartbeat from core client for 30 sec - exiting 18:10:58 (4580): No heartbeat from core client for 30 sec - exiting 18:10:59 (4580): No heartbeat from core client for 30 sec - exiting 18:11:00 (4580): No heartbeat from core client for 30 sec - exiting 18:11:01 (4580): No heartbeat from core client for 30 sec - exiting 18:11:02 (4580): No heartbeat from core client for 30 sec - exiting 18:11:03 (4580): No heartbeat from core client for 30 sec - exiting 18:11:04 (4580): No heartbeat from core client for 30 sec - exiting 18:11:05 (4580): No heartbeat from core client for 30 sec - exiting 18:11:06 (4580): No heartbeat from core client for 30 sec - exiting 18:11:07 (4580): No heartbeat from core client for 30 sec - exiting 18:11:08 (4580): No heartbeat from core client for 30 sec - exiting 18:11:09 (4580): No heartbeat from core client for 30 sec - exiting 18:11:10 (4580): No heartbeat from core client for 30 sec - exiting 18:11:11 (4580): No heartbeat from core client for 30 sec - exiting 18:11:12 (4580): No heartbeat from core client for 30 sec - exiting 18:11:13 (4580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:34 (4684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:16:35 (4684): No heartbeat from core client for 30 sec - exiting 18:17:22 (5080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:42 (3700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:18:43 (3700): No heartbeat from core client for 30 sec - exiting 18:18:44 (3700): No heartbeat from core client for 30 sec - exiting 18:19:45 (4996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:24 (4792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:41 (4792): No heartbeat from core client for 30 sec - exiting 21:08:48 (4792): No heartbeat from core client for 30 sec - exiting 21:08:51 (4792): No heartbeat from core client for 30 sec - exiting 21:08:52 (4792): No heartbeat from core client for 30 sec - exiting 21:08:53 (4792): No heartbeat from core client for 30 sec - exiting 21:12:27 (4680): No heartbeat from core client for 30 sec - exiting 21:12:28 (4680): No heartbeat from core client for 30 sec - exiting 21:12:29 (4680): No heartbeat from core client for 30 sec - exiting 21:12:30 (4680): No heartbeat from core client for 30 sec - exiting 21:12:31 (4680): No heartbeat from core client for 30 sec - exiting 21:12:32 (4680): No heartbeat from core client for 30 sec - exiting 21:12:33 (4680): No heartbeat from core client for 30 sec - exiting 21:12:34 (4680): No heartbeat from core client for 30 sec - exiting 21:12:35 (4680): No heartbeat from core client for 30 sec - exiting 21:12:36 (4680): No heartbeat from core client for 30 sec - exiting 21:12:37 (4680): No heartbeat from core client for 30 sec - exiting 21:12:38 (4680): No heartbeat from core client for 30 sec - exiting 21:12:39 (4680): No heartbeat from core client for 30 sec - exiting 21:12:40 (4680): No heartbeat from core client for 30 sec - exiting 21:12:41 (4680): No heartbeat from core client for 30 sec - exiting 21:12:42 (4680): No heartbeat from core client for 30 sec - exiting 21:12:43 (4680): No heartbeat from core client for 30 sec - exiting 21:12:44 (4680): No heartbeat from core client for 30 sec - exiting 21:12:45 (4680): No heartbeat from core client for 30 sec - exiting 21:12:47 (4680): No heartbeat from core client for 30 sec - exiting 21:12:48 (4680): No heartbeat from core client for 30 sec - exiting 21:12:49 (4680): No heartbeat from core client for 30 sec - exiting 21:12:50 (4680): No heartbeat from core client for 30 sec - exiting 21:12:51 (4680): No heartbeat from core client for 30 sec - exiting 21:12:52 (4680): No heartbeat from core client for 30 sec - exiting 21:12:53 (4680): No heartbeat from core client for 30 sec - exiting 21:12:54 (4680): No heartbeat from core client for 30 sec - exiting 21:12:55 (4680): No heartbeat from core client for 30 sec - exiting 21:12:56 (4680): No heartbeat from core client for 30 sec - exiting 21:12:57 (4680): No heartbeat from core client for 30 sec - exiting 21:12:58 (4680): No heartbeat from core client for 30 sec - exiting 21:12:59 (4680): No heartbeat from core client for 30 sec - exiting 21:13:00 (4680): No heartbeat from core client for 30 sec - exiting 21:13:01 (4680): No heartbeat from core client for 30 sec - exiting 21:13:02 (4680): No heartbeat from core client for 30 sec - exiting 21:13:03 (4680): No heartbeat from core client for 30 sec - exiting 21:13:04 (4680): No heartbeat from core client for 30 sec - exiting 21:13:05 (4680): No heartbeat from core client for 30 sec - exiting 21:13:06 (4680): No heartbeat from core client for 30 sec - exiting 21:13:07 (4680): No heartbeat from core client for 30 sec - exiting 21:13:08 (4680): No heartbeat from core client for 30 sec - exiting 21:13:09 (4680): No heartbeat from core client for 30 sec - exiting 21:13:10 (4680): No heartbeat from core client for 30 sec - exiting 21:13:11 (4680): No heartbeat from core client for 30 sec - exiting 21:13:12 (4680): No heartbeat from core client for 30 sec - exiting 21:13:13 (4680): No heartbeat from core client for 30 sec - exiting 21:13:14 (4680): No heartbeat from core client for 30 sec - exiting 21:13:15 (4680): No heartbeat from core client for 30 sec - exiting 21:13:16 (4680): No heartbeat from core client for 30 sec - exiting 21:13:17 (4680): No heartbeat from core client for 30 sec - exiting 21:13:18 (4680): No heartbeat from core client for 30 sec - exiting 21:13:19 (4680): No heartbeat from core client for 30 sec - exiting 21:13:20 (4680): No heartbeat from core client for 30 sec - exiting 21:13:21 (4680): No heartbeat from core client for 30 sec - exiting 21:13:22 (4680): No heartbeat from core client for 30 sec - exiting 21:13:23 (4680): No heartbeat from core client for 30 sec - exiting 21:13:24 (4680): No heartbeat from core client for 30 sec - exiting 21:13:25 (4680): No heartbeat from core client for 30 sec - exiting 21:13:26 (4680): No heartbeat from core client for 30 sec - exiting 21:13:27 (4680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6092, iMonCtr=1 Model crash detected, will try to restart... 21:11:03 (4632): No heartbeat from core client for 30 sec - exiting 21:11:04 (4632): No heartbeat from core client for 30 sec - exiting 21:11:05 (4632): No heartbeat from core client for 30 sec - exiting 21:11:06 (4632): No heartbeat from core client for 30 sec - exiting 21:11:07 (4632): No heartbeat from core client for 30 sec - exiting 21:11:08 (4632): No heartbeat from core client for 30 sec - exiting 21:11:09 (4632): No heartbeat from core client for 30 sec - exiting 21:11:10 (4632): No heartbeat from core client for 30 sec - exiting 21:11:11 (4632): No heartbeat from core client for 30 sec - exiting 21:11:12 (4632): No heartbeat from core client for 30 sec - exiting 21:11:13 (4632): No heartbeat from core client for 30 sec - exiting 21:11:14 (4632): No heartbeat from core client for 30 sec - exiting 21:11:15 (4632): No heartbeat from core client for 30 sec - exiting 21:11:16 (4632): No heartbeat from core client for 30 sec - exiting 21:11:17 (4632): No heartbeat from core client for 30 sec - exiting 21:11:18 (4632): No heartbeat from core client for 30 sec - exiting 21:11:19 (4632): No heartbeat from core client for 30 sec - exiting 21:11:20 (4632): No heartbeat from core client for 30 sec - exiting 21:11:21 (4632): No heartbeat from core client for 30 sec - exiting 21:11:22 (4632): No heartbeat from core client for 30 sec - exiting 21:11:23 (4632): No heartbeat from core client for 30 sec - exiting 21:11:24 (4632): No heartbeat from core client for 30 sec - exiting 21:11:25 (4632): No heartbeat from core client for 30 sec - exiting 21:11:26 (4632): No heartbeat from core client for 30 sec - exiting 21:11:27 (4632): No heartbeat from core client for 30 sec - exiting 21:11:28 (4632): No heartbeat from core client for 30 sec - exiting 21:11:29 (4632): No heartbeat from core client for 30 sec - exiting 21:11:30 (4632): No heartbeat from core client for 30 sec - exiting 21:11:31 (4632): No heartbeat from core client for 30 sec - exiting 21:11:32 (4632): No heartbeat from core client for 30 sec - exiting 21:11:33 (4632): No heartbeat from core client for 30 sec - exiting 21:11:34 (4632): No heartbeat from core client for 30 sec - exiting 21:11:35 (4632): No heartbeat from core client for 30 sec - exiting 21:11:36 (4632): No heartbeat from core client for 30 sec - exiting 21:11:37 (4632): No heartbeat from core client for 30 sec - exiting 21:11:38 (4632): No heartbeat from core client for 30 sec - exiting 21:11:39 (4632): No heartbeat from core client for 30 sec - exiting 21:11:40 (4632): No heartbeat from core client for 30 sec - exiting 21:11:41 (4632): No heartbeat from core client for 30 sec - exiting 21:11:42 (4632): No heartbeat from core client for 30 sec - exiting 21:11:43 (4632): No heartbeat from core client for 30 sec - exiting 21:11:44 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:11:45 (4632): No heartbeat from core client for 30 sec - exiting 21:11:46 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3884, iMonCtr=1 Model crash detected, will try to restart... 18:07:41 (5092): No heartbeat from core client for 30 sec - exiting 18:07:42 (5092): No heartbeat from core client for 30 sec - exiting 18:07:43 (5092): No heartbeat from core client for 30 sec - exiting 18:07:44 (5092): No heartbeat from core client for 30 sec - exiting 18:07:45 (5092): No heartbeat from core client for 30 sec - exiting 18:07:46 (5092): No heartbeat from core client for 30 sec - exiting 18:07:47 (5092): No heartbeat from core client for 30 sec - exiting 18:07:48 (5092): No heartbeat from core client for 30 sec - exiting 18:07:49 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5564, iMonCtr=1 Model crash detected, will try to restart... 08:33:45 (4360): No heartbeat from core client for 30 sec - exiting 08:33:46 (4360): No heartbeat from core client for 30 sec - exiting 08:33:47 (4360): No heartbeat from core client for 30 sec - exiting 08:33:48 (4360): No heartbeat from core client for 30 sec - exiting 08:33:49 (4360): No heartbeat from core client for 30 sec - exiting 08:33:50 (4360): No heartbeat from core client for 30 sec - exiting 08:33:51 (4360): No heartbeat from core client for 30 sec - exiting 08:33:52 (4360): No heartbeat from core client for 30 sec - exiting 08:33:53 (4360): No heartbeat from core client for 30 sec - exiting 08:33:54 (4360): No heartbeat from core client for 30 sec - exiting 08:33:55 (4360): No heartbeat from core client for 30 sec - exiting 08:33:56 (4360): No heartbeat from core client for 30 sec - exiting 08:33:57 (4360): No heartbeat from core client for 30 sec - exiting 08:33:58 (4360): No heartbeat from core client for 30 sec - exiting 08:33:59 (4360): No heartbeat from core client for 30 sec - exiting 08:34:00 (4360): No heartbeat from core client for 30 sec - exiting 08:34:01 (4360): No heartbeat from core client for 30 sec - exiting 08:34:02 (4360): No heartbeat from core client for 30 sec - exiting 08:34:03 (4360): No heartbeat from core client for 30 sec - exiting 08:34:04 (4360): No heartbeat from core client for 30 sec - exiting 08:34:05 (4360): No heartbeat from core client for 30 sec - exiting 08:34:06 (4360): No heartbeat from core client for 30 sec - exiting 08:34:07 (4360): No heartbeat from core client for 30 sec - exiting 08:34:08 (4360): No heartbeat from core client for 30 sec - exiting 08:34:09 (4360): No heartbeat from core client for 30 sec - exiting 08:34:10 (4360): No heartbeat from core client for 30 sec - exiting 08:34:11 (4360): No heartbeat from core client for 30 sec - exiting 08:34:12 (4360): No heartbeat from core client for 30 sec - exiting 08:34:13 (4360): No heartbeat from core client for 30 sec - exiting 08:34:15 (4360): No heartbeat from core client for 30 sec - exiting 08:34:16 (4360): No heartbeat from core client for 30 sec - exiting 08:34:17 (4360): No heartbeat from core client for 30 sec - exiting 08:34:18 (4360): No heartbeat from core client for 30 sec - exiting 08:34:19 (4360): No heartbeat from core client for 30 sec - exiting 08:34:20 (4360): No heartbeat from core client for 30 sec - exiting 08:34:21 (4360): No heartbeat from core client for 30 sec - exiting 08:34:22 (4360): No heartbeat from core client for 30 sec - exiting 08:34:23 (4360): No heartbeat from core client for 30 sec - exiting 08:34:24 (4360): No heartbeat from core client for 30 sec - exiting 08:34:25 (4360): No heartbeat from core client for 30 sec - exiting 08:34:26 (4360): No heartbeat from core client for 30 sec - exiting 08:34:27 (4360): No heartbeat from core client for 30 sec - exiting 08:34:28 (4360): No heartbeat from core client for 30 sec - exiting 08:34:29 (4360): No heartbeat from core client for 30 sec - exiting 08:34:30 (4360): No heartbeat from core client for 30 sec - exiting 08:34:31 (4360): No heartbeat from core client for 30 sec - exiting 08:34:32 (4360): No heartbeat from core client for 30 sec - exiting 08:34:33 (4360): No heartbeat from core client for 30 sec - exiting 08:34:34 (4360): No heartbeat from core client for 30 sec - exiting 08:34:35 (4360): No heartbeat from core client for 30 sec - exiting 08:34:36 (4360): No heartbeat from core client for 30 sec - exiting 08:34:37 (4360): No heartbeat from core client for 30 sec - exiting 08:34:38 (4360): No heartbeat from core client for 30 sec - exiting 08:34:39 (4360): No heartbeat from core client for 30 sec - exiting 08:34:40 (4360): No heartbeat from core client for 30 sec - exiting 08:34:41 (4360): No heartbeat from core client for 30 sec - exiting 08:34:42 (4360): No heartbeat from core client for 30 sec - exiting 08:34:43 (4360): No heartbeat from core client for 30 sec - exiting 08:34:44 (4360): No heartbeat from core client for 30 sec - exiting 08:34:45 (4360): No heartbeat from core client for 30 sec - exiting 08:34:46 (4360): No heartbeat from core client for 30 sec - exiting 08:34:47 (4360): No heartbeat from core client for 30 sec - exiting 08:34:48 (4360): No heartbeat from core client for 30 sec - exiting 08:34:49 (4360): No heartbeat from core client for 30 sec - exiting 08:34:50 (4360): No heartbeat from core client for 30 sec - exiting 08:34:51 (4360): No heartbeat from core client for 30 sec - exiting 08:34:52 (4360): No heartbeat from core client for 30 sec - exiting 08:34:53 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4840, iMonCtr=1 Model crash detected, will try to restart... 21:15:36 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:15:37 (4352): No heartbeat from core client for 30 sec - exiting 21:16:25 (4808): No heartbeat from core client for 30 sec - exiting 21:16:26 (4808): No heartbeat from core client for 30 sec - exiting 21:16:27 (4808): No heartbeat from core client for 30 sec - exiting 21:16:28 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:17:38 (1740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:18:14 (2224): No heartbeat from core client for 30 sec - exiting 21:18:15 (2224): No heartbeat from core client for 30 sec - exiting 21:18:16 (2224): No heartbeat from core client for 30 sec - exiting 21:18:17 (2224): No heartbeat from core client for 30 sec - exiting 21:18:18 (2224): No heartbeat from core client for 30 sec - exiting 21:18:19 (2224): No heartbeat from core client for 30 sec - exiting 21:18:20 (2224): No heartbeat from core client for 30 sec - exiting 21:18:21 (2224): No heartbeat from core client for 30 sec - exiting 21:18:22 (2224): No heartbeat from core client for 30 sec - exiting 21:18:23 (2224): No heartbeat from core client for 30 sec - exiting 21:18:24 (2224): No heartbeat from core client for 30 sec - exiting 21:18:25 (2224): No heartbeat from core client for 30 sec - exiting 21:18:26 (2224): No heartbeat from core client for 30 sec - exiting 21:18:27 (2224): No heartbeat from core client for 30 sec - exiting 21:18:28 (2224): No heartbeat from core client for 30 sec - exiting 21:18:29 (2224): No heartbeat from core client for 30 sec - exiting 21:18:30 (2224): No heartbeat from core client for 30 sec - exiting 21:18:31 (2224): No heartbeat from core client for 30 sec - exiting 21:18:32 (2224): No heartbeat from core client for 30 sec - exiting 21:18:33 (2224): No heartbeat from core client for 30 sec - exiting 21:18:34 (2224): No heartbeat from core client for 30 sec - exiting 21:18:35 (2224): No heartbeat from core client for 30 sec - exiting 21:18:36 (2224): No heartbeat from core client for 30 sec - exiting 21:18:37 (2224): No heartbeat from core client for 30 sec - exiting 21:18:38 (2224): No heartbeat from core client for 30 sec - exiting 21:18:39 (2224): No heartbeat from core client for 30 sec - exiting 21:18:40 (2224): No heartbeat from core client for 30 sec - exiting 21:18:41 (2224): No heartbeat from core client for 30 sec - exiting 21:18:42 (2224): No heartbeat from core client for 30 sec - exiting 21:18:43 (2224): No heartbeat from core client for 30 sec - exiting 21:18:44 (2224): No heartbeat from core client for 30 sec - exiting 21:18:45 (2224): No heartbeat from core client for 30 sec - exiting 21:18:46 (2224): No heartbeat from core client for 30 sec - exiting 21:18:47 (2224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:19:35 (5896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:20:17 (852): No heartbeat from core client for 30 sec - exiting 21:20:18 (852): No heartbeat from core client for 30 sec - exiting 21:20:19 (852): No heartbeat from core client for 30 sec - exiting 21:20:20 (852): No heartbeat from core client for 30 sec - exiting 21:20:21 (852): No heartbeat from core client for 30 sec - exiting 21:20:22 (852): No heartbeat from core client for 30 sec - exiting 21:20:23 (852): No heartbeat from core client for 30 sec - exiting 21:20:25 (852): No heartbeat from core client for 30 sec - exiting 21:20:26 (852): No heartbeat from core client for 30 sec - exiting 21:20:28 (852): No heartbeat from core client for 30 sec - exiting 21:20:29 (852): No heartbeat from core client for 30 sec - exiting 21:20:31 (852): No heartbeat from core client for 30 sec - exiting 21:20:32 (852): No heartbeat from core client for 30 sec - exiting 21:20:33 (852): No heartbeat from core client for 30 sec - exiting 21:20:35 (852): No heartbeat from core client for 30 sec - exiting 21:20:36 (852): No heartbeat from core client for 30 sec - exiting 21:20:37 (852): No heartbeat from core client for 30 sec - exiting 21:20:38 (852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:24:20 (5692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:25:23 (5116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:25:55 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5912, iMonCtr=1 Model crash detected, will try to restart... 18:26:07 (2528): No heartbeat from core client for 30 sec - exiting 18:26:08 (2528): No heartbeat from core client for 30 sec - exiting 18:26:09 (2528): No heartbeat from core client for 30 sec - exiting 18:26:10 (2528): No heartbeat from core client for 30 sec - exiting 18:26:11 (2528): No heartbeat from core client for 30 sec - exiting 18:26:12 (2528): No heartbeat from core client for 30 sec - exiting 18:26:13 (2528): No heartbeat from core client for 30 sec - exiting 18:26:14 (2528): No heartbeat from core client for 30 sec - exiting 18:26:15 (2528): No heartbeat from core client for 30 sec - exiting 18:26:16 (2528): No heartbeat from core client for 30 sec - exiting 18:26:17 (2528): No heartbeat from core client for 30 sec - exiting 18:26:18 (2528): No heartbeat from core client for 30 sec - exiting 18:26:19 (2528): No heartbeat from core client for 30 sec - exiting 18:26:20 (2528): No heartbeat from core client for 30 sec - exiting 18:26:21 (2528): No heartbeat from core client for 30 sec - exiting 18:26:22 (2528): No heartbeat from core client for 30 sec - exiting 18:26:23 (2528): No heartbeat from core client for 30 sec - exiting 18:26:24 (2528): No heartbeat from core client for 30 sec - exiting 18:26:25 (2528): No heartbeat from core client for 30 sec - exiting 18:26:26 (2528): No heartbeat from core client for 30 sec - exiting 18:26:27 (2528): No heartbeat from core client for 30 sec - exiting 18:26:28 (2528): No heartbeat from core client for 30 sec - exiting 18:26:29 (2528): No heartbeat from core client for 30 sec - exiting 18:26:30 (2528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:26:31 (2528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 18:36:05 (3980): No heartbeat from core client for 30 sec - exiting 18:36:06 (3980): No heartbeat from core client for 30 sec - exiting 18:36:07 (3980): No heartbeat from core client for 30 sec - exiting 18:36:08 (3980): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3168, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77792AB1 read attempt to address 0x40B8B680 Engaging BOINC Windows Runtime Debugger... 06:18:18 (4264): No heartbeat from core client for 30 sec - exiting 06:18:19 (4264): No heartbeat from core client for 30 sec - exiting 06:18:20 (4264): No heartbeat from core client for 30 sec - exiting 06:18:21 (4264): No heartbeat from core client for 30 sec - exiting 06:18:22 (4264): No heartbeat from core client for 30 sec - exiting 06:18:23 (4264): No heartbeat from core client for 30 sec - exiting 06:18:24 (4264): No heartbeat from core client for 30 sec - exiting 06:18:25 (4264): No heartbeat from core client for 30 sec - exiting 06:18:26 (4264): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77792AB1 read attempt to address 0x40B8B680 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_o1xv_2020_40_007856806/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 May 2012 01:24:30	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	259,200	630,486	2.4324
15 May 2012 17:39:00	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	233,280	566,854	2.4299
13 May 2012 20:37:37	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	207,360	504,684	2.4339
13 May 2012 00:52:59	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	181,440	440,819	2.4296
10 May 2012 03:29:14	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	155,520	377,424	2.4269
23 Apr 2012 21:36:34	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	129,600	313,558	2.4194
22 Apr 2012 14:33:33	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	103,680	250,425	2.4154
21 Apr 2012 18:32:12	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	77,760	187,286	2.4085
20 Apr 2012 22:59:33	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	51,840	124,232	2.3965
19 Apr 2012 04:23:25	1135421	14358873	hadcm3n_o1xv_2020_40_007856806_0	25,920	62,269	2.4024