Task 16028432

Name	hadcm3n_n31s_1960_40_008394777_1
Workunit	8545636
Created	20 Sep 2013, 14:09:25 UTC
Sent	20 Sep 2013, 15:31:36 UTC
Report deadline	20 Dec 2013, 22:58:47 UTC
Received	1 Nov 2013, 8:38:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1276284
Run time	11 days 16 hours 54 min 1 sec
CPU time	9 days 6 hours 9 min 3 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... 15:50:07 (1768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:15:48 (1840): No heartbeat from core client for 30 sec - exiting 16:15:49 (1840): No heartbeat from core client for 30 sec - exiting 16:15:50 (1840): No heartbeat from core client for 30 sec - exiting 16:15:51 (1840): No heartbeat from core client for 30 sec - exiting 16:15:53 (1840): No heartbeat from core client for 30 sec - exiting 16:15:54 (1840): No heartbeat from core client for 30 sec - exiting 16:15:55 (1840): No heartbeat from core client for 30 sec - exiting 16:15:56 (1840): No heartbeat from core client for 30 sec - exiting 16:15:57 (1840): No heartbeat from core client for 30 sec - exiting 16:15:58 (1840): No heartbeat from core client for 30 sec - exiting 16:15:59 (1840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:30:27 (4332): No heartbeat from core client for 30 sec - exiting 07:30:28 (4332): No heartbeat from core client for 30 sec - exiting 07:30:29 (4332): No heartbeat from core client for 30 sec - exiting 07:30:30 (4332): No heartbeat from core client for 30 sec - exiting 07:30:31 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:30:32 (4332): No heartbeat from core client for 30 sec - exiting 08:29:25 (292): No heartbeat from core client for 30 sec - exiting 08:29:26 (292): No heartbeat from core client for 30 sec - exiting 08:29:27 (292): No heartbeat from core client for 30 sec - exiting 08:29:28 (292): No heartbeat from core client for 30 sec - exiting 08:29:29 (292): No heartbeat from core client for 30 sec - exiting 08:29:30 (292): No heartbeat from core client for 30 sec - exiting 08:29:32 (292): No heartbeat from core client for 30 sec - exiting 08:29:33 (292): No heartbeat from core client for 30 sec - exiting 08:29:34 (292): No heartbeat from core client for 30 sec - exiting 08:29:35 (292): No heartbeat from core client for 30 sec - exiting 08:29:36 (292): No heartbeat from core client for 30 sec - exiting 08:29:37 (292): No heartbeat from core client for 30 sec - exiting 08:29:38 (292): No heartbeat from core client for 30 sec - exiting 08:29:40 (292): No heartbeat from core client for 30 sec - exiting 08:29:41 (292): No heartbeat from core client for 30 sec - exiting 08:29:42 (292): No heartbeat from core client for 30 sec - exiting 08:29:43 (292): No heartbeat from core client for 30 sec - exiting 08:29:44 (292): No heartbeat from core client for 30 sec - exiting 08:29:45 (292): No heartbeat from core client for 30 sec - exiting 08:29:46 (292): No heartbeat from core client for 30 sec - exiting 08:29:47 (292): No heartbeat from core client for 30 sec - exiting 08:29:48 (292): No heartbeat from core client for 30 sec - exiting 08:29:49 (292): No heartbeat from core client for 30 sec - exiting 08:29:50 (292): No heartbeat from core client for 30 sec - exiting 08:29:52 (292): No heartbeat from core client for 30 sec - exiting 08:29:53 (292): No heartbeat from core client for 30 sec - exiting 08:29:54 (292): No heartbeat from core client for 30 sec - exiting 08:29:55 (292): No heartbeat from core client for 30 sec - exiting 08:29:56 (292): No heartbeat from core client for 30 sec - exiting 08:29:57 (292): No heartbeat from core client for 30 sec - exiting 08:29:58 (292): No heartbeat from core client for 30 sec - exiting 08:29:59 (292): No heartbeat from core client for 30 sec - exiting 08:30:00 (292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:43:52 (4724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:04:27 (3624): No heartbeat from core client for 30 sec - exiting 09:04:28 (3624): No heartbeat from core client for 30 sec - exiting 09:04:29 (3624): No heartbeat from core client for 30 sec - exiting 09:04:31 (3624): No heartbeat from core client for 30 sec - exiting 09:04:32 (3624): No heartbeat from core client for 30 sec - exiting 09:04:33 (3624): No heartbeat from core client for 30 sec - exiting 09:04:34 (3624): No heartbeat from core client for 30 sec - exiting 09:04:35 (3624): No heartbeat from core client for 30 sec - exiting 09:04:36 (3624): No heartbeat from core client for 30 sec - exiting 09:04:37 (3624): No heartbeat from core client for 30 sec - exiting 09:04:38 (3624): No heartbeat from core client for 30 sec - exiting 09:04:39 (3624): No heartbeat from core client for 30 sec - exiting 09:04:40 (3624): No heartbeat from core client for 30 sec - exiting 09:04:41 (3624): No heartbeat from core client for 30 sec - exiting 09:04:43 (3624): No heartbeat from core client for 30 sec - exiting 09:04:44 (3624): No heartbeat from core client for 30 sec - exiting 09:04:45 (3624): No heartbeat from core client for 30 sec - exiting 09:04:46 (3624): No heartbeat from core client for 30 sec - exiting 09:04:47 (3624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5348, iMonCtr=1 Model crash detected, will try to restart... 16:52:20 (4732): No heartbeat from core client for 30 sec - exiting 16:52:21 (4732): No heartbeat from core client for 30 sec - exiting 16:52:22 (4732): No heartbeat from core client for 30 sec - exiting 16:52:23 (4732): No heartbeat from core client for 30 sec - exiting 16:52:24 (4732): No heartbeat from core client for 30 sec - exiting 16:52:25 (4732): No heartbeat from core client for 30 sec - exiting 16:52:26 (4732): No heartbeat from core client for 30 sec - exiting 16:52:28 (4732): No heartbeat from core client for 30 sec - exiting 16:52:29 (4732): No heartbeat from core client for 30 sec - exiting 16:52:30 (4732): No heartbeat from core client for 30 sec - exiting 16:52:31 (4732): No heartbeat from core client for 30 sec - exiting 16:52:32 (4732): No heartbeat from core client for 30 sec - exiting 16:52:33 (4732): No heartbeat from core client for 30 sec - exiting 16:52:34 (4732): No heartbeat from core client for 30 sec - exiting 16:52:35 (4732): No heartbeat from core client for 30 sec - exiting 16:52:36 (4732): No heartbeat from core client for 30 sec - exiting 16:52:37 (4732): No heartbeat from core client for 30 sec - exiting 16:52:38 (4732): No heartbeat from core client for 30 sec - exiting 16:52:40 (4732): No heartbeat from core client for 30 sec - exiting 16:52:41 (4732): No heartbeat from core client for 30 sec - exiting 16:52:42 (4732): No heartbeat from core client for 30 sec - exiting 16:52:43 (4732): No heartbeat from core client for 30 sec - exiting 16:52:44 (4732): No heartbeat from core client for 30 sec - exiting 16:52:45 (4732): No heartbeat from core client for 30 sec - exiting 16:52:46 (4732): No heartbeat from core client for 30 sec - exiting 16:52:47 (4732): No heartbeat from core client for 30 sec - exiting 16:52:48 (4732): No heartbeat from core client for 30 sec - exiting 16:52:49 (4732): No heartbeat from core client for 30 sec - exiting 16:52:50 (4732): No heartbeat from core client for 30 sec - exiting 16:52:52 (4732): No heartbeat from core client for 30 sec - exiting 16:52:53 (4732): No heartbeat from core client for 30 sec - exiting 16:52:54 (4732): No heartbeat from core client for 30 sec - exiting 16:52:55 (4732): No heartbeat from core client for 30 sec - exiting 16:52:56 (4732): No heartbeat from core client for 30 sec - exiting 16:52:57 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:52:57 (5316): No heartbeat from core client for 30 sec - exiting 14:52:58 (5316): No heartbeat from core client for 30 sec - exiting 14:52:59 (5316): No heartbeat from core client for 30 sec - exiting 14:53:01 (5316): No heartbeat from core client for 30 sec - exiting 14:53:02 (5316): No heartbeat from core client for 30 sec - exiting 14:53:03 (5316): No heartbeat from core client for 30 sec - exiting 14:53:04 (5316): No heartbeat from core client for 30 sec - exiting 14:53:05 (5316): No heartbeat from core client for 30 sec - exiting 14:53:06 (5316): No heartbeat from core client for 30 sec - exiting 14:53:07 (5316): No heartbeat from core client for 30 sec - exiting 14:53:08 (5316): No heartbeat from core client for 30 sec - exiting 14:53:09 (5316): No heartbeat from core client for 30 sec - exiting 14:53:10 (5316): No heartbeat from core client for 30 sec - exiting 14:53:11 (5316): No heartbeat from core client for 30 sec - exiting 14:53:13 (5316): No heartbeat from core client for 30 sec - exiting 14:53:14 (5316): No heartbeat from core client for 30 sec - exiting 14:53:15 (5316): No heartbeat from core client for 30 sec - exiting 14:53:16 (5316): No heartbeat from core client for 30 sec - exiting 14:53:17 (5316): No heartbeat from core client for 30 sec - exiting 14:53:18 (5316): No heartbeat from core client for 30 sec - exiting 14:53:19 (5316): No heartbeat from core client for 30 sec - exiting 14:53:20 (5316): No heartbeat from core client for 30 sec - exiting 14:53:22 (5316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:06:17 (1920): No heartbeat from core client for 30 sec - exiting 09:06:18 (1920): No heartbeat from core client for 30 sec - exiting 09:06:19 (1920): No heartbeat from core client for 30 sec - exiting 09:06:20 (1920): No heartbeat from core client for 30 sec - exiting 09:06:21 (1920): No heartbeat from core client for 30 sec - exiting 09:06:22 (1920): No heartbeat from core client for 30 sec - exiting 09:06:23 (1920): No heartbeat from core client for 30 sec - exiting 09:06:25 (1920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:31:29 (4360): No heartbeat from core client for 30 sec - exiting 07:31:30 (4360): No heartbeat from core client for 30 sec - exiting 07:31:31 (4360): No heartbeat from core client for 30 sec - exiting 07:31:32 (4360): No heartbeat from core client for 30 sec - exiting 07:31:33 (4360): No heartbeat from core client for 30 sec - exiting 07:31:34 (4360): No heartbeat from core client for 30 sec - exiting 07:31:36 (4360): No heartbeat from core client for 30 sec - exiting 07:31:37 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:37:14 (5552): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5768, iMonCtr=1 Model crash detected, will try to restart... 10:29:29 (3704): No heartbeat from core client for 30 sec - exiting 10:29:30 (3704): No heartbeat from core client for 30 sec - exiting 10:29:31 (3704): No heartbeat from core client for 30 sec - exiting 10:29:32 (3704): No heartbeat from core client for 30 sec - exiting 10:29:34 (3704): No heartbeat from core client for 30 sec - exiting 10:29:35 (3704): No heartbeat from core client for 30 sec - exiting 10:29:36 (3704): No heartbeat from core client for 30 sec - exiting 10:29:37 (3704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:29:38 (3704): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4608, iMonCtr=1 Model crash detected, will try to restart... 18:11:25 (3792): No heartbeat from core client for 30 sec - exiting 18:11:26 (3792): No heartbeat from core client for 30 sec - exiting 18:11:27 (3792): No heartbeat from core client for 30 sec - exiting 18:11:28 (3792): No heartbeat from core client for 30 sec - exiting 18:11:29 (3792): No heartbeat from core client for 30 sec - exiting 18:11:31 (3792): No heartbeat from core client for 30 sec - exiting 18:11:32 (3792): No heartbeat from core client for 30 sec - exiting 18:11:33 (3792): No heartbeat from core client for 30 sec - exiting 18:11:34 (3792): No heartbeat from core client for 30 sec - exiting 18:11:35 (3792): No heartbeat from core client for 30 sec - exiting 18:11:36 (3792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5592, iMonCtr=1 Model crash detected, will try to restart... 14:45:27 (2304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:39:51 (4500): No heartbeat from core client for 30 sec - exiting 19:39:53 (4500): No heartbeat from core client for 30 sec - exiting 19:39:54 (4500): No heartbeat from core client for 30 sec - exiting 19:39:55 (4500): No heartbeat from core client for 30 sec - exiting 19:39:56 (4500): No heartbeat from core client for 30 sec - exiting 19:39:58 (4500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 15:16:54 (4448): No heartbeat from core client for 30 sec - exiting 15:16:55 (4448): No heartbeat from core client for 30 sec - exiting 15:16:56 (4448): No heartbeat from core client for 30 sec - exiting 15:16:57 (4448): No heartbeat from core client for 30 sec - exiting 15:16:58 (4448): No heartbeat from core client for 30 sec - exiting 15:16:59 (4448): No heartbeat from core client for 30 sec - exiting 15:17:00 (4448): No heartbeat from core client for 30 sec - exiting 15:17:01 (4448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 12:16:15 (5904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4996, iMonCtr=1 Model crash detected, will try to restart... 15:53:03 (4920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:59:32 (5028): No heartbeat from core client for 30 sec - exiting 16:59:33 (5028): No heartbeat from core client for 30 sec - exiting 16:59:35 (5028): No heartbeat from core client for 30 sec - exiting 16:59:36 (5028): No heartbeat from core client for 30 sec - exiting 16:59:37 (5028): No heartbeat from core client for 30 sec - exiting 16:59:38 (5028): No heartbeat from core client for 30 sec - exiting 16:59:39 (5028): No heartbeat from core client for 30 sec - exiting 16:59:40 (5028): No heartbeat from core client for 30 sec - exiting 16:59:41 (5028): No heartbeat from core client for 30 sec - exiting 16:59:42 (5028): No heartbeat from core client for 30 sec - exiting 16:59:43 (5028): No heartbeat from core client for 30 sec - exiting 16:59:44 (5028): No heartbeat from core client for 30 sec - exiting 16:59:45 (5028): No heartbeat from core client for 30 sec - exiting 16:59:47 (5028): No heartbeat from core client for 30 sec - exiting 16:59:48 (5028): No heartbeat from core client for 30 sec - exiting 16:59:49 (5028): No heartbeat from core client for 30 sec - exiting 16:59:50 (5028): No heartbeat from core client for 30 sec - exiting 16:59:51 (5028): No heartbeat from core client for 30 sec - exiting 16:59:52 (5028): No heartbeat from core client for 30 sec - exiting 16:59:53 (5028): No heartbeat from core client for 30 sec - exiting 16:59:54 (5028): No heartbeat from core client for 30 sec - exiting 16:59:55 (5028): No heartbeat from core client for 30 sec - exiting 16:59:57 (5028): No heartbeat from core client for 30 sec - exiting 16:59:58 (5028): No heartbeat from core client for 30 sec - exiting 16:59:59 (5028): No heartbeat from core client for 30 sec - exiting 17:00:00 (5028): No heartbeat from core client for 30 sec - exiting 17:00:01 (5028): No heartbeat from core client for 30 sec - exiting 17:00:02 (5028): No heartbeat from core client for 30 sec - exiting 17:00:03 (5028): No heartbeat from core client for 30 sec - exiting 17:00:05 (5028): No heartbeat from core client for 30 sec - exiting 17:00:06 (5028): No heartbeat from core client for 30 sec - exiting 17:00:07 (5028): No heartbeat from core client for 30 sec - exiting 17:00:08 (5028): No heartbeat from core client for 30 sec - exiting 17:00:09 (5028): No heartbeat from core client for 30 sec - exiting 17:00:10 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:11 (5028): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 17:01:07 (1220): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 19:04:09 (4784): No heartbeat from core client for 30 sec - exiting 19:04:10 (4784): No heartbeat from core client for 30 sec - exiting 19:04:11 (4784): No heartbeat from core client for 30 sec - exiting 19:04:12 (4784): No heartbeat from core client for 30 sec - exiting 19:04:13 (4784): No heartbeat from core client for 30 sec - exiting 19:04:15 (4784): No heartbeat from core client for 30 sec - exiting 19:04:16 (4784): No heartbeat from core client for 30 sec - exiting 19:04:17 (4784): No heartbeat from core client for 30 sec - exiting 19:04:18 (4784): No heartbeat from core client for 30 sec - exiting 19:04:19 (4784): No heartbeat from core client for 30 sec - exiting 19:04:20 (4784): No heartbeat from core client for 30 sec - exiting 19:04:21 (4784): No heartbeat from core client for 30 sec - exiting 19:04:22 (4784): No heartbeat from core client for 30 sec - exiting 19:04:23 (4784): No heartbeat from core client for 30 sec - exiting 19:04:24 (4784): No heartbeat from core client for 30 sec - exiting 19:04:26 (4784): No heartbeat from core client for 30 sec - exiting 19:04:27 (4784): No heartbeat from core client for 30 sec - exiting 19:04:28 (4784): No heartbeat from core client for 30 sec - exiting 19:04:29 (4784): No heartbeat from core client for 30 sec - exiting 19:04:30 (4784): No heartbeat from core client for 30 sec - exiting 19:04:31 (4784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5696, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:49:49 (4560): No heartbeat from core client for 30 sec - exiting 18:49:50 (4560): No heartbeat from core client for 30 sec - exiting 18:49:51 (4560): No heartbeat from core client for 30 sec - exiting 18:49:52 (4560): No heartbeat from core client for 30 sec - exiting 18:49:53 (4560): No heartbeat from core client for 30 sec - exiting 18:49:54 (4560): No heartbeat from core client for 30 sec - exiting 18:49:55 (4560): No heartbeat from core client for 30 sec - exiting 18:49:56 (4560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:49:57 (4560): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 18:52:06 (4320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:00:22 (5148): No heartbeat from core client for 30 sec - exiting 19:00:23 (5148): No heartbeat from core client for 30 sec - exiting 19:00:24 (5148): No heartbeat from core client for 30 sec - exiting 19:00:25 (5148): No heartbeat from core client for 30 sec - exiting 19:00:27 (5148): No heartbeat from core client for 30 sec - exiting 19:00:28 (5148): No heartbeat from core client for 30 sec - exiting 19:00:29 (5148): No heartbeat from core client for 30 sec - exiting 19:00:30 (5148): No heartbeat from core client for 30 sec - exiting 19:00:31 (5148): No heartbeat from core client for 30 sec - exiting 19:00:32 (5148): No heartbeat from core client for 30 sec - exiting 19:00:33 (5148): No heartbeat from core client for 30 sec - exiting 19:00:34 (5148): No heartbeat from core client for 30 sec - exiting 19:00:35 (5148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:12:10 (4944): No heartbeat from core client for 30 sec - exiting 17:12:12 (4944): No heartbeat from core client for 30 sec - exiting 17:12:13 (4944): No heartbeat from core client for 30 sec - exiting 17:12:14 (4944): No heartbeat from core client for 30 sec - exiting 17:12:15 (4944): No heartbeat from core client for 30 sec - exiting 17:12:16 (4944): No heartbeat from core client for 30 sec - exiting 17:12:17 (4944): No heartbeat from core client for 30 sec - exiting 17:12:18 (4944): No heartbeat from core client for 30 sec - exiting 17:12:19 (4944): No heartbeat from core client for 30 sec - exiting 17:12:20 (4944): No heartbeat from core client for 30 sec - exiting 17:12:21 (4944): No heartbeat from core client for 30 sec - exiting 17:12:22 (4944): No heartbeat from core client for 30 sec - exiting 17:12:23 (4944): No heartbeat from core client for 30 sec - exiting 17:12:24 (4944): No heartbeat from core client for 30 sec - exiting 17:12:26 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:01:26 (4532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5776, iMonCtr=1 Model crash detected, will try to restart... 12:22:02 (3532): No heartbeat from core client for 30 sec - exiting 12:22:03 (3532): No heartbeat from core client for 30 sec - exiting 12:22:04 (3532): No heartbeat from core client for 30 sec - exiting 12:22:05 (3532): No heartbeat from core client for 30 sec - exiting 12:22:07 (3532): No heartbeat from core client for 30 sec - exiting 12:22:08 (3532): No heartbeat from core client for 30 sec - exiting 12:22:09 (3532): No heartbeat from core client for 30 sec - exiting 12:22:10 (3532): No heartbeat from core client for 30 sec - exiting 12:22:11 (3532): No heartbeat from core client for 30 sec - exiting 12:22:12 (3532): No heartbeat from core client for 30 sec - exiting 12:22:13 (3532): No heartbeat from core client for 30 sec - exiting 12:22:14 (3532): No heartbeat from core client for 30 sec - exiting 12:22:15 (3532): No heartbeat from core client for 30 sec - exiting 12:22:16 (3532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:21:35 (2008): No heartbeat from core client for 30 sec - exiting 19:21:36 (2008): No heartbeat from core client for 30 sec - exiting 19:21:37 (2008): No heartbeat from core client for 30 sec - exiting 19:21:38 (2008): No heartbeat from core client for 30 sec - exiting 19:21:39 (2008): No heartbeat from core client for 30 sec - exiting 19:21:40 (2008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4856, iMonCtr=1 Model crash detected, will try to restart... 18:03:49 (876): No heartbeat from core client for 30 sec - exiting 18:03:51 (876): No heartbeat from core client for 30 sec - exiting 18:03:52 (876): No heartbeat from core client for 30 sec - exiting 18:03:53 (876): No heartbeat from core client for 30 sec - exiting 18:03:54 (876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:34:29 (5864): No heartbeat from core client for 30 sec - exiting 20:34:31 (5864): No heartbeat from core client for 30 sec - exiting 20:34:32 (5864): No heartbeat from core client for 30 sec - exiting 20:34:33 (5864): No heartbeat from core client for 30 sec - exiting 20:34:34 (5864): No heartbeat from core client for 30 sec - exiting 20:34:35 (5864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77BF59C7 read attempt to address 0x40BC367B Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77786A0D read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Cannot serialize file C:\ProgramData\BOINC/projects/climateprediction.net/hadcm3n_n31s_1960_40_008394777/dataout/shmem_restart.day Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Oct 2013 09:43:42	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	259,200	765,290	2.9525
24 Oct 2013 19:31:10	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	233,280	693,356	2.9722
22 Oct 2013 13:55:49	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	207,360	621,155	2.9955
19 Oct 2013 12:31:43	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	181,440	546,839	3.0139
14 Oct 2013 17:31:45	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	155,520	474,845	3.0533
10 Oct 2013 07:45:31	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	129,600	403,846	3.1161
03 Oct 2013 15:39:42	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	103,680	328,404	3.1675
29 Sep 2013 16:52:05	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	77,760	244,270	3.1413
27 Sep 2013 15:36:28	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	51,840	162,308	3.1309
23 Sep 2013 16:40:38	1276284	16028432	hadcm3n_n31s_1960_40_008394777_1	25,920	82,339	3.1767