Task 15764301

Name	hadcm3n_3ffp_1940_40_008258580_4
Workunit	8413704
Created	6 May 2013, 17:51:43 UTC
Sent	6 May 2013, 17:51:53 UTC
Report deadline	6 Aug 2013, 1:19:04 UTC
Received	15 Jun 2013, 12:06:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-226 (0xFFFFFF1E) ERR_TOO_MANY_EXITS
Computer ID	1226353
Run time	30 days 8 hours 28 min 23 sec
CPU time	15 days 19 hours 2 min 46 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.47 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> too many exit(0)s </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:27:48 (5016): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:42:05 (3776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:14:14 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:14:16 (4384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:43:32 (2352): No heartbeat from core client for 30 sec - exiting 04:43:33 (2352): No heartbeat from core client for 30 sec - exiting 04:43:34 (2352): No heartbeat from core client for 30 sec - exiting 04:43:36 (2352): No heartbeat from core client for 30 sec - exiting 04:43:37 (2352): No heartbeat from core client for 30 sec - exiting 04:43:38 (2352): No heartbeat from core client for 30 sec - exiting 04:43:40 (2352): No heartbeat from core client for 30 sec - exiting 04:43:41 (2352): No heartbeat from core client for 30 sec - exiting 04:43:42 (2352): No heartbeat from core client for 30 sec - exiting 04:43:43 (2352): No heartbeat from core client for 30 sec - exiting 04:43:44 (2352): No heartbeat from core client for 30 sec - exiting 04:43:45 (2352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:20 (4740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:48:21 (4740): No heartbeat from core client for 30 sec - exiting 04:49:44 (5552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:07:01 (6396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:07:44 (3340): No heartbeat from core client for 30 sec - exiting 05:07:45 (3340): No heartbeat from core client for 30 sec - exiting 05:07:46 (3340): No heartbeat from core client for 30 sec - exiting 05:07:47 (3340): No heartbeat from core client for 30 sec - exiting 05:07:48 (3340): No heartbeat from core client for 30 sec - exiting 05:07:49 (3340): No heartbeat from core client for 30 sec - exiting 05:07:50 (3340): No heartbeat from core client for 30 sec - exiting 05:07:51 (3340): No heartbeat from core client for 30 sec - exiting 05:07:52 (3340): No heartbeat from core client for 30 sec - exiting 05:07:53 (3340): No heartbeat from core client for 30 sec - exiting 05:07:54 (3340): No heartbeat from core client for 30 sec - exiting 05:07:55 (3340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:13:07 (1132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:14:20 (1420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:27:16 (10028): No heartbeat from core client for 30 sec - exiting 05:27:17 (10028): No heartbeat from core client for 30 sec - exiting 05:27:18 (10028): No heartbeat from core client for 30 sec - exiting 05:27:19 (10028): No heartbeat from core client for 30 sec - exiting 05:27:20 (10028): No heartbeat from core client for 30 sec - exiting 05:27:21 (10028): No heartbeat from core client for 30 sec - exiting 05:27:22 (10028): No heartbeat from core client for 30 sec - exiting 05:27:23 (10028): No heartbeat from core client for 30 sec - exiting 05:27:24 (10028): No heartbeat from core client for 30 sec - exiting 05:27:25 (10028): No heartbeat from core client for 30 sec - exiting 05:27:26 (10028): No heartbeat from core client for 30 sec - exiting 05:27:27 (10028): No heartbeat from core client for 30 sec - exiting 05:27:28 (10028): No heartbeat from core client for 30 sec - exiting 05:27:30 (10028): No heartbeat from core client for 30 sec - exiting 05:27:31 (10028): No heartbeat from core client for 30 sec - exiting 05:27:32 (10028): No heartbeat from core client for 30 sec - exiting 05:27:33 (10028): No heartbeat from core client for 30 sec - exiting 05:27:34 (10028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:27:35 (10028): No heartbeat from core client for 30 sec - exiting 05:27:36 (10028): No heartbeat from core client for 30 sec - exiting 05:27:37 (10028): No heartbeat from core client for 30 sec - exiting 05:29:30 (8204): No heartbeat from core client for 30 sec - exiting 05:29:32 (8204): No heartbeat from core client for 30 sec - exiting 05:29:33 (8204): No heartbeat from core client for 30 sec - exiting 05:29:34 (8204): No heartbeat from core client for 30 sec - exiting 05:29:35 (8204): No heartbeat from core client for 30 sec - exiting 05:29:36 (8204): No heartbeat from core client for 30 sec - exiting 05:29:37 (8204): No heartbeat from core client for 30 sec - exiting 05:29:38 (8204): No heartbeat from core client for 30 sec - exiting 05:29:39 (8204): No heartbeat from core client for 30 sec - exiting 05:29:40 (8204): No heartbeat from core client for 30 sec - exiting 05:29:41 (8204): No heartbeat from core client for 30 sec - exiting 05:29:42 (8204): No heartbeat from core client for 30 sec - exiting 05:29:43 (8204): No heartbeat from core client for 30 sec - exiting 05:29:44 (8204): No heartbeat from core client for 30 sec - exiting 05:29:46 (8204): No heartbeat from core client for 30 sec - exiting 05:29:47 (8204): No heartbeat from core client for 30 sec - exiting 05:29:48 (8204): No heartbeat from core client for 30 sec - exiting 05:29:49 (8204): No heartbeat from core client for 30 sec - exiting 05:29:50 (8204): No heartbeat from core client for 30 sec - exiting 05:29:51 (8204): No heartbeat from core client for 30 sec - exiting 05:29:52 (8204): No heartbeat from core client for 30 sec - exiting 05:29:53 (8204): No heartbeat from core client for 30 sec - exiting 05:29:54 (8204): No heartbeat from core client for 30 sec - exiting 05:29:55 (8204): No heartbeat from core client for 30 sec - exiting 05:29:56 (8204): No heartbeat from core client for 30 sec - exiting 05:29:57 (8204): No heartbeat from core client for 30 sec - exiting 05:29:58 (8204): No heartbeat from core client for 30 sec - exiting 05:29:59 (8204): No heartbeat from core client for 30 sec - exiting 05:30:00 (8204): No heartbeat from core client for 30 sec - exiting 05:30:01 (8204): No heartbeat from core client for 30 sec - exiting 05:30:02 (8204): No heartbeat from core client for 30 sec - exiting 05:30:04 (8204): No heartbeat from core client for 30 sec - exiting 05:30:05 (8204): No heartbeat from core client for 30 sec - exiting 05:30:06 (8204): No heartbeat from core client for 30 sec - exiting 05:30:07 (8204): No heartbeat from core client for 30 sec - exiting 05:30:08 (8204): No heartbeat from core client for 30 sec - exiting 05:30:09 (8204): No heartbeat from core client for 30 sec - exiting 05:30:10 (8204): No heartbeat from core client for 30 sec - exiting 05:30:11 (8204): No heartbeat from core client for 30 sec - exiting 05:30:12 (8204): No heartbeat from core client for 30 sec - exiting 05:30:13 (8204): No heartbeat from core client for 30 sec - exiting 05:30:14 (8204): No heartbeat from core client for 30 sec - exiting 05:30:15 (8204): No heartbeat from core client for 30 sec - exiting 05:30:16 (8204): No heartbeat from core client for 30 sec - exiting 05:30:17 (8204): No heartbeat from core client for 30 sec - exiting 05:30:18 (8204): No heartbeat from core client for 30 sec - exiting 05:30:19 (8204): No heartbeat from core client for 30 sec - exiting 05:30:20 (8204): No heartbeat from core client for 30 sec - exiting 05:30:22 (8204): No heartbeat from core client for 30 sec - exiting 05:30:23 (8204): No heartbeat from core client for 30 sec - exiting 05:30:25 (8204): No heartbeat from core client for 30 sec - exiting 05:30:26 (8204): No heartbeat from core client for 30 sec - exiting 05:30:28 (8204): No heartbeat from core client for 30 sec - exiting 05:30:29 (8204): No heartbeat from core client for 30 sec - exiting 05:30:30 (8204): No heartbeat from core client for 30 sec - exiting 05:30:31 (8204): No heartbeat from core client for 30 sec - exiting 05:30:32 (8204): No heartbeat from core client for 30 sec - exiting 05:30:33 (8204): No heartbeat from core client for 30 sec - exiting 05:30:34 (8204): No heartbeat from core client for 30 sec - exiting 05:30:35 (8204): No heartbeat from core client for 30 sec - exiting 05:30:36 (8204): No heartbeat from core client for 30 sec - exiting 05:30:37 (8204): No heartbeat from core client for 30 sec - exiting 05:30:38 (8204): No heartbeat from core client for 30 sec - exiting 05:30:40 (8204): No heartbeat from core client for 30 sec - exiting 05:30:41 (8204): No heartbeat from core client for 30 sec - exiting 05:30:42 (8204): No heartbeat from core client for 30 sec - exiting 05:30:43 (8204): No heartbeat from core client for 30 sec - exiting 05:30:44 (8204): No heartbeat from core client for 30 sec - exiting 05:30:45 (8204): No heartbeat from core client for 30 sec - exiting 05:30:46 (8204): No heartbeat from core client for 30 sec - exiting 05:30:48 (8204): No heartbeat from core client for 30 sec - exiting 05:30:49 (8204): No heartbeat from core client for 30 sec - exiting 05:30:50 (8204): No heartbeat from core client for 30 sec - exiting 05:30:51 (8204): No heartbeat from core client for 30 sec - exiting 05:30:52 (8204): No heartbeat from core client for 30 sec - exiting 05:30:53 (8204): No heartbeat from core client for 30 sec - exiting 05:30:54 (8204): No heartbeat from core client for 30 sec - exiting 05:30:55 (8204): No heartbeat from core client for 30 sec - exiting 05:30:56 (8204): No heartbeat from core client for 30 sec - exiting 05:30:57 (8204): No heartbeat from core client for 30 sec - exiting 05:30:58 (8204): No heartbeat from core client for 30 sec - exiting 05:30:59 (8204): No heartbeat from core client for 30 sec - exiting 05:31:00 (8204): No heartbeat from core client for 30 sec - exiting 05:31:01 (8204): No heartbeat from core client for 30 sec - exiting 05:31:02 (8204): No heartbeat from core client for 30 sec - exiting 05:31:03 (8204): No heartbeat from core client for 30 sec - exiting 05:31:04 (8204): No heartbeat from core client for 30 sec - exiting 05:31:05 (8204): No heartbeat from core client for 30 sec - exiting 05:31:06 (8204): No heartbeat from core client for 30 sec - exiting 05:31:07 (8204): No heartbeat from core client for 30 sec - exiting 05:31:08 (8204): No heartbeat from core client for 30 sec - exiting 05:31:09 (8204): No heartbeat from core client for 30 sec - exiting 05:31:10 (8204): No heartbeat from core client for 30 sec - exiting 05:31:12 (8204): No heartbeat from core client for 30 sec - exiting 05:31:13 (8204): No heartbeat from core client for 30 sec - exiting 05:31:14 (8204): No heartbeat from core client for 30 sec - exiting 05:31:15 (8204): No heartbeat from core client for 30 sec - exiting 05:31:16 (8204): No heartbeat from core client for 30 sec - exiting 05:31:17 (8204): No heartbeat from core client for 30 sec - exiting 05:31:18 (8204): No heartbeat from core client for 30 sec - exiting 05:31:19 (8204): No heartbeat from core client for 30 sec - exiting 05:31:21 (8204): No heartbeat from core client for 30 sec - exiting 05:31:22 (8204): No heartbeat from core client for 30 sec - exiting 05:31:23 (8204): No heartbeat from core client for 30 sec - exiting 05:31:24 (8204): No heartbeat from core client for 30 sec - exiting 05:31:25 (8204): No heartbeat from core client for 30 sec - exiting 05:31:26 (8204): No heartbeat from core client for 30 sec - exiting 05:31:27 (8204): No heartbeat from core client for 30 sec - exiting 05:31:28 (8204): No heartbeat from core client for 30 sec - exiting 05:31:30 (8204): No heartbeat from core client for 30 sec - exiting 05:31:31 (8204): No heartbeat from core client for 30 sec - exiting 05:31:32 (8204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:25 (10196): No heartbeat from core client for 30 sec - exiting 05:32:26 (10196): No heartbeat from core client for 30 sec - exiting 05:32:28 (10196): No heartbeat from core client for 30 sec - exiting 05:32:29 (10196): No heartbeat from core client for 30 sec - exiting 05:32:30 (10196): No heartbeat from core client for 30 sec - exiting 05:32:31 (10196): No heartbeat from core client for 30 sec - exiting 05:32:32 (10196): No heartbeat from core client for 30 sec - exiting 05:32:33 (10196): No heartbeat from core client for 30 sec - exiting 05:32:36 (10196): No heartbeat from core client for 30 sec - exiting 05:32:37 (10196): No heartbeat from core client for 30 sec - exiting 05:32:38 (10196): No heartbeat from core client for 30 sec - exiting 05:32:39 (10196): No heartbeat from core client for 30 sec - exiting 05:32:40 (10196): No heartbeat from core client for 30 sec - exiting 05:32:41 (10196): No heartbeat from core client for 30 sec - exiting 05:32:42 (10196): No heartbeat from core client for 30 sec - exiting 05:32:43 (10196): No heartbeat from core client for 30 sec - exiting 05:32:44 (10196): No heartbeat from core client for 30 sec - exiting 05:32:45 (10196): No heartbeat from core client for 30 sec - exiting 05:32:46 (10196): No heartbeat from core client for 30 sec - exiting 05:32:47 (10196): No heartbeat from core client for 30 sec - exiting 05:32:49 (10196): No heartbeat from core client for 30 sec - exiting 05:32:50 (10196): No heartbeat from core client for 30 sec - exiting 05:32:51 (10196): No heartbeat from core client for 30 sec - exiting 05:32:52 (10196): No heartbeat from core client for 30 sec - exiting 05:32:54 (10196): No heartbeat from core client for 30 sec - exiting 05:32:55 (10196): No heartbeat from core client for 30 sec - exiting 05:32:56 (10196): No heartbeat from core client for 30 sec - exiting 05:32:57 (10196): No heartbeat from core client for 30 sec - exiting 05:32:58 (10196): No heartbeat from core client for 30 sec - exiting 05:33:00 (10196): No heartbeat from core client for 30 sec - exiting 05:33:01 (10196): No heartbeat from core client for 30 sec - exiting 05:33:03 (10196): No heartbeat from core client for 30 sec - exiting 05:33:04 (10196): No heartbeat from core client for 30 sec - exiting 05:33:05 (10196): No heartbeat from core client for 30 sec - exiting 05:33:06 (10196): No heartbeat from core client for 30 sec - exiting 05:33:07 (10196): No heartbeat from core client for 30 sec - exiting 05:33:08 (10196): No heartbeat from core client for 30 sec - exiting 05:33:17 (10196): No heartbeat from core client for 30 sec - exiting 05:33:18 (10196): No heartbeat from core client for 30 sec - exiting 05:33:19 (10196): No heartbeat from core client for 30 sec - exiting 05:33:20 (10196): No heartbeat from core client for 30 sec - exiting 05:33:21 (10196): No heartbeat from core client for 30 sec - exiting 05:33:22 (10196): No heartbeat from core client for 30 sec - exiting 05:33:23 (10196): No heartbeat from core client for 30 sec - exiting 05:33:24 (10196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:37:06 (9156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:37:08 (9156): No heartbeat from core client for 30 sec - exiting 05:49:38 (6872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:03:16 (9660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:04:57 (2276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:18:34 (9144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5352, iMonCtr=1 Model crash detected, will try to restart... 06:31:36 (8176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:34:07 (3060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7452, iMonCtr=1 Model crash detected, will try to restart... 15:32:53 (7624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:59:11 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... 05:18:08 (2200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:18:09 (2200): No heartbeat from core client for 30 sec - exiting 05:18:10 (2200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:57:27 (6032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:28:55 (9164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5512, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2144, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:37:40 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:37:41 (2784): No heartbeat from core client for 30 sec - exiting 19:39:40 (1244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:39 (9396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:52:51 (9396): No heartbeat from core client for 30 sec - exiting 19:52:52 (9396): No heartbeat from core client for 30 sec - exiting 19:52:54 (9396): No heartbeat from core client for 30 sec - exiting 19:55:49 (7228): No heartbeat from core client for 30 sec - exiting 19:55:50 (7228): No heartbeat from core client for 30 sec - exiting 19:55:51 (7228): No heartbeat from core client for 30 sec - exiting 19:55:52 (7228): No heartbeat from core client for 30 sec - exiting 19:55:53 (7228): No heartbeat from core client for 30 sec - exiting 19:55:54 (7228): No heartbeat from core client for 30 sec - exiting 19:55:55 (7228): No heartbeat from core client for 30 sec - exiting 19:55:56 (7228): No heartbeat from core client for 30 sec - exiting 19:55:57 (7228): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:55:58 (7228): No heartbeat from core client for 30 sec - exiting 19:55:59 (7228): No heartbeat from core client for 30 sec - exiting 20:12:37 (4944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:12:38 (4944): No heartbeat from core client for 30 sec - exiting 20:12:39 (4944): No heartbeat from core client for 30 sec - exiting 20:12:40 (4944): No heartbeat from core client for 30 sec - exiting 20:12:41 (4944): No heartbeat from core client for 30 sec - exiting 20:31:38 (2032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:31:51 (2032): No heartbeat from core client for 30 sec - exiting 20:31:54 (2032): No heartbeat from core client for 30 sec - exiting 20:31:55 (2032): No heartbeat from core client for 30 sec - exiting 20:31:56 (2032): No heartbeat from core client for 30 sec - exiting 20:31:57 (2032): No heartbeat from core client for 30 sec - exiting 20:31:58 (2032): No heartbeat from core client for 30 sec - exiting 20:31:59 (2032): No heartbeat from core client for 30 sec - exiting 20:32:00 (2032): No heartbeat from core client for 30 sec - exiting 20:32:01 (2032): No heartbeat from core client for 30 sec - exiting 20:32:02 (2032): No heartbeat from core client for 30 sec - exiting 20:32:03 (2032): No heartbeat from core client for 30 sec - exiting 20:32:04 (2032): No heartbeat from core client for 30 sec - exiting 20:32:05 (2032): No heartbeat from core client for 30 sec - exiting 20:32:06 (2032): No heartbeat from core client for 30 sec - exiting 20:32:07 (2032): No heartbeat from core client for 30 sec - exiting 20:32:08 (2032): No heartbeat from core client for 30 sec - exiting 20:32:09 (2032): No heartbeat from core client for 30 sec - exiting 20:32:10 (2032): No heartbeat from core client for 30 sec - exiting 20:32:11 (2032): No heartbeat from core client for 30 sec - exiting 20:32:12 (2032): No heartbeat from core client for 30 sec - exiting 20:32:13 (2032): No heartbeat from core client for 30 sec - exiting 20:32:14 (2032): No heartbeat from core client for 30 sec - exiting 20:32:15 (2032): No heartbeat from core client for 30 sec - exiting 20:32:16 (2032): No heartbeat from core client for 30 sec - exiting 20:32:17 (2032): No heartbeat from core client for 30 sec - exiting 01:50:51 (7112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:50:54 (7112): No heartbeat from core client for 30 sec - exiting 01:50:55 (7112): No heartbeat from core client for 30 sec - exiting 01:50:56 (7112): No heartbeat from core client for 30 sec - exiting 01:50:57 (7112): No heartbeat from core client for 30 sec - exiting 01:50:58 (7112): No heartbeat from core client for 30 sec - exiting 01:50:59 (7112): No heartbeat from core client for 30 sec - exiting 01:55:45 (4508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:04:35 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:04:36 (2720): No heartbeat from core client for 30 sec - exiting 05:04:37 (2720): No heartbeat from core client for 30 sec - exiting 05:04:38 (2720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7556, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:15:58 (7244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:44:41 (9148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:44:42 (9148): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3208, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:05:29 (10660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:05:30 (10660): No heartbeat from core client for 30 sec - exiting 21:05:31 (10660): No heartbeat from core client for 30 sec - exiting 21:56:11 (8036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:56:12 (8036): No heartbeat from core client for 30 sec - exiting 21:56:13 (8036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 23:27:56 (9336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:23 (7576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:13:29 (7576): No heartbeat from core client for 30 sec - exiting 01:13:30 (7576): No heartbeat from core client for 30 sec - exiting 02:54:56 (11284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:16 (9168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:22:17 (9168): No heartbeat from core client for 30 sec - exiting 05:53:49 (7496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:53:50 (7496): No heartbeat from core client for 30 sec - exiting 05:56:26 (6776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6056, iMonCtr=1 Model crash detected, will try to restart... 03:38:28 (8836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:38 (9288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3592, iMonCtr=1 Model crash detected, will try to restart... 06:04:25 (3568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:04:27 (3568): No heartbeat from core client for 30 sec - exiting 06:04:28 (3568): No heartbeat from core client for 30 sec - exiting 11:13:08 (6900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:13:09 (6900): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 00:12:42 (7884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:12:44 (7884): No heartbeat from core client for 30 sec - exiting 02:50:46 (6236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:54:02 (7724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Jun 2013 06:48:04	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	518,400	1,310,467	2.5279
11 Jun 2013 11:34:28	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	492,480	1,244,497	2.5270
10 Jun 2013 06:16:50	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	466,560	1,181,923	2.5333
08 Jun 2013 16:42:23	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	440,640	1,119,151	2.5398
06 Jun 2013 22:55:54	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	414,720	1,052,131	2.5370
05 Jun 2013 09:20:39	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	388,800	985,863	2.5357
03 Jun 2013 00:26:50	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	362,880	920,741	2.5373
31 May 2013 22:19:40	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	336,960	855,428	2.5387
30 May 2013 04:57:43	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	311,040	789,891	2.5395
28 May 2013 01:15:54	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	285,120	723,237	2.5366
24 May 2013 12:13:59	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	259,200	663,109	2.5583
22 May 2013 04:51:55	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	233,280	600,530	2.5743
20 May 2013 21:38:25	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	207,360	537,052	2.5899
17 May 2013 02:32:53	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	181,440	475,109	2.6185
15 May 2013 17:18:51	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	155,520	409,852	2.6354
14 May 2013 05:37:34	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	129,600	344,759	2.6602
12 May 2013 07:45:41	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	103,680	280,577	2.7062
11 May 2013 06:37:20	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	77,760	216,530	2.7846
10 May 2013 04:45:02	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	51,840	152,127	2.9345
08 May 2013 15:35:36	1226353	15764301	hadcm3n_3ffp_1940_40_008258580_4	25,920	78,780	3.0394