Task 15813929

Name	hadcm3n_o0cg_1940_40_008382431_0
Workunit	8533290
Created	1 Jun 2013, 4:49:45 UTC
Sent	2 Jun 2013, 17:46:50 UTC
Report deadline	2 Sep 2013, 1:14:01 UTC
Received	8 Jul 2013, 21:31:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1072815
Run time	10 days 16 hours 45 min 18 sec
CPU time	10 days 7 hours 36 min 57 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6884, iMonCtr=1 Model crash detected, will try to restart... 21:06:23 (5232): No heartbeat from core client for 30 sec - exiting 21:06:24 (5232): No heartbeat from core client for 30 sec - exiting 21:06:25 (5232): No heartbeat from core client for 30 sec - exiting 21:06:26 (5232): No heartbeat from core client for 30 sec - exiting 21:06:27 (5232): No heartbeat from core client for 30 sec - exiting 21:06:28 (5232): No heartbeat from core client for 30 sec - exiting 21:06:29 (5232): No heartbeat from core client for 30 sec - exiting 21:06:30 (5232): No heartbeat from core client for 30 sec - exiting 21:06:31 (5232): No heartbeat from core client for 30 sec - exiting 21:06:32 (5232): No heartbeat from core client for 30 sec - exiting 21:06:34 (5232): No heartbeat from core client for 30 sec - exiting 21:06:35 (5232): No heartbeat from core client for 30 sec - exiting 21:06:36 (5232): No heartbeat from core client for 30 sec - exiting 21:06:37 (5232): No heartbeat from core client for 30 sec - exiting 21:06:38 (5232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:06:39 (5232): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 19:53:02 (6992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:54:06 (6300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4056, iMonCtr=1 Model crash detected, will try to restart... 20:16:08 (2656): No heartbeat from core client for 30 sec - exiting 20:16:09 (2656): No heartbeat from core client for 30 sec - exiting 20:16:10 (2656): No heartbeat from core client for 30 sec - exiting 20:16:11 (2656): No heartbeat from core client for 30 sec - exiting 20:16:12 (2656): No heartbeat from core client for 30 sec - exiting 20:16:13 (2656): No heartbeat from core client for 30 sec - exiting 20:16:14 (2656): No heartbeat from core client for 30 sec - exiting 20:16:15 (2656): No heartbeat from core client for 30 sec - exiting 20:16:16 (2656): No heartbeat from core client for 30 sec - exiting 20:16:17 (2656): No heartbeat from core client for 30 sec - exiting 20:16:18 (2656): No heartbeat from core client for 30 sec - exiting 20:16:19 (2656): No heartbeat from core client for 30 sec - exiting 20:16:20 (2656): No heartbeat from core client for 30 sec - exiting 20:16:22 (2656): No heartbeat from core client for 30 sec - exiting 20:16:23 (2656): No heartbeat from core client for 30 sec - exiting 20:16:24 (2656): No heartbeat from core client for 30 sec - exiting 20:16:25 (2656): No heartbeat from core client for 30 sec - exiting 20:16:26 (2656): No heartbeat from core client for 30 sec - exiting 20:16:27 (2656): No heartbeat from core client for 30 sec - exiting 20:16:28 (2656): No heartbeat from core client for 30 sec - exiting 20:16:29 (2656): No heartbeat from core client for 30 sec - exiting 20:16:30 (2656): No heartbeat from core client for 30 sec - exiting 20:16:31 (2656): No heartbeat from core client for 30 sec - exiting 20:16:32 (2656): No heartbeat from core client for 30 sec - exiting 20:16:33 (2656): No heartbeat from core client for 30 sec - exiting 20:16:34 (2656): No heartbeat from core client for 30 sec - exiting 20:16:35 (2656): No heartbeat from core client for 30 sec - exiting 20:16:36 (2656): No heartbeat from core client for 30 sec - exiting 20:16:37 (2656): No heartbeat from core client for 30 sec - exiting 20:16:38 (2656): No heartbeat from core client for 30 sec - exiting 20:16:39 (2656): No heartbeat from core client for 30 sec - exiting 20:16:40 (2656): No heartbeat from core client for 30 sec - exiting 20:16:41 (2656): No heartbeat from core client for 30 sec - exiting 20:16:42 (2656): No heartbeat from core client for 30 sec - exiting 20:16:43 (2656): No heartbeat from core client for 30 sec - exiting 20:16:44 (2656): No heartbeat from core client for 30 sec - exiting 20:16:45 (2656): No heartbeat from core client for 30 sec - exiting 20:16:46 (2656): No heartbeat from core client for 30 sec - exiting 20:16:47 (2656): No heartbeat from core client for 30 sec - exiting 20:16:48 (2656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:22:26 (6612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6560, iMonCtr=1 Model crash detected, will try to restart... 08:27:52 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:27:53 (5368): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:33:21 (5784): No heartbeat from core client for 30 sec - exiting 12:33:22 (5784): No heartbeat from core client for 30 sec - exiting 12:33:23 (5784): No heartbeat from core client for 30 sec - exiting 12:33:25 (5784): No heartbeat from core client for 30 sec - exiting 12:33:26 (5784): No heartbeat from core client for 30 sec - exiting 12:33:27 (5784): No heartbeat from core client for 30 sec - exiting 12:33:28 (5784): No heartbeat from core client for 30 sec - exiting 12:33:29 (5784): No heartbeat from core client for 30 sec - exiting 12:33:30 (5784): No heartbeat from core client for 30 sec - exiting 12:33:31 (5784): No heartbeat from core client for 30 sec - exiting 12:33:32 (5784): No heartbeat from core client for 30 sec - exiting 12:33:33 (5784): No heartbeat from core client for 30 sec - exiting 12:33:34 (5784): No heartbeat from core client for 30 sec - exiting 12:33:35 (5784): No heartbeat from core client for 30 sec - exiting 12:33:37 (5784): No heartbeat from core client for 30 sec - exiting 12:33:38 (5784): No heartbeat from core client for 30 sec - exiting 12:33:39 (5784): No heartbeat from core client for 30 sec - exiting 12:33:40 (5784): No heartbeat from core client for 30 sec - exiting 12:33:41 (5784): No heartbeat from core client for 30 sec - exiting 12:33:42 (5784): No heartbeat from core client for 30 sec - exiting 12:33:43 (5784): No heartbeat from core client for 30 sec - exiting 12:33:44 (5784): No heartbeat from core client for 30 sec - exiting 12:33:45 (5784): No heartbeat from core client for 30 sec - exiting 12:33:46 (5784): No heartbeat from core client for 30 sec - exiting 12:33:47 (5784): No heartbeat from core client for 30 sec - exiting 12:33:49 (5784): No heartbeat from core client for 30 sec - exiting 12:33:50 (5784): No heartbeat from core client for 30 sec - exiting 12:33:51 (5784): No heartbeat from core client for 30 sec - exiting 12:33:52 (5784): No heartbeat from core client for 30 sec - exiting 12:33:53 (5784): No heartbeat from core client for 30 sec - exiting 12:33:54 (5784): No heartbeat from core client for 30 sec - exiting 12:33:55 (5784): No heartbeat from core client for 30 sec - exiting 12:33:56 (5784): No heartbeat from core client for 30 sec - exiting 12:33:57 (5784): No heartbeat from core client for 30 sec - exiting 12:33:58 (5784): No heartbeat from core client for 30 sec - exiting 12:34:00 (5784): No heartbeat from core client for 30 sec - exiting 12:34:01 (5784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:34:02 (5784): No heartbeat from core client for 30 sec - exiting 12:34:03 (5784): No heartbeat from core client for 30 sec - exiting 12:34:04 (5784): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2968, iMonCtr=1 Model crash detected, will try to restart... 11:31:04 (2372): No heartbeat from core client for 30 sec - exiting 11:31:06 (2372): No heartbeat from core client for 30 sec - exiting 11:31:07 (2372): No heartbeat from core client for 30 sec - exiting 11:31:08 (2372): No heartbeat from core client for 30 sec - exiting 11:31:09 (2372): No heartbeat from core client for 30 sec - exiting 11:31:10 (2372): No heartbeat from core client for 30 sec - exiting 11:31:11 (2372): No heartbeat from core client for 30 sec - exiting 11:31:12 (2372): No heartbeat from core client for 30 sec - exiting 11:31:13 (2372): No heartbeat from core client for 30 sec - exiting 11:31:14 (2372): No heartbeat from core client for 30 sec - exiting 11:31:15 (2372): No heartbeat from core client for 30 sec - exiting 11:31:16 (2372): No heartbeat from core client for 30 sec - exiting 11:31:18 (2372): No heartbeat from core client for 30 sec - exiting 11:31:19 (2372): No heartbeat from core client for 30 sec - exiting 11:31:20 (2372): No heartbeat from core client for 30 sec - exiting 11:31:21 (2372): No heartbeat from core client for 30 sec - exiting 11:31:22 (2372): No heartbeat from core client for 30 sec - exiting 11:31:23 (2372): No heartbeat from core client for 30 sec - exiting 11:31:24 (2372): No heartbeat from core client for 30 sec - exiting 11:31:25 (2372): No heartbeat from core client for 30 sec - exiting 11:31:26 (2372): No heartbeat from core client for 30 sec - exiting 11:31:27 (2372): No heartbeat from core client for 30 sec - exiting 11:31:29 (2372): No heartbeat from core client for 30 sec - exiting 11:31:30 (2372): No heartbeat from core client for 30 sec - exiting 11:31:31 (2372): No heartbeat from core client for 30 sec - exiting 11:31:32 (2372): No heartbeat from core client for 30 sec - exiting 11:31:33 (2372): No heartbeat from core client for 30 sec - exiting 11:31:34 (2372): No heartbeat from core client for 30 sec - exiting 11:31:35 (2372): No heartbeat from core client for 30 sec - exiting 11:31:36 (2372): No heartbeat from core client for 30 sec - exiting 11:31:37 (2372): No heartbeat from core client for 30 sec - exiting 11:31:38 (2372): No heartbeat from core client for 30 sec - exiting 11:31:39 (2372): No heartbeat from core client for 30 sec - exiting 11:31:41 (2372): No heartbeat from core client for 30 sec - exiting 11:31:42 (2372): No heartbeat from core client for 30 sec - exiting 11:31:43 (2372): No heartbeat from core client for 30 sec - exiting 11:31:44 (2372): No heartbeat from core client for 30 sec - exiting 11:31:45 (2372): No heartbeat from core client for 30 sec - exiting 11:31:46 (2372): No heartbeat from core client for 30 sec - exiting 11:31:47 (2372): No heartbeat from core client for 30 sec - exiting 11:31:48 (2372): No heartbeat from core client for 30 sec - exiting 11:31:49 (2372): No heartbeat from core client for 30 sec - exiting 11:31:50 (2372): No heartbeat from core client for 30 sec - exiting 11:31:51 (2372): No heartbeat from core client for 30 sec - exiting 11:31:53 (2372): No heartbeat from core client for 30 sec - exiting 11:31:54 (2372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:03:18 (4560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:48:18 (5552): No heartbeat from core client for 30 sec - exiting 18:48:19 (5552): No heartbeat from core client for 30 sec - exiting 18:48:20 (5552): No heartbeat from core client for 30 sec - exiting 18:48:21 (5552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5316, iMonCtr=1 Model crash detected, will try to restart... 18:34:57 (6460): No heartbeat from core client for 30 sec - exiting 18:34:58 (6460): No heartbeat from core client for 30 sec - exiting 18:34:59 (6460): No heartbeat from core client for 30 sec - exiting 18:35:00 (6460): No heartbeat from core client for 30 sec - exiting 18:35:01 (6460): No heartbeat from core client for 30 sec - exiting 18:35:02 (6460): No heartbeat from core client for 30 sec - exiting 18:35:03 (6460): No heartbeat from core client for 30 sec - exiting 18:35:04 (6460): No heartbeat from core client for 30 sec - exiting 18:35:05 (6460): No heartbeat from core client for 30 sec - exiting 18:35:06 (6460): No heartbeat from core client for 30 sec - exiting 18:35:07 (6460): No heartbeat from core client for 30 sec - exiting 18:35:08 (6460): No heartbeat from core client for 30 sec - exiting 18:35:09 (6460): No heartbeat from core client for 30 sec - exiting 18:35:10 (6460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:11 (6460): No heartbeat from core client for 30 sec - exiting 18:35:12 (6460): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7092, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3676, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:13:31 (6648): No heartbeat from core client for 30 sec - exiting 19:13:32 (6648): No heartbeat from core client for 30 sec - exiting 19:13:33 (6648): No heartbeat from core client for 30 sec - exiting 19:13:34 (6648): No heartbeat from core client for 30 sec - exiting 19:13:35 (6648): No heartbeat from core client for 30 sec - exiting 19:13:36 (6648): No heartbeat from core client for 30 sec - exiting 19:13:37 (6648): No heartbeat from core client for 30 sec - exiting 19:13:38 (6648): No heartbeat from core client for 30 sec - exiting 19:13:39 (6648): No heartbeat from core client for 30 sec - exiting 19:13:41 (6648): No heartbeat from core client for 30 sec - exiting 19:13:42 (6648): No heartbeat from core client for 30 sec - exiting 19:13:43 (6648): No heartbeat from core client for 30 sec - exiting 19:13:44 (6648): No heartbeat from core client for 30 sec - exiting 19:13:45 (6648): No heartbeat from core client for 30 sec - exiting 19:13:46 (6648): No heartbeat from core client for 30 sec - exiting 19:13:47 (6648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:14:47 (7036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:07:57 (6016): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:35 (6780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:25:34 (4472): No heartbeat from core client for 30 sec - exiting 19:25:36 (4472): No heartbeat from core client for 30 sec - exiting 19:25:37 (4472): No heartbeat from core client for 30 sec - exiting 19:25:38 (4472): No heartbeat from core client for 30 sec - exiting 19:25:39 (4472): No heartbeat from core client for 30 sec - exiting 19:25:40 (4472): No heartbeat from core client for 30 sec - exiting 19:25:41 (4472): No heartbeat from core client for 30 sec - exiting 19:25:42 (4472): No heartbeat from core client for 30 sec - exiting 19:25:43 (4472): No heartbeat from core client for 30 sec - exiting 19:25:44 (4472): No heartbeat from core client for 30 sec - exiting 19:25:45 (4472): No heartbeat from core client for 30 sec - exiting 19:25:46 (4472): No heartbeat from core client for 30 sec - exiting 19:25:48 (4472): No heartbeat from core client for 30 sec - exiting 19:25:49 (4472): No heartbeat from core client for 30 sec - exiting 19:25:50 (4472): No heartbeat from core client for 30 sec - exiting 19:25:51 (4472): No heartbeat from core client for 30 sec - exiting 19:25:52 (4472): No heartbeat from core client for 30 sec - exiting 19:25:53 (4472): No heartbeat from core client for 30 sec - exiting 19:25:54 (4472): No heartbeat from core client for 30 sec - exiting 19:25:55 (4472): No heartbeat from core client for 30 sec - exiting 19:25:56 (4472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:51:57 (6996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:39:40 (5960): No heartbeat from core client for 30 sec - exiting 11:39:41 (5960): No heartbeat from core client for 30 sec - exiting 11:39:42 (5960): No heartbeat from core client for 30 sec - exiting 11:39:43 (5960): No heartbeat from core client for 30 sec - exiting 11:39:44 (5960): No heartbeat from core client for 30 sec - exiting 11:40:18 (5960): No heartbeat from core client for 30 sec - exiting 11:40:19 (5960): No heartbeat from core client for 30 sec - exiting 11:40:20 (5960): No heartbeat from core client for 30 sec - exiting 11:40:22 (5960): No heartbeat from core client for 30 sec - exiting 11:40:23 (5960): No heartbeat from core client for 30 sec - exiting 11:40:24 (5960): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:40:25 (5960): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:48:40 (4312): No heartbeat from core client for 30 sec - exiting 11:48:41 (4312): No heartbeat from core client for 30 sec - exiting 11:48:42 (4312): No heartbeat from core client for 30 sec - exiting 11:48:43 (4312): No heartbeat from core client for 30 sec - exiting 11:48:44 (4312): No heartbeat from core client for 30 sec - exiting 11:48:46 (4312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:02:37 (4708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:20:25 (6380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:43:55 (8064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:33:37 (4720): No heartbeat from core client for 30 sec - exiting 18:33:38 (4720): No heartbeat from core client for 30 sec - exiting 18:33:39 (4720): No heartbeat from core client for 30 sec - exiting 18:33:40 (4720): No heartbeat from core client for 30 sec - exiting 18:33:41 (4720): No heartbeat from core client for 30 sec - exiting 18:33:42 (4720): No heartbeat from core client for 30 sec - exiting 18:33:44 (4720): No heartbeat from core client for 30 sec - exiting 18:33:45 (4720): No heartbeat from core client for 30 sec - exiting 18:33:46 (4720): No heartbeat from core client for 30 sec - exiting 18:33:47 (4720): No heartbeat from core client for 30 sec - exiting 18:33:48 (4720): No heartbeat from core client for 30 sec - exiting 18:33:49 (4720): No heartbeat from core client for 30 sec - exiting 18:33:50 (4720): No heartbeat from core client for 30 sec - exiting 18:33:51 (4720): No heartbeat from core client for 30 sec - exiting 18:33:52 (4720): No heartbeat from core client for 30 sec - exiting 18:33:53 (4720): No heartbeat from core client for 30 sec - exiting 18:33:54 (4720): No heartbeat from core client for 30 sec - exiting 18:33:56 (4720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:55:52 (6780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:00:33 (6164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:01:15 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:10:30 (1312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:10:31 (1312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:31:15 (6344): No heartbeat from core client for 30 sec - exiting 18:31:16 (6344): No heartbeat from core client for 30 sec - exiting 18:31:17 (6344): No heartbeat from core client for 30 sec - exiting 18:31:18 (6344): No heartbeat from core client for 30 sec - exiting 18:31:19 (6344): No heartbeat from core client for 30 sec - exiting 18:31:21 (6344): No heartbeat from core client for 30 sec - exiting 18:31:22 (6344): No heartbeat from core client for 30 sec - exiting 18:31:23 (6344): No heartbeat from core client for 30 sec - exiting 18:31:24 (6344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:21:45 (5320): No heartbeat from core client for 30 sec - exiting 08:21:46 (5320): No heartbeat from core client for 30 sec - exiting 08:21:47 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:23:00 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5332, iMonCtr=1 Model crash detected, will try to restart... 12:37:01 (6012): No heartbeat from core client for 30 sec - exiting 12:37:02 (6012): No heartbeat from core client for 30 sec - exiting 12:37:03 (6012): No heartbeat from core client for 30 sec - exiting 12:37:04 (6012): No heartbeat from core client for 30 sec - exiting 12:37:05 (6012): No heartbeat from core client for 30 sec - exiting 12:37:06 (6012): No heartbeat from core client for 30 sec - exiting 12:37:07 (6012): No heartbeat from core client for 30 sec - exiting 12:37:08 (6012): No heartbeat from core client for 30 sec - exiting 12:37:09 (6012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:37:10 (6012): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:26:24 (5456): No heartbeat from core client for 30 sec - exiting 20:26:26 (5456): No heartbeat from core client for 30 sec - exiting 20:26:27 (5456): No heartbeat from core client for 30 sec - exiting 20:26:28 (5456): No heartbeat from core client for 30 sec - exiting 20:26:29 (5456): No heartbeat from core client for 30 sec - exiting 20:26:30 (5456): No heartbeat from core client for 30 sec - exiting 20:26:31 (5456): No heartbeat from core client for 30 sec - exiting 20:26:32 (5456): No heartbeat from core client for 30 sec - exiting 20:26:33 (5456): No heartbeat from core client for 30 sec - exiting 20:26:34 (5456): No heartbeat from core client for 30 sec - exiting 20:26:35 (5456): No heartbeat from core client for 30 sec - exiting 20:26:36 (5456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
08 Jul 2013 21:35:28	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	518,400	891,409	1.7195
07 Jul 2013 14:42:14	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	492,480	847,300	1.7205
06 Jul 2013 15:24:16	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	466,560	803,464	1.7221
06 Jul 2013 05:17:54	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	440,640	759,474	1.7236
04 Jul 2013 18:41:56	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	414,720	714,904	1.7238
02 Jul 2013 11:00:25	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	388,800	670,731	1.7251
02 Jul 2013 10:11:34	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	362,880	626,293	1.7259
28 Jun 2013 07:42:58	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	336,960	581,368	1.7253
24 Jun 2013 21:19:40	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	311,040	536,912	1.7262
23 Jun 2013 11:58:04	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	285,120	492,857	1.7286
21 Jun 2013 17:21:16	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	259,200	448,172	1.7291
19 Jun 2013 00:02:13	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	233,280	403,712	1.7306
16 Jun 2013 21:14:09	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	207,360	359,726	1.7348
15 Jun 2013 18:46:22	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	181,440	315,611	1.7395
12 Jun 2013 18:49:40	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	155,520	271,040	1.7428
09 Jun 2013 23:48:47	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	129,600	226,578	1.7483
08 Jun 2013 22:20:26	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	103,680	181,334	1.7490
07 Jun 2013 23:03:52	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	77,760	134,524	1.7300
07 Jun 2013 09:55:59	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	51,840	89,715	1.7306
04 Jun 2013 20:59:05	1072815	15813929	hadcm3n_o0cg_1940_40_008382431_0	25,920	44,631	1.7219