Task 13554735

Name	hadcm3n_yh9k_1900_40_007524514_1
Workunit	7721989
Created	28 Oct 2011, 13:30:26 UTC
Sent	30 Oct 2011, 21:46:54 UTC
Report deadline	30 Jan 2012, 5:14:05 UTC
Received	31 Dec 2011, 5:22:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1100341
Run time	14 days 20 hours 7 min 5 sec
CPU time	10 days 6 hours 13 min 51 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.18 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code 193 (0xc1) </message> <stderr_txt> 17:34:48 (3504): No heartbeat from core client for 30 sec - exiting 17:34:49 (3504): No heartbeat from core client for 30 sec - exiting 17:34:50 (3504): No heartbeat from core client for 30 sec - exiting 17:34:51 (3504): No heartbeat from core client for 30 sec - exiting 17:34:52 (3504): No heartbeat from core client for 30 sec - exiting 17:34:53 (3504): No heartbeat from core client for 30 sec - exiting 17:34:54 (3504): No heartbeat from core client for 30 sec - exiting 17:34:55 (3504): No heartbeat from core client for 30 sec - exiting 17:34:56 (3504): No heartbeat from core client for 30 sec - exiting 17:34:57 (3504): No heartbeat from core client for 30 sec - exiting 17:34:58 (3504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3188, iMonCtr=1 Model crash detected, will try to restart... 18:10:02 (3516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3632, iMonCtr=1 Model crash detected, will try to restart... 20:02:13 (908): No heartbeat from core client for 30 sec - exiting 20:02:14 (908): No heartbeat from core client for 30 sec - exiting 20:02:16 (908): No heartbeat from core client for 30 sec - exiting 20:02:17 (908): No heartbeat from core client for 30 sec - exiting 20:02:18 (908): No heartbeat from core client for 30 sec - exiting 20:02:19 (908): No heartbeat from core client for 30 sec - exiting 20:02:20 (908): No heartbeat from core client for 30 sec - exiting 20:02:21 (908): No heartbeat from core client for 30 sec - exiting 20:02:22 (908): No heartbeat from core client for 30 sec - exiting 20:02:24 (908): No heartbeat from core client for 30 sec - exiting 20:02:25 (908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:02:26 (908): No heartbeat from core client for 30 sec - exiting 08:00:09 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:47:41 (3764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C17:35:35 (3848): No heartbeat from core client for 30 sec - exiting 17:35:37 (3848): No heartbeat from core client for 30 sec - exiting 17:35:38 (3848): No heartbeat from core client for 30 sec - exiting 17:35:39 (3848): No heartbeat from core client for 30 sec - exiting 17:35:40 (3848): No heartbeat from core client for 30 sec - exiting 17:35:41 (3848): No heartbeat from core client for 30 sec - exiting 17:35:42 (3848): No heartbeat from core client for 30 sec - exiting 17:35:43 (3848): No heartbeat from core client for 30 sec - exiting 17:35:45 (3848): No heartbeat from core client for 30 sec - exiting 17:35:46 (3848): No heartbeat from core client for 30 sec - exiting 17:35:47 (3848): No heartbeat from core client for 30 sec - exiting 17:35:48 (3848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4440, iMonCtr=1 Model crash detected, will try to restart... 17:13:22 (2280): No heartbeat from core client for 30 sec - exiting 17:13:24 (2280): No heartbeat from core client for 30 sec - exiting 17:13:25 (2280): No heartbeat from core client for 30 sec - exiting 17:13:26 (2280): No heartbeat from core client for 30 sec - exiting 17:13:27 (2280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4656, iMonCtr=1 Model crash detected, will try to restart... 20:47:03 (2456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:30:13 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:49:50 (3652): No heartbeat from core client for 30 sec - exiting 17:49:51 (3652): No heartbeat from core client for 30 sec - exiting 17:49:52 (3652): No heartbeat from core client for 30 sec - exiting 17:49:53 (3652): No heartbeat from core client for 30 sec - exiting 17:49:55 (3652): No heartbeat from core client for 30 sec - exiting 17:49:56 (3652): No heartbeat from core client for 30 sec - exiting 17:49:57 (3652): No heartbeat from core client for 30 sec - exiting 17:49:58 (3652): No heartbeat from core client for 30 sec - exiting 17:50:00 (3652): No heartbeat from core client for 30 sec - exiting 17:50:01 (3652): No heartbeat from core client for 30 sec - exiting 17:50:04 (3652): No heartbeat from core client for 30 sec - exiting 17:50:05 (3652): No heartbeat from core client for 30 sec - exiting 17:50:07 (3652): No heartbeat from core client for 30 sec - exiting 17:50:08 (3652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... C21:30:32 (2348): No heartbeat from core client for 30 sec - exiting 21:30:35 (2348): No heartbeat from core client for 30 sec - exiting 21:30:36 (2348): No heartbeat from core client for 30 sec - exiting 21:30:37 (2348): No heartbeat from core client for 30 sec - exiting 21:30:38 (2348): No heartbeat from core client for 30 sec - exiting 21:30:39 (2348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:54:17 (3868): No heartbeat from core client for 30 sec - exiting 21:54:18 (3868): No heartbeat from core client for 30 sec - exiting 21:54:20 (3868): No heartbeat from core client for 30 sec - exiting 21:54:21 (3868): No heartbeat from core client for 30 sec - exiting 21:54:22 (3868): No heartbeat from core client for 30 sec - exiting 21:54:23 (3868): No heartbeat from core client for 30 sec - exiting 21:54:24 (3868): No heartbeat from core client for 30 sec - exiting 21:54:25 (3868): No heartbeat from core client for 30 sec - exiting 21:54:28 (3868): No heartbeat from core client for 30 sec - exiting 21:54:30 (3868): No heartbeat from core client for 30 sec - exiting 21:54:31 (3868): No heartbeat from core client for 30 sec - exiting 21:54:33 (3868): No heartbeat from core client for 30 sec - exiting 21:54:36 (3868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:00:38 (1940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:41:01 (3852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:26 (2664): No heartbeat from core client for 30 sec - exiting 22:19:31 (2664): No heartbeat from core client for 30 sec - exiting 22:19:32 (2664): No heartbeat from core client for 30 sec - exiting 22:19:33 (2664): No heartbeat from core client for 30 sec - exiting 22:19:35 (2664): No heartbeat from core client for 30 sec - exiting 22:19:36 (2664): No heartbeat from core client for 30 sec - exiting 22:19:37 (2664): No heartbeat from core client for 30 sec - exiting 22:19:38 (2664): No heartbeat from core client for 30 sec - exiting 22:19:39 (2664): No heartbeat from core client for 30 sec - exiting 22:19:40 (2664): No heartbeat from core client for 30 sec - exiting 22:19:41 (2664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1364, iMonCtr=1 Model crash detected, will try to restart... 07:58:02 (2968): No heartbeat from core client for 30 sec - exiting 07:58:03 (2968): No heartbeat from core client for 30 sec - exiting 07:58:05 (2968): No heartbeat from core client for 30 sec - exiting 07:58:06 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3280, iMonCtr=1 Model crash detected, will try to restart... 19:03:47 (4072): No heartbeat from core client for 30 sec - exiting 19:03:48 (4072): No heartbeat from core client for 30 sec - exiting 19:03:49 (4072): No heartbeat from core client for 30 sec - exiting 19:03:50 (4072): No heartbeat from core client for 30 sec - exiting 19:03:51 (4072): No heartbeat from core client for 30 sec - exiting 19:03:52 (4072): No heartbeat from core client for 30 sec - exiting 19:03:54 (4072): No heartbeat from core client for 30 sec - exiting 19:03:55 (4072): No heartbeat from core client for 30 sec - exiting 19:03:56 (4072): No heartbeat from core client for 30 sec - exiting 19:03:57 (4072): No heartbeat from core client for 30 sec - exiting 19:03:58 (4072): No heartbeat from core client for 30 sec - exiting 19:03:59 (4072): No heartbeat from core client for 30 sec - exiting 19:04:00 (4072): No heartbeat from core client for 30 sec - exiting 19:04:01 (4072): No heartbeat from core client for 30 sec - exiting 19:04:03 (4072): No heartbeat from core client for 30 sec - exiting 19:04:04 (4072): No heartbeat from core client for 30 sec - exiting 19:04:05 (4072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:04:06 (4072): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4512, iMonCtr=1 Model crash detected, will try to restart... 10:09:08 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=844, iMonCtr=1 Model crash detected, will try to restart... 18:58:20 (3344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:09:11 (2972): No heartbeat from core client for 30 sec - exiting 19:09:12 (2972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:58:22 (3888): No heartbeat from core client for 30 sec - exiting 16:58:24 (3888): No heartbeat from core client for 30 sec - exiting 16:58:25 (3888): No heartbeat from core client for 30 sec - exiting 16:58:26 (3888): No heartbeat from core client for 30 sec - exiting 16:58:27 (3888): No heartbeat from core client for 30 sec - exiting 16:58:28 (3888): No heartbeat from core client for 30 sec - exiting 16:58:30 (3888): No heartbeat from core client for 30 sec - exiting 16:58:31 (3888): No heartbeat from core client for 30 sec - exiting 16:58:32 (3888): No heartbeat from core client for 30 sec - exiting 16:58:33 (3888): No heartbeat from core client for 30 sec - exiting 16:58:34 (3888): No heartbeat from core client for 30 sec - exiting 16:58:35 (3888): No heartbeat from core client for 30 sec - exiting 16:58:36 (3888): No heartbeat from core client for 30 sec - exiting 16:58:37 (3888): No heartbeat from core client for 30 sec - exiting 16:58:38 (3888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4528, iMonCtr=1 Model crash detected, will try to restart... 17:40:01 (1468): No heartbeat from core client for 30 sec - exiting 17:40:02 (1468): No heartbeat from core client for 30 sec - exiting 17:40:03 (1468): No heartbeat from core client for 30 sec - exiting 17:40:04 (1468): No heartbeat from core client for 30 sec - exiting 17:40:05 (1468): No heartbeat from core client for 30 sec - exiting 17:40:06 (1468): No heartbeat from core client for 30 sec - exiting 17:40:08 (1468): No heartbeat from core client for 30 sec - exiting 17:40:09 (1468): No heartbeat from core client for 30 sec - exiting 17:40:10 (1468): No heartbeat from core client for 30 sec - exiting 17:40:11 (1468): No heartbeat from core client for 30 sec - exiting 17:40:12 (1468): No heartbeat from core client for 30 sec - exiting 17:40:14 (1468): No heartbeat from core client for 30 sec - exiting 17:40:15 (1468): No heartbeat from core client for 30 sec - exiting 17:40:16 (1468): No heartbeat from core client for 30 sec - exiting 17:40:17 (1468): No heartbeat from core client for 30 sec - exiting 17:40:18 (1468): No heartbeat from core client for 30 sec - exiting 17:40:19 (1468): No heartbeat from core client for 30 sec - exiting 17:40:20 (1468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:02:55 (3328): No heartbeat from core client for 30 sec - exiting 08:02:56 (3328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3812, iMonCtr=1 Model crash detected, will try to restart... 09:04:44 (664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6072, iMonCtr=1 Model crash detected, will try to restart... 16:25:35 (2832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4888, iMonCtr=1 Model crash detected, will try to restart... 17:56:53 (3180): No heartbeat from core client for 30 sec - exiting 17:56:54 (3180): No heartbeat from core client for 30 sec - exiting 17:56:55 (3180): No heartbeat from core client for 30 sec - exiting 17:56:56 (3180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=176, iMonCtr=1 Model crash detected, will try to restart... 19:04:04 (3292): No heartbeat from core client for 30 sec - exiting 19:04:05 (3292): No heartbeat from core client for 30 sec - exiting 19:04:07 (3292): No heartbeat from core client for 30 sec - exiting 19:04:08 (3292): No heartbeat from core client for 30 sec - exiting 19:04:09 (3292): No heartbeat from core client for 30 sec - exiting 19:04:10 (3292): No heartbeat from core client for 30 sec - exiting 19:04:11 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=488, iMonCtr=1 Model crash detected, will try to restart... 17:53:26 (3488): No heartbeat from core client for 30 sec - exiting 17:53:27 (3488): No heartbeat from core client for 30 sec - exiting 17:53:28 (3488): No heartbeat from core client for 30 sec - exiting 17:53:29 (3488): No heartbeat from core client for 30 sec - exiting 17:53:30 (3488): No heartbeat from core client for 30 sec - exiting 17:53:31 (3488): No heartbeat from core client for 30 sec - exiting 17:53:32 (3488): No heartbeat from core client for 30 sec - exiting 17:53:33 (3488): No heartbeat from core client for 30 sec - exiting 17:53:35 (3488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:53:36 (3488): No heartbeat from core client for 30 sec - exiting 18:54:30 (2180): No heartbeat from core client for 30 sec - exiting 18:54:31 (2180): No heartbeat from core client for 30 sec - exiting 18:54:32 (2180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:13:26 (3308): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3772, iMonCtr=1 Model crash detected, will try to restart... 19:05:58 (3916): No heartbeat from core client for 30 sec - exiting 19:06:00 (3916): No heartbeat from core client for 30 sec - exiting 19:06:01 (3916): No heartbeat from core client for 30 sec - exiting 19:06:02 (3916): No heartbeat from core client for 30 sec - exiting 19:06:03 (3916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:20 (2192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:17:43 (1912): No heartbeat from core client for 30 sec - exiting 11:17:45 (1912): No heartbeat from core client for 30 sec - exiting 11:17:46 (1912): No heartbeat from core client for 30 sec - exiting 11:17:47 (1912): No heartbeat from core client for 30 sec - exiting 11:17:48 (1912): No heartbeat from core client for 30 sec - exiting 11:17:49 (1912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3676, iMonCtr=1 Model crash detected, will try to restart... 12:09:52 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:45:48 (1680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:00:44 (176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:00:53 (1180): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4320, iMonCtr=1 Model crash detected, will try to restart... 17:58:17 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:28 (176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... 20:36:36 (992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:21:22 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:28:29 (3932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:21 (1136): No heartbeat from core client for 30 sec - exiting 08:01:22 (1136): No heartbeat from core client for 30 sec - exiting 08:01:23 (1136): No heartbeat from core client for 30 sec - exiting 08:01:25 (1136): No heartbeat from core client for 30 sec - exiting 08:01:26 (1136): No heartbeat from core client for 30 sec - exiting 08:01:27 (1136): No heartbeat from core client for 30 sec - exiting 08:01:28 (1136): No heartbeat from core client for 30 sec - exiting 08:01:29 (1136): No heartbeat from core client for 30 sec - exiting 08:01:30 (1136): No heartbeat from core client for 30 sec - exiting 08:01:31 (1136): No heartbeat from core client for 30 sec - exiting 08:01:32 (1136): No heartbeat from core client for 30 sec - exiting 08:01:33 (1136): No heartbeat from core client for 30 sec - exiting 08:01:35 (1136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:39 (2076): No heartbeat from core client for 30 sec - exiting 21:30:40 (2076): No heartbeat from core client for 30 sec - exiting 21:30:41 (2076): No heartbeat from core client for 30 sec - exiting 21:30:42 (2076): No heartbeat from core client for 30 sec - exiting 21:30:44 (2076): No heartbeat from core client for 30 sec - exiting 21:30:45 (2076): No heartbeat from core client for 30 sec - exiting 21:30:46 (2076): No heartbeat from core client for 30 sec - exiting 21:30:48 (2076): No heartbeat from core client for 30 sec - exiting 21:30:49 (2076): No heartbeat from core client for 30 sec - exiting 21:30:52 (2076): No heartbeat from core client for 30 sec - exiting 21:30:54 (2076): No heartbeat from core client for 30 sec - exiting 21:30:55 (2076): No heartbeat from core client for 30 sec - exiting 21:30:59 (2076): No heartbeat from core client for 30 sec - exiting 21:31:00 (2076): No heartbeat from core client for 30 sec - exiting 21:31:01 (2076): No heartbeat from core client for 30 sec - exiting 21:31:03 (2076): No heartbeat from core client for 30 sec - exiting 21:31:04 (2076): No heartbeat from core client for 30 sec - exiting 21:31:05 (2076): No heartbeat from core client for 30 sec - exiting 21:31:06 (2076): No heartbeat from core client for 30 sec - exiting 21:31:07 (2076): No heartbeat from core client for 30 sec - exiting 21:31:08 (2076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5248, iMonCtr=1 Model crash detected, will try to restart... 18:36:57 (2684): No heartbeat from core client for 30 sec - exiting 18:36:59 (2684): No heartbeat from core client for 30 sec - exiting 18:37:00 (2684): No heartbeat from core client for 30 sec - exiting 18:37:01 (2684): No heartbeat from core client for 30 sec - exiting 18:37:02 (2684): No heartbeat from core client for 30 sec - exiting 18:37:03 (2684): No heartbeat from core client for 30 sec - exiting 18:37:04 (2684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:37:05 (2684): No heartbeat from core client for 30 sec - exiting 21:21:52 (5000): No heartbeat from core client for 30 sec - exiting 21:21:54 (5000): No heartbeat from core client for 30 sec - exiting 21:21:55 (5000): No heartbeat from core client for 30 sec - exiting 21:21:56 (5000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:21:57 (5000): No heartbeat from core client for 30 sec - exiting 21:21:58 (5000): No heartbeat from core client for 30 sec - exiting 21:21:59 (5000): No heartbeat from core client for 30 sec - exiting 21:22:00 (5000): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4624, iMonCtr=1 Model crash detected, will try to restart... 21:28:21 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:42 (1992): No heartbeat from core client for 30 sec - exiting 21:42:44 (1992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:25:54 (3496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:06:39 (5072): No heartbeat from core client for 30 sec - exiting 12:06:40 (5072): No heartbeat from core client for 30 sec - exiting 12:06:41 (5072): No heartbeat from core client for 30 sec - exiting 12:06:42 (5072): No heartbeat from core client for 30 sec - exiting 12:06:43 (5072): No heartbeat from core client for 30 sec - exiting 12:06:44 (5072): No heartbeat from core client for 30 sec - exiting 12:06:46 (5072): No heartbeat from core client for 30 sec - exiting 12:06:47 (5072): No heartbeat from core client for 30 sec - exiting 12:06:48 (5072): No heartbeat from core client for 30 sec - exiting 12:06:49 (5072): No heartbeat from core client for 30 sec - exiting 12:06:50 (5072): No heartbeat from core client for 30 sec - exiting 12:06:51 (5072): No heartbeat from core client for 30 sec - exiting 12:06:52 (5072): No heartbeat from core client for 30 sec - exiting 12:06:54 (5072): No heartbeat from core client for 30 sec - exiting 12:06:55 (5072): No heartbeat from core client for 30 sec - exiting 12:06:56 (5072): No heartbeat from core client for 30 sec - exiting 12:06:57 (5072): No heartbeat from core client for 30 sec - exiting 12:06:58 (5072): No heartbeat from core client for 30 sec - exiting 12:06:59 (5072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:26 (2588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:55:27 (3896): No heartbeat from core client for 30 sec - exiting 07:55:28 (3896): No heartbeat from core client for 30 sec - exiting 07:55:29 (3896): No heartbeat from core client for 30 sec - exiting 07:55:30 (3896): No heartbeat from core client for 30 sec - exiting 07:55:31 (3896): No heartbeat from core client for 30 sec - exiting 07:55:32 (3896): No heartbeat from core client for 30 sec - exiting 07:55:33 (3896): No heartbeat from core client for 30 sec - exiting 07:55:34 (3896): No heartbeat from core client for 30 sec - exiting 07:55:35 (3896): No heartbeat from core client for 30 sec - exiting 07:55:36 (3896): No heartbeat from core client for 30 sec - exiting 07:55:37 (3896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3432, iMonCtr=1 Model crash detected, will try to restart... 18:56:41 (3196): No heartbeat from core client for 30 sec - exiting 18:56:42 (3196): No heartbeat from core client for 30 sec - exiting 18:56:44 (3196): No heartbeat from core client for 30 sec - exiting 18:56:45 (3196): No heartbeat from core client for 30 sec - exiting 18:56:47 (3196): No heartbeat from core client for 30 sec - exiting 18:56:48 (3196): No heartbeat from core client for 30 sec - exiting 18:56:49 (3196): No heartbeat from core client for 30 sec - exiting 18:56:50 (3196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6032, iMonCtr=1 Model crash detected, will try to restart... 18:42:09 (5152): No heartbeat from core client for 30 sec - exiting 18:42:10 (5152): No heartbeat from core client for 30 sec - exiting 18:42:11 (5152): No heartbeat from core client for 30 sec - exiting 18:42:13 (5152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4544, iMonCtr=1 Model crash detected, will try to restart... 21:03:51 (2664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:07:02 (3312): No heartbeat from core client for 30 sec - exiting 18:07:04 (3312): No heartbeat from core client for 30 sec - exiting 18:07:05 (3312): No heartbeat from core client for 30 sec - exiting 18:07:06 (3312): No heartbeat from core client for 30 sec - exiting 18:07:07 (3312): No heartbeat from core client for 30 sec - exiting 18:07:09 (3312): No heartbeat from core client for 30 sec - exiting 18:07:13 (3312): No heartbeat from core client for 30 sec - exiting 18:07:17 (3312): No heartbeat from core client for 30 sec - exiting 18:07:19 (3312): No heartbeat from core client for 30 sec - exiting 18:07:20 (3312): No heartbeat from core client for 30 sec - exiting 18:07:21 (3312): No heartbeat from core client for 30 sec - exiting 18:07:22 (3312): No heartbeat from core client for 30 sec - exiting 18:07:23 (3312): No heartbeat from core client for 30 sec - exiting 18:07:25 (3312): No heartbeat from core client for 30 sec - exiting 18:07:26 (3312): No heartbeat from core client for 30 sec - exiting 18:07:27 (3312): No heartbeat from core client for 30 sec - exiting 18:07:28 (3312): No heartbeat from core client for 30 sec - exiting 18:07:29 (3312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:17:57 (3116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:44 (3940): No heartbeat from core client for 30 sec - exiting 19:21:46 (3940): No heartbeat from core client for 30 sec - exiting 19:21:47 (3940): No heartbeat from core client for 30 sec - exiting 19:21:48 (3940): No heartbeat from core client for 30 sec - exiting 19:21:49 (3940): No heartbeat from core client for 30 sec - exiting 19:21:50 (3940): No heartbeat from core client for 30 sec - exiting 19:21:51 (3940): No heartbeat from core client for 30 sec - exiting 19:21:52 (3940): No heartbeat from core client for 30 sec - exiting 19:21:53 (3940): No heartbeat from core client for 30 sec - exiting 19:21:54 (3940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:43 (3136): No heartbeat from core client for 30 sec - exiting 08:08:44 (3136): No heartbeat from core client for 30 sec - exiting 08:08:46 (3136): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1308, iMonCtr=1 Model crash detected, will try to restart... 21:55:00 (1724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3368, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4592, iMonCtr=1 Model crash detected, will try to restart... 02:01:52 (2904): No heartbeat from core client for 30 sec - exiting 02:01:53 (2904): No heartbeat from core client for 30 sec - exiting 02:01:54 (2904): No heartbeat from core client for 30 sec - exiting 02:01:55 (2904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1480, iMonCtr=1 Model crash detected, will try to restart... 14:04:00 (1780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:35:02 (3080): No heartbeat from core client for 30 sec - exiting 12:35:04 (3080): No heartbeat from core client for 30 sec - exiting 12:35:05 (3080): No heartbeat from core client for 30 sec - exiting 12:35:06 (3080): No heartbeat from core client for 30 sec - exiting 12:35:07 (3080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:35:08 (3080): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3108, iMonCtr=1 Model crash detected, will try to restart... 17:10:30 (3680): No heartbeat from core client for 30 sec - exiting 17:10:31 (3680): No heartbeat from core client for 30 sec - exiting 17:10:32 (3680): No heartbeat from core client for 30 sec - exiting 17:10:33 (3680): No heartbeat from core client for 30 sec - exiting 17:10:34 (3680): No heartbeat from core client for 30 sec - exiting 17:10:35 (3680): No heartbeat from core client for 30 sec - exiting 17:10:37 (3680): No heartbeat from core client for 30 sec - exiting 17:10:38 (3680): No heartbeat from core client for 30 sec - exiting 17:10:39 (3680): No heartbeat from core client for 30 sec - exiting 17:10:40 (3680): No heartbeat from core client for 30 sec - exiting 17:10:41 (3680): No heartbeat from core client for 30 sec - exiting 17:10:42 (3680): No heartbeat from core client for 30 sec - exiting 17:10:43 (3680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:35:26 (2680): No heartbeat from core client for 30 sec - exiting 21:35:27 (2680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1000, iMonCtr=1 Model crash detected, will try to restart... 17:40:11 (852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller::20:26:57 (3820): No heartbeat from core client for 30 sec - exiting 20:26:58 (3820): No heartbeat from core client for 30 sec - exiting 20:26:59 (3820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:12:15 (1616): No heartbeat from core client for 30 sec - exiting 21:12:16 (1616): No heartbeat from core client for 30 sec - exiting 21:12:17 (1616): No heartbeat from core client for 30 sec - exiting 21:12:18 (1616): No heartbeat from core client for 30 sec - exiting 21:12:20 (1616): No heartbeat from core client for 30 sec - exiting 21:12:21 (1616): No heartbeat from core client for 30 sec - exiting 21:12:22 (1616): No heartbeat from core client for 30 sec - exiting 21:12:23 (1616): No heartbeat from core client for 30 sec - exiting 21:12:24 (1616): No heartbeat from core client for 30 sec - exiting 21:12:25 (1616): No heartbeat from core client for 30 sec - exiting 21:12:26 (1616): No heartbeat from core client for 30 sec - exiting 21:12:27 (1616): No heartbeat from core client for 30 sec - exiting 21:12:28 (1616): No heartbeat from core client for 30 sec - exiting 21:12:29 (1616): No heartbeat from core client for 30 sec - exiting 21:12:30 (1616): No heartbeat from core client for 30 sec - exiting 21:12:31 (1616): No heartbeat from core client for 30 sec - exiting 21:12:33 (1616): No heartbeat from core client for 30 sec - exiting 21:12:34 (1616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5928, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 08:05:53 (2852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=304, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1516, iMonCtr=1 Model crash detected, will try to restart... 08:18:27 (3004): No heartbeat from core client for 30 sec - exiting 08:18:28 (3004): No heartbeat from core client for 30 sec - exiting 08:18:29 (3004): No heartbeat from core client for 30 sec - exiting 08:18:31 (3004): No heartbeat from core client for 30 sec - exiting 08:18:32 (3004): No heartbeat from core client for 30 sec - exiting 08:18:33 (3004): No heartbeat from core client for 30 sec - exiting 08:18:34 (3004): No heartbeat from core client for 30 sec - exiting 08:18:35 (3004): No heartbeat from core client for 30 sec - exiting 08:18:36 (3004): No heartbeat from core client for 30 sec - exiting 08:18:38 (3004): No heartbeat from core client for 30 sec - exiting 08:18:39 (3004): No heartbeat from core client for 30 sec - exiting 08:18:40 (3004): No heartbeat from core client for 30 sec - exiting 08:18:41 (3004): No heartbeat from core client for 30 sec - exiting 08:18:43 (3004): No heartbeat from core client for 30 sec - exiting 08:18:44 (3004): No heartbeat from core client for 30 sec - exiting 08:18:45 (3004): No heartbeat from core client for 30 sec - exiting 08:18:46 (3004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:50:20 (1440): No heartbeat from core client for 30 sec - exiting 14:50:22 (1440): No heartbeat from core client for 30 sec - exiting 14:50:23 (1440): No heartbeat from core client for 30 sec - exiting 14:50:24 (1440): No heartbeat from core client for 30 sec - exiting 14:50:25 (1440): No heartbeat from core client for 30 sec - exiting 14:50:26 (1440): No heartbeat from core client for 30 sec - exiting 14:50:27 (1440): No heartbeat from core client for 30 sec - exiting 14:50:28 (1440): No heartbeat from core client for 30 sec - exiting 14:50:30 (1440): No heartbeat from core client for 30 sec - exiting 14:50:31 (1440): No heartbeat from core client for 30 sec - exiting 14:50:32 (1440): No heartbeat from core client for 30 sec - exiting 14:50:33 (1440): No heartbeat from core client for 30 sec - exiting 14:50:34 (1440): No heartbeat from core client for 30 sec - exiting 14:50:36 (1440): No heartbeat from core client for 30 sec - exiting 14:50:37 (1440): No heartbeat from core client for 30 sec - exiting 14:50:38 (1440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:56 (3160): No heartbeat from core client for 30 sec - exiting 17:49:57 (3160): No heartbeat from core client for 30 sec - exiting 17:49:58 (3160): No heartbeat from core client for 30 sec - exiting 17:49:59 (3160): No heartbeat from core client for 30 sec - exiting 17:50:01 (3160): No heartbeat from core client for 30 sec - exiting 17:50:02 (3160): No heartbeat from core client for 30 sec - exiting 17:50:03 (3160): No heartbeat from core client for 30 sec - exiting 17:50:04 (3160): No heartbeat from core client for 30 sec - exiting 17:50:05 (3160): No heartbeat from core client for 30 sec - exiting 17:50:06 (3160): No heartbeat from core client for 30 sec - exiting 17:50:07 (3160): No heartbeat from core client for 30 sec - exiting 17:50:08 (3160): No heartbeat from core client for 30 sec - exiting 17:50:09 (3160): No heartbeat from core client for 30 sec - exiting 17:50:10 (3160): No heartbeat from core client for 30 sec - exiting 17:50:12 (3160): No heartbeat from core client for 30 sec - exiting 17:50:13 (3160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:54:11 (2564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:12:34 (3920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1260, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 12:05:07 (1588): No heartbeat from core client for 30 sec - exiting 12:05:09 (1588): No heartbeat from core client for 30 sec - exiting 12:05:10 (1588): No heartbeat from core client for 30 sec - exiting 12:05:11 (1588): No heartbeat from core client for 30 sec - exiting 12:05:12 (1588): No heartbeat from core client for 30 sec - exiting 12:05:13 (1588): No heartbeat from core client for 30 sec - exiting 12:05:14 (1588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Dec 2011 05:24:03	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	259,200	886,421	3.4198
18 Dec 2011 21:04:42	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	233,280	794,656	3.4064
10 Dec 2011 20:33:16	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	207,360	705,161	3.4007
03 Dec 2011 15:17:55	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	181,440	614,929	3.3892
27 Nov 2011 15:42:32	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	155,520	524,915	3.3752
21 Nov 2011 23:31:20	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	129,600	436,548	3.3684
19 Nov 2011 14:45:22	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	103,680	346,836	3.3453
15 Nov 2011 20:43:36	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	77,760	261,314	3.3605
15 Nov 2011 20:43:36	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	51,840	174,178	3.3599
06 Nov 2011 17:58:17	1100341	13554735	hadcm3n_yh9k_1900_40_007524514_1	25,920	88,332	3.4079