Task 16069904

Name	hadcm3n_oczj_1900_40_008471954_1
Workunit	8622793
Created	18 Oct 2013, 14:45:09 UTC
Sent	18 Oct 2013, 14:45:36 UTC
Report deadline	17 Jan 2014, 22:12:47 UTC
Received	24 Dec 2013, 15:39:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1186013
Run time	18 days 7 hours 10 min 29 sec
CPU time	13 days 13 hours 14 min 2 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.30 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code 193 (0xc1) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8096, iMonCtr=1 Model crash detected, will try to restart... 10:53:14 (828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... 18:46:48 (3640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... C10:13:14 (4224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:13:16 (4224): No heartbeat from core client for 30 sec - exiting 10:13:17 (4224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5416, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5416, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4276, iMonCtr=1 Model crash detected, will try to restart... 13:51:23 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:51:25 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4384, iMonCtr=1 Model crash detected, will try to restart... 18:55:39 (1656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7676, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:34:15 (5404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:35:33 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1252, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4932, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... 18:55:46 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:55:48 (4460): No heartbeat from core client for 30 sec - exiting 18:55:49 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6284, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6296, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6296, iMonCtr=1 Model crash detected, will try to restart... 21:42:32 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:42:34 (5368): No heartbeat from core client for 30 sec - exiting 21:42:35 (5368): No heartbeat from core client for 30 sec - exiting 21:42:36 (5368): No heartbeat from core client for 30 sec - exiting 11:11:27 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:11:28 (5428): No heartbeat from core client for 30 sec - exiting 11:11:29 (5428): No heartbeat from core client for 30 sec - exiting 11:11:30 (5428): No heartbeat from core client for 30 sec - exiting 11:11:31 (5428): No heartbeat from core client for 30 sec - exiting 11:11:32 (5428): No heartbeat from core client for 30 sec - exiting 11:11:33 (5428): No heartbeat from core client for 30 sec - exiting 11:11:34 (5428): No heartbeat from core client for 30 sec - exiting 11:11:35 (5428): No heartbeat from core client for 30 sec - exiting 11:11:36 (5428): No heartbeat from core client for 30 sec - exiting 11:11:38 (5428): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3360, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4900, iMonCtr=1 Model crash detected, will try to restart... 17:47:16 (5452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4008, iMonCtr=1 Model crash detected, will try to restart... 11:16:53 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:16:54 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7104, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5016, iMonCtr=1 Model crash detected, will try to restart... 01:15:15 (4348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7416, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2120, iMonCtr=1 Model crash detected, will try to restart... 18:42:33 (2100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7492, iMonCtr=1 Model crash detected, will try to restart... 00:44:01 (4804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5548, iMonCtr=1 Model crash detected, will try to restart... 19:09:23 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:37:55 (7288): No heartbeat from core client for 30 sec - exiting 22:37:56 (7288): No heartbeat from core client for 30 sec - exiting 22:37:57 (7288): No heartbeat from core client for 30 sec - exiting 22:37:58 (7288): No heartbeat from core client for 30 sec - exiting 22:37:59 (7288): No heartbeat from core client for 30 sec - exiting 22:38:01 (7288): No heartbeat from core client for 30 sec - exiting 22:38:02 (7288): No heartbeat from core client for 30 sec - exiting 22:38:03 (7288): No heartbeat from core client for 30 sec - exiting 22:38:04 (7288): No heartbeat from core client for 30 sec - exiting 22:38:05 (7288): No heartbeat from core client for 30 sec - exiting 22:38:06 (7288): No heartbeat from core client for 30 sec - exiting 22:38:07 (7288): No heartbeat from core client for 30 sec - exiting 22:38:08 (7288): No heartbeat from core client for 30 sec - exiting 22:38:09 (7288): No heartbeat from core client for 30 sec - exiting 22:38:11 (7288): No heartbeat from core client for 30 sec - exiting 22:38:12 (7288): No heartbeat from core client for 30 sec - exiting 22:38:13 (7288): No heartbeat from core client for 30 sec - exiting 22:38:14 (7288): No heartbeat from core client for 30 sec - exiting 22:38:15 (7288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:16 (7288): No heartbeat from core client for 30 sec - exiting 22:38:17 (7288): No heartbeat from core client for 30 sec - exiting 22:38:18 (7288): No heartbeat from core client for 30 sec - exiting 22:38:19 (7288): No heartbeat from core client for 30 sec - exiting 22:38:20 (7288): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5704, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7048, iMonCtr=1 Model crash detected, will try to restart... 12:44:11 (5428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4488, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4488, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4488, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8080, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4784, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:18:11 (6256): No heartbeat from core client for 30 sec - exiting 20:18:12 (6256): No heartbeat from core client for 30 sec - exiting 20:18:13 (6256): No heartbeat from core client for 30 sec - exiting 20:18:14 (6256): No heartbeat from core client for 30 sec - exiting 20:18:15 (6256): No heartbeat from core client for 30 sec - exiting 20:18:16 (6256): No heartbeat from core client for 30 sec - exiting 20:18:18 (6256): No heartbeat from core client for 30 sec - exiting 20:18:19 (6256): No heartbeat from core client for 30 sec - exiting 20:18:20 (6256): No heartbeat from core client for 30 sec - exiting 20:18:21 (6256): No heartbeat from core client for 30 sec - exiting 20:18:22 (6256): No heartbeat from core client for 30 sec - exiting 20:18:23 (6256): No heartbeat from core client for 30 sec - exiting 20:18:24 (6256): No heartbeat from core client for 30 sec - exiting 20:18:25 (6256): No heartbeat from core client for 30 sec - exiting 20:18:26 (6256): No heartbeat from core client for 30 sec - exiting 20:18:27 (6256): No heartbeat from core client for 30 sec - exiting 20:18:28 (6256): No heartbeat from core client for 30 sec - exiting 20:18:30 (6256): No heartbeat from core client for 30 sec - exiting 20:18:31 (6256): No heartbeat from core client for 30 sec - exiting 20:18:32 (6256): No heartbeat from core client for 30 sec - exiting 20:18:33 (6256): No heartbeat from core client for 30 sec - exiting 20:18:34 (6256): No heartbeat from core client for 30 sec - exiting 20:18:35 (6256): No heartbeat from core client for 30 sec - exiting 20:18:36 (6256): No heartbeat from core client for 30 sec - exiting 20:18:37 (6256): No heartbeat from core client for 30 sec - exiting 20:18:38 (6256): No heartbeat from core client for 30 sec - exiting 20:18:39 (6256): No heartbeat from core client for 30 sec - exiting 20:18:40 (6256): No heartbeat from core client for 30 sec - exiting 20:18:41 (6256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6420, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5020, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5512, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4972, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3676, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4988, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5500, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 10:44:11 (200): No heartbeat from core client for 30 sec - exiting 10:44:12 (200): No heartbeat from core client for 30 sec - exiting 10:44:13 (200): No heartbeat from core client for 30 sec - exiting 10:44:14 (200): No heartbeat from core client for 30 sec - exiting 10:44:16 (200): No heartbeat from core client for 30 sec - exiting 10:44:17 (200): No heartbeat from core client for 30 sec - exiting 10:44:18 (200): No heartbeat from core client for 30 sec - exiting 10:44:19 (200): No heartbeat from core client for 30 sec - exiting 10:44:20 (200): No heartbeat from core client for 30 sec - exiting 10:44:21 (200): No heartbeat from core client for 30 sec - exiting 10:44:22 (200): No heartbeat from core client for 30 sec - exiting 10:44:23 (200): No heartbeat from core client for 30 sec - exiting 10:44:24 (200): No heartbeat from core client for 30 sec - exiting 10:44:25 (200): No heartbeat from core client for 30 sec - exiting 10:44:27 (200): No heartbeat from core client for 30 sec - exiting 10:44:28 (200): No heartbeat from core client for 30 sec - exiting 10:44:29 (200): No heartbeat from core client for 30 sec - exiting 10:44:30 (200): No heartbeat from core client for 30 sec - exiting 10:44:31 (200): No heartbeat from core client for 30 sec - exiting 10:44:32 (200): No heartbeat from core client for 30 sec - exiting 10:44:33 (200): No heartbeat from core client for 30 sec - exiting 10:44:34 (200): No heartbeat from core client for 30 sec - exiting 10:44:35 (200): No heartbeat from core client for 30 sec - exiting 10:44:36 (200): No heartbeat from core client for 30 sec - exiting 10:44:37 (200): No heartbeat from core client for 30 sec - exiting 10:44:38 (200): No heartbeat from core client for 30 sec - exiting 10:44:39 (200): No heartbeat from core client for 30 sec - exiting 10:44:40 (200): No heartbeat from core client for 30 sec - exiting 10:44:42 (200): No heartbeat from core client for 30 sec - exiting 10:44:43 (200): No heartbeat from core client for 30 sec - exiting 10:44:44 (200): No heartbeat from core client for 30 sec - exiting 10:44:45 (200): No heartbeat from core client for 30 sec - exiting 10:44:46 (200): No heartbeat from core client for 30 sec - exiting 10:44:47 (200): No heartbeat from core client for 30 sec - exiting 10:44:48 (200): No heartbeat from core client for 30 sec - exiting 10:44:49 (200): No heartbeat from core client for 30 sec - exiting 10:44:50 (200): No heartbeat from core client for 30 sec - exiting 10:44:51 (200): No heartbeat from core client for 30 sec - exiting 10:44:52 (200): No heartbeat from core client for 30 sec - exiting 10:44:53 (200): No heartbeat from core client for 30 sec - exiting 10:44:54 (200): No heartbeat from core client for 30 sec - exiting 10:44:55 (200): No heartbeat from core client for 30 sec - exiting 10:44:56 (200): No heartbeat from core client for 30 sec - exiting 10:44:57 (200): No heartbeat from core client for 30 sec - exiting 10:44:58 (200): No heartbeat from core client for 30 sec - exiting 10:44:59 (200): No heartbeat from core client for 30 sec - exiting 10:45:00 (200): No heartbeat from core client for 30 sec - exiting 10:45:01 (200): No heartbeat from core client for 30 sec - exiting 10:45:02 (200): No heartbeat from core client for 30 sec - exiting 10:45:03 (200): No heartbeat from core client for 30 sec - exiting 10:45:04 (200): No heartbeat from core client for 30 sec - exiting 10:45:05 (200): No heartbeat from core client for 30 sec - exiting 10:45:06 (200): No heartbeat from core client for 30 sec - exiting 10:45:07 (200): No heartbeat from core client for 30 sec - exiting 10:45:08 (200): No heartbeat from core client for 30 sec - exiting 10:45:09 (200): No heartbeat from core client for 30 sec - exiting 10:45:10 (200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:45:12 (200): No heartbeat from core client for 30 sec - exiting 10:45:13 (200): No heartbeat from core client for 30 sec - exiting 10:45:14 (200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:56:24 (7556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:56:26 (7556): No heartbeat from core client for 30 sec - exiting 22:56:27 (7556): No heartbeat from core client for 30 sec - exiting 22:56:28 (7556): No heartbeat from core client for 30 sec - exiting 22:56:29 (7556): No heartbeat from core client for 30 sec - exiting 22:56:30 (7556): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8780, iMonCtr=1 Model crash dController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4208, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4628, iMonCtr=1 Model crash detected, will try to restart... 12:55:22 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5188, iMonCtr=1 Model crash detected, will try to restart... 01:53:49 (4856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5704, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4764, iMonCtr=1 Model crash detected, will try to restart... 19:12:09 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:11:42 (6176): No heartbeat from core client for 30 sec - exiting 21:11:43 (6176): No heartbeat from core client for 30 sec - exiting 21:11:44 (6176): No heartbeat from core client for 30 sec - exiting 21:11:45 (6176): No heartbeat from core client for 30 sec - exiting 21:11:46 (6176): No heartbeat from core client for 30 sec - exiting 21:11:47 (6176): No heartbeat from core client for 30 sec - exiting 21:12:19 (6176): No heartbeat from core client for 30 sec - exiting 21:12:20 (6176): No heartbeat from core client for 30 sec - exiting 21:12:21 (6176): No heartbeat from core client for 30 sec - exiting 21:12:22 (6176): No heartbeat from core client for 30 sec - exiting 21:12:23 (6176): No heartbeat from core client for 30 sec - exiting 21:12:24 (6176): No heartbeat from core client for 30 sec - exiting 21:12:25 (6176): No heartbeat from core client for 30 sec - exiting 21:12:26 (6176): No heartbeat from core client for 30 sec - exiting 21:12:27 (6176): No heartbeat from core client for 30 sec - exiting 21:12:28 (6176): No heartbeat from core client for 30 sec - exiting 21:12:29 (6176): No heartbeat from core client for 30 sec - exiting 21:12:30 (6176): No heartbeat from core client for 30 sec - exiting 21:12:31 (6176): No heartbeat from core client for 30 sec - exiting 21:12:32 (6176): No heartbeat from core client for 30 sec - exiting 21:12:33 (6176): No heartbeat from core client for 30 sec - exiting 21:12:34 (6176): No heartbeat from core client for 30 sec - exiting 21:12:35 (6176): No heartbeat from core client for 30 sec - exiting 21:12:36 (6176): No heartbeat from core client for 30 sec - exiting 21:12:37 (6176): No heartbeat from core client for 30 sec - exiting 21:12:38 (6176): No heartbeat from core client for 30 sec - exiting 21:12:39 (6176): No heartbeat from core client for 30 sec - exiting 21:12:41 (6176): No heartbeat from core client for 30 sec - exiting 21:12:42 (6176): No heartbeat from core client for 30 sec - exiting 21:12:43 (6176): No heartbeat from core client for 30 sec - exiting 21:12:44 (6176): No heartbeat from core client for 30 sec - exiting 21:12:45 (6176): No heartbeat from core client for 30 sec - exiting 21:12:46 (6176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:43 (4808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:45:45 (4808): No heartbeat from core client for 30 sec - exiting 18:45:46 (4808): No heartbeat from core client for 30 sec - exiting 18:51:23 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6088, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4976, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4976, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4764, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:50:43 (9084): No heartbeat from core client for 30 sec - exiting 17:50:52 (9084): No heartbeat from core client for 30 sec - exiting 17:50:53 (9084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4472, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5288, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4184, iMonCtr=1 Model crash detected, will try to restart... 19:49:50 (3420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5216, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4256, iMonCtr=1 Model crash detected, will try to restart... 01:36:12 (5012): No heartbeat from core client for 30 sec - exiting 01:36:13 (5012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4808, iMonCtr=1 Model crash detected, will try to restart... 23:16:32 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4792, iMonCtr=1 Model crash detected, will try to restart... 22:38:45 (4716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4924, iMonCtr=1 Model crash detected, will try to restart... 19:21:25 (5112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4648, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3640, iMonCtr=1 Model crash detected, will try to restart... 19:21:14 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:26:20 (3832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5228, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 16:40:09 (6792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:40:18 (6792): No heartbeat from core client for 30 sec - exiting 16:40:19 (6792): No heartbeat from core client for 30 sec - exiting 16:40:20 (6792): No heartbeat from core client for 30 sec - exiting 16:40:21 (6792): No heartbeat from core client for 30 sec - exiting 16:40:22 (6792): No heartbeat from core client for 30 sec - exiting 16:40:23 (6792): No heartbeat from core client for 30 sec - exiting 16:40:24 (6792): No heartbeat from core client for 30 sec - exiting 16:40:25 (6792): No heartbeat from core client for 30 sec - exiting 16:40:26 (6792): No heartbeat from core client for 30 sec - exiting 16:40:27 (6792): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5724, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7272, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3304, iMonCtr=1 Model crash detected, will try to restart... Signal 11 received, exiting... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Dec 2013 15:43:10	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	259,200	1,170,828	4.5171
18 Dec 2013 00:24:06	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	233,280	1,054,931	4.5222
09 Dec 2013 22:42:47	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	207,360	938,842	4.5276
03 Dec 2013 21:41:45	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	181,440	824,295	4.5431
28 Nov 2013 20:36:29	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	155,520	709,402	4.5615
20 Nov 2013 20:50:14	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	129,600	590,014	4.5526
14 Nov 2013 23:58:53	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	103,680	474,576	4.5773
08 Nov 2013 23:44:07	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	77,760	357,805	4.6014
01 Nov 2013 18:42:44	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	51,840	238,347	4.5977
26 Oct 2013 12:28:40	1186013	16069904	hadcm3n_oczj_1900_40_008471954_1	25,920	119,539	4.6118