Task 16452064

Name	hadam3p_anz_o0ix_2012_1_008620906_0
Workunit	8767418
Created	2 Apr 2014, 16:42:42 UTC
Sent	21 Apr 2014, 6:45:29 UTC
Report deadline	3 Apr 2015, 12:05:29 UTC
Received	3 Aug 2014, 18:18:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1281494
Run time	7 days 10 hours 46 min 35 sec
CPU time	6 days 8 hours 26 min 39 sec
Validate state	Invalid
Credit	3,987.46
Device peak FLOPS	3.14 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <stderr_txt> Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6252, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2824, selfPID=2824, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7488, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7576, selfPID=6928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5852, selfPID=4128, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=364, selfPID=4248, iMonCtr=1 Model crash detected, will try to restart... 17:48:31 (4828): No heartbeat from core client for 30 sec - exiting 17:48:32 (4828): No heartbeat from core client for 30 sec - exiting 17:48:34 (4828): No heartbeat from core client for 30 sec - exiting 17:48:35 (4828): No heartbeat from core client for 30 sec - exiting 17:48:36 (4828): No heartbeat from core client for 30 sec - exiting 17:48:37 (4828): No heartbeat from core client for 30 sec - exiting 17:48:38 (4828): No heartbeat from core client for 30 sec - exiting 17:48:39 (4828): No heartbeat from core client for 30 sec - exiting 17:48:40 (4828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:50:48 (2700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4828, selfPID=4828, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8400, selfPID=3304, iMonCtr=1 Model crash detected, will try to restart... 13:42:01 (5468): No heartbeat from core client for 30 sec - exiting 13:42:02 (5468): No heartbeat from core client for 30 sec - exiting 13:42:03 (5468): No heartbeat from core client for 30 sec - exiting 13:42:04 (5468): No heartbeat from core client for 30 sec - exiting 13:42:05 (5468): No heartbeat from core client for 30 sec - exiting 13:42:06 (5468): No heartbeat from core client for 30 sec - exiting 13:42:07 (5468): No heartbeat from core client for 30 sec - exiting 13:42:08 (5468): No heartbeat from core client for 30 sec - exiting 13:42:09 (5468): No heartbeat from core client for 30 sec - exiting 13:42:10 (5468): No heartbeat from core client for 30 sec - exiting 13:42:11 (5468): No heartbeat from core client for 30 sec - exiting 13:42:12 (5468): No heartbeat from core client for 30 sec - exiting 13:42:13 (5468): No heartbeat from core client for 30 sec - exiting 13:42:14 (5468): No heartbeat from core client for 30 sec - exiting 13:42:15 (5468): No heartbeat from core client for 30 sec - exiting 13:42:16 (5468): No heartbeat from core client for 30 sec - exiting 13:42:17 (5468): No heartbeat from core client for 30 sec - exiting 13:42:18 (5468): No heartbeat from core client for 30 sec - exiting 13:42:19 (5468): No heartbeat from core client for 30 sec - exiting 13:42:20 (5468): No heartbeat from core client for 30 sec - exiting 13:42:21 (5468): No heartbeat from core client for 30 sec - exiting 13:42:22 (5468): No heartbeat from core client for 30 sec - exiting 13:42:23 (5468): No heartbeat from core client for 30 sec - exiting 13:42:24 (5468): No heartbeat from core client for 30 sec - exiting 13:42:25 (5468): No heartbeat from core client for 30 sec - exiting 13:42:26 (5468): No heartbeat from core client for 30 sec - exiting 13:42:27 (5468): No heartbeat from core client for 30 sec - exiting 13:42:28 (5468): No heartbeat from core client for 30 sec - exiting 13:42:29 (5468): No heartbeat from core client for 30 sec - exiting 13:42:30 (5468): No heartbeat from core client for 30 sec - exiting 13:42:31 (5468): No heartbeat from core client for 30 sec - exiting 13:42:32 (5468): No heartbeat from core client for 30 sec - exiting 13:42:33 (5468): No heartbeat from core client for 30 sec - exiting 13:42:34 (5468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8248, selfPID=6892, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:52:07 (9108): No heartbeat from core client for 30 sec - exiting 20:52:08 (9108): No heartbeat from core client for 30 sec - exiting 20:52:09 (9108): No heartbeat from core client for 30 sec - exiting 20:52:10 (9108): No heartbeat from core client for 30 sec - exiting 20:52:11 (9108): No heartbeat from core client for 30 sec - exiting 20:52:12 (9108): No heartbeat from core client for 30 sec - exiting 20:52:13 (9108): No heartbeat from core client for 30 sec - exiting 20:52:14 (9108): No heartbeat from core client for 30 sec - exiting 20:52:15 (9108): No heartbeat from core client for 30 sec - exiting 20:52:16 (9108): No heartbeat from core client for 30 sec - exiting 20:52:17 (9108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:05:48 (1744): No heartbeat from core client for 30 sec - exiting 20:05:49 (1744): No heartbeat from core client for 30 sec - exiting 20:05:50 (1744): No heartbeat from core client for 30 sec - exiting 20:05:52 (1744): No heartbeat from core client for 30 sec - exiting 20:05:53 (1744): No heartbeat from core client for 30 sec - exiting 20:05:54 (1744): No heartbeat from core client for 30 sec - exiting 20:05:55 (1744): No heartbeat from core client for 30 sec - exiting 20:05:56 (1744): No heartbeat from core client for 30 sec - exiting 20:05:57 (1744): No heartbeat from core client for 30 sec - exiting 20:05:58 (1744): No heartbeat from core client for 30 sec - exiting 20:05:59 (1744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9292, selfPID=8780, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1876, selfPID=4828, iMonCtr=1 Model crash detected, will try to restart... 17:19:41 (5780): No heartbeat from core client for 30 sec - exiting 17:19:42 (5780): No heartbeat from core client for 30 sec - exiting 17:19:43 (5780): No heartbeat from core client for 30 sec - exiting 17:19:45 (5780): No heartbeat from core client for 30 sec - exiting 17:19:46 (5780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4972, selfPID=4460, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6212, selfPID=1376, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 17:22:53 (368): No heartbeat from core client for 30 sec - exiting 17:22:54 (368): No heartbeat from core client for 30 sec - exiting 17:22:55 (368): No heartbeat from core client for 30 sec - exiting 17:22:56 (368): No heartbeat from core client for 30 sec - exiting 17:22:57 (368): No heartbeat from core client for 30 sec - exiting 17:22:58 (368): No heartbeat from core client for 30 sec - exiting 17:22:59 (368): No heartbeat from core client for 30 sec - exiting 17:23:00 (368): No heartbeat from core client for 30 sec - exiting 17:23:01 (368): No heartbeat from core client for 30 sec - exiting 17:23:02 (368): No heartbeat from core client for 30 sec - exiting 17:23:03 (368): No heartbeat from core client for 30 sec - exiting 17:23:04 (368): No heartbeat from core client for 30 sec - exiting 17:23:05 (368): No heartbeat from core client for 30 sec - exiting 17:23:06 (368): No heartbeat from core client for 30 sec - exiting 17:23:07 (368): No heartbeat from core client for 30 sec - exiting 17:23:08 (368): No heartbeat from core client for 30 sec - exiting 17:23:09 (368): No heartbeat from core client for 30 sec - exiting 17:23:10 (368): No heartbeat from core client for 30 sec - exiting 17:23:11 (368): No heartbeat from core client for 30 sec - exiting 17:23:12 (368): No heartbeat from core client for 30 sec - exiting 17:23:13 (368): No heartbeat from core client for 30 sec - exiting 17:23:14 (368): No heartbeat from core client for 30 sec - exiting 17:23:15 (368): No heartbeat from core client for 30 sec - exiting 17:23:16 (368): No heartbeat from core client for 30 sec - exiting 17:23:17 (368): No heartbeat from core client for 30 sec - exiting 17:23:18 (368): No heartbeat from core client for 30 sec - exiting 17:23:19 (368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7396, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7440, selfPID=7148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6188, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8856, selfPID=700, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7388, selfPID=5676, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 17:22:05 (5504): No heartbeat from core client for 30 sec - exiting 17:22:06 (5504): No heartbeat from core client for 30 sec - exiting 17:22:07 (5504): No heartbeat from core client for 30 sec - exiting 17:22:08 (5504): No heartbeat from core client for 30 sec - exiting 17:22:09 (5504): No heartbeat from core client for 30 sec - exiting 17:22:10 (5504): No heartbeat from core client for 30 sec - exiting 17:22:11 (5504): No heartbeat from core client for 30 sec - exiting 17:22:12 (5504): No heartbeat from core client for 30 sec - exiting 17:22:14 (5504): No heartbeat from core client for 30 sec - exiting 17:22:15 (5504): No heartbeat from core client for 30 sec - exiting 17:22:16 (5504): No heartbeat from core client for 30 sec - exiting 17:22:17 (5504): No heartbeat from core client for 30 sec - exiting 17:22:18 (5504): No heartbeat from core client for 30 sec - exiting 17:22:19 (5504): No heartbeat from core client for 30 sec - exiting 17:22:20 (5504): No heartbeat from core client for 30 sec - exiting 17:22:21 (5504): No heartbeat from core client for 30 sec - exiting 17:22:22 (5504): No heartbeat from core client for 30 sec - exiting 17:22:23 (5504): No heartbeat from core client for 30 sec - exiting 17:22:24 (5504): No heartbeat from core client for 30 sec - exiting 17:22:26 (5504): No heartbeat from core client for 30 sec - exiting 17:22:27 (5504): No heartbeat from core client for 30 sec - exiting 17:22:28 (5504): No heartbeat from core client for 30 sec - exiting 17:22:29 (5504): No heartbeat from core client for 30 sec - exiting 17:22:30 (5504): No heartbeat from core client for 30 sec - exiting 17:22:31 (5504): No heartbeat from core client for 30 sec - exiting 17:22:32 (5504): No heartbeat from core client for 30 sec - exiting 17:22:33 (5504): No heartbeat from core client for 30 sec - exiting 17:22:34 (5504): No heartbeat from core client for 30 sec - exiting 17:22:35 (5504): No heartbeat from core client for 30 sec - exiting 17:22:36 (5504): No heartbeat from core client for 30 sec - exiting 17:22:38 (5504): No heartbeat from core client for 30 sec - exiting 17:22:39 (5504): No heartbeat from core client for 30 sec - exiting 17:22:40 (5504): No heartbeat from core client for 30 sec - exiting 17:22:41 (5504): No heartbeat from core client for 30 sec - exiting 17:22:42 (5504): No heartbeat from core client for 30 sec - exiting 17:22:43 (5504): No heartbeat from core client for 30 sec - exiting 17:22:44 (5504): No heartbeat from core client for 30 sec - exiting 17:22:45 (5504): No heartbeat from core client for 30 sec - exiting 17:22:46 (5504): No heartbeat from core client for 30 sec - exiting 17:22:47 (5504): No heartbeat from core client for 30 sec - exiting 17:22:49 (5504): No heartbeat from core client for 30 sec - exiting 17:22:50 (5504): No heartbeat from core client for 30 sec - exiting 17:22:51 (5504): No heartbeat from core client for 30 sec - exiting 17:22:52 (5504): No heartbeat from core client for 30 sec - exiting 17:22:53 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6304, selfPID=6304, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7388, selfPID=4008, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9432, selfPID=3104, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7828, selfPID=6368, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Precis Restart file copy #1 failed on o0ixga.dal33a0 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2952, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6848, selfPID=5004, iMonCtr=1 Model crash detected, will try to restart... 14:23:00 (5304): No heartbeat from core client for 30 sec - exiting 14:23:01 (5304): No heartbeat from core client for 30 sec - exiting 14:23:02 (5304): No heartbeat from core client for 30 sec - exiting 14:23:03 (5304): No heartbeat from core client for 30 sec - exiting 14:23:04 (5304): No heartbeat from core client for 30 sec - exiting 14:23:05 (5304): No heartbeat from core client for 30 sec - exiting 14:23:06 (5304): No heartbeat from core client for 30 sec - exiting 14:23:07 (5304): No heartbeat from core client for 30 sec - exiting 14:23:08 (5304): No heartbeat from core client for 30 sec - exiting 14:23:09 (5304): No heartbeat from core client for 30 sec - exiting 14:23:10 (5304): No heartbeat from core client for 30 sec - exiting 14:23:11 (5304): No heartbeat from core client for 30 sec - exiting 14:23:12 (5304): No heartbeat from core client for 30 sec - exiting 14:23:13 (5304): No heartbeat from core client for 30 sec - exiting 14:23:14 (5304): No heartbeat from core client for 30 sec - exiting 14:23:15 (5304): No heartbeat from core client for 30 sec - exiting 14:23:16 (5304): No heartbeat from core client for 30 sec - exiting 14:23:17 (5304): No heartbeat from core client for 30 sec - exiting 14:23:18 (5304): No heartbeat from core client for 30 sec - exiting 14:23:19 (5304): No heartbeat from core client for 30 sec - exiting 14:23:20 (5304): No heartbeat from core client for 30 sec - exiting 14:23:21 (5304): No heartbeat from core client for 30 sec - exiting 14:23:22 (5304): No heartbeat from core client for 30 sec - exiting 14:23:23 (5304): No heartbeat from core client for 30 sec - exiting 14:23:24 (5304): No heartbeat from core client for 30 sec - exiting 14:23:25 (5304): No heartbeat from core client for 30 sec - exiting 14:23:26 (5304): No heartbeat from core client for 30 sec - exiting 14:23:27 (5304): No heartbeat from core client for 30 sec - exiting 14:23:28 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2428, selfPID=2428, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5404, selfPID=3088, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6316, selfPID=7008, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8764, selfPID=6368, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3336, selfPID=2044, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5432, selfPID=5444, iMonCtr=1 Model crash detected, will try to restart... 17:23:09 (6584): No heartbeat from core client for 30 sec - exiting 17:23:11 (6584): No heartbeat from core client for 30 sec - exiting 17:23:12 (6584): No heartbeat from core client for 30 sec - exiting 17:23:13 (6584): No heartbeat from core client for 30 sec - exiting 17:23:14 (6584): No heartbeat from core client for 30 sec - exiting 17:23:15 (6584): No heartbeat from core client for 30 sec - exiting 17:23:16 (6584): No heartbeat from core client for 30 sec - exiting 17:23:17 (6584): No heartbeat from core client for 30 sec - exiting 17:23:18 (6584): No heartbeat from core client for 30 sec - exiting 17:23:19 (6584): No heartbeat from core client for 30 sec - exiting 17:23:20 (6584): No heartbeat from core client for 30 sec - exiting 17:23:21 (6584): No heartbeat from core client for 30 sec - exiting 17:23:23 (6584): No heartbeat from core client for 30 sec - exiting 17:23:24 (6584): No heartbeat from core client for 30 sec - exiting 17:23:25 (6584): No heartbeat from core client for 30 sec - exiting 17:23:26 (6584): No heartbeat from core client for 30 sec - exiting 17:23:27 (6584): No heartbeat from core client for 30 sec - exiting 17:26:00 (6584): No heartbeat from core client for 30 sec - exiting 17:26:01 (6584): No heartbeat from core client for 30 sec - exiting 17:26:02 (6584): No heartbeat from core client for 30 sec - exiting 17:26:03 (6584): No heartbeat from core client for 30 sec - exiting 17:26:04 (6584): No heartbeat from core client for 30 sec - exiting 17:26:05 (6584): No heartbeat from core client for 30 sec - exiting 17:26:06 (6584): No heartbeat from core client for 30 sec - exiting 17:26:07 (6584): No heartbeat from core client for 30 sec - exiting 17:26:09 (6584): No heartbeat from core client for 30 sec - exiting 17:26:10 (6584): No heartbeat from core client for 30 sec - exiting 17:26:11 (6584): No heartbeat from core client for 30 sec - exiting 17:26:12 (6584): No heartbeat from core client for 30 sec - exiting 17:26:13 (6584): No heartbeat from core client for 30 sec - exiting 17:26:14 (6584): No heartbeat from core client for 30 sec - exiting 17:26:15 (6584): No heartbeat from core client for 30 sec - exiting 17:26:16 (6584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3260, selfPID=3260, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7596, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5336, selfPID=3736, iMonCtr=1 Model crash detected, will try to restart... 12:27:53 (6544): No heartbeat from core client for 30 sec - exiting 12:27:54 (6544): No heartbeat from core client for 30 sec - exiting 12:27:55 (6544): No heartbeat from core client for 30 sec - exiting 12:27:56 (6544): No heartbeat from core client for 30 sec - exiting 12:27:57 (6544): No heartbeat from core client for 30 sec - exiting 12:27:58 (6544): No heartbeat from core client for 30 sec - exiting 12:27:59 (6544): No heartbeat from core client for 30 sec - exiting 12:28:00 (6544): No heartbeat from core client for 30 sec - exiting 12:28:01 (6544): No heartbeat from core client for 30 sec - exiting 12:28:03 (6544): No heartbeat from core client for 30 sec - exiting 12:28:04 (6544): No heartbeat from core client for 30 sec - exiting 12:28:05 (6544): No heartbeat from core client for 30 sec - exiting 12:28:06 (6544): No heartbeat from core client for 30 sec - exiting 12:28:07 (6544): No heartbeat from core client for 30 sec - exiting 12:28:08 (6544): No heartbeat from core client for 30 sec - exiting 12:28:09 (6544): No heartbeat from core client for 30 sec - exiting 12:28:10 (6544): No heartbeat from core client for 30 sec - exiting 12:28:11 (6544): No heartbeat from core client for 30 sec - exiting 12:28:12 (6544): No heartbeat from core client for 30 sec - exiting 12:28:13 (6544): No heartbeat from core client for 30 sec - exiting 12:28:15 (6544): No heartbeat from core client for 30 sec - exiting 12:28:16 (6544): No heartbeat from core client for 30 sec - exiting 12:28:17 (6544): No heartbeat from core client for 30 sec - exiting 12:28:18 (6544): No heartbeat from core client for 30 sec - exiting 12:28:19 (6544): No heartbeat from core client for 30 sec - exiting 12:28:20 (6544): No heartbeat from core client for 30 sec - exiting 12:28:21 (6544): No heartbeat from core client for 30 sec - exiting 12:28:22 (6544): No heartbeat from core client for 30 sec - exiting 12:28:23 (6544): No heartbeat from core client for 30 sec - exiting 12:28:24 (6544): No heartbeat from core client for 30 sec - exiting 12:28:25 (6544): No heartbeat from core client for 30 sec - exiting 12:28:27 (6544): No heartbeat from core client for 30 sec - exiting 12:28:28 (6544): No heartbeat from core client for 30 sec - exiting 12:28:29 (6544): No heartbeat from core client for 30 sec - exiting 12:28:30 (6544): No heartbeat from core client for 30 sec - exiting 12:28:31 (6544): No heartbeat from core client for 30 sec - exiting 12:28:32 (6544): No heartbeat from core client for 30 sec - exiting 12:28:33 (6544): No heartbeat from core client for 30 sec - exiting 12:28:34 (6544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=976, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5928, selfPID=5780, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6444, selfPID=3192, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6276, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7144, selfPID=8820, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadam3p_anz_o0ix_2012_1_008620906/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/hadam3p_anz_o0ix_2012_1_008620906/dataout/region_restart.day after 11 attempts Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/xaakm.pipe_dummy 2048 Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/xaakg.pipe_dummy 2048 Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_o0ix_2012_1_008620906_0_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o0ix_2012_1_008620906_0_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o0ix_2012_1_008620906_0_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o0ix_2012_1_008620906_0_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jul 2014 04:43:15	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	92,459	499,477	5.4021
05 Jul 2014 01:33:56	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	80,939	426,028	5.2636
29 Jun 2014 01:14:31	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	69,419	379,638	5.4688
28 Jun 2014 02:53:11	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	57,899	336,302	5.8084
23 Jun 2014 21:35:32	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	46,379	292,893	6.3152
10 Jun 2014 09:02:49	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	34,859	222,951	6.3958
27 May 2014 01:39:45	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	23,339	149,101	6.3885
10 May 2014 02:01:54	1281494	16452064	hadam3p_anz_o0ix_2012_1_008620906_0	11,819	75,353	6.3756