Task 13474364

Name	hadam3p_saf_288g_1997_1_007485449_0
Workunit	7682924
Created	11 Oct 2011, 12:38:22 UTC
Sent	11 Oct 2011, 12:38:41 UTC
Report deadline	22 Sep 2012, 17:58:41 UTC
Received	17 Nov 2012, 16:05:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1166994
Run time	5 days 3 hours 51 min 35 sec
CPU time	4 days 14 hours 6 min 32 sec
Validate state	Invalid
Credit	1,122.95
Device peak FLOPS	2.10 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Southern Africa v6.09 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... 21:30:38 (4672): No heartbeat from core client for 30 sec - exiting 21:30:39 (4672): No heartbeat from core client for 30 sec - exiting 21:30:40 (4672): No heartbeat from core client for 30 sec - exiting 21:30:41 (4672): No heartbeat from core client for 30 sec - exiting 21:30:42 (4672): No heartbeat from core client for 30 sec - exiting 21:30:43 (4672): No heartbeat from core client for 30 sec - exiting 21:30:44 (4672): No heartbeat from core client for 30 sec - exiting 21:30:45 (4672): No heartbeat from core client for 30 sec - exiting 21:30:46 (4672): No heartbeat from core client for 30 sec - exiting 21:30:47 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9112, selfPID=11116, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7564, selfPID=7564, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN prCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9072, selfPID=9072, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11304, selfPID=11304, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11464, selfPID=11464, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11712, selfPID=11712, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1328, selfPID=1328, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:05:54 (5060): Can't acquire lockfile (32) - waiting 35s 03:06:55 (11200): Can't acquire lockfile (32) - waiting 35s CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDNCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11752, selfPID=3352, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not runniCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=16452, selfPID=16452, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=21016, selfPID=21016, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=41612, selfPID=74180, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9352, selfPID=5324, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=24604, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=24616, selfPID=21116, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 18:28:40 (2620): No heartbeat from core client for 30 sec - exiting 18:28:41 (2620): No heartbeat from core client for 30 sec - exiting 18:28:43 (2620): No heartbeat from core client for 30 sec - exiting 18:28:44 (2620): No heartbeat from core client for 30 sec - exiting 18:28:45 (2620): No heartbeat from core client for 30 sec - exiting 18:28:46 (2620): No heartbeat from core client for 30 sec - exiting 18:28:47 (2620): No heartbeat from core client for 30 sec - exiting 18:28:48 (2620): No heartbeat from core client for 30 sec - exiting 18:28:49 (2620): No heartbeat from core client for 30 sec - exiting 18:28:50 (2620): No heartbeat from core client for 30 sec - exiting 18:28:51 (2620): No heartbeat from core client for 30 sec - exiting 18:28:52 (2620): No heartbeat from core client for 30 sec - exiting 18:28:53 (2620): No heartbeat from core client for 30 sec - exiting 18:28:55 (2620): No heartbeat from core client for 30 sec - exiting 18:28:56 (2620): No heartbeat from core client for 30 sec - exiting 18:28:57 (2620): No heartbeat from core client for 30 sec - exiting 18:28:58 (2620): No heartbeat from core client for 30 sec - exiting 18:28:59 (2620): No heartbeat from core client for 30 sec - exiting 18:29:00 (2620): No heartbeat from core client for 30 sec - exiting 18:29:01 (2620): No heartbeat from core client for 30 sec - exiting 18:29:02 (2620): No heartbeat from core client for 30 sec - exiting 18:29:03 (2620): No heartbeat from core client for 30 sec - exiting 18:29:04 (2620): No heartbeat from core client for 30 sec - exiting 18:29:05 (2620): No heartbeat from core client for 30 sec - exiting 18:29:07 (2620): No heartbeat from core client for 30 sec - exiting 18:29:08 (2620): No heartbeat from core client for 30 sec - exiting 18:29:09 (2620): No heartbeat from core client for 30 sec - exiting 18:29:10 (2620): No heartbeat from core client for 30 sec - exiting 18:29:11 (2620): No heartbeat from core client for 30 sec - exiting 18:29:12 (2620): No heartbeat from core client for 30 sec - exiting 18:29:13 (2620): No heartbeat from core client for 30 sec - exiting 18:29:14 (2620): No heartbeat from core client for 30 sec - exiting 18:29:15 (2620): No heartbeat from core client for 30 sec - exiting 18:29:16 (2620): No heartbeat from core client for 30 sec - exiting 18:29:18 (2620): No heartbeat from core client for 30 sec - exiting 18:29:19 (2620): No heartbeat from core client for 30 sec - exiting 18:29:20 (2620): No heartbeat from core client for 30 sec - exiting 18:29:21 (2620): No heartbeat from core client for 30 sec - exiting 18:29:22 (2620): No heartbeat from core client for 30 sec - exiting 18:29:23 (2620): No heartbeat from core client for 30 sec - exiting 18:29:24 (2620): No heartbeat from core client for 30 sec - exiting 18:29:26 (2620): No heartbeat from core client for 30 sec - exiting 18:29:27 (2620): No heartbeat from core client for 30 sec - exiting 18:29:28 (2620): No heartbeat from core client for 30 sec - exiting 18:29:29 (2620): No heartbeat from core client for 30 sec - exiting 18:29:30 (2620): No heartbeat from core client for 30 sec - exiting 18:29:31 (2620): No heartbeat from core client for 30 sec - exiting 18:29:32 (2620): No heartbeat from core client for 30 sec - exiting 18:29:33 (2620): No heartbeat from core client for 30 sec - exiting 18:29:34 (2620): No heartbeat from core client for 30 sec - exiting 18:29:35 (2620): No heartbeat from core client for 30 sec - exiting 18:29:37 (2620): No heartbeat from core client for 30 sec - exiting 18:29:38 (2620): No heartbeat from core client for 30 sec - exiting 18:29:39 (2620): No heartbeat from core client for 30 sec - exiting 18:29:40 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5448, selfPID=4704, iMonCtr=1 Model crash detected, will try to restart... 21:14:26 (3364): No heartbeat from core client for 30 sec - exiting 21:14:27 (3364): No heartbeat from core client for 30 sec - exiting 21:14:28 (3364): No heartbeat from core client for 30 sec - exiting 21:14:29 (3364): No heartbeat from core client for 30 sec - exiting 21:14:30 (3364): No heartbeat from core client for 30 sec - exiting 21:14:31 (3364): No heartbeat from core client for 30 sec - exiting 21:14:32 (3364): No heartbeat from core client for 30 sec - exiting 21:14:33 (3364): No heartbeat from core client for 30 sec - exiting 21:14:34 (3364): No heartbeat from core client for 30 sec - exiting 21:14:35 (3364): No heartbeat from core client for 30 sec - exiting 21:14:37 (3364): No heartbeat from core client for 30 sec - exiting 21:14:38 (3364): No heartbeat from core client for 30 sec - exiting 21:14:39 (3364): No heartbeat from core client for 30 sec - exiting 21:14:40 (3364): No heartbeat from core client for 30 sec - exiting 21:14:41 (3364): No heartbeat from core client for 30 sec - exiting 21:14:42 (3364): No heartbeat from core client for 30 sec - exiting 21:14:43 (3364): No heartbeat from core client for 30 sec - exiting 21:14:44 (3364): No heartbeat from core client for 30 sec - exiting 21:14:45 (3364): No heartbeat from core client for 30 sec - exiting 21:14:46 (3364): No heartbeat from core client for 30 sec - exiting 21:14:47 (3364): No heartbeat from core client for 30 sec - exiting 21:14:49 (3364): No heartbeat from core client for 30 sec - exiting 21:14:50 (3364): No heartbeat from core client for 30 sec - exiting 21:14:51 (3364): No heartbeat from core client for 30 sec - exiting 21:14:52 (3364): No heartbeat from core client for 30 sec - exiting 21:14:53 (3364): No heartbeat from core client for 30 sec - exiting 21:14:54 (3364): No heartbeat from core client for 30 sec - exiting 21:14:55 (3364): No heartbeat from core client for 30 sec - exiting 21:14:56 (3364): No heartbeat from core client for 30 sec - exiting 21:14:57 (3364): No heartbeat from core client for 30 sec - exiting 21:14:58 (3364): No heartbeat from core client for 30 sec - exiting 21:14:59 (3364): No heartbeat from core client for 30 sec - exiting 21:15:01 (3364): No heartbeat from core client for 30 sec - exiting 21:15:02 (3364): No heartbeat from core client for 30 sec - exiting 21:15:03 (3364): No heartbeat from core client for 30 sec - exiting 21:15:04 (3364): No heartbeat from core client for 30 sec - exiting 21:15:05 (3364): No heartbeat from core client for 30 sec - exiting 21:15:06 (3364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20032, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2424, selfPID=6012, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 23:29:00 (5764): No heartbeat from core client for 30 sec - exiting 23:29:01 (5764): No heartbeat from core client for 30 sec - exiting 23:29:02 (5764): No heartbeat from core client for 30 sec - exiting 23:29:03 (5764): No heartbeat from core client for 30 sec - exiting 23:29:05 (5764): No heartbeat from core client for 30 sec - exiting 23:29:06 (5764): No heartbeat from core client for 30 sec - exiting 23:29:07 (5764): No heartbeat from core client for 30 sec - exiting 23:29:08 (5764): No heartbeat from core client for 30 sec - exiting 23:29:09 (5764): No heartbeat from core client for 30 sec - exiting 23:29:10 (5764): No heartbeat from core client for 30 sec - exiting 23:29:11 (5764): No heartbeat from core client for 30 sec - exiting 23:29:12 (5764): No heartbeat from core client for 30 sec - exiting 23:29:13 (5764): No heartbeat from core client for 30 sec - exiting 23:29:14 (5764): No heartbeat from core client for 30 sec - exiting 23:29:15 (5764): No heartbeat from core client for 30 sec - exiting 23:29:17 (5764): No heartbeat from core client for 30 sec - exiting 23:29:18 (5764): No heartbeat from core client for 30 sec - exiting 23:29:19 (5764): No heartbeat from core client for 30 sec - exiting 23:29:20 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5508, selfPID=3844, iMonCtr=1 Model crash detected, will try to restart... 23:41:05 (6036): No heartbeat from core client for 30 sec - exiting 23:41:06 (6036): No heartbeat from core client for 30 sec - exiting 23:41:07 (6036): No heartbeat from core client for 30 sec - exiting 23:41:08 (6036): No heartbeat from core client for 30 sec - exiting 23:41:09 (6036): No heartbeat from core client for 30 sec - exiting 23:41:11 (6036): No heartbeat from core client for 30 sec - exiting 23:41:12 (6036): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7032, selfPID=6588, iMonCtr=1 Model crash detected, will try to restart... 23:50:32 (3072): No heartbeat from core client for 30 sec - exiting 23:50:35 (3072): No heartbeat from core client for 30 sec - exiting 23:50:36 (3072): No heartbeat from core client for 30 sec - exiting 23:50:37 (3072): No heartbeat from core client for 30 sec - exiting 23:50:38 (3072): No heartbeat from core client for 30 sec - exiting 23:50:39 (3072): No heartbeat from core client for 30 sec - exiting 23:50:40 (3072): No heartbeat from core client for 30 sec - exiting 23:50:41 (3072): No heartbeat from core client for 30 sec - exiting 23:50:42 (3072): No heartbeat from core client for 30 sec - exiting 23:50:43 (3072): No heartbeat from core client for 30 sec - exiting 23:50:44 (3072): No heartbeat from core client for 30 sec - exiting 23:50:45 (3072): No heartbeat from core client for 30 sec - exiting 23:50:47 (3072): No heartbeat from core client for 30 sec - exiting 23:50:48 (3072): No heartbeat from core client for 30 sec - exiting 23:50:49 (3072): No heartbeat from core client for 30 sec - exiting 23:50:50 (3072): No heartbeat from core client for 30 sec - exiting 23:50:51 (3072): No heartbeat from core client for 30 sec - exiting 23:50:52 (3072): No heartbeat from core client for 30 sec - exiting 23:50:53 (3072): No heartbeat from core client for 30 sec - exiting 23:50:54 (3072): No heartbeat from core client for 30 sec - exiting 23:50:55 (3072): No heartbeat from core client for 30 sec - exiting 23:52:13 (3072): No heartbeat from core client for 30 sec - exiting 23:52:14 (3072): No heartbeat from core client for 30 sec - exiting 23:52:15 (3072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:03:31 (3360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_saf_288g_1997_1_007485449_0_7.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_288g_1997_1_007485449_0_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_288g_1997_1_007485449_0_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_288g_1997_1_007485449_0_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_288g_1997_1_007485449_0_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_saf_288g_1997_1_007485449_0_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Nov 2012 18:15:13	1166994	13474364	hadam3p_saf_288g_1997_1_007485449_0	69,224	362,415	5.2354
14 Nov 2012 03:38:28	1166994	13474364	hadam3p_saf_288g_1997_1_007485449_0	69,216	361,704	5.2257
11 Nov 2012 19:54:31	1166994	13474364	hadam3p_saf_288g_1997_1_007485449_0	57,696	298,818	5.1792
09 Nov 2012 21:28:56	1166994	13474364	hadam3p_saf_288g_1997_1_007485449_0	46,176	238,310	5.1609
08 Nov 2012 07:54:27	1166994	13474364	hadam3p_saf_288g_1997_1_007485449_0	34,656	173,073	4.9940
07 Nov 2012 09:57:48	1166994	13474364	hadam3p_saf_288g_1997_1_007485449_0	23,136	105,393	4.5554
16 Oct 2011 19:02:30	1166994	13474364	hadam3p_saf_288g_1997_1_007485449_0	11,616	45,685	3.9329