Task 16459024

Name	hadam3p_anz_o8dg_2012_1_008627634_0
Workunit	8774146
Created	2 Apr 2014, 19:41:26 UTC
Sent	14 Apr 2014, 9:20:16 UTC
Report deadline	27 Mar 2015, 14:40:16 UTC
Received	10 May 2014, 4:48:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1322841
Run time	7 days 17 hours 30 min 45 sec
CPU time	7 days 10 hours 35 min 51 sec
Validate state	Invalid
Credit	3,490.64
Device peak FLOPS	2.75 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.3.11</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6736, selfPID=12808, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6652, selfPID=4512, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=6732, iMonCtr=1 09:55:43 (3528): No heartbeat from core client for 30 sec - exiting 09:55:44 (3528): No heartbeat from core client for 30 sec - exiting 09:55:45 (3528): No heartbeat from core client for 30 sec - exiting 09:55:46 (3528): No heartbeat from core client for 30 sec - exiting 09:55:48 (3528): No heartbeat from core client for 30 sec - exiting 09:55:49 (3528): No heartbeat from core client for 30 sec - exiting 09:55:50 (3528): No heartbeat from core client for 30 sec - exiting 09:55:51 (3528): No heartbeat from core client for 30 sec - exiting 09:55:52 (3528): No heartbeat from core client for 30 sec - exiting 09:55:53 (3528): No heartbeat from core client for 30 sec - exiting 09:55:54 (3528): No heartbeat from core client for 30 sec - exiting 09:55:55 (3528): No heartbeat from core client for 30 sec - exiting 09:55:56 (3528): No heartbeat from core client for 30 sec - exiting 09:55:57 (3528): No heartbeat from core client for 30 sec - exiting 09:55:58 (3528): No heartbeat from core client for 30 sec - exiting 09:56:00 (3528): No heartbeat from core client for 30 sec - exiting 09:56:01 (3528): No heartbeat from core client for 30 sec - exiting 09:56:02 (3528): No heartbeat from core client for 30 sec - exiting 09:56:03 (3528): No heartbeat from core client for 30 sec - exiting 09:56:04 (3528): No heartbeat from core client for 30 sec - exiting 09:56:05 (3528): No heartbeat from core client for 30 sec - exiting 09:56:06 (3528): No heartbeat from core client for 30 sec - exiting 09:56:07 (3528): No heartbeat from core client for 30 sec - exiting 09:56:08 (3528): No heartbeat from core client for 30 sec - exiting 09:56:09 (3528): No heartbeat from core client for 30 sec - exiting 09:56:10 (3528): No heartbeat from core client for 30 sec - exiting 09:56:12 (3528): No heartbeat from core client for 30 sec - exiting Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:26:09 (4948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7448, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=584, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7688, selfPID=2336, iMonCtr=1 Model crash detected, will try to restart... 09:49:30 (4532): No heartbeat from core client for 30 sec - exiting 09:49:31 (4532): No heartbeat from core client for 30 sec - exiting 09:49:32 (4532): No heartbeat from core client for 30 sec - exiting 09:49:33 (4532): No heartbeat from core client for 30 sec - exiting 09:49:35 (4532): No heartbeat from core client for 30 sec - exiting 09:49:36 (4532): No heartbeat from core client for 30 sec - exiting 09:49:37 (4532): No heartbeat from core client for 30 sec - exiting 09:49:38 (4532): No heartbeat from core client for 30 sec - exiting 09:49:39 (4532): No heartbeat from core client for 30 sec - exiting 09:49:40 (4532): No heartbeat from core client for 30 sec - exiting 09:49:41 (4532): No heartbeat from core client for 30 sec - exiting 09:49:42 (4532): No heartbeat from core client for 30 sec - exiting 09:49:43 (4532): No heartbeat from core client for 30 sec - exiting 09:49:44 (4532): No heartbeat from core client for 30 sec - exiting 09:49:45 (4532): No heartbeat from core client for 30 sec - exiting 09:49:47 (4532): No heartbeat from core client for 30 sec - exiting 09:49:48 (4532): No heartbeat from core client for 30 sec - exiting 09:49:49 (4532): No heartbeat from core client for 30 sec - exiting 09:49:50 (4532): No heartbeat from core client for 30 sec - exiting 09:49:51 (4532): No heartbeat from core client for 30 sec - exiting 09:49:52 (4532): No heartbeat from core client for 30 sec - exiting 09:49:53 (4532): No heartbeat from core client for 30 sec - exiting 09:49:54 (4532): No heartbeat from core client for 30 sec - exiting 09:49:55 (4532): No heartbeat from core client for 30 sec - exiting 09:49:56 (4532): No heartbeat from core client for 30 sec - exiting 09:49:58 (4532): No heartbeat from core client for 30 sec - exiting 09:49:59 (4532): No heartbeat from core client for 30 sec - exiting Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=5688, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6824, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5952, selfPID=6032, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 15:30:03 (8668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2960, selfPID=9500, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6224, selfPID=5604, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3952, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11936, selfPID=8480, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6592, selfPID=4564, iMonCtr=1 Model crash detected, will try to restart... 13:43:00 (4588): No heartbeat from core client for 30 sec - exiting 13:43:02 (4588): No heartbeat from core client for 30 sec - exiting 13:43:03 (4588): No heartbeat from core client for 30 sec - exiting 13:43:04 (4588): No heartbeat from core client for 30 sec - exiting 13:43:05 (4588): No heartbeat from core client for 30 sec - exiting 13:43:06 (4588): No heartbeat from core client for 30 sec - exiting 13:43:07 (4588): No heartbeat from core client for 30 sec - exiting 13:43:08 (4588): No heartbeat from core client for 30 sec - exiting 13:43:09 (4588): No heartbeat from core client for 30 sec - exiting 13:43:10 (4588): No heartbeat from core client for 30 sec - exiting 13:43:11 (4588): No heartbeat from core client for 30 sec - exiting 13:43:12 (4588): No heartbeat from core client for 30 sec - exiting 13:43:13 (4588): No heartbeat from core client for 30 sec - exiting 13:43:14 (4588): No heartbeat from core client for 30 sec - exiting 13:43:15 (4588): No heartbeat from core client for 30 sec - exiting 13:43:16 (4588): No heartbeat from core client for 30 sec - exiting 13:43:17 (4588): No heartbeat from core client for 30 sec - exiting 13:43:18 (4588): No heartbeat from core client for 30 sec - exiting 13:43:19 (4588): No heartbeat from core client for 30 sec - exiting 13:43:20 (4588): No heartbeat from core client for 30 sec - exiting 13:43:21 (4588): No heartbeat from core client for 30 sec - exiting 13:43:22 (4588): No heartbeat from core client for 30 sec - exiting 13:43:23 (4588): No heartbeat from core client for 30 sec - exiting 13:43:24 (4588): No heartbeat from core client for 30 sec - exiting 13:43:25 (4588): No heartbeat from core client for 30 sec - exiting 13:43:26 (4588): No heartbeat from core client for 30 sec - exiting 13:43:27 (4588): No heartbeat from core client for 30 sec - exiting 13:43:28 (4588): No heartbeat from core client for 30 sec - exiting 13:43:29 (4588): No heartbeat from core client for 30 sec - exiting 13:43:30 (4588): No heartbeat from core client for 30 sec - exiting 13:43:31 (4588): No heartbeat from core client for 30 sec - exiting 13:43:32 (4588): No heartbeat from core client for 30 sec - exiting 13:43:33 (4588): No heartbeat from core client for 30 sec - exiting 13:43:34 (4588): No heartbeat from core client for 30 sec - exiting 13:43:35 (4588): No heartbeat from core client for 30 sec - exiting 13:43:36 (4588): No heartbeat from core client for 30 sec - exiting 13:43:37 (4588): No heartbeat from core client for 30 sec - exiting 13:43:38 (4588): No heartbeat from core client for 30 sec - exiting 13:43:39 (4588): No heartbeat from core client for 30 sec - exiting 13:43:40 (4588): No heartbeat from core client for 30 sec - exiting 13:43:41 (4588): No heartbeat from core client for 30 sec - exiting 13:43:42 (4588): No heartbeat from core client for 30 sec - exiting 13:43:43 (4588): No heartbeat from core client for 30 sec - exiting 13:43:44 (4588): No heartbeat from core client for 30 sec - exiting 13:43:45 (4588): No heartbeat from core client for 30 sec - exiting 13:43:46 (4588): No heartbeat from core client for 30 sec - exiting 13:43:47 (4588): No heartbeat from core client for 30 sec - exiting 13:43:48 (4588): No heartbeat from core client for 30 sec - exiting 13:43:49 (4588): No heartbeat from core client for 30 sec - exiting 13:43:50 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4352, selfPID=2804, iMonCtr=1 Model crash detected, will try to restart... 06:27:01 (7964): No heartbeat from core client for 30 sec - exiting 06:27:02 (7964): No heartbeat from core client for 30 sec - exiting 06:27:03 (7964): No heartbeat from core client for 30 sec - exiting 06:27:04 (7964): No heartbeat from core client for 30 sec - exiting 06:27:05 (7964): No heartbeat from core client for 30 sec - exiting 06:27:06 (7964): No heartbeat from core client for 30 sec - exiting 06:27:07 (7964): No heartbeat from core client for 30 sec - exiting 06:27:08 (7964): No heartbeat from core client for 30 sec - exiting 06:27:09 (7964): No heartbeat from core client for 30 sec - exiting 06:27:10 (7964): No heartbeat from core client for 30 sec - exiting 06:27:11 (7964): No heartbeat from core client for 30 sec - exiting 06:27:13 (7964): No heartbeat from core client for 30 sec - exiting 06:27:14 (7964): No heartbeat from core client for 30 sec - exiting 06:27:15 (7964): No heartbeat from core client for 30 sec - exiting 06:27:16 (7964): No heartbeat from core client for 30 sec - exiting 06:27:17 (7964): No heartbeat from core client for 30 sec - exiting 06:27:18 (7964): No heartbeat from core client for 30 sec - exiting 06:27:19 (7964): No heartbeat from core client for 30 sec - exiting 06:27:20 (7964): No heartbeat from core client for 30 sec - exiting 06:27:21 (7964): No heartbeat from core client for 30 sec - exiting 06:27:22 (7964): No heartbeat from core client for 30 sec - exiting 06:27:23 (7964): No heartbeat from core client for 30 sec - exiting 06:27:25 (7964): No heartbeat from core client for 30 sec - exiting 06:27:26 (7964): No heartbeat from core client for 30 sec - exiting 06:27:27 (7964): No heartbeat from core client for 30 sec - exiting 06:27:28 (7964): No heartbeat from core client for 30 sec - exiting 06:27:29 (7964): No heartbeat from core client for 30 sec - exiting 06:27:30 (7964): No heartbeat from core client for 30 sec - exiting 06:27:31 (7964): No heartbeat from core client for 30 sec - exiting 06:27:32 (7964): No heartbeat from core client for 30 sec - exiting 06:27:33 (7964): No heartbeat from core client for 30 sec - exiting 06:27:34 (7964): No heartbeat from core client for 30 sec - exiting 06:27:35 (7964): No heartbeat from core client for 30 sec - exiting 06:27:37 (7964): No heartbeat from core client for 30 sec - exiting 06:27:38 (7964): No heartbeat from core client for 30 sec - exiting 06:27:39 (7964): No heartbeat from core client for 30 sec - exiting 06:27:40 (7964): No heartbeat from core client for 30 sec - exiting 06:27:41 (7964): No heartbeat from core client for 30 sec - exiting 06:27:42 (7964): No heartbeat from core client for 30 sec - exiting 06:27:43 (7964): No heartbeat from core client for 30 sec - exiting 06:27:44 (7964): No heartbeat from core client for 30 sec - exiting 06:27:45 (7964): No heartbeat from core client for 30 sec - exiting 06:27:46 (7964): No heartbeat from core client for 30 sec - exiting 06:27:47 (7964): No heartbeat from core client for 30 sec - exiting 06:27:49 (7964): No heartbeat from core client for 30 sec - exiting 06:27:50 (7964): No heartbeat from core client for 30 sec - exiting 06:27:51 (7964): No heartbeat from core client for 30 sec - exiting 06:27:52 (7964): No heartbeat from core client for 30 sec - exiting 06:27:53 (7964): No heartbeat from core client for 30 sec - exiting 06:27:54 (7964): No heartbeat from core client for 30 sec - exiting 06:27:55 (7964): No heartbeat from core client for 30 sec - exiting 06:27:56 (7964): No heartbeat from core client for 30 sec - exiting 06:27:57 (7964): No heartbeat from core client for 30 sec - exiting 06:27:58 (7964): No heartbeat from core client for 30 sec - exiting 06:27:59 (7964): No heartbeat from core client for 30 sec - exiting 06:28:01 (7964): No heartbeat from core client for 30 sec - exiting 06:28:02 (7964): No heartbeat from core client for 30 sec - exiting 06:28:03 (7964): No heartbeat from core client for 30 sec - exiting 06:28:04 (7964): No heartbeat from core client for 30 sec - exiting 06:28:05 (7964): No heartbeat from core client for 30 sec - exiting 06:28:06 (7964): No heartbeat from core client for 30 sec - exiting 06:28:07 (7964): No heartbeat from core client for 30 sec - exiting 06:28:08 (7964): No heartbeat from core client for 30 sec - exiting 06:28:09 (7964): No heartbeat from core client for 30 sec - exiting 06:28:10 (7964): No heartbeat from core client for 30 sec - exiting 06:28:11 (7964): No heartbeat from core client for 30 sec - exiting 06:28:13 (7964): No heartbeat from core client for 30 sec - exiting 06:28:14 (7964): No heartbeat from core client for 30 sec - exiting 06:28:15 (7964): No heartbeat from core client for 30 sec - exiting 06:28:16 (7964): No heartbeat from core client for 30 sec - exiting 06:28:17 (7964): No heartbeat from core client for 30 sec - exiting 06:28:18 (7964): No heartbeat from core client for 30 sec - exiting 06:28:19 (7964): No heartbeat from core client for 30 sec - exiting 06:28:20 (7964): No heartbeat from core client for 30 sec - exiting 06:28:21 (7964): No heartbeat from core client for 30 sec - exiting 06:28:22 (7964): No heartbeat from core client for 30 sec - exiting 06:28:23 (7964): No heartbeat from core client for 30 sec - exiting 06:28:25 (7964): No heartbeat from core client for 30 sec - exiting 06:28:26 (7964): No heartbeat from core client for 30 sec - exiting 06:28:27 (7964): No heartbeat from core client for 30 sec - exiting 06:28:28 (7964): No heartbeat from core client for 30 sec - exiting 06:28:29 (7964): No heartbeat from core client for 30 sec - exiting 06:28:30 (7964): No heartbeat from core client for 30 sec - exiting 06:28:31 (7964): No heartbeat from core client for 30 sec - exiting 06:28:32 (7964): No heartbeat from core client for 30 sec - exiting 06:28:33 (7964): No heartbeat from core client for 30 sec - exiting 06:28:34 (7964): No heartbeat from core client for 30 sec - exiting 06:28:36 (7964): No heartbeat from core client for 30 sec - exiting 06:28:37 (7964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9312, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9668, selfPID=5300, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=8152, iMonCtr=1 10:12:25 (1296): No heartbeat from core client for 30 sec - exiting 10:12:26 (1296): No heartbeat from core client for 30 sec - exiting 10:12:27 (1296): No heartbeat from core client for 30 sec - exiting 10:12:28 (1296): No heartbeat from core client for 30 sec - exiting 10:12:29 (1296): No heartbeat from core client for 30 sec - exiting 10:12:30 (1296): No heartbeat from core client for 30 sec - exiting 10:12:31 (1296): No heartbeat from core client for 30 sec - exiting 10:12:32 (1296): No heartbeat from core client for 30 sec - exiting 10:12:33 (1296): No heartbeat from core client for 30 sec - exiting 10:12:34 (1296): No heartbeat from core client for 30 sec - exiting 10:12:35 (1296): No heartbeat from core client for 30 sec - exiting 10:12:36 (1296): No heartbeat from core client for 30 sec - exiting 10:12:37 (1296): No heartbeat from core client for 30 sec - exiting 10:12:38 (1296): No heartbeat from core client for 30 sec - exiting 10:12:39 (1296): No heartbeat from core client for 30 sec - exiting 10:12:40 (1296): No heartbeat from core client for 30 sec - exiting 10:12:41 (1296): No heartbeat from core client for 30 sec - exiting 10:12:42 (1296): No heartbeat from core client for 30 sec - exiting 10:12:43 (1296): No heartbeat from core client for 30 sec - exiting Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:15:07 (7160): No heartbeat from core client for 30 sec - exiting 20:15:08 (7160): No heartbeat from core client for 30 sec - exiting 20:15:09 (7160): No heartbeat from core client for 30 sec - exiting 20:15:10 (7160): No heartbeat from core client for 30 sec - exiting 20:15:11 (7160): No heartbeat from core client for 30 sec - exiting 20:15:12 (7160): No heartbeat from core client for 30 sec - exiting 20:15:13 (7160): No heartbeat from core client for 30 sec - exiting 20:15:14 (7160): No heartbeat from core client for 30 sec - exiting 20:15:15 (7160): No heartbeat from core client for 30 sec - exiting 20:15:16 (7160): No heartbeat from core client for 30 sec - exiting 20:15:18 (7160): No heartbeat from core client for 30 sec - exiting 20:15:19 (7160): No heartbeat from core client for 30 sec - exiting 20:15:20 (7160): No heartbeat from core client for 30 sec - exiting 20:15:21 (7160): No heartbeat from core client for 30 sec - exiting 20:15:22 (7160): No heartbeat from core client for 30 sec - exiting 20:15:23 (7160): No heartbeat from core client for 30 sec - exiting 20:15:24 (7160): No heartbeat from core client for 30 sec - exiting 20:15:25 (7160): No heartbeat from core client for 30 sec - exiting 20:15:26 (7160): No heartbeat from core client for 30 sec - exiting 20:15:27 (7160): No heartbeat from core client for 30 sec - exiting 20:15:29 (7160): No heartbeat from core client for 30 sec - exiting 20:15:30 (7160): No heartbeat from core client for 30 sec - exiting 20:15:31 (7160): No heartbeat from core client for 30 sec - exiting 20:15:32 (7160): No heartbeat from core client for 30 sec - exiting 20:15:33 (7160): No heartbeat from core client for 30 sec - exiting 20:15:34 (7160): No heartbeat from core client for 30 sec - exiting 20:15:35 (7160): No heartbeat from core client for 30 sec - exiting 20:15:36 (7160): No heartbeat from core client for 30 sec - exiting 20:15:37 (7160): No heartbeat from core client for 30 sec - exiting 20:15:38 (7160): No heartbeat from core client for 30 sec - exiting 20:15:39 (7160): No heartbeat from core client for 30 sec - exiting 20:15:41 (7160): No heartbeat from core client for 30 sec - exiting 20:15:42 (7160): No heartbeat from core client for 30 sec - exiting 20:15:43 (7160): No heartbeat from core client for 30 sec - exiting 20:15:44 (7160): No heartbeat from core client for 30 sec - exiting 20:15:45 (7160): No heartbeat from core client for 30 sec - exiting 20:15:46 (7160): No heartbeat from core client for 30 sec - exiting 20:15:47 (7160): No heartbeat from core client for 30 sec - exiting 20:15:48 (7160): No heartbeat from core client for 30 sec - exiting 20:15:49 (7160): No heartbeat from core client for 30 sec - exiting 20:15:50 (7160): No heartbeat from core client for 30 sec - exiting 20:15:51 (7160): No heartbeat from core client for 30 sec - exiting 20:15:53 (7160): No heartbeat from core client for 30 sec - exiting 20:15:54 (7160): No heartbeat from core client for 30 sec - exiting 20:15:55 (7160): No heartbeat from core client for 30 sec - exiting 20:15:56 (7160): No heartbeat from core client for 30 sec - exiting 20:15:57 (7160): No heartbeat from core client for 30 sec - exiting 20:15:58 (7160): No heartbeat from core client for 30 sec - exiting 20:15:59 (7160): No heartbeat from core client for 30 sec - exiting 20:16:00 (7160): No heartbeat from core client for 30 sec - exiting Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=1316, iMonCtr=1 CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10140, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10160, selfPID=5388, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 10:06:11 (5380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5404, iMonCtr=2 Model crash detected, will try to restart... 18:59:42 (9448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:36:01 (288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:50 (10436): No heartbeat from core client for 30 sec - exiting 20:36:51 (10436): No heartbeat from core client for 30 sec - exiting 20:36:52 (10436): No heartbeat from core client for 30 sec - exiting 20:36:53 (10436): No heartbeat from core client for 30 sec - exiting 20:36:54 (10436): No heartbeat from core client for 30 sec - exiting 20:36:55 (10436): No heartbeat from core client for 30 sec - exiting 20:36:56 (10436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2912, selfPID=6744, iMonCtr=1 Model crash detected, will try to restart... 09:39:32 (3568): No heartbeat from core client for 30 sec - exiting 09:39:33 (3568): No heartbeat from core client for 30 sec - exiting 09:39:34 (3568): No heartbeat from core client for 30 sec - exiting 09:39:35 (3568): No heartbeat from core client for 30 sec - exiting 09:39:36 (3568): No heartbeat from core client for 30 sec - exiting 09:39:37 (3568): No heartbeat from core client for 30 sec - exiting 09:39:39 (3568): No heartbeat from core client for 30 sec - exiting 09:39:40 (3568): No heartbeat from core client for 30 sec - exiting 09:39:41 (3568): No heartbeat from core client for 30 sec - exiting 09:39:42 (3568): No heartbeat from core client for 30 sec - exiting 09:39:43 (3568): No heartbeat from core client for 30 sec - exiting 09:39:44 (3568): No heartbeat from core client for 30 sec - exiting 09:39:45 (3568): No heartbeat from core client for 30 sec - exiting 09:39:46 (3568): No heartbeat from core client for 30 sec - exiting 09:39:47 (3568): No heartbeat from core client for 30 sec - exiting Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=9052, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7600, selfPID=8228, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8124, iMonCtr=2 Model crash detected, will try to restart... 10:04:34 (4560): No heartbeat from core client for 30 sec - exiting 10:04:35 (4560): No heartbeat from core client for 30 sec - exiting 10:04:36 (4560): No heartbeat from core client for 30 sec - exiting 10:04:37 (4560): No heartbeat from core client for 30 sec - exiting 10:04:38 (4560): No heartbeat from core client for 30 sec - exiting 10:04:39 (4560): No heartbeat from core client for 30 sec - exiting 10:04:40 (4560): No heartbeat from core client for 30 sec - exiting 10:04:41 (4560): No heartbeat from core client for 30 sec - exiting 10:04:42 (4560): No heartbeat from core client for 30 sec - exiting 10:04:43 (4560): No heartbeat from core client for 30 sec - exiting 10:04:44 (4560): No heartbeat from core client for 30 sec - exiting 10:04:45 (4560): No heartbeat from core client for 30 sec - exiting 10:04:46 (4560): No heartbeat from core client for 30 sec - exiting 10:04:47 (4560): No heartbeat from core client for 30 sec - exiting 10:04:48 (4560): No heartbeat from core client for 30 sec - exiting 10:04:49 (4560): No heartbeat from core client for 30 sec - exiting 10:04:50 (4560): No heartbeat from core client for 30 sec - exiting 10:04:51 (4560): No heartbeat from core client for 30 sec - exiting 10:04:52 (4560): No heartbeat from core client for 30 sec - exiting 10:04:53 (4560): No heartbeat from core client for 30 sec - exiting 10:04:54 (4560): No heartbeat from core client for 30 sec - exiting 10:04:55 (4560): No heartbeat from core client for 30 sec - exiting 10:04:56 (4560): No heartbeat from core client for 30 sec - exiting 10:04:57 (4560): No heartbeat from core client for 30 sec - exiting 10:04:58 (4560): No heartbeat from core client for 30 sec - exiting 10:04:59 (4560): No heartbeat from core client for 30 sec - exiting 10:05:00 (4560): No heartbeat from core client for 30 sec - exiting 10:05:01 (4560): No heartbeat from core client for 30 sec - exiting 10:05:02 (4560): No heartbeat from core client for 30 sec - exiting 10:05:03 (4560): No heartbeat from core client for 30 sec - exiting 10:05:04 (4560): No heartbeat from core client for 30 sec - exiting 10:05:05 (4560): No heartbeat from core client for 30 sec - exiting 10:05:06 (4560): No heartbeat from core client for 30 sec - exiting 10:05:07 (4560): No heartbeat from core client for 30 sec - exiting 10:05:08 (4560): No heartbeat from core client for 30 sec - exiting 10:05:09 (4560): No heartbeat from core client for 30 sec - exiting 10:05:10 (4560): No heartbeat from core client for 30 sec - exiting 10:05:11 (4560): No heartbeat from core client for 30 sec - exiting 10:05:12 (4560): No heartbeat from core client for 30 sec - exiting 10:05:13 (4560): No heartbeat from core client for 30 sec - exiting 10:05:14 (4560): No heartbeat from core client for 30 sec - exiting 10:05:15 (4560): No heartbeat from core client for 30 sec - exiting 10:05:16 (4560): No heartbeat from core client for 30 sec - exiting 10:05:17 (4560): No heartbeat from core client for 30 sec - exiting 10:05:18 (4560): No heartbeat from core client for 30 sec - exiting 10:05:19 (4560): No heartbeat from core client for 30 sec - exiting 10:05:20 (4560): No heartbeat from core client for 30 sec - exiting 10:05:21 (4560): No heartbeat from core client for 30 sec - exiting 10:05:22 (4560): No heartbeat from core client for 30 sec - exiting 10:05:23 (4560): No heartbeat from core client for 30 sec - exiting 10:05:24 (4560): No heartbeat from core client for 30 sec - exiting 10:05:25 (4560): No heartbeat from core client for 30 sec - exiting 10:05:26 (4560): No heartbeat from core client for 30 sec - exiting 10:05:27 (4560): No heartbeat from core client for 30 sec - exiting 10:05:28 (4560): No heartbeat from core client for 30 sec - exiting 10:05:29 (4560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9016, selfPID=3396, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9136, selfPID=4868, iMonCtr=1 Model crash detected, will try to restart... 12:13:18 (4376): No heartbeat from core client for 30 sec - exiting 12:13:24 (4376): No heartbeat from core client for 30 sec - exiting 12:13:25 (4376): No heartbeat from core client for 30 sec - exiting 12:13:26 (4376): No heartbeat from core client for 30 sec - exiting 12:13:27 (4376): No heartbeat from core client for 30 sec - exiting 12:13:28 (4376): No heartbeat from core client for 30 sec - exiting 12:13:29 (4376): No heartbeat from core client for 30 sec - exiting 12:13:30 (4376): No heartbeat from core client for 30 sec - exiting 12:13:31 (4376): No heartbeat from core client for 30 sec - exiting 12:13:32 (4376): No heartbeat from core client for 30 sec - exiting 12:13:33 (4376): No heartbeat from core client for 30 sec - exiting 12:13:35 (4376): No heartbeat from core client for 30 sec - exiting 12:13:36 (4376): No heartbeat from core client for 30 sec - exiting 12:13:37 (4376): No heartbeat from core client for 30 sec - exiting 12:13:38 (4376): No heartbeat from core client for 30 sec - exiting 12:13:39 (4376): No heartbeat from core client for 30 sec - exiting 12:13:40 (4376): No heartbeat from core client for 30 sec - exiting 12:13:41 (4376): No heartbeat from core client for 30 sec - exiting 12:13:42 (4376): No heartbeat from core client for 30 sec - exiting 12:13:43 (4376): No heartbeat from core client for 30 sec - exiting 12:13:44 (4376): No heartbeat from core client for 30 sec - exiting 12:13:45 (4376): No heartbeat from core client for 30 sec - exiting 12:13:47 (4376): No heartbeat from core client for 30 sec - exiting 12:13:48 (4376): No heartbeat from core client for 30 sec - exiting 12:13:49 (4376): No heartbeat from core client for 30 sec - exiting 12:13:50 (4376): No heartbeat from core client for 30 sec - exiting 12:13:51 (4376): No heartbeat from core client for 30 sec - exiting 12:13:52 (4376): No heartbeat from core client for 30 sec - exiting 12:13:53 (4376): No heartbeat from core client for 30 sec - exiting 12:13:54 (4376): No heartbeat from core client for 30 sec - exiting 12:13:55 (4376): No heartbeat from core client for 30 sec - exiting 12:13:56 (4376): No heartbeat from core client for 30 sec - exiting 12:13:58 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3316, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9836, selfPID=10672, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=7728, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4656, selfPID=4656, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4656, selfPID=1816, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_o8dg_2012_1_008627634_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o8dg_2012_1_008627634_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o8dg_2012_1_008627634_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o8dg_2012_1_008627634_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o8dg_2012_1_008627634_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 May 2014 12:19:06	1322841	16459024	hadam3p_anz_o8dg_2012_1_008627634_0	80,939	617,582	7.6302
07 May 2014 14:24:48	1322841	16459024	hadam3p_anz_o8dg_2012_1_008627634_0	69,419	536,891	7.7341
02 May 2014 12:29:58	1322841	16459024	hadam3p_anz_o8dg_2012_1_008627634_0	57,899	449,171	7.7578
29 Apr 2014 22:30:47	1322841	16459024	hadam3p_anz_o8dg_2012_1_008627634_0	46,379	359,280	7.7466
26 Apr 2014 17:19:33	1322841	16459024	hadam3p_anz_o8dg_2012_1_008627634_0	34,859	267,959	7.6869
22 Apr 2014 15:24:15	1322841	16459024	hadam3p_anz_o8dg_2012_1_008627634_0	23,339	177,586	7.6090
16 Apr 2014 17:40:48	1322841	16459024	hadam3p_anz_o8dg_2012_1_008627634_0	11,819	90,111	7.6242