Task 16650222

Name	hadam3p_anz_r0tp_2012_1_008730683_1
Workunit	8876661
Created	29 May 2014, 3:52:51 UTC
Sent	29 May 2014, 3:53:08 UTC
Report deadline	11 May 2015, 9:13:08 UTC
Received	21 Sep 2014, 19:27:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1310670
Run time	7 days 15 hours 25 min 56 sec
CPU time	6 days 14 hours 36 min 24 sec
Validate state	Invalid
Credit	2,993.82
Device peak FLOPS	2.17 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6384, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3204, selfPID=4500, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7888, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8004, selfPID=3840, iMonCtr=1 Model crash detected, will try to restart... 08:58:04 (6288): No heartbeat from core client for 30 sec - exiting 08:58:05 (6288): No heartbeat from core client for 30 sec - exiting 08:58:06 (6288): No heartbeat from core client for 30 sec - exiting 08:58:07 (6288): No heartbeat from core client for 30 sec - exiting 08:58:08 (6288): No heartbeat from core client for 30 sec - exiting 08:58:09 (6288): No heartbeat from core client for 30 sec - exiting 08:58:10 (6288): No heartbeat from core client for 30 sec - exiting 08:58:11 (6288): No heartbeat from core client for 30 sec - exiting 08:58:12 (6288): No heartbeat from core client for 30 sec - exiting 08:58:13 (6288): No heartbeat from core client for 30 sec - exiting 08:58:14 (6288): No heartbeat from core client for 30 sec - exiting 08:58:16 (6288): No heartbeat from core client for 30 sec - exiting 08:58:17 (6288): No heartbeat from core client for 30 sec - exiting 08:58:18 (6288): No heartbeat from core client for 30 sec - exiting 08:58:19 (6288): No heartbeat from core client for 30 sec - exiting 08:58:20 (6288): No heartbeat from core client for 30 sec - exiting 08:58:21 (6288): No heartbeat from core client for 30 sec - exiting 08:58:22 (6288): No heartbeat from core client for 30 sec - exiting 08:58:23 (6288): No heartbeat from core client for 30 sec - exiting 08:58:24 (6288): No heartbeat from core client for 30 sec - exiting 08:58:25 (6288): No heartbeat from core client for 30 sec - exiting 08:58:26 (6288): No heartbeat from core client for 30 sec - exiting 08:58:27 (6288): No heartbeat from core client for 30 sec - exiting 08:58:28 (6288): No heartbeat from core client for 30 sec - exiting 08:58:29 (6288): No heartbeat from core client for 30 sec - exiting 08:58:30 (6288): No heartbeat from core client for 30 sec - exiting 08:58:31 (6288): No heartbeat from core client for 30 sec - exiting 08:58:32 (6288): No heartbeat from core client for 30 sec - exiting 08:58:33 (6288): No heartbeat from core client for 30 sec - exiting 08:58:34 (6288): No heartbeat from core client for 30 sec - exiting 08:58:35 (6288): No heartbeat from core client for 30 sec - exiting 08:58:36 (6288): No heartbeat from core client for 30 sec - exiting 08:58:37 (6288): No heartbeat from core client for 30 sec - exiting 08:58:38 (6288): No heartbeat from core client for 30 sec - exiting 08:58:39 (6288): No heartbeat from core client for 30 sec - exiting 08:58:40 (6288): No heartbeat from core client for 30 sec - exiting 08:58:41 (6288): No heartbeat from core client for 30 sec - exiting 08:58:42 (6288): No heartbeat from core client for 30 sec - exiting 08:58:43 (6288): No heartbeat from core client for 30 sec - exiting 08:58:44 (6288): No heartbeat from core client for 30 sec - exiting 08:58:45 (6288): No heartbeat from core client for 30 sec - exiting 08:58:46 (6288): No heartbeat from core client for 30 sec - exiting 08:58:47 (6288): No heartbeat from core client for 30 sec - exiting 08:58:48 (6288): No heartbeat from core client for 30 sec - exiting 08:58:49 (6288): No heartbeat from core client for 30 sec - exiting 08:58:50 (6288): No heartbeat from core client for 30 sec - exiting 08:58:51 (6288): No heartbeat from core client for 30 sec - exiting 08:58:52 (6288): No heartbeat from core client for 30 sec - exiting 08:58:53 (6288): No heartbeat from core client for 30 sec - exiting 08:58:54 (6288): No heartbeat from core client for 30 sec - exiting 08:58:55 (6288): No heartbeat from core client for 30 sec - exiting 08:58:56 (6288): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6988, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7468, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9852, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5072, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9932, selfPID=776, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4120, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6820, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5736, iMonCtr=2 Global WorkeController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1852, iMonCtr=2 Model crash detected, will try to restart... CGlobal WController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=608, selfPID=6944, iMonCtr=1 Model crash detected, will try to restart... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10484, selfPID=6360, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8196, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7200, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7032, selfPID=7336, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6276, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4768, selfPID=10064, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9348, selfPID=9848, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7736, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4188, selfPID=6760, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6232, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1152, selfPID=6364, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10140, selfPID=10988, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5572, iMonCtr=2 06:24:19 (9200): No heartbeat from core client for 30 sec - exiting 06:24:20 (9200): No heartbeat from core client for 30 sec - exiting 06:24:21 (9200): No heartbeat from core client for 30 sec - exiting 06:24:22 (9200): No heartbeat from core client for 30 sec - exiting 06:24:23 (9200): No heartbeat from core client for 30 sec - exiting 06:24:24 (9200): No heartbeat from core client for 30 sec - exiting 06:24:25 (9200): No heartbeat from core client for 30 sec - exiting 06:24:26 (9200): No heartbeat from core client for 30 sec - exiting 06:24:27 (9200): No heartbeat from core client for 30 sec - exiting 06:24:28 (9200): No heartbeat from core client for 30 sec - exiting 06:24:29 (9200): No heartbeat from core client for 30 sec - exiting 06:24:30 (9200): No heartbeat from core client for 30 sec - exiting 06:24:31 (9200): No heartbeat from core client for 30 sec - exiting 06:24:32 (9200): No heartbeat from core client for 30 sec - exiting 06:24:33 (9200): No heartbeat from core client for 30 sec - exiting 06:24:34 (9200): No heartbeat from core client for 30 sec - exiting 06:24:35 (9200): No heartbeat from core client for 30 sec - exiting 06:24:36 (9200): No heartbeat from core client for 30 sec - exiting 06:24:37 (9200): No heartbeat from core client for 30 sec - exiting 06:24:38 (9200): No heartbeat from core client for 30 sec - exiting 06:24:39 (9200): No heartbeat from core client for 30 sec - exiting 06:24:40 (9200): No heartbeat from core client for 30 sec - exiting 06:24:41 (9200): No heartbeat from core client for 30 sec - exiting 06:24:42 (9200): No heartbeat from core client for 30 sec - exiting 06:24:43 (9200): No heartbeat from core client for 30 sec - exiting 06:24:44 (9200): No heartbeat from core client for 30 sec - exiting 06:24:45 (9200): No heartbeat from core client for 30 sec - exiting 06:24:46 (9200): No heartbeat from core client for 30 sec - exiting 06:24:47 (9200): No heartbeat from core client for 30 sec - exiting 06:24:48 (9200): No heartbeat from core client for 30 sec - exiting 06:24:49 (9200): No heartbeat from core client for 30 sec - exiting 06:24:50 (9200): No heartbeat from core client for 30 sec - exiting 06:24:51 (9200): No heartbeat from core client for 30 sec - exiting 06:24:52 (9200): No heartbeat from core client for 30 sec - exiting 06:24:53 (9200): No heartbeat from core client for 30 sec - exiting 06:24:54 (9200): No heartbeat from core client for 30 sec - exiting 06:24:55 (9200): No heartbeat from core client for 30 sec - exiting 06:24:56 (9200): No heartbeat from core client for 30 sec - exiting 06:24:57 (9200): No heartbeat from core client for 30 sec - exiting 06:24:58 (9200): No heartbeat from core client for 30 sec - exiting 06:24:59 (9200): No heartbeat from core client for 30 sec - exiting 06:25:00 (9200): No heartbeat from core client for 30 sec - exiting 06:25:01 (9200): No heartbeat from core client for 30 sec - exiting 06:25:02 (9200): No heartbeat from core client for 30 sec - exiting 06:25:03 (9200): No heartbeat from core client for 30 sec - exiting 06:25:04 (9200): No heartbeat from core client for 30 sec - exiting 06:25:05 (9200): No heartbeat from core client for 30 sec - exiting 06:25:06 (9200): No heartbeat from core client for 30 sec - exiting 06:25:07 (9200): No heartbeat from core client for 30 sec - exiting 06:25:08 (9200): No heartbeat from core client for 30 sec - exiting 06:25:09 (9200): No heartbeat from core client for 30 sec - exiting 06:25:10 (9200): No heartbeat from core client for 30 sec - exiting 06:25:11 (9200): No heartbeat from core client for 30 sec - exiting 06:25:12 (9200): No heartbeat from core client for 30 sec - exiting 06:25:13 (9200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7740, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7880, selfPID=7532, iMonCtr=1 Model crash detected, will try to restart... 20:53:36 (7232): No heartbeat from core client for 30 sec - exiting 20:53:37 (7232): No heartbeat from core client for 30 sec - exiting 20:53:38 (7232): No heartbeat from core client for 30 sec - exiting 20:53:39 (7232): No heartbeat from core client for 30 sec - exiting 20:53:40 (7232): No heartbeat from core client for 30 sec - exiting 20:53:42 (7232): No heartbeat from core client for 30 sec - exiting 20:53:43 (7232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6028, selfPID=9400, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10952, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3208, selfPID=10816, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5160, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=976, selfPID=2936, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1792, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2368, selfPID=10220, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5520, selfPID=9712, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2460, selfPID=1232, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10796, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11200, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10844, selfPID=5820, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5288, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11248, selfPID=6920, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11248, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5684, selfPID=4780, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7724, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2868, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5604, selfPID=3792, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11372, selfPID=6988, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9140, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5268, selfPID=8120, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9156, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7664, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5668, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6496, selfPID=7912, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1692, selfPID=2424, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5528, selfPID=9164, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2748, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5128, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1788, selfPID=8972, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8184, selfPID=7488, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6208, selfPID=4700, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3356, selfPID=9816, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 16:52:00 (2796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:52:02 (2796): No heartbeat from core client for 30 sec - exiting 16:52:03 (2796): No heartbeat from core client for 30 sec - exiting 16:52:04 (2796): No heartbeat from core client for 30 sec - exiting 16:52:55 (1160): No heartbeat from core client for 30 sec - exiting 16:52:56 (1160): No heartbeat from core client for 30 sec - exiting 16:52:57 (1160): No heartbeat from core client for 30 sec - exiting 16:52:58 (1160): No heartbeat from core client for 30 sec - exiting 16:53:00 (1160): No heartbeat from core client for 30 sec - exiting 16:53:01 (1160): No heartbeat from core client for 30 sec - exiting 16:53:02 (1160): No heartbeat from core client for 30 sec - exiting 16:53:03 (1160): No heartbeat from core client for 30 sec - exiting 16:53:04 (1160): No heartbeat from core client for 30 sec - exiting 16:53:05 (1160): No heartbeat from core client for 30 sec - exiting 16:53:06 (1160): No heartbeat from core client for 30 sec - exiting 16:53:07 (1160): No heartbeat from core client for 30 sec - exiting 16:53:08 (1160): No heartbeat from core client for 30 sec - exiting 16:53:09 (1160): No heartbeat from core client for 30 sec - exiting 16:53:10 (1160): No heartbeat from core client for 30 sec - exiting 16:53:12 (1160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4232, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1640, selfPID=2576, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=348, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8812, selfPID=2976, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8868, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8884, selfPID=4440, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7096, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8148, selfPID=7864, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8260, iMonCtr=2 Model crash detected, will try to restart... 23:44:53 (4140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7196, iMonCtr=2 23:50:10 (9832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9664, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=13304, iMonCtr=2 Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10140, selfPID=11964, iMonCtr=1 Model crash detected, will try to restart... 17:22:38 (11328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:23:12 (7400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8164, selfPID=8044, iMonCtr=1 Model crash detected, will try to restart... 21:17:34 (12080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11052, selfPID=8032, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6052, iMonCtr=2 Model crash detected, will try to restart... 19:58:46 (11916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:47 (11916): No heartbeat from core client for 30 sec - exiting 19:58:48 (11916): No heartbeat from core client for 30 sec - exiting Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7956, selfPID=7956, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8160, iMonCtr=2 02:53:53 (2452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10972, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10476, selfPID=4800, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7560, selfPID=7608, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11584, selfPID=6892, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3272, selfPID=12080, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6628, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11372, iMonCtr=2 Model crash detected, will try to restart... 06:45:08 (1572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:57:13 (3556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:16 (10640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11412, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10060, selfPID=11752, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6060, selfPID=7772, iMonCtr=1 Model crash detected, will try to restart... 17:01:28 (8172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7900, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10064, selfPID=5132, iMonCtr=1 Model crash detected, will try to restart... GController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4788, selfPID=5448, iMonCtr=1 Model crash detected, will try to restart... 05:26:11 (8004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:43:45 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7360, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5408, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4584, selfPID=6784, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7036, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4864, selfPID=4916, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5156, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1408, selfPID=4884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7992, selfPID=1452, iMonCtr=1 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8200, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4876, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4244, selfPID=7012, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5548, selfPID=1576, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7256, selfPID=5736, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5848, iMonCtr=2 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9508, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_r0tp_2012_1_008730683_1_7.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_r0tp_2012_1_008730683_1_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_r0tp_2012_1_008730683_1_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_r0tp_2012_1_008730683_1_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_r0tp_2012_1_008730683_1_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_r0tp_2012_1_008730683_1_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Sep 2014 03:24:36	1310670	16650222	hadam3p_anz_r0tp_2012_1_008730683_1	69,419	553,268	7.9700
25 Aug 2014 04:35:59	1310670	16650222	hadam3p_anz_r0tp_2012_1_008730683_1	57,899	464,512	8.0228
15 Aug 2014 10:47:47	1310670	16650222	hadam3p_anz_r0tp_2012_1_008730683_1	46,379	373,939	8.0627
27 Jul 2014 02:54:56	1310670	16650222	hadam3p_anz_r0tp_2012_1_008730683_1	34,859	283,674	8.1378
19 Jun 2014 09:46:37	1310670	16650222	hadam3p_anz_r0tp_2012_1_008730683_1	23,339	197,160	8.4477
10 Jun 2014 09:02:58	1310670	16650222	hadam3p_anz_r0tp_2012_1_008730683_1	11,819	100,069	8.4668