Task 18661833

Name	hadam3p_anz_u297_2013_1_010000619_0
Workunit	9998977
Created	3 Jul 2015, 10:33:20 UTC
Sent	3 Jul 2015, 17:37:10 UTC
Report deadline	14 Jun 2016, 22:57:10 UTC
Received	12 Jul 2015, 23:31:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1365097
Run time	2 days 7 hours 49 min 3 sec
CPU time	1 days 15 hours 13 min 42 sec
Validate state	Invalid
Credit	1,503.36
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.4.42</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5568, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3880, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2744, selfPID=3540, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5988, selfPID=5588, iMonCtr=1 Model crash detected, will try to restart... CGntrolobal ::orCPDN:p rPoN process is not running, exitinRetVal = 1 = checkPeckPI, selfselfPID=4132, iCtr=2 r=ode l crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5148, selfPID=4052, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5252, selfPID=1204, iMonCtr=1 Model crash detected, will try to restart... 13:32:56 (4588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4620, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3792, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4324, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5776, selfPID=2960, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2792, iMonCtr=2 Model crash detected, will try to restart... C00:12:48 (4800): No heartbeat from core client for 30 sec - exiting 00:12:49 (4800): No heartbeat from core client for 30 sec - exiting 00:12:51 (4800): No heartbeat from core client for 30 sec - exiting 00:12:52 (4800): No heartbeat from core client for 30 sec - exiting 00:12:53 (4800): No heartbeat from core client for 30 sec - exiting 00:12:54 (4800): No heartbeat from core client for 30 sec - exiting 00:12:55 (4800): No heartbeat from core client for 30 sec - exiting 00:12:56 (4800): No heartbeat from core client for 30 sec - exiting 00:12:57 (4800): No heartbeat from core client for 30 sec - exiting 00:12:58 (4800): No heartbeat from core client for 30 sec - exiting 00:12:59 (4800): No heartbeat from core client for 30 sec - exiting 00:13:00 (4800): No heartbeat from core client for 30 sec - exiting 00:13:01 (4800): No heartbeat from core client for 30 sec - exiting 00:13:03 (4800): No heartbeat from core client for 30 sec - exiting 00:13:04 (4800): No heartbeat from core client for 30 sec - exiting 00:13:05 (4800): No heartbeat from core client for 30 sec - exiting 00:13:06 (4800): No heartbeat from core client for 30 sec - exiting 00:13:07 (4800): No heartbeat from core client for 30 sec - exiting 00:13:08 (4800): No heartbeat from core client for 30 sec - exiting 00:13:09 (4800): No heartbeat from core client for 30 sec - exiting 00:13:10 (4800): No heartbeat from core client for 30 sec - exiting 00:13:11 (4800): No heartbeat from core client for 30 sec - exiting 00:13:12 (4800): No heartbeat from core client for 30 sec - exiting 00:13:14 (4800): No heartbeat from core client for 30 sec - exiting 00:13:15 (4800): No heartbeat from core client for 30 sec - exiting 00:13:16 (4800): No heartbeat from core client for 30 sec - exiting 00:13:17 (4800): No heartbeat from core client for 30 sec - exiting 00:13:18 (4800): No heartbeat from core client for 30 sec - exiting 00:13:19 (4800): No heartbeat from core client for 30 sec - exiting 00:13:20 (4800): No heartbeat from core client for 30 sec - exiting 00:13:21 (4800): No heartbeat from core client for 30 sec - exiting 00:13:22 (4800): No heartbeat from core client for 30 sec - exiting 00:13:24 (4800): No heartbeat from core client for 30 sec - exiting 00:13:25 (4800): No heartbeat from core client for 30 sec - exiting 00:13:26 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:26:22 (4408): No heartbeat from core client for 30 sec - exiting 00:26:23 (4408): No heartbeat from core client for 30 sec - exiting 00:26:24 (4408): No heartbeat from core client for 30 sec - exiting 00:26:25 (4408): No heartbeat from core client for 30 sec - exiting 00:26:26 (4408): No heartbeat from core client for 30 sec - exiting 00:26:27 (4408): No heartbeat from core client for 30 sec - exiting 00:26:28 (4408): No heartbeat from core client for 30 sec - exiting 00:26:29 (4408): No heartbeat from core client for 30 sec - exiting 00:26:30 (4408): No heartbeat from core client for 30 sec - exiting 00:26:31 (4408): No heartbeat from core client for 30 sec - exiting 00:26:32 (4408): No heartbeat from core client for 30 sec - exiting 00:26:33 (4408): No heartbeat from core client for 30 sec - exiting 00:26:34 (4408): No heartbeat from core client for 30 sec - exiting 00:26:35 (4408): No heartbeat from core client for 30 sec - exiting 00:26:36 (4408): No heartbeat from core client for 30 sec - exiting 00:26:37 (4408): No heartbeat from core client for 30 sec - exiting 00:26:38 (4408): No heartbeat from core client for 30 sec - exiting 00:26:39 (4408): No heartbeat from core client for 30 sec - exiting 00:26:40 (4408): No heartbeat from core client for 30 sec - exiting 00:26:41 (4408): No heartbeat from core client for 30 sec - exiting 00:26:42 (4408): No heartbeat from core client for 30 sec - exiting 00:26:43 (4408): No heartbeat from core client for 30 sec - exiting 00:26:44 (4408): No heartbeat from core client for 30 sec - exiting 00:26:45 (4408): No heartbeat from core client for 30 sec - exiting 00:26:46 (4408): No heartbeat from core client for 30 sec - exiting 00:26:47 (4408): No heartbeat from core client for 30 sec - exiting 00:26:48 (4408): No heartbeat from core client for 30 sec - exiting 00:26:49 (4408): No heartbeat from core client for 30 sec - exiting 00:26:50 (4408): No heartbeat from core client for 30 sec - exiting 00:26:51 (4408): No heartbeat from core client for 30 sec - exiting 00:26:52 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:31:50 (4440): No heartbeat from core client for 30 sec - exiting 01:31:51 (4440): No heartbeat from core client for 30 sec - exiting 01:31:52 (4440): No heartbeat from core client for 30 sec - exiting 01:31:53 (4440): No heartbeat from core client for 30 sec - exiting 01:31:54 (4440): No heartbeat from core client for 30 sec - exiting 01:31:55 (4440): No heartbeat from core client for 30 sec - exiting 01:31:56 (4440): No heartbeat from core client for 30 sec - exiting 01:31:57 (4440): No heartbeat from core client for 30 sec - exiting 01:31:58 (4440): No heartbeat from core client for 30 sec - exiting 01:32:00 (4440): No heartbeat from core client for 30 sec - exiting 01:32:01 (4440): No heartbeat from core client for 30 sec - exiting 01:32:02 (4440): No heartbeat from core client for 30 sec - exiting 01:32:03 (4440): No heartbeat from core client for 30 sec - exiting 01:32:04 (4440): No heartbeat from core client for 30 sec - exiting 01:32:05 (4440): No heartbeat from core client for 30 sec - exiting 01:32:06 (4440): No heartbeat from core client for 30 sec - exiting 01:32:07 (4440): No heartbeat from core client for 30 sec - exiting 01:32:08 (4440): No heartbeat from core client for 30 sec - exiting 01:32:09 (4440): No heartbeat from core client for 30 sec - exiting 01:32:10 (4440): No heartbeat from core client for 30 sec - exiting 01:32:12 (4440): No heartbeat from core client for 30 sec - exiting 01:32:13 (4440): No heartbeat from core client for 30 sec - exiting 01:32:14 (4440): No heartbeat from core client for 30 sec - exiting 01:32:15 (4440): No heartbeat from core client for 30 sec - exiting 01:32:16 (4440): No heartbeat from core client for 30 sec - exiting 01:32:17 (4440): No heartbeat from core client for 30 sec - exiting 01:32:18 (4440): No heartbeat from core client for 30 sec - exiting 01:32:19 (4440): No heartbeat from core client for 30 sec - exiting 01:32:20 (4440): No heartbeat from core client for 30 sec - exiting 01:32:21 (4440): No heartbeat from core client for 30 sec - exiting 01:32:23 (4440): No heartbeat from core client for 30 sec - exiting 01:32:24 (4440): No heartbeat from core client for 30 sec - exiting 01:32:25 (4440): No heartbeat from core client for 30 sec - exiting 01:32:26 (4440): No heartbeat from core client for 30 sec - exiting 01:32:27 (4440): No heartbeat from core client for 30 sec - exiting 01:32:28 (4440): No heartbeat from core client for 30 sec - exiting 01:32:29 (4440): No heartbeat from core client for 30 sec - exiting 01:32:30 (4440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:49 (4612): No heartbeat from core client for 30 sec - exiting 02:01:50 (4612): No heartbeat from core client for 30 sec - exiting 02:01:51 (4612): No heartbeat from core client for 30 sec - exiting 02:01:52 (4612): No heartbeat from core client for 30 sec - exiting 02:01:53 (4612): No heartbeat from core client for 30 sec - exiting 02:01:55 (4612): No heartbeat from core client for 30 sec - exiting 02:01:56 (4612): No heartbeat from core client for 30 sec - exiting 02:01:57 (4612): No heartbeat from core client for 30 sec - exiting 02:01:58 (4612): No heartbeat from core client for 30 sec - exiting 02:01:59 (4612): No heartbeat from core client for 30 sec - exiting 02:02:00 (4612): No heartbeat from core client for 30 sec - exiting 02:02:01 (4612): No heartbeat from core client for 30 sec - exiting 02:02:02 (4612): No heartbeat from core client for 30 sec - exiting 02:02:03 (4612): No heartbeat from core client for 30 sec - exiting 02:02:04 (4612): No heartbeat from core client for 30 sec - exiting 02:02:06 (4612): No heartbeat from core client for 30 sec - exiting 02:02:07 (4612): No heartbeat from core client for 30 sec - exiting 02:02:08 (4612): No heartbeat from core client for 30 sec - exiting 02:02:09 (4612): No heartbeat from core client for 30 sec - exiting 02:02:10 (4612): No heartbeat from core client for 30 sec - exiting 02:02:11 (4612): No heartbeat from core client for 30 sec - exiting 02:02:12 (4612): No heartbeat from core client for 30 sec - exiting 02:02:13 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5708, selfPID=5708, iMonCtr=2 03:09:49 (4860): No heartbeat from core client for 30 sec - exiting 03:09:50 (4860): No heartbeat from core client for 30 sec - exiting 03:09:51 (4860): No heartbeat from core client for 30 sec - exiting 03:09:52 (4860): No heartbeat from core client for 30 sec - exiting 03:09:53 (4860): No heartbeat from core client for 30 sec - exiting 03:09:54 (4860): No heartbeat from core client for 30 sec - exiting 03:09:55 (4860): No heartbeat from core client for 30 sec - exiting 03:09:57 (4860): No heartbeat from core client for 30 sec - exiting 03:09:58 (4860): No heartbeat from core client for 30 sec - exiting 03:09:59 (4860): No heartbeat from core client for 30 sec - exiting 03:10:00 (4860): No heartbeat from core client for 30 sec - exiting 03:10:01 (4860): No heartbeat from core client for 30 sec - exiting 03:10:02 (4860): No heartbeat from core client for 30 sec - exiting 03:10:03 (4860): No heartbeat from core client for 30 sec - exiting 03:10:04 (4860): No heartbeat from core client for 30 sec - exiting 03:10:05 (4860): No heartbeat from core client for 30 sec - exiting 03:10:06 (4860): No heartbeat from core client for 30 sec - exiting 03:10:07 (4860): No heartbeat from core client for 30 sec - exiting 03:10:09 (4860): No heartbeat from core client for 30 sec - exiting 03:10:10 (4860): No heartbeat from core client for 30 sec - exiting 03:10:11 (4860): No heartbeat from core client for 30 sec - exiting 03:10:12 (4860): No heartbeat from core client for 30 sec - exiting 03:10:13 (4860): No heartbeat from core client for 30 sec - exiting 03:10:14 (4860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=2 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5684, selfPID=3276, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1284, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2748, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=152, selfPID=4000, iMonCtr=1 Model crash detected, will try to restart... 22:34:11 (1004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker22:34:13 (1004): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3592, selfPID=5292, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4968, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5008, selfPID=2496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4852, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=6132, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=928, selfPID=4248, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_4.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_5.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u297_2013_1_010000619_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Jul 2015 23:28:10	1365097	18661833	hadam3p_anz_u297_2013_1_010000619_0	34,859	115,592	3.3160
07 Jul 2015 20:04:16	1365097	18661833	hadam3p_anz_u297_2013_1_010000619_0	23,339	79,916	3.4241
04 Jul 2015 22:01:24	1365097	18661833	hadam3p_anz_u297_2013_1_010000619_0	11,819	39,234	3.3196