Task 16579784

Name	hadam3p_anz_o9ng_2012_1_008629290_2
Workunit	8775802
Created	23 Apr 2014, 5:40:00 UTC
Sent	23 Apr 2014, 5:57:23 UTC
Report deadline	5 Apr 2015, 11:17:23 UTC
Received	5 May 2014, 21:09:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1322841
Run time	3 days 5 hours 28 min 4 sec
CPU time	3 days 2 hours 25 min 16 sec
Validate state	Invalid
Credit	1,006.54
Device peak FLOPS	2.76 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.3.11</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9536, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1940, iMonCtr=2 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7248, iMonCtr=2 13:43:01 (4880): No heartbeat from core client for 30 sec - exiting 13:43:02 (4880): No heartbeat from core client for 30 sec - exiting 13:43:03 (4880): No heartbeat from core client for 30 sec - exiting 13:43:04 (4880): No heartbeat from core client for 30 sec - exiting 13:43:05 (4880): No heartbeat from core client for 30 sec - exiting 13:43:06 (4880): No heartbeat from core client for 30 sec - exiting 13:43:07 (4880): No heartbeat from core client for 30 sec - exiting 13:43:08 (4880): No heartbeat from core client for 30 sec - exiting 13:43:09 (4880): No heartbeat from core client for 30 sec - exiting 13:43:10 (4880): No heartbeat from core client for 30 sec - exiting 13:43:11 (4880): No heartbeat from core client for 30 sec - exiting 13:43:12 (4880): No heartbeat from core client for 30 sec - exiting 13:43:13 (4880): No heartbeat from core client for 30 sec - exiting 13:43:14 (4880): No heartbeat from core client for 30 sec - exiting 13:43:15 (4880): No heartbeat from core client for 30 sec - exiting 13:43:16 (4880): No heartbeat from core client for 30 sec - exiting 13:43:17 (4880): No heartbeat from core client for 30 sec - exiting 13:43:18 (4880): No heartbeat from core client for 30 sec - exiting 13:43:19 (4880): No heartbeat from core client for 30 sec - exiting 13:43:20 (4880): No heartbeat from core client for 30 sec - exiting 13:43:21 (4880): No heartbeat from core client for 30 sec - exiting 13:43:22 (4880): No heartbeat from core client for 30 sec - exiting 13:43:23 (4880): No heartbeat from core client for 30 sec - exiting 13:43:24 (4880): No heartbeat from core client for 30 sec - exiting 13:43:25 (4880): No heartbeat from core client for 30 sec - exiting 13:43:26 (4880): No heartbeat from core client for 30 sec - exiting 13:43:27 (4880): No heartbeat from core client for 30 sec - exiting 13:43:28 (4880): No heartbeat from core client for 30 sec - exiting 13:43:29 (4880): No heartbeat from core client for 30 sec - exiting 13:43:30 (4880): No heartbeat from core client for 30 sec - exiting 13:43:31 (4880): No heartbeat from core client for 30 sec - exiting 13:43:32 (4880): No heartbeat from core client for 30 sec - exiting 13:43:33 (4880): No heartbeat from core client for 30 sec - exiting 13:43:34 (4880): No heartbeat from core client for 30 sec - exiting 13:43:35 (4880): No heartbeat from core client for 30 sec - exiting 13:43:36 (4880): No heartbeat from core client for 30 sec - exiting 13:43:37 (4880): No heartbeat from core client for 30 sec - exiting 13:43:38 (4880): No heartbeat from core client for 30 sec - exiting 13:43:39 (4880): No heartbeat from core client for 30 sec - exiting 13:43:40 (4880): No heartbeat from core client for 30 sec - exiting 13:43:41 (4880): No heartbeat from core client for 30 sec - exiting 13:43:42 (4880): No heartbeat from core client for 30 sec - exiting 13:43:43 (4880): No heartbeat from core client for 30 sec - exiting 13:43:44 (4880): No heartbeat from core client for 30 sec - exiting 13:43:45 (4880): No heartbeat from core client for 30 sec - exiting 13:43:46 (4880): No heartbeat from core client for 30 sec - exiting 13:43:47 (4880): No heartbeat from core client for 30 sec - exiting 13:43:48 (4880): No heartbeat from core client for 30 sec - exiting 13:43:49 (4880): No heartbeat from core client for 30 sec - exiting 13:43:50 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7980, iMonCtr=2 Model crash detected, will try to restart... 06:27:01 (5556): No heartbeat from core client for 30 sec - exiting 06:27:03 (5556): No heartbeat from core client for 30 sec - exiting 06:27:04 (5556): No heartbeat from core client for 30 sec - exiting 06:27:05 (5556): No heartbeat from core client for 30 sec - exiting 06:27:06 (5556): No heartbeat from core client for 30 sec - exiting 06:27:07 (5556): No heartbeat from core client for 30 sec - exiting 06:27:08 (5556): No heartbeat from core client for 30 sec - exiting 06:27:09 (5556): No heartbeat from core client for 30 sec - exiting 06:27:10 (5556): No heartbeat from core client for 30 sec - exiting 06:27:11 (5556): No heartbeat from core client for 30 sec - exiting 06:27:12 (5556): No heartbeat from core client for 30 sec - exiting 06:27:13 (5556): No heartbeat from core client for 30 sec - exiting 06:27:15 (5556): No heartbeat from core client for 30 sec - exiting 06:27:16 (5556): No heartbeat from core client for 30 sec - exiting 06:27:17 (5556): No heartbeat from core client for 30 sec - exiting 06:27:18 (5556): No heartbeat from core client for 30 sec - exiting 06:27:19 (5556): No heartbeat from core client for 30 sec - exiting 06:27:20 (5556): No heartbeat from core client for 30 sec - exiting 06:27:21 (5556): No heartbeat from core client for 30 sec - exiting 06:27:22 (5556): No heartbeat from core client for 30 sec - exiting 06:27:23 (5556): No heartbeat from core client for 30 sec - exiting 06:27:24 (5556): No heartbeat from core client for 30 sec - exiting 06:27:25 (5556): No heartbeat from core client for 30 sec - exiting 06:27:27 (5556): No heartbeat from core client for 30 sec - exiting 06:27:28 (5556): No heartbeat from core client for 30 sec - exiting 06:27:29 (5556): No heartbeat from core client for 30 sec - exiting 06:27:30 (5556): No heartbeat from core client for 30 sec - exiting 06:27:31 (5556): No heartbeat from core client for 30 sec - exiting 06:27:32 (5556): No heartbeat from core client for 30 sec - exiting 06:27:33 (5556): No heartbeat from core client for 30 sec - exiting 06:27:34 (5556): No heartbeat from core client for 30 sec - exiting 06:27:35 (5556): No heartbeat from core client for 30 sec - exiting 06:27:36 (5556): No heartbeat from core client for 30 sec - exiting 06:27:37 (5556): No heartbeat from core client for 30 sec - exiting 06:27:39 (5556): No heartbeat from core client for 30 sec - exiting 06:27:40 (5556): No heartbeat from core client for 30 sec - exiting 06:27:41 (5556): No heartbeat from core client for 30 sec - exiting 06:27:42 (5556): No heartbeat from core client for 30 sec - exiting 06:27:43 (5556): No heartbeat from core client for 30 sec - exiting 06:27:44 (5556): No heartbeat from core client for 30 sec - exiting 06:27:45 (5556): No heartbeat from core client for 30 sec - exiting 06:27:46 (5556): No heartbeat from core client for 30 sec - exiting 06:27:47 (5556): No heartbeat from core client for 30 sec - exiting 06:27:48 (5556): No heartbeat from core client for 30 sec - exiting 06:27:49 (5556): No heartbeat from core client for 30 sec - exiting 06:27:51 (5556): No heartbeat from core client for 30 sec - exiting 06:27:52 (5556): No heartbeat from core client for 30 sec - exiting 06:27:53 (5556): No heartbeat from core client for 30 sec - exiting 06:27:54 (5556): No heartbeat from core client for 30 sec - exiting 06:27:55 (5556): No heartbeat from core client for 30 sec - exiting 06:27:56 (5556): No heartbeat from core client for 30 sec - exiting 06:27:57 (5556): No heartbeat from core client for 30 sec - exiting 06:27:58 (5556): No heartbeat from core client for 30 sec - exiting 06:27:59 (5556): No heartbeat from core client for 30 sec - exiting 06:28:00 (5556): No heartbeat from core client for 30 sec - exiting 06:28:01 (5556): No heartbeat from core client for 30 sec - exiting 06:28:03 (5556): No heartbeat from core client for 30 sec - exiting 06:28:04 (5556): No heartbeat from core client for 30 sec - exiting 06:28:05 (5556): No heartbeat from core client for 30 sec - exiting 06:28:06 (5556): No heartbeat from core client for 30 sec - exiting 06:28:07 (5556): No heartbeat from core client for 30 sec - exiting 06:28:08 (5556): No heartbeat from core client for 30 sec - exiting 06:28:09 (5556): No heartbeat from core client for 30 sec - exiting 06:28:10 (5556): No heartbeat from core client for 30 sec - exiting 06:28:11 (5556): No heartbeat from core client for 30 sec - exiting 06:28:12 (5556): No heartbeat from core client for 30 sec - exiting 06:28:14 (5556): No heartbeat from core client for 30 sec - exiting 06:28:15 (5556): No heartbeat from core client for 30 sec - exiting 06:28:16 (5556): No heartbeat from core client for 30 sec - exiting 06:28:17 (5556): No heartbeat from core client for 30 sec - exiting 06:28:18 (5556): No heartbeat from core client for 30 sec - exiting 06:28:19 (5556): No heartbeat from core client for 30 sec - exiting 06:28:20 (5556): No heartbeat from core client for 30 sec - exiting 06:28:21 (5556): No heartbeat from core client for 30 sec - exiting 06:28:22 (5556): No heartbeat from core client for 30 sec - exiting 06:28:23 (5556): No heartbeat from core client for 30 sec - exiting 06:28:24 (5556): No heartbeat from core client for 30 sec - exiting 06:28:26 (5556): No heartbeat from core client for 30 sec - exiting 06:28:27 (5556): No heartbeat from core client for 30 sec - exiting 06:28:28 (5556): No heartbeat from core client for 30 sec - exiting 06:28:29 (5556): No heartbeat from core client for 30 sec - exiting 06:28:30 (5556): No heartbeat from core client for 30 sec - exiting 06:28:31 (5556): No heartbeat from core client for 30 sec - exiting 06:28:32 (5556): No heartbeat from core client for 30 sec - exiting 06:28:33 (5556): No heartbeat from core client for 30 sec - exiting 06:28:34 (5556): No heartbeat from core client for 30 sec - exiting 06:28:35 (5556): No heartbeat from core client for 30 sec - exiting 06:28:36 (5556): No heartbeat from core client for 30 sec - exiting 06:28:38 (5556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8684, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9320, iMonCtr=2 Leaving CPDN_Main::Monitor... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:12:24 (844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7804, iMonCtr=2 Model crash detected, will try to restart... 20:15:06 (7180): No heartbeat from core client for 30 sec - exiting 20:15:07 (7180): No heartbeat from core client for 30 sec - exiting 20:15:08 (7180): No heartbeat from core client for 30 sec - exiting 20:15:09 (7180): No heartbeat from core client for 30 sec - exiting 20:15:10 (7180): No heartbeat from core client for 30 sec - exiting 20:15:11 (7180): No heartbeat from core client for 30 sec - exiting 20:15:12 (7180): No heartbeat from core client for 30 sec - exiting 20:15:14 (7180): No heartbeat from core client for 30 sec - exiting 20:15:15 (7180): No heartbeat from core client for 30 sec - exiting 20:15:16 (7180): No heartbeat from core client for 30 sec - exiting 20:15:17 (7180): No heartbeat from core client for 30 sec - exiting 20:15:18 (7180): No heartbeat from core client for 30 sec - exiting 20:15:19 (7180): No heartbeat from core client for 30 sec - exiting 20:15:20 (7180): No heartbeat from core client for 30 sec - exiting 20:15:21 (7180): No heartbeat from core client for 30 sec - exiting 20:15:22 (7180): No heartbeat from core client for 30 sec - exiting 20:15:23 (7180): No heartbeat from core client for 30 sec - exiting 20:15:25 (7180): No heartbeat from core client for 30 sec - exiting 20:15:26 (7180): No heartbeat from core client for 30 sec - exiting 20:15:27 (7180): No heartbeat from core client for 30 sec - exiting 20:15:28 (7180): No heartbeat from core client for 30 sec - exiting 20:15:29 (7180): No heartbeat from core client for 30 sec - exiting 20:15:30 (7180): No heartbeat from core client for 30 sec - exiting 20:15:31 (7180): No heartbeat from core client for 30 sec - exiting 20:15:32 (7180): No heartbeat from core client for 30 sec - exiting 20:15:33 (7180): No heartbeat from core client for 30 sec - exiting 20:15:34 (7180): No heartbeat from core client for 30 sec - exiting 20:15:35 (7180): No heartbeat from core client for 30 sec - exiting 20:15:37 (7180): No heartbeat from core client for 30 sec - exiting 20:15:38 (7180): No heartbeat from core client for 30 sec - exiting 20:15:39 (7180): No heartbeat from core client for 30 sec - exiting 20:15:40 (7180): No heartbeat from core client for 30 sec - exiting 20:15:41 (7180): No heartbeat from core client for 30 sec - exiting 20:15:42 (7180): No heartbeat from core client for 30 sec - exiting 20:15:43 (7180): No heartbeat from core client for 30 sec - exiting 20:15:44 (7180): No heartbeat from core client for 30 sec - exiting 20:15:45 (7180): No heartbeat from core client for 30 sec - exiting 20:15:46 (7180): No heartbeat from core client for 30 sec - exiting 20:15:47 (7180): No heartbeat from core client for 30 sec - exiting 20:15:49 (7180): No heartbeat from core client for 30 sec - exiting 20:15:50 (7180): No heartbeat from core client for 30 sec - exiting 20:15:51 (7180): No heartbeat from core client for 30 sec - exiting 20:15:52 (7180): No heartbeat from core client for 30 sec - exiting 20:15:53 (7180): No heartbeat from core client for 30 sec - exiting 20:15:54 (7180): No heartbeat from core client for 30 sec - exiting 20:15:55 (7180): No heartbeat from core client for 30 sec - exiting 20:15:56 (7180): No heartbeat from core client for 30 sec - exiting 20:15:57 (7180): No heartbeat from core client for 30 sec - exiting 20:15:58 (7180): No heartbeat from core client for 30 sec - exiting 20:15:59 (7180): No heartbeat from core client for 30 sec - exiting 20:16:01 (7180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9708, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7180, selfPID=9292, iMonCtr=1 Model crash detected, will try to restart... 10:06:12 (5464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8648, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8716, selfPID=8260, iMonCtr=1 Model crash detected, will try to restart... 18:59:43 (424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:36:01 (9328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:50 (10616): No heartbeat from core client for 30 sec - exiting 20:36:51 (10616): No heartbeat from core client for 30 sec - exiting 20:36:52 (10616): No heartbeat from core client for 30 sec - exiting 20:36:53 (10616): No heartbeat from core client for 30 sec - exiting 20:36:54 (10616): No heartbeat from core client for 30 sec - exiting 20:36:55 (10616): No heartbeat from core client for 30 sec - exiting 20:36:56 (10616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=10776, selfPID=10776, iMonCtr=2 Global Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=8856, iMonCtr=1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3060, selfPID=2852, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_3.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_4.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_5.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o9ng_2012_1_008629290_2_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 May 2014 09:52:45	1322841	16579784	hadam3p_anz_o9ng_2012_1_008629290_2	23,339	184,033	7.8852
28 Apr 2014 12:11:49	1322841	16579784	hadam3p_anz_o9ng_2012_1_008629290_2	11,819	92,816	7.8531