Task 16640878

Name	hadam3p_anz_o4o9_2012_1_008622839_1
Workunit	8769351
Created	16 May 2014, 15:20:58 UTC
Sent	16 May 2014, 15:31:09 UTC
Report deadline	28 Apr 2015, 20:51:09 UTC
Received	14 Jun 2014, 8:58:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1307130
Run time	1 days 18 hours 2 min 39 sec
CPU time	1 days 17 hours 39 min 39 sec
Validate state	Invalid
Credit	1,006.54
Device peak FLOPS	2.51 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5364, selfPID=4148, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5452, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... 16:09:42 (4360): No heartbeat from core client for 30 sec - exiting 16:09:43 (4360): No heartbeat from core client for 30 sec - exiting 16:09:44 (4360): No heartbeat from core client for 30 sec - exiting 16:09:45 (4360): No heartbeat from core client for 30 sec - exiting 16:09:46 (4360): No heartbeat from core client for 30 sec - exiting 16:09:47 (4360): No heartbeat from core client for 30 sec - exiting 16:09:48 (4360): No heartbeat from core client for 30 sec - exiting 16:09:49 (4360): No heartbeat from core client for 30 sec - exiting 16:09:50 (4360): No heartbeat from core client for 30 sec - exiting 16:09:51 (4360): No heartbeat from core client for 30 sec - exiting 16:09:52 (4360): No heartbeat from core client for 30 sec - exiting 16:09:54 (4360): No heartbeat from core client for 30 sec - exiting 16:09:55 (4360): No heartbeat from core client for 30 sec - exiting 16:09:56 (4360): No heartbeat from core client for 30 sec - exiting 16:09:57 (4360): No heartbeat from core client for 30 sec - exiting 16:09:58 (4360): No heartbeat from core client for 30 sec - exiting 16:09:59 (4360): No heartbeat from core client for 30 sec - exiting 16:10:00 (4360): No heartbeat from core client for 30 sec - exiting 16:10:01 (4360): No heartbeat from core client for 30 sec - exiting 16:10:02 (4360): No heartbeat from core client for 30 sec - exiting 16:10:03 (4360): No heartbeat from core client for 30 sec - exiting 16:10:04 (4360): No heartbeat from core client for 30 sec - exiting 16:10:06 (4360): No heartbeat from core client for 30 sec - exiting 16:10:07 (4360): No heartbeat from core client for 30 sec - exiting 16:10:08 (4360): No heartbeat from core client for 30 sec - exiting 16:10:09 (4360): No heartbeat from core client for 30 sec - exiting 16:10:10 (4360): No heartbeat from core client for 30 sec - exiting 16:10:11 (4360): No heartbeat from core client for 30 sec - exiting 16:10:12 (4360): No heartbeat from core client for 30 sec - exiting 16:10:13 (4360): No heartbeat from core client for 30 sec - exiting 16:10:14 (4360): No heartbeat from core client for 30 sec - exiting 16:10:15 (4360): No heartbeat from core client for 30 sec - exiting 16:10:16 (4360): No heartbeat from core client for 30 sec - exiting 16:10:18 (4360): No heartbeat from core client for 30 sec - exiting 16:10:19 (4360): No heartbeat from core client for 30 sec - exiting 16:10:20 (4360): No heartbeat from core client for 30 sec - exiting 16:10:21 (4360): No heartbeat from core client for 30 sec - exiting 16:10:22 (4360): No heartbeat from core client for 30 sec - exiting 16:10:23 (4360): No heartbeat from core client for 30 sec - exiting 16:10:24 (4360): No heartbeat from core client for 30 sec - exiting 16:10:25 (4360): No heartbeat from core client for 30 sec - exiting 16:10:26 (4360): No heartbeat from core client for 30 sec - exiting 16:10:27 (4360): No heartbeat from core client for 30 sec - exiting 16:10:28 (4360): No heartbeat from core client for 30 sec - exiting 16:10:30 (4360): No heartbeat from core client for 30 sec - exiting 16:10:31 (4360): No heartbeat from core client for 30 sec - exiting 16:10:32 (4360): No heartbeat from core client for 30 sec - exiting 16:10:33 (4360): No heartbeat from core client for 30 sec - exiting 16:10:34 (4360): No heartbeat from core client for 30 sec - exiting 16:10:35 (4360): No heartbeat from core client for 30 sec - exiting 16:10:36 (4360): No heartbeat from core client for 30 sec - exiting 16:10:37 (4360): No heartbeat from core client for 30 sec - exiting 16:10:38 (4360): No heartbeat from core client for 30 sec - exiting 16:10:39 (4360): No heartbeat from core client for 30 sec - exiting 16:10:40 (4360): No heartbeat from core client for 30 sec - exiting 16:10:42 (4360): No heartbeat from core client for 30 sec - exiting 16:10:43 (4360): No heartbeat from core client for 30 sec - exiting 16:10:44 (4360): No heartbeat from core client for 30 sec - exiting 16:10:45 (4360): No heartbeat from core client for 30 sec - exiting 16:10:46 (4360): No heartbeat from core client for 30 sec - exiting 16:10:47 (4360): No heartbeat from core client for 30 sec - exiting 16:10:48 (4360): No heartbeat from core client for 30 sec - exiting 16:10:49 (4360): No heartbeat from core client for 30 sec - exiting 16:10:50 (4360): No heartbeat from core client for 30 sec - exiting 16:10:51 (4360): No heartbeat from core client for 30 sec - exiting 16:10:52 (4360): No heartbeat from core client for 30 sec - exiting 16:10:54 (4360): No heartbeat from core client for 30 sec - exiting 16:10:55 (4360): No heartbeat from core client for 30 sec - exiting 16:10:56 (4360): No heartbeat from core client for 30 sec - exiting 16:10:57 (4360): No heartbeat from core client for 30 sec - exiting 16:10:58 (4360): No heartbeat from core client for 30 sec - exiting 16:10:59 (4360): No heartbeat from core client for 30 sec - exiting 16:11:00 (4360): No heartbeat from core client for 30 sec - exiting 16:11:01 (4360): No heartbeat from core client for 30 sec - exiting 16:11:02 (4360): No heartbeat from core client for 30 sec - exiting 16:11:03 (4360): No heartbeat from core client for 30 sec - exiting 16:11:04 (4360): No heartbeat from core client for 30 sec - exiting 16:11:06 (4360): No heartbeat from core client for 30 sec - exiting 16:11:07 (4360): No heartbeat from core client for 30 sec - exiting 16:11:08 (4360): No heartbeat from core client for 30 sec - exiting 16:11:09 (4360): No heartbeat from core client for 30 sec - exiting 16:11:10 (4360): No heartbeat from core client for 30 sec - exiting 16:11:11 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No Process HaCPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2880, selfPID=4088, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4688, selfPID=4060, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4300, selfPID=3256, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4776, selfPID=4416, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2668, iMonCtr=2 Model crash detected, will try to restart... 18:14:45 (5424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4648, selfPID=4648, iMonCtr=2 16:11:32 (2768): No heartbeat from core client for 30 sec - exiting 16:11:33 (2768): No heartbeat from core client for 30 sec - exiting 16:11:34 (2768): No heartbeat from core client for 30 sec - exiting 16:11:35 (2768): No heartbeat from core client for 30 sec - exiting 16:11:36 (2768): No heartbeat from core client for 30 sec - exiting 16:11:37 (2768): No heartbeat from core client for 30 sec - exiting 16:11:38 (2768): No heartbeat from core client for 30 sec - exiting 16:11:40 (2768): No heartbeat from core client for 30 sec - exiting 16:11:41 (2768): No heartbeat from core client for 30 sec - exiting 16:11:42 (2768): No heartbeat from core client for 30 sec - exiting 16:11:43 (2768): No heartbeat from core client for 30 sec - exiting 16:11:44 (2768): No heartbeat from core client for 30 sec - exiting 16:11:45 (2768): No heartbeat from core client for 30 sec - exiting 16:11:46 (2768): No heartbeat from core client for 30 sec - exiting 16:11:47 (2768): No heartbeat from core client for 30 sec - exiting 16:11:48 (2768): No heartbeat from core client for 30 sec - exiting 16:11:49 (2768): No heartbeat from core client for 30 sec - exiting 16:11:50 (2768): No heartbeat from core client for 30 sec - exiting 16:11:52 (2768): No heartbeat from core client for 30 sec - exiting 16:11:53 (2768): No heartbeat from core client for 30 sec - exiting 16:11:54 (2768): No heartbeat from core client for 30 sec - exiting 16:11:55 (2768): No heartbeat from core client for 30 sec - exiting 16:11:56 (2768): No heartbeat from core client for 30 sec - exiting 16:11:57 (2768): No heartbeat from core client for 30 sec - exiting 16:11:58 (2768): No heartbeat from core client for 30 sec - exiting 16:11:59 (2768): No heartbeat from core client for 30 sec - exiting 16:12:00 (2768): No heartbeat from core client for 30 sec - exiting 16:12:01 (2768): No heartbeat from core client for 30 sec - exiting 16:12:02 (2768): No heartbeat from core client for 30 sec - exiting 16:12:04 (2768): No heartbeat from core client for 30 sec - exiting 16:12:05 (2768): No heartbeat from core client for 30 sec - exiting 16:12:06 (2768): No heartbeat from core client for 30 sec - exiting 16:12:07 (2768): No heartbeat from core client for 30 sec - exiting 16:12:08 (2768): No heartbeat from core client for 30 sec - exiting 16:12:09 (2768): No heartbeat from core client for 30 sec - exiting 16:12:10 (2768): No heartbeat from core client for 30 sec - exiting 16:12:11 (2768): No heartbeat from core client for 30 sec - exiting 16:12:12 (2768): No heartbeat from core client for 30 sec - exiting 16:12:13 (2768): No heartbeat from core client for 30 sec - exiting 16:12:14 (2768): No heartbeat from core client for 30 sec - exiting 16:12:16 (2768): No heartbeat from core client for 30 sec - exiting 16:12:17 (2768): No heartbeat from core client for 30 sec - exiting 16:12:18 (2768): No heartbeat from core client for 30 sec - exiting 16:12:19 (2768): No heartbeat from core client for 30 sec - exiting 16:12:20 (2768): No heartbeat from core client for 30 sec - exiting 16:12:21 (2768): No heartbeat from core client for 30 sec - exiting 16:12:22 (2768): No heartbeat from core client for 30 sec - exiting 16:12:23 (2768): No heartbeat from core client for 30 sec - exiting 16:12:24 (2768): No heartbeat from core client for 30 sec - exiting 16:12:25 (2768): No heartbeat from core client for 30 sec - exiting 16:12:26 (2768): No heartbeat from core client for 30 sec - exiting 16:12:28 (2768): No heartbeat from core client for 30 sec - exiting 16:12:29 (2768): No heartbeat from core client for 30 sec - exiting 16:12:30 (2768): No heartbeat from core client for 30 sec - exiting 16:12:31 (2768): No heartbeat from core client for 30 sec - exiting 16:12:32 (2768): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3692, iMonCtr=2 Model crash detected, will try to restart... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3280, iMonCtr=2 Leaving CPDN_Main::Monitor... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5920, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5880, selfPID=5196, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3112, selfPID=4108, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4396, iMonCtr=2 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... GCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3324, selfPID=3324, iMonCtr=2 16:08:39 (2616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file D:\BOINC\data/projects/climateprediction.net/hadam3p_anz_o4o9_2012_1_008622839/dataout/atmos_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/xaakm.pipe_dummy 2048 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=0, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6080, selfPID=4216, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_3.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_4.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_5.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_o4o9_2012_1_008622839_1_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Jun 2014 17:17:37	1307130	16640878	hadam3p_anz_o4o9_2012_1_008622839_1	23,339	138,601	5.9386
23 May 2014 06:09:53	1307130	16640878	hadam3p_anz_o4o9_2012_1_008622839_1	11,819	71,935	6.0864