Task 19193133

Name	hadam3p_anz_h12h_200612_12_289_010255628_0
Workunit	10255628
Created	14 Jan 2016, 12:29:28 UTC
Sent	14 Jan 2016, 14:10:04 UTC
Report deadline	26 Dec 2016, 19:30:04 UTC
Received	4 Feb 2016, 11:37:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1323943
Run time	3 days 21 hours 56 min 32 sec
CPU time	3 days 9 hours 54 min 17 sec
Validate state	Invalid
Credit	2,993.82
Device peak FLOPS	2.99 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... 13:58:26 (8396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:19:07 (4444): No heartbeat from core client for 30 sec - exiting 08:19:08 (4444): No heartbeat from core client for 30 sec - exiting 08:19:09 (4444): No heartbeat from core client for 30 sec - exiting 08:19:10 (4444): No heartbeat from core client for 30 sec - exiting 08:19:11 (4444): No heartbeat from core client for 30 sec - exiting 08:19:12 (4444): No heartbeat from core client for 30 sec - exiting 08:19:13 (4444): No heartbeat from core client for 30 sec - exiting 08:19:14 (4444): No heartbeat from core client for 30 sec - exiting 08:19:16 (4444): No heartbeat from core client for 30 sec - exiting 08:19:17 (4444): No heartbeat from core client for 30 sec - exiting 08:19:18 (4444): No heartbeat from core client for 30 sec - exiting 08:19:19 (4444): No heartbeat from core client for 30 sec - exiting 08:19:20 (4444): No heartbeat from core client for 30 sec - exiting 08:19:21 (4444): No heartbeat from core client for 30 sec - exiting 08:19:22 (4444): No heartbeat from core client for 30 sec - exiting 08:19:23 (4444): No heartbeat from core client for 30 sec - exiting 08:19:24 (4444): No heartbeat from core client for 30 sec - exiting 08:19:25 (4444): No heartbeat from core client for 30 sec - exiting 08:19:27 (4444): No heartbeat from core client for 30 sec - exiting 08:19:28 (4444): No heartbeat from core client for 30 sec - exiting 08:19:29 (4444): No heartbeat from core client for 30 sec - exiting 08:19:30 (4444): No heartbeat from core client for 30 sec - exiting 08:19:31 (4444): No heartbeat from core client for 30 sec - exiting 08:19:32 (4444): No heartbeat from core client for 30 sec - exiting 08:19:33 (4444): No heartbeat from core client for 30 sec - exiting 08:19:34 (4444): No heartbeat from core client for 30 sec - exiting 08:19:35 (4444): No heartbeat from core client for 30 sec - exiting 08:19:36 (4444): No heartbeat from core client for 30 sec - exiting 08:19:37 (4444): No heartbeat from core client for 30 sec - exiting 08:19:39 (4444): No heartbeat from core client for 30 sec - exiting 08:19:40 (4444): No heartbeat from core client for 30 sec - exiting 08:19:41 (4444): No heartbeat from core client for 30 sec - exiting 08:19:42 (4444): No heartbeat from core client for 30 sec - exiting 08:19:43 (4444): No heartbeat from core client for 30 sec - exiting 08:19:44 (4444): No heartbeat from core client for 30 sec - exiting 08:19:45 (4444): No heartbeat from core client for 30 sec - exiting 08:19:46 (4444): No heartbeat from core client for 30 sec - exiting 08:19:47 (4444): No heartbeat from core client for 30 sec - exiting 08:19:48 (4444): No heartbeat from core client for 30 sec - exiting 08:19:49 (4444): No heartbeat from core client for 30 sec - exiting 08:19:51 (4444): No heartbeat from core client for 30 sec - exiting 08:19:52 (4444): No heartbeat from core client for 30 sec - exiting 08:19:53 (4444): No heartbeat from core client for 30 sec - exiting 08:19:54 (4444): No heartbeat from core client for 30 sec - exiting 08:19:55 (4444): No heartbeat from core client for 30 sec - exiting 08:19:56 (4444): No heartbeat from core client for 30 sec - exiting 08:19:57 (4444): No heartbeat from core client for 30 sec - exiting 08:19:58 (4444): No heartbeat from core client for 30 sec - exiting 08:19:59 (4444): No heartbeat from core client for 30 sec - exiting 08:20:00 (4444): No heartbeat from core client for 30 sec - exiting 08:20:01 (4444): No heartbeat from core client for 30 sec - exiting 08:20:03 (4444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:04 (4444): No heartbeat from core client for 30 sec - exiting 08:32:41 (7200): No heartbeat from core client for 30 sec - exiting 08:32:42 (7200): No heartbeat from core client for 30 sec - exiting 08:32:43 (7200): No heartbeat from core client for 30 sec - exiting 08:32:44 (7200): No heartbeat from core client for 30 sec - exiting 08:32:45 (7200): No heartbeat from core client for 30 sec - exiting 08:32:46 (7200): No heartbeat from core client for 30 sec - exiting 08:32:48 (7200): No heartbeat from core client for 30 sec - exiting 08:32:49 (7200): No heartbeat from core client for 30 sec - exiting 08:32:50 (7200): No heartbeat from core client for 30 sec - exiting 08:32:51 (7200): No heartbeat from core client for 30 sec - exiting 08:32:52 (7200): No heartbeat from core client for 30 sec - exiting 08:32:53 (7200): No heartbeat from core client for 30 sec - exiting 08:32:54 (7200): No heartbeat from core client for 30 sec - exiting 08:32:55 (7200): No heartbeat from core client for 30 sec - exiting 08:32:56 (7200): No heartbeat from core client for 30 sec - exiting 08:32:57 (7200): No heartbeat from core client for 30 sec - exiting 08:32:58 (7200): No heartbeat from core client for 30 sec - exiting 08:33:00 (7200): No heartbeat from core client for 30 sec - exiting 08:33:01 (7200): No heartbeat from core client for 30 sec - exiting 08:33:02 (7200): No heartbeat from core client for 30 sec - exiting 08:33:03 (7200): No heartbeat from core client for 30 sec - exiting 08:33:04 (7200): No heartbeat from core client for 30 sec - exiting 08:33:05 (7200): No heartbeat from core client for 30 sec - exiting 08:33:06 (7200): No heartbeat from core client for 30 sec - exiting 08:33:07 (7200): No heartbeat from core client for 30 sec - exiting 08:33:08 (7200): No heartbeat from core client for 30 sec - exiting 08:33:09 (7200): No heartbeat from core client for 30 sec - exiting 08:33:10 (7200): No heartbeat from core client for 30 sec - exiting 08:33:12 (7200): No heartbeat from core client for 30 sec - exiting 08:33:13 (7200): No heartbeat from core client for 30 sec - exiting 08:33:14 (7200): No heartbeat from core client for 30 sec - exiting 08:33:15 (7200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:33:16 (7200): No heartbeat from core client for 30 sec - exiting 08:33:17 (7200): No heartbeat from core client for 30 sec - exiting 08:33:18 (7200): No heartbeat from core client for 30 sec - exiting 08:33:19 (7200): No heartbeat from core client for 30 sec - exiting 08:33:20 (7200): No heartbeat from core client for 30 sec - exiting 08:43:52 (5332): No heartbeat from core client for 30 sec - exiting 08:43:53 (5332): No heartbeat from core client for 30 sec - exiting 08:43:54 (5332): No heartbeat from core client for 30 sec - exiting 08:43:56 (5332): No heartbeat from core client for 30 sec - exiting 08:43:57 (5332): No heartbeat from core client for 30 sec - exiting 08:43:58 (5332): No heartbeat from core client for 30 sec - exiting 08:43:59 (5332): No heartbeat from core client for 30 sec - exiting 08:44:00 (5332): No heartbeat from core client for 30 sec - exiting 08:44:01 (5332): No heartbeat from core client for 30 sec - exiting 08:44:02 (5332): No heartbeat from core client for 30 sec - exiting 08:44:03 (5332): No heartbeat from core client for 30 sec - exiting 08:44:04 (5332): No heartbeat from core client for 30 sec - exiting 08:44:06 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:07 (5332): No heartbeat from core client for 30 sec - exiting 08:44:08 (5332): No heartbeat from core client for 30 sec - exiting 08:45:17 (6932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:00:00 (1448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:16:02 (7160): No heartbeat from core client for 30 sec - exiting 14:16:03 (7160): No heartbeat from core client for 30 sec - exiting 14:16:04 (7160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:47 (1452): No heartbeat from core client for 30 sec - exiting 08:44:49 (1452): No heartbeat from core client for 30 sec - exiting 08:44:50 (1452): No heartbeat from core client for 30 sec - exiting 08:44:51 (1452): No heartbeat from core client for 30 sec - exiting 08:44:52 (1452): No heartbeat from core client for 30 sec - exiting 08:44:53 (1452): No heartbeat from core client for 30 sec - exiting 08:44:54 (1452): No heartbeat from core client for 30 sec - exiting 08:44:55 (1452): No heartbeat from core client for 30 sec - exiting 08:44:56 (1452): No heartbeat from core client for 30 sec - exiting 08:44:57 (1452): No heartbeat from core client for 30 sec - exiting 08:44:58 (1452): No heartbeat from core client for 30 sec - exiting 08:44:59 (1452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:45:01 (1452): No heartbeat from core client for 30 sec - exiting 08:45:02 (1452): No heartbeat from core client for 30 sec - exiting 08:45:03 (1452): No heartbeat from core client for 30 sec - exiting 08:45:04 (1452): No heartbeat from core client for 30 sec - exiting 08:47:07 (7756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6420, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 08:06:07 (3996): No heartbeat from core client for 30 sec - exiting 08:06:08 (3996): No heartbeat from core client for 30 sec - exiting 08:06:09 (3996): No heartbeat from core client for 30 sec - exiting 08:06:10 (3996): No heartbeat from core client for 30 sec - exiting 08:06:11 (3996): No heartbeat from core client for 30 sec - exiting 08:06:12 (3996): No heartbeat from core client for 30 sec - exiting 08:06:13 (3996): No heartbeat from core client for 30 sec - exiting 08:06:14 (3996): No heartbeat from core client for 30 sec - exiting 08:06:15 (3996): No heartbeat from core client for 30 sec - exiting 08:06:16 (3996): No heartbeat from core client for 30 sec - exiting 08:06:18 (3996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:59:03 (6440): No heartbeat from core client for 30 sec - exiting 08:59:04 (6440): No heartbeat from core client for 30 sec - exiting 08:59:05 (6440): No heartbeat from core client for 30 sec - exiting 08:59:07 (6440): No heartbeat from core client for 30 sec - exiting 08:59:08 (6440): No heartbeat from core client for 30 sec - exiting 08:59:09 (6440): No heartbeat from core client for 30 sec - exiting 08:59:10 (6440): No heartbeat from core client for 30 sec - exiting 08:59:11 (6440): No heartbeat from core client for 30 sec - exiting 08:59:12 (6440): No heartbeat from core client for 30 sec - exiting 08:59:13 (6440): No heartbeat from core client for 30 sec - exiting 08:59:14 (6440): No heartbeat from core client for 30 sec - exiting 08:59:15 (6440): No heartbeat from core client for 30 sec - exiting 08:59:16 (6440): No heartbeat from core client for 30 sec - exiting 08:59:17 (6440): No heartbeat from core client for 30 sec - exiting 08:59:19 (6440): No heartbeat from core client for 30 sec - exiting 08:59:20 (6440): No heartbeat from core client for 30 sec - exiting 08:59:21 (6440): No heartbeat from core client for 30 sec - exiting 08:59:22 (6440): No heartbeat from core client for 30 sec - exiting 08:59:23 (6440): No heartbeat from core client for 30 sec - exiting 08:59:24 (6440): No heartbeat from core client for 30 sec - exiting 08:59:25 (6440): No heartbeat from core client for 30 sec - exiting 08:59:26 (6440): No heartbeat from core client for 30 sec - exiting 08:59:27 (6440): No heartbeat from core client for 30 sec - exiting 08:59:28 (6440): No heartbeat from core client for 30 sec - exiting 08:59:29 (6440): No heartbeat from core client for 30 sec - exiting 08:59:31 (6440): No heartbeat from core client for 30 sec - exiting 08:59:32 (6440): No heartbeat from core client for 30 sec - exiting 08:59:33 (6440): No heartbeat from core client for 30 sec - exiting 08:59:34 (6440): No heartbeat from core client for 30 sec - exiting 08:59:35 (6440): No heartbeat from core client for 30 sec - exiting 08:59:36 (6440): No heartbeat from core client for 30 sec - exiting 08:59:37 (6440): No heartbeat from core client for 30 sec - exiting 08:59:38 (6440): No heartbeat from core client for 30 sec - exiting 08:59:39 (6440): No heartbeat from core client for 30 sec - exiting 08:59:40 (6440): No heartbeat from core client for 30 sec - exiting 08:59:41 (6440): No heartbeat from core client for 30 sec - exiting 08:59:43 (6440): No heartbeat from core client for 30 sec - exiting 08:59:44 (6440): No heartbeat from core client for 30 sec - exiting 08:59:45 (6440): No heartbeat from core client for 30 sec - exiting 08:59:46 (6440): No heartbeat from core client for 30 sec - exiting 08:59:47 (6440): No heartbeat from core client for 30 sec - exiting 08:59:48 (6440): No heartbeat from core client for 30 sec - exiting 08:59:49 (6440): No heartbeat from core client for 30 sec - exiting 08:59:50 (6440): No heartbeat from core client for 30 sec - exiting 08:59:51 (6440): No heartbeat from core client for 30 sec - exiting 08:59:52 (6440): No heartbeat from core client for 30 sec - exiting 08:59:53 (6440): No heartbeat from core client for 30 sec - exiting 08:59:55 (6440): No heartbeat from core client for 30 sec - exiting 08:59:56 (6440): No heartbeat from core client for 30 sec - exiting 08:59:57 (6440): No heartbeat from core client for 30 sec - exiting 08:59:58 (6440): No heartbeat from core client for 30 sec - exiting 08:59:59 (6440): No heartbeat from core client for 30 sec - exiting 09:00:00 (6440): No heartbeat from core client for 30 sec - exiting 09:00:01 (6440): No heartbeat from core client for 30 sec - exiting 09:00:02 (6440): No heartbeat from core client for 30 sec - exiting 09:00:03 (6440): No heartbeat from core client for 30 sec - exiting 09:00:04 (6440): No heartbeat from core client for 30 sec - exiting 09:00:06 (6440): No heartbeat from core client for 30 sec - exiting 09:00:07 (6440): No heartbeat from core client for 30 sec - exiting 09:00:08 (6440): No heartbeat from core client for 30 sec - exiting 09:00:09 (6440): No heartbeat from core client for 30 sec - exiting 09:00:10 (6440): No heartbeat from core client for 30 sec - exiting 09:00:11 (6440): No heartbeat from core client for 30 sec - exiting 09:00:12 (6440): No heartbeat from core client for 30 sec - exiting 09:00:13 (6440): No heartbeat from core client for 30 sec - exiting 09:00:14 (6440): No heartbeat from core client for 30 sec - exiting 09:00:15 (6440): No heartbeat from core client for 30 sec - exiting 09:00:16 (6440): No heartbeat from core client for 30 sec - exiting 09:00:18 (6440): No heartbeat from core client for 30 sec - exiting 09:00:19 (6440): No heartbeat from core client for 30 sec - exiting 09:00:20 (6440): No heartbeat from core client for 30 sec - exiting 09:00:21 (6440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:00:22 (6440): No heartbeat from core client for 30 sec - exiting 09:00:23 (6440): No heartbeat from core client for 30 sec - exiting Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8120, selfPID=8120, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 09:38:46 (8144): No heartbeat from core client for 30 sec - exiting 09:38:47 (8144): No heartbeat from core client for 30 sec - exiting 09:38:48 (8144): No heartbeat from core client for 30 sec - exiting 09:38:49 (8144): No heartbeat from core client for 30 sec - exiting 09:38:50 (8144): No heartbeat from core client for 30 sec - exiting 09:38:51 (8144): No heartbeat from core client for 30 sec - exiting 09:38:52 (8144): No heartbeat from core client for 30 sec - exiting 09:38:53 (8144): No heartbeat from core client for 30 sec - exiting 09:38:55 (8144): No heartbeat from core client for 30 sec - exiting 09:38:56 (8144): No heartbeat from core client for 30 sec - exiting 09:38:57 (8144): No heartbeat from core client for 30 sec - exiting 09:38:58 (8144): No heartbeat from core client for 30 sec - exiting 09:38:59 (8144): No heartbeat from core client for 30 sec - exiting 09:39:00 (8144): No heartbeat from core client for 30 sec - exiting 09:39:01 (8144): No heartbeat from core client for 30 sec - exiting 09:39:02 (8144): No heartbeat from core client for 30 sec - exiting 09:39:03 (8144): No heartbeat from core client for 30 sec - exiting 09:39:04 (8144): No heartbeat from core client for 30 sec - exiting 09:39:06 (8144): No heartbeat from core client for 30 sec - exiting 09:39:07 (8144): No heartbeat from core client for 30 sec - exiting 09:39:08 (8144): No heartbeat from core client for 30 sec - exiting 09:39:09 (8144): No heartbeat from core client for 30 sec - exiting 09:39:10 (8144): No heartbeat from core client for 30 sec - exiting 09:39:11 (8144): No heartbeat from core client for 30 sec - exiting 09:39:12 (8144): No heartbeat from core client for 30 sec - exiting 09:39:13 (8144): No heartbeat from core client for 30 sec - exiting 09:39:14 (8144): No heartbeat from core client for 30 sec - exiting 09:39:15 (8144): No heartbeat from core client for 30 sec - exiting 09:39:16 (8144): No heartbeat from core client for 30 sec - exiting 09:39:18 (8144): No heartbeat from core client for 30 sec - exiting 09:39:19 (8144): No heartbeat from core client for 30 sec - exiting 09:39:20 (8144): No heartbeat from core client for 30 sec - exiting 09:39:21 (8144): No heartbeat from core client for 30 sec - exiting 09:39:22 (8144): No heartbeat from core client for 30 sec - exiting 09:39:23 (8144): No heartbeat from core client for 30 sec - exiting 09:39:24 (8144): No heartbeat from core client for 30 sec - exiting 09:39:25 (8144): No heartbeat from core client for 30 sec - exiting 09:39:26 (8144): No heartbeat from core client for 30 sec - exiting 09:39:27 (8144): No heartbeat from core client for 30 sec - exiting 09:39:29 (8144): No heartbeat from core client for 30 sec - exiting 09:39:30 (8144): No heartbeat from core client for 30 sec - exiting 09:39:31 (8144): No heartbeat from core client for 30 sec - exiting 09:39:32 (8144): No heartbeat from core client for 30 sec - exiting 09:39:33 (8144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:39:34 (8144): No heartbeat from core client for 30 sec - exiting 09:39:35 (8144): No heartbeat from core client for 30 sec - exiting 09:40:40 (7020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:22:31 (8644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1508, selfPID=3844, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_h12h_200612_12_289_010255628_0_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_h12h_200612_12_289_010255628_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_h12h_200612_12_289_010255628_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_h12h_200612_12_289_010255628_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_h12h_200612_12_289_010255628_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_h12h_200612_12_289_010255628_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Feb 2016 19:08:35	1323943	19193133	hadam3p_anz_h12h_200612_12_289_010255628_0	69,419	266,537	3.8395
28 Jan 2016 15:41:15	1323943	19193133	hadam3p_anz_h12h_200612_12_289_010255628_0	57,899	219,017	3.7827
26 Jan 2016 15:47:27	1323943	19193133	hadam3p_anz_h12h_200612_12_289_010255628_0	46,379	172,639	3.7224
21 Jan 2016 18:53:39	1323943	19193133	hadam3p_anz_h12h_200612_12_289_010255628_0	34,859	130,213	3.7354
20 Jan 2016 13:51:07	1323943	19193133	hadam3p_anz_h12h_200612_12_289_010255628_0	23,339	88,377	3.7867
18 Jan 2016 15:38:29	1323943	19193133	hadam3p_anz_h12h_200612_12_289_010255628_0	11,819	44,990	3.8066