Task 13955027

Name	hadam3p_eu_88qa_2002_1_007661339_1
Workunit	7816426
Created	23 Jan 2012, 14:55:48 UTC
Sent	23 Jan 2012, 14:56:09 UTC
Report deadline	4 Jan 2013, 20:16:09 UTC
Received	5 Mar 2012, 17:48:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1191213
Run time	2 days 5 hours 51 min 2 sec
CPU time	2 days 2 hours 20 min 45 sec
Validate state	Invalid
Credit	1,591.48
Device peak FLOPS	3.32 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <stderr_txt> 17:24:57 (3604): No heartbeat from core client for 30 sec - exiting 17:24:58 (3604): No heartbeat from core client for 30 sec - exiting 17:24:59 (3604): No heartbeat from core client for 30 sec - exiting 17:25:00 (3604): No heartbeat from core client for 30 sec - exiting 17:25:01 (3604): No heartbeat from core client for 30 sec - exiting 17:25:02 (3604): No heartbeat from core client for 30 sec - exiting 17:25:03 (3604): No heartbeat from core client for 30 sec - exiting 17:25:04 (3604): No heartbeat from core client for 30 sec - exiting 17:25:06 (3604): No heartbeat from core client for 30 sec - exiting 17:25:07 (3604): No heartbeat from core client for 30 sec - exiting 17:25:08 (3604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4928, selfPID=4928, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=2 19:06:11 (4032): No heartbeat from core client for 30 sec - exiting 19:06:12 (4032): No heartbeat from core client for 30 sec - exiting 19:06:13 (4032): No heartbeat from core client for 30 sec - exiting 19:06:14 (4032): No heartbeat from core client for 30 sec - exiting 19:06:16 (4032): No heartbeat from core client for 30 sec - exiting 19:06:17 (4032): No heartbeat from core client for 30 sec - exiting 19:06:18 (4032): No heartbeat from core client for 30 sec - exiting 19:06:19 (4032): No heartbeat from core client for 30 sec - exiting 19:06:20 (4032): No heartbeat from core client for 30 sec - exiting 19:06:21 (4032): No heartbeat from core client for 30 sec - exiting 19:06:22 (4032): No heartbeat from core client for 30 sec - exiting 19:06:23 (4032): No heartbeat from core client for 30 sec - exiting 19:06:24 (4032): No heartbeat from core client for 30 sec - exiting 19:06:25 (4032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:52:07 (3536): No heartbeat from core client for 30 sec - exiting 20:52:08 (3536): No heartbeat from core client for 30 sec - exiting 20:52:09 (3536): No heartbeat from core client for 30 sec - exiting 20:52:10 (3536): No heartbeat from core client for 30 sec - exiting 20:52:11 (3536): No heartbeat from core client for 30 sec - exiting 20:52:12 (3536): No heartbeat from core client for 30 sec - exiting 20:52:13 (3536): No heartbeat from core client for 30 sec - exiting 20:52:15 (3536): No heartbeat from core client for 30 sec - exiting 20:52:16 (3536): No heartbeat from core client for 30 sec - exiting 20:52:17 (3536): No heartbeat from core client for 30 sec - exiting 20:52:18 (3536): No heartbeat from core client for 30 sec - exiting 20:52:19 (3536): No heartbeat from core client for 30 sec - exiting 20:52:20 (3536): No heartbeat from core client for 30 sec - exiting 20:52:21 (3536): No heartbeat from core client for 30 sec - exiting 20:52:22 (3536): No heartbeat from core client for 30 sec - exiting 20:52:56 (3536): No heartbeat from core client for 30 sec - exiting 20:52:57 (3536): No heartbeat from core client for 30 sec - exiting 20:52:58 (3536): No heartbeat from core client for 30 sec - exiting 20:52:59 (3536): No heartbeat from core client for 30 sec - exiting 20:53:00 (3536): No heartbeat from core client for 30 sec - exiting 20:53:01 (3536): No heartbeat from core client for 30 sec - exiting 20:53:02 (3536): No heartbeat from core client for 30 sec - exiting 20:53:03 (3536): No heartbeat from core client for 30 sec - exiting 20:53:04 (3536): No heartbeat from core client for 30 sec - exiting 20:53:05 (3536): No heartbeat from core client for 30 sec - exiting 20:53:06 (3536): No heartbeat from core client for 30 sec - exiting 20:53:08 (3536): No heartbeat from core client for 30 sec - exiting 20:53:09 (3536): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:14:22 (4204): No heartbeat from core client for 30 sec - exiting 10:14:25 (4204): No heartbeat from core client for 30 sec - exiting 10:14:26 (4204): No heartbeat from core client for 30 sec - exiting 10:14:27 (4204): No heartbeat from core client for 30 sec - exiting 10:14:28 (4204): No heartbeat from core client for 30 sec - exiting 10:14:29 (4204): No heartbeat from core client for 30 sec - exiting 10:14:30 (4204): No heartbeat from core client for 30 sec - exiting 10:14:31 (4204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:26:32 (3692): No heartbeat from core client for 30 sec - exiting 19:26:33 (3692): No heartbeat from core client for 30 sec - exiting 19:26:34 (3692): No heartbeat from core client for 30 sec - exiting 19:26:35 (3692): No heartbeat from core client for 30 sec - exiting 19:26:36 (3692): No heartbeat from core client for 30 sec - exiting 19:27:10 (3692): No heartbeat from core client for 30 sec - exiting 19:27:11 (3692): No heartbeat from core client for 30 sec - exiting 19:27:12 (3692): No heartbeat from core client for 30 sec - exiting 19:27:13 (3692): No heartbeat from core client for 30 sec - exiting 19:27:14 (3692): No heartbeat from core client for 30 sec - exiting 19:27:15 (3692): No heartbeat from core client for 30 sec - exiting 19:27:16 (3692): No heartbeat from core client for 30 sec - exiting 19:27:18 (3692): No heartbeat from core client for 30 sec - exiting 19:27:19 (3692): No heartbeat from core client for 30 sec - exiting 19:27:20 (3692): No heartbeat from core client for 30 sec - exiting 19:27:21 (3692): No heartbeat from core client for 30 sec - exiting 19:27:22 (3692): No heartbeat from core client for 30 sec - exiting 19:27:23 (3692): No heartbeat from core client for 30 sec - exiting 19:27:24 (3692): No heartbeat from core client for 30 sec - exiting 19:27:25 (3692): No heartbeat from core client for 30 sec - exiting 19:27:26 (3692): No heartbeat from core client for 30 sec - exiting 19:27:27 (3692): No heartbeat from core client for 30 sec - exiting 19:27:28 (3692): No heartbeat from core client for 30 sec - exiting 19:27:30 (3692): No heartbeat from core client for 30 sec - exiting 19:27:31 (3692): No heartbeat from core client for 30 sec - exiting 19:27:32 (3692): No heartbeat from core client for 30 sec - exiting 19:27:33 (3692): No heartbeat from core client for 30 sec - exiting 19:27:34 (3692): No heartbeat from core client for 30 sec - exiting 19:27:35 (3692): No heartbeat from core client for 30 sec - exiting 19:27:36 (3692): No heartbeat from core client for 30 sec - exiting 19:27:37 (3692): No heartbeat from core client for 30 sec - exiting 19:27:38 (3692): No heartbeat from core client for 30 sec - exiting 19:27:39 (3692): No heartbeat from core client for 30 sec - exiting 19:27:40 (3692): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:28:29 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3292, iMonCtr=2 08:57:21 (3972): No heartbeat from core client for 30 sec - exiting 08:57:22 (3972): No heartbeat from core client for 30 sec - exiting 08:57:23 (3972): No heartbeat from core client for 30 sec - exiting 08:57:24 (3972): No heartbeat from core client for 30 sec - exiting 08:57:25 (3972): No heartbeat from core client for 30 sec - exiting 08:57:26 (3972): No heartbeat from core client for 30 sec - exiting 08:57:28 (3972): No heartbeat from core client for 30 sec - exiting 08:58:01 (3972): No heartbeat from core client for 30 sec - exiting 08:58:02 (3972): No heartbeat from core client for 30 sec - exiting 08:58:03 (3972): No heartbeat from core client for 30 sec - exiting 08:58:04 (3972): No heartbeat from core client for 30 sec - exiting 08:58:05 (3972): No heartbeat from core client for 30 sec - exiting 08:58:06 (3972): No heartbeat from core client for 30 sec - exiting 08:58:08 (3972): No heartbeat from core client for 30 sec - exiting 08:58:09 (3972): No heartbeat from core client for 30 sec - exiting 08:58:10 (3972): No heartbeat from core client for 30 sec - exiting 08:58:11 (3972): No heartbeat from core client for 30 sec - exiting 08:58:12 (3972): No heartbeat from core client for 30 sec - exiting 08:58:13 (3972): No heartbeat from core client for 30 sec - exiting 08:58:14 (3972): No heartbeat from core client for 30 sec - exiting 08:58:15 (3972): No heartbeat from core client for 30 sec - exiting 08:58:16 (3972): No heartbeat from core client for 30 sec - exiting 08:58:17 (3972): No heartbeat from core client for 30 sec - exiting 08:58:18 (3972): No heartbeat from core client for 30 sec - exiting 08:58:20 (3972): No heartbeat from core client for 30 sec - exiting 08:58:21 (3972): No heartbeat from core client for 30 sec - exiting 08:58:22 (3972): No heartbeat from core client for 30 sec - exiting 08:58:23 (3972): No heartbeat from core client for 30 sec - exiting 08:58:24 (3972): No heartbeat from core client for 30 sec - exiting 08:58:25 (3972): No heartbeat from core client for 30 sec - exiting 08:58:26 (3972): No heartbeat from core client for 30 sec - exiting 08:58:27 (3972): No heartbeat from core client for 30 sec - exiting 08:58:28 (3972): No heartbeat from core client for 30 sec - exiting 08:58:29 (3972): No heartbeat from core client for 30 sec - exiting 08:58:30 (3972): No heartbeat from core client for 30 sec - exiting 08:58:32 (3972): No heartbeat from core client for 30 sec - exiting 08:58:33 (3972): No heartbeat from core client for 30 sec - exiting 08:58:34 (3972): No heartbeat from core client for 30 sec - exiting 08:58:35 (3972): No heartbeat from core client for 30 sec - exiting 08:58:36 (3972): No heartbeat from core client for 30 sec - exiting 08:58:37 (3972): No heartbeat from core client for 30 sec - exiting 08:58:38 (3972): No heartbeat from core client for 30 sec - exiting 08:58:39 (3972): No heartbeat from core client for 30 sec - exiting 08:58:40 (3972): No heartbeat from core client for 30 sec - exiting 08:58:41 (3972): No heartbeat from core client for 30 sec - exiting 08:58:42 (3972): No heartbeat from core client for 30 sec - exiting 08:58:44 (3972): No heartbeat from core client for 30 sec - exiting 08:58:45 (3972): No heartbeat from core client for 30 sec - exiting 08:58:46 (3972): No heartbeat from core client for 30 sec - exiting 08:58:47 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5488, selfPID=3972, iMonCtr=1 Model crash detected, will try to restart... 08:50:28 (3836): No heartbeat from core client for 30 sec - exiting 08:50:29 (3836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:41:25 (4220): No heartbeat from core client for 30 sec - exiting 18:41:26 (4220): No heartbeat from core client for 30 sec - exiting 18:41:27 (4220): No heartbeat from core client for 30 sec - exiting 18:41:28 (4220): No heartbeat from core client for 30 sec - exiting 18:41:29 (4220): No heartbeat from core client for 30 sec - exiting 18:41:30 (4220): No heartbeat from core client for 30 sec - exiting 18:41:31 (4220): No heartbeat from core client for 30 sec - exiting 18:41:32 (4220): No heartbeat from core client for 30 sec - exiting 18:41:34 (4220): No heartbeat from core client for 30 sec - exiting 18:41:35 (4220): No heartbeat from core client for 30 sec - exiting 18:41:36 (4220): No heartbeat from core client for 30 sec - exiting 18:41:37 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:46:29 (4140): No heartbeat from core client for 30 sec - exiting 17:46:30 (4140): No heartbeat from core client for 30 sec - exiting 17:46:32 (4140): No heartbeat from core client for 30 sec - exiting 17:46:33 (4140): No heartbeat from core client for 30 sec - exiting 17:47:06 (4140): No heartbeat from core client for 30 sec - exiting 17:47:08 (4140): No heartbeat from core client for 30 sec - exiting 17:47:09 (4140): No heartbeat from core client for 30 sec - exiting 17:47:10 (4140): No heartbeat from core client for 30 sec - exiting 17:47:11 (4140): No heartbeat from core client for 30 sec - exiting 17:47:12 (4140): No heartbeat from core client for 30 sec - exiting 17:47:13 (4140): No heartbeat from core client for 30 sec - exiting 17:47:14 (4140): No heartbeat from core client for 30 sec - exiting 17:47:15 (4140): No heartbeat from core client for 30 sec - exiting 17:47:16 (4140): No heartbeat from core client for 30 sec - exiting 17:47:17 (4140): No heartbeat from core client for 30 sec - exiting 17:47:18 (4140): No heartbeat from core client for 30 sec - exiting 17:47:20 (4140): No heartbeat from core client for 30 sec - exiting 17:47:21 (4140): No heartbeat from core client for 30 sec - exiting 17:47:22 (4140): No heartbeat from core client for 30 sec - exiting 17:47:23 (4140): No heartbeat from core client for 30 sec - exiting 17:47:24 (4140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:30:49 (3408): No heartbeat from core client for 30 sec - exiting 18:30:50 (3408): No heartbeat from core client for 30 sec - exiting 18:30:51 (3408): No heartbeat from core client for 30 sec - exiting 18:30:52 (3408): No heartbeat from core client for 30 sec - exiting 18:30:53 (3408): No heartbeat from core client for 30 sec - exiting 18:30:54 (3408): No heartbeat from core client for 30 sec - exiting 18:30:55 (3408): No heartbeat from core client for 30 sec - exiting 18:30:57 (3408): No heartbeat from core client for 30 sec - exiting 18:30:58 (3408): No heartbeat from core client for 30 sec - exiting 18:30:59 (3408): No heartbeat from core client for 30 sec - exiting 18:31:00 (3408): No heartbeat from core client for 30 sec - exiting 18:31:01 (3408): No heartbeat from core client for 30 sec - exiting 18:31:02 (3408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/xaakg.pipe_dummy 2048 Leaving CPDN_Main::Monitor... zip error: Could not create output file (was replacing the original zip file) Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_eu_88qa_2002_1_007661339_1_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_88qa_2002_1_007661339_1_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_88qa_2002_1_007661339_1_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_88qa_2002_1_007661339_1_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Mar 2012 20:21:57	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	92,256	177,112	1.9198
03 Mar 2012 19:23:18	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	80,736	155,563	1.9268
02 Mar 2012 19:22:50	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	69,216	133,769	1.9326
02 Mar 2012 11:46:59	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	57,696	111,823	1.9381
01 Mar 2012 16:00:22	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	46,176	89,070	1.9289
01 Mar 2012 09:26:46	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	34,656	66,263	1.9120
29 Feb 2012 14:23:40	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	23,136	44,112	1.9066
28 Feb 2012 19:28:06	1191213	13955027	hadam3p_eu_88qa_2002_1_007661339_1	11,616	22,456	1.9332