Task 16227232

Name	hadcm3n_849t_1980_40_008463493_3
Workunit	8614332
Created	14 Jan 2014, 4:40:21 UTC
Sent	14 Jan 2014, 5:05:46 UTC
Report deadline	15 Apr 2014, 12:32:57 UTC
Received	1 Feb 2014, 17:50:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	25 (0x00000019) Unknown error code
Computer ID	1359502
Run time	7 days 3 hours 32 min 7 sec
CPU time	7 days 3 hours 27 min 17 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.17 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The drive cannot locate a specific area or track on the disk. (0x19) - exit code 25 (0x19) </message> <stderr_txt> 02:33:33 (5940): No heartbeat from core client for 30 sec - exiting 02:33:34 (5940): No heartbeat from core client for 30 sec - exiting 02:33:35 (5940): No heartbeat from core client for 30 sec - exiting 02:33:36 (5940): No heartbeat from core client for 30 sec - exiting 02:33:37 (5940): No heartbeat from core client for 30 sec - exiting 02:33:38 (5940): No heartbeat from core client for 30 sec - exiting 02:33:39 (5940): No heartbeat from core client for 30 sec - exiting 02:33:40 (5940): No heartbeat from core client for 30 sec - exiting 02:33:41 (5940): No heartbeat from core client for 30 sec - exiting 02:33:42 (5940): No heartbeat from core client for 30 sec - exiting 02:33:43 (5940): No heartbeat from core client for 30 sec - exiting 02:33:44 (5940): No heartbeat from core client for 30 sec - exiting 02:33:45 (5940): No heartbeat from core client for 30 sec - exiting 02:33:46 (5940): No heartbeat from core client for 30 sec - exiting 02:33:47 (5940): No heartbeat from core client for 30 sec - exiting 02:33:49 (5940): No heartbeat from core client for 30 sec - exiting 02:33:50 (5940): No heartbeat from core client for 30 sec - exiting 02:33:51 (5940): No heartbeat from core client for 30 sec - exiting 02:33:52 (5940): No heartbeat from core client for 30 sec - exiting 02:33:53 (5940): No heartbeat from core client for 30 sec - exiting 02:33:54 (5940): No heartbeat from core client for 30 sec - exiting 02:33:55 (5940): No heartbeat from core client for 30 sec - exiting 02:33:56 (5940): No heartbeat from core client for 30 sec - exiting 02:33:57 (5940): No heartbeat from core client for 30 sec - exiting 02:33:58 (5940): No heartbeat from core client for 30 sec - exiting 02:33:59 (5940): No heartbeat from core client for 30 sec - exiting 02:34:00 (5940): No heartbeat from core client for 30 sec - exiting 02:34:01 (5940): No heartbeat from core client for 30 sec - exiting 02:34:02 (5940): No heartbeat from core client for 30 sec - exiting 02:34:03 (5940): No heartbeat from core client for 30 sec - exiting 02:34:04 (5940): No heartbeat from core client for 30 sec - exiting 02:34:05 (5940): No heartbeat from core client for 30 sec - exiting 02:34:06 (5940): No heartbeat from core client for 30 sec - exiting 02:34:07 (5940): No heartbeat from core client for 30 sec - exiting 02:34:08 (5940): No heartbeat from core client for 30 sec - exiting 02:34:09 (5940): No heartbeat from core client for 30 sec - exiting 02:34:10 (5940): No heartbeat from core client for 30 sec - exiting 02:34:11 (5940): No heartbeat from core client for 30 sec - exiting 02:34:12 (5940): No heartbeat from core client for 30 sec - exiting 02:34:13 (5940): No heartbeat from core client for 30 sec - exiting 02:34:14 (5940): No heartbeat from core client for 30 sec - exiting 02:34:15 (5940): No heartbeat from core client for 30 sec - exiting 02:34:16 (5940): No heartbeat from core client for 30 sec - exiting 02:34:17 (5940): No heartbeat from core client for 30 sec - exiting 02:34:18 (5940): No heartbeat from core client for 30 sec - exiting 02:34:19 (5940): No heartbeat from core client for 30 sec - exiting 02:34:20 (5940): No heartbeat from core client for 30 sec - exiting 02:34:21 (5940): No heartbeat from core client for 30 sec - exiting 02:34:22 (5940): No heartbeat from core client for 30 sec - exiting 02:34:23 (5940): No heartbeat from core client for 30 sec - exiting 02:34:24 (5940): No heartbeat from core client for 30 sec - exiting 02:34:25 (5940): No heartbeat from core client for 30 sec - exiting 02:34:26 (5940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4216, iMonCtr=1 Model crash detected, will try to restart... 19:45:55 (5204): No heartbeat from core client for 30 sec - exiting 19:45:56 (5204): No heartbeat from core client for 30 sec - exiting 19:45:57 (5204): No heartbeat from core client for 30 sec - exiting 19:45:58 (5204): No heartbeat from core client for 30 sec - exiting 19:46:00 (5204): No heartbeat from core client for 30 sec - exiting 19:46:01 (5204): No heartbeat from core client for 30 sec - exiting 19:46:02 (5204): No heartbeat from core client for 30 sec - exiting 19:46:03 (5204): No heartbeat from core client for 30 sec - exiting 19:46:04 (5204): No heartbeat from core client for 30 sec - exiting 19:46:05 (5204): No heartbeat from core client for 30 sec - exiting 19:46:06 (5204): No heartbeat from core client for 30 sec - exiting 19:46:07 (5204): No heartbeat from core client for 30 sec - exiting 19:46:08 (5204): No heartbeat from core client for 30 sec - exiting 19:46:09 (5204): No heartbeat from core client for 30 sec - exiting 19:46:10 (5204): No heartbeat from core client for 30 sec - exiting 19:46:11 (5204): No heartbeat from core client for 30 sec - exiting 19:46:12 (5204): No heartbeat from core client for 30 sec - exiting 19:46:13 (5204): No heartbeat from core client for 30 sec - exiting 19:46:14 (5204): No heartbeat from core client for 30 sec - exiting 19:46:15 (5204): No heartbeat from core client for 30 sec - exiting 19:46:16 (5204): No heartbeat from core client for 30 sec - exiting 19:46:17 (5204): No heartbeat from core client for 30 sec - exiting 19:46:18 (5204): No heartbeat from core client for 30 sec - exiting 19:46:19 (5204): No heartbeat from core client for 30 sec - exiting 19:46:20 (5204): No heartbeat from core client for 30 sec - exiting 19:46:21 (5204): No heartbeat from core client for 30 sec - exiting 19:46:22 (5204): No heartbeat from core client for 30 sec - exiting 19:46:23 (5204): No heartbeat from core client for 30 sec - exiting 19:46:24 (5204): No heartbeat from core client for 30 sec - exiting 19:46:25 (5204): No heartbeat from core client for 30 sec - exiting 19:46:26 (5204): No heartbeat from core client for 30 sec - exiting 19:46:27 (5204): No heartbeat from core client for 30 sec - exiting 19:46:28 (5204): No heartbeat from core client for 30 sec - exiting 19:46:29 (5204): No heartbeat from core client for 30 sec - exiting 19:46:30 (5204): No heartbeat from core client for 30 sec - exiting 19:46:31 (5204): No heartbeat from core client for 30 sec - exiting 19:46:32 (5204): No heartbeat from core client for 30 sec - exiting 19:46:33 (5204): No heartbeat from core client for 30 sec - exiting 19:46:34 (5204): No heartbeat from core client for 30 sec - exiting 19:46:35 (5204): No heartbeat from core client for 30 sec - exiting 19:46:36 (5204): No heartbeat from core client for 30 sec - exiting 19:46:37 (5204): No heartbeat from core client for 30 sec - exiting 19:46:38 (5204): No heartbeat from core client for 30 sec - exiting 19:46:39 (5204): No heartbeat from core client for 30 sec - exiting 19:46:40 (5204): No heartbeat from core client for 30 sec - exiting 19:46:41 (5204): No heartbeat from core client for 30 sec - exiting 19:46:42 (5204): No heartbeat from core client for 30 sec - exiting 19:46:43 (5204): No heartbeat from core client for 30 sec - exiting 19:46:45 (5204): No heartbeat from core client for 30 sec - exiting 19:46:46 (5204): No heartbeat from core client for 30 sec - exiting 19:46:47 (5204): No heartbeat from core client for 30 sec - exiting 19:46:48 (5204): No heartbeat from core client for 30 sec - exiting 19:46:49 (5204): No heartbeat from core client for 30 sec - exiting 19:46:50 (5204): No heartbeat from core client for 30 sec - exiting 19:46:51 (5204): No heartbeat from core client for 30 sec - exiting 19:46:52 (5204): No heartbeat from core client for 30 sec - exiting 19:46:53 (5204): No heartbeat from core client for 30 sec - exiting 19:46:54 (5204): No heartbeat from core client for 30 sec - exiting 19:46:55 (5204): No heartbeat from core client for 30 sec - exiting 19:46:56 (5204): No heartbeat from core client for 30 sec - exiting 19:46:57 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:48:21 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:49:22 (5180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:50:45 (2400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2768, iMonCtr=1 Model crash detected, will try to restart... 18:07:23 (4376): No heartbeat from core client for 30 sec - exiting 18:07:25 (4376): No heartbeat from core client for 30 sec - exiting 18:07:26 (4376): No heartbeat from core client for 30 sec - exiting 18:07:27 (4376): No heartbeat from core client for 30 sec - exiting 18:07:28 (4376): No heartbeat from core client for 30 sec - exiting 18:07:29 (4376): No heartbeat from core client for 30 sec - exiting 18:07:30 (4376): No heartbeat from core client for 30 sec - exiting 18:07:31 (4376): No heartbeat from core client for 30 sec - exiting 18:07:32 (4376): No heartbeat from core client for 30 sec - exiting 18:07:33 (4376): No heartbeat from core client for 30 sec - exiting 18:07:34 (4376): No heartbeat from core client for 30 sec - exiting 18:07:35 (4376): No heartbeat from core client for 30 sec - exiting 18:07:37 (4376): No heartbeat from core client for 30 sec - exiting 18:07:38 (4376): No heartbeat from core client for 30 sec - exiting 18:07:39 (4376): No heartbeat from core client for 30 sec - exiting 18:07:40 (4376): No heartbeat from core client for 30 sec - exiting 18:07:41 (4376): No heartbeat from core client for 30 sec - exiting 18:07:42 (4376): No heartbeat from core client for 30 sec - exiting 18:07:43 (4376): No heartbeat from core client for 30 sec - exiting 18:07:44 (4376): No heartbeat from core client for 30 sec - exiting 18:07:45 (4376): No heartbeat from core client for 30 sec - exiting 18:07:46 (4376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:09:32 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:06 (2024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jan 2014 16:27:10	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	259,200	566,906	2.1871
30 Jan 2014 16:27:09	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	233,280	510,173	2.1870
30 Jan 2014 16:27:09	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	207,360	454,214	2.1905
27 Jan 2014 23:29:24	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	181,440	397,690	2.1919
27 Jan 2014 04:26:52	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	155,520	340,748	2.1910
26 Jan 2014 04:31:20	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	129,600	284,805	2.1976
24 Jan 2014 20:54:46	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	103,680	228,878	2.2075
23 Jan 2014 09:15:23	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	77,760	172,589	2.2195
22 Jan 2014 17:10:56	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	51,840	115,690	2.2317
21 Jan 2014 11:15:25	1295173	16227232	hadcm3n_849t_1980_40_008463493_3	25,920	58,317	2.2499