Task 16012303

Name	hadcm3n_7z1x_1980_40_008456712_1
Workunit	8607568
Created	11 Sep 2013, 18:35:08 UTC
Sent	11 Sep 2013, 19:19:30 UTC
Report deadline	12 Dec 2013, 2:46:41 UTC
Received	13 Jan 2014, 14:01:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	25 (0x00000019) Unknown error code
Computer ID	1267417
Run time	17 days 19 hours 2 min 21 sec
CPU time	16 days 21 hours 59 min 58 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The drive cannot locate a specific area or track on the disk. (0x19) - exit code 25 (0x19) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5172, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6744, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:00:06 (7480): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:23:07 (568): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 17:15:21 (7512): No heartbeat from core client for 30 sec - exiting 17:15:22 (7512): No heartbeat from core client for 30 sec - exiting 17:15:23 (7512): No heartbeat from core client for 30 sec - exiting 17:15:24 (7512): No heartbeat from core client for 30 sec - exiting 17:15:25 (7512): No heartbeat from core client for 30 sec - exiting 17:15:26 (7512): No heartbeat from core client for 30 sec - exiting 17:15:27 (7512): No heartbeat from core client for 30 sec - exiting 17:15:28 (7512): No heartbeat from core client for 30 sec - exiting 17:15:29 (7512): No heartbeat from core client for 30 sec - exiting 17:15:30 (7512): No heartbeat from core client for 30 sec - exiting 17:15:31 (7512): No heartbeat from core client for 30 sec - exiting 17:15:32 (7512): No heartbeat from core client for 30 sec - exiting 17:15:33 (7512): No heartbeat from core client for 30 sec - exiting 17:15:34 (7512): No heartbeat from core client for 30 sec - exiting 17:15:35 (7512): No heartbeat from core client for 30 sec - exiting 17:15:36 (7512): No heartbeat from core client for 30 sec - exiting 17:15:37 (7512): No heartbeat from core client for 30 sec - exiting 17:15:38 (7512): No heartbeat from core client for 30 sec - exiting 17:15:39 (7512): No heartbeat from core client for 30 sec - exiting 17:15:40 (7512): No heartbeat from core client for 30 sec - exiting 17:15:41 (7512): No heartbeat from core client for 30 sec - exiting 17:15:42 (7512): No heartbeat from core client for 30 sec - exiting 17:15:43 (7512): No heartbeat from core client for 30 sec - exiting 17:15:44 (7512): No heartbeat from core client for 30 sec - exiting 17:15:45 (7512): No heartbeat from core client for 30 sec - exiting 17:15:46 (7512): No heartbeat from core client for 30 sec - exiting 17:15:47 (7512): No heartbeat from core client for 30 sec - exiting 17:15:48 (7512): No heartbeat from core client for 30 sec - exiting 17:15:49 (7512): No heartbeat from core client for 30 sec - exiting 17:15:50 (7512): No heartbeat from core client for 30 sec - exiting 17:15:51 (7512): No heartbeat from core client for 30 sec - exiting 17:15:52 (7512): No heartbeat from core client for 30 sec - exiting 17:15:53 (7512): No heartbeat from core client for 30 sec - exiting 17:15:54 (7512): No heartbeat from core client for 30 sec - exiting 17:15:55 (7512): No heartbeat from core client for 30 sec - exiting 17:15:56 (7512): No heartbeat from core client for 30 sec - exiting 17:15:57 (7512): No heartbeat from core client for 30 sec - exiting 17:15:58 (7512): No heartbeat from core client for 30 sec - exiting 17:15:59 (7512): No heartbeat from core client for 30 sec - exiting 17:16:00 (7512): No heartbeat from core client for 30 sec - exiting 17:16:01 (7512): No heartbeat from core client for 30 sec - exiting 17:16:02 (7512): No heartbeat from core client for 30 sec - exiting 17:16:03 (7512): No heartbeat from core client for 30 sec - exiting 17:16:04 (7512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:45:20 (7264): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:29:59 (8308): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:42:52 (9568): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:30:55 (7380): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8692, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:32:52 (7556): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:49:42 (5372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:49:43 (5372): No heartbeat from core client for 30 sec - exiting 17:49:44 (5372): No heartbeat from core client for 30 sec - exiting 17:49:45 (5372): No heartbeat from core client for 30 sec - exiting 17:49:46 (5372): No heartbeat from core client for 30 sec - exiting 17:49:47 (5372): No heartbeat from core client for 30 sec - exiting 17:49:48 (5372): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 17:19:52 (7580): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4372, iMonCtr=1 Model crash detected, will try to restart... 09:17:24 (7992): No heartbeat from core client for 30 sec - exiting 09:17:26 (7992): No heartbeat from core client for 30 sec - exiting 09:17:27 (7992): No heartbeat from core client for 30 sec - exiting 09:17:28 (7992): No heartbeat from core client for 30 sec - exiting 09:17:29 (7992): No heartbeat from core client for 30 sec - exiting 09:17:30 (7992): No heartbeat from core client for 30 sec - exiting 09:17:31 (7992): No heartbeat from core client for 30 sec - exiting 09:17:32 (7992): No heartbeat from core client for 30 sec - exiting 09:17:33 (7992): No heartbeat from core client for 30 sec - exiting 09:17:34 (7992): No heartbeat from core client for 30 sec - exiting 09:17:35 (7992): No heartbeat from core client for 30 sec - exiting 09:17:36 (7992): No heartbeat from core client for 30 sec - exiting 09:17:37 (7992): No heartbeat from core client for 30 sec - exiting 09:17:38 (7992): No heartbeat from core client for 30 sec - exiting 09:17:39 (7992): No heartbeat from core client for 30 sec - exiting 09:17:40 (7992): No heartbeat from core client for 30 sec - exiting 09:17:41 (7992): No heartbeat from core client for 30 sec - exiting 09:17:42 (7992): No heartbeat from core client for 30 sec - exiting 09:17:43 (7992): No heartbeat from core client for 30 sec - exiting 09:17:44 (7992): No heartbeat from core client for 30 sec - exiting 09:17:45 (7992): No heartbeat from core client for 30 sec - exiting 09:17:46 (7992): No heartbeat from core client for 30 sec - exiting 09:17:47 (7992): No heartbeat from core client for 30 sec - exiting 09:17:48 (7992): No heartbeat from core client for 30 sec - exiting 09:17:49 (7992): No heartbeat from core client for 30 sec - exiting 09:17:50 (7992): No heartbeat from core client for 30 sec - exiting 09:17:51 (7992): No heartbeat from core client for 30 sec - exiting 09:17:52 (7992): No heartbeat from core client for 30 sec - exiting 09:17:53 (7992): No heartbeat from core client for 30 sec - exiting 09:17:54 (7992): No heartbeat from core client for 30 sec - exiting 09:17:55 (7992): No heartbeat from core client for 30 sec - exiting 09:17:56 (7992): No heartbeat from core client for 30 sec - exiting 09:17:57 (7992): No heartbeat from core client for 30 sec - exiting 09:17:58 (7992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:01:29 (8112): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 10:01:48 (8112): No heartbeat from core client for 30 sec - exiting 10:01:49 (8112): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 09:19:52 (6912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:30:53 (6500): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 14:30:54 (6500): No heartbeat from core client for 30 sec - exiting 14:30:55 (6500): No heartbeat from core client for 30 sec - exiting 14:30:56 (6500): No heartbeat from core client for 30 sec - exiting 14:30:57 (6500): No heartbeat from core client for 30 sec - exiting 14:30:58 (6500): No heartbeat from core client for 30 sec - exiting 14:30:59 (6500): No heartbeat from core client for 30 sec - exiting 14:31:00 (6500): No heartbeat from core client for 30 sec - exiting 14:31:01 (6500): No heartbeat from core client for 30 sec - exiting 14:31:02 (6500): No heartbeat from core client for 30 sec - exiting 14:31:03 (6500): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:56:57 (7528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:16:05 (5316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:24:23 (5872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:59:42 (7548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7188, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:19:57 (6468): No heartbeat from core client for 30 sec - exiting 03:19:58 (6468): No heartbeat from core client for 30 sec - exiting 03:19:59 (6468): No heartbeat from core client for 30 sec - exiting 03:20:00 (6468): No heartbeat from core client for 30 sec - exiting 03:20:01 (6468): No heartbeat from core client for 30 sec - exiting 03:20:03 (6468): No heartbeat from core client for 30 sec - exiting 03:20:04 (6468): No heartbeat from core client for 30 sec - exiting 03:20:05 (6468): No heartbeat from core client for 30 sec - exiting 03:20:06 (6468): No heartbeat from core client for 30 sec - exiting 03:20:07 (6468): No heartbeat from core client for 30 sec - exiting 03:20:08 (6468): No heartbeat from core client for 30 sec - exiting 03:20:09 (6468): No heartbeat from core client for 30 sec - exiting 03:20:10 (6468): No heartbeat from core client for 30 sec - exiting 03:20:11 (6468): No heartbeat from core client for 30 sec - exiting 03:20:12 (6468): No heartbeat from core client for 30 sec - exiting 03:20:13 (6468): No heartbeat from core client for 30 sec - exiting 03:20:15 (6468): No heartbeat from core client for 30 sec - exiting 03:20:16 (6468): No heartbeat from core client for 30 sec - exiting 03:20:17 (6468): No heartbeat from core client for 30 sec - exiting 03:20:18 (6468): No heartbeat from core client for 30 sec - exiting 03:20:19 (6468): No heartbeat from core client for 30 sec - exiting 03:20:20 (6468): No heartbeat from core client for 30 sec - exiting 03:20:21 (6468): No heartbeat from core client for 30 sec - exiting 03:20:22 (6468): No heartbeat from core client for 30 sec - exiting 03:20:23 (6468): No heartbeat from core client for 30 sec - exiting 03:20:24 (6468): No heartbeat from core client for 30 sec - exiting 03:20:26 (6468): No heartbeat from core client for 30 sec - exiting 03:20:27 (6468): No heartbeat from core client for 30 sec - exiting 03:20:28 (6468): No heartbeat from core client for 30 sec - exiting 03:20:29 (6468): No heartbeat from core client for 30 sec - exiting 03:20:30 (6468): No heartbeat from core client for 30 sec - exiting 03:20:31 (6468): No heartbeat from core client for 30 sec - exiting 03:20:32 (6468): No heartbeat from core client for 30 sec - exiting 03:20:33 (6468): No heartbeat from core client for 30 sec - exiting 03:20:34 (6468): No heartbeat from core client for 30 sec - exiting 03:20:35 (6468): No heartbeat from core client for 30 sec - exiting 03:20:36 (6468): No heartbeat from core client for 30 sec - exiting 03:20:37 (6468): No heartbeat from core client for 30 sec - exiting 03:20:39 (6468): No heartbeat from core client for 30 sec - exiting 03:20:40 (6468): No heartbeat from core client for 30 sec - exiting 03:20:41 (6468): No heartbeat from core client for 30 sec - exiting 03:20:42 (6468): No heartbeat from core client for 30 sec - exiting 03:20:43 (6468): No heartbeat from core client for 30 sec - exiting 03:20:44 (6468): No heartbeat from core client for 30 sec - exiting 03:20:45 (6468): No heartbeat from core client for 30 sec - exiting 03:20:46 (6468): No heartbeat from core client for 30 sec - exiting 03:20:47 (6468): No heartbeat from core client for 30 sec - exiting 03:20:48 (6468): No heartbeat from core client for 30 sec - exiting 03:20:49 (6468): No heartbeat from core client for 30 sec - exiting 03:20:50 (6468): No heartbeat from core client for 30 sec - exiting 03:20:51 (6468): No heartbeat from core client for 30 sec - exiting 03:20:52 (6468): No heartbeat from core client for 30 sec - exiting 03:20:53 (6468): No heartbeat from core client for 30 sec - exiting 03:20:54 (6468): No heartbeat from core client for 30 sec - exiting 03:20:56 (6468): No heartbeat from core client for 30 sec - exiting 03:20:57 (6468): No heartbeat from core client for 30 sec - exiting 03:20:58 (6468): No heartbeat from core client for 30 sec - exiting 03:20:59 (6468): No heartbeat from core client for 30 sec - exiting 03:21:00 (6468): No heartbeat from core client for 30 sec - exiting 03:21:01 (6468): No heartbeat from core client for 30 sec - exiting 03:21:02 (6468): No heartbeat from core client for 30 sec - exiting 03:21:03 (6468): No heartbeat from core client for 30 sec - exiting 03:21:04 (6468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:43 (7404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:20:33 (6452): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:07:36 (4124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Jan 2014 17:08:30	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	1,010,880	1,453,205	1.4376
12 Jan 2014 07:23:08	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	984,960	1,418,093	1.4397
11 Jan 2014 19:25:06	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	959,040	1,379,433	1.4383
11 Jan 2014 08:32:34	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	933,120	1,340,357	1.4364
09 Jan 2014 18:27:31	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	907,200	1,304,593	1.4380
07 Jan 2014 20:47:07	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	881,280	1,268,813	1.4397
06 Jan 2014 18:53:48	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	855,360	1,231,481	1.4397
02 Jan 2014 20:52:45	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	829,440	1,194,234	1.4398
30 Dec 2013 20:16:58	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	803,520	1,158,206	1.4414
24 Dec 2013 16:14:52	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	777,600	1,121,508	1.4423
23 Dec 2013 09:21:19	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	751,680	1,082,736	1.4404
22 Dec 2013 22:03:17	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	725,760	1,042,957	1.4371
22 Dec 2013 12:15:22	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	699,840	1,008,319	1.4408
22 Dec 2013 02:07:48	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	673,920	973,085	1.4439
21 Dec 2013 15:54:14	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	648,000	936,996	1.4460
20 Dec 2013 17:16:44	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	622,080	900,883	1.4482
19 Dec 2013 05:39:28	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	596,160	863,186	1.4479
18 Dec 2013 14:39:02	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	570,240	825,819	1.4482
12 Dec 2013 16:26:43	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	544,320	788,721	1.4490
09 Dec 2013 19:26:43	1267417	16012303	hadcm3n_7z1x_1980_40_008456712_1	518,400	752,005	1.4506