Task 15690315

Name	hadcm3n_4byd_1940_40_008310235_4
Workunit	8461370
Created	28 Mar 2013, 18:59:28 UTC
Sent	28 Mar 2013, 23:16:23 UTC
Report deadline	28 Jun 2013, 6:43:34 UTC
Received	24 Jan 2014, 22:12:52 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	25 (0x00000019) Unknown error code
Computer ID	1070147
Run time	25 days 5 hours 17 min 44 sec
CPU time	25 days 5 hours 17 min 44 sec
Validate state	Invalid
Credit	7,153.92
Device peak FLOPS	2.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.2.28</core_client_version> <![CDATA[ <message> The drive cannot locate a specific area or track on the disk. (0x19) - exit code 25 (0x19) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 12:35:45 (7492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:19:05 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:06 (4304): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6748, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3628, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5908, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5324, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5324, iMonCtr=1 Model crash detected, will try to restart... 08:48:13 (4428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4380, iMonCtr=1 Model crash detected, will try to restart... CSuspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4bydko.pje7c10 Error converting file to netcdf: dataout/4bydko.pie7c10 Error converting file to netcdf: dataout/4bydko.pfe7c10 Error converting file to netcdf: dataout/4bydka.phe7c10 Error converting file to netcdf: dataout/4bydka.pge7c10 Error converting file to netcdf: dataout/4bydka.pee7c10 Error converting file to netcdf: dataout/4bydka.pde7c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8056, iMonCtr=1 Model crash detected, will try to restart... 18:55:57 (8008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:56:00 (8008): No heartbeat from core client for 30 sec - exiting 18:56:01 (8008): No heartbeat from core client for 30 sec - exiting 18:56:02 (8008): No heartbeat from core client for 30 sec - exiting 18:56:03 (8008): No heartbeat from core client for 30 sec - exiting 18:56:04 (8008): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7432, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4120, iMonCtr=1 Model crash detected, will try to restart... 15:45:11 (7724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7452, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6992, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8024, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5676, iMonCtr=1 Model crash detected, will try to restart... C16:44:07 (4112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7488, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8296, iMonCtr=1 Model crash detected, will try to restart... 01:35:21 (1532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7912, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1792, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=260, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12240, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12240, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12240, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4452, iMonCtr=1 Model crash detected, will try to restart... 14:32:15 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:32:17 (5720): No heartbeat from core client for 30 sec - exiting 14:32:18 (5720): No heartbeat from core client for 30 sec - exiting 18:35:51 (10120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:35:52 (10120): No heartbeat from core client for 30 sec - exiting 18:35:53 (10120): No heartbeat from core client for 30 sec - exiting 18:35:54 (10120): No heartbeat from core client for 30 sec - exiting 19:40:49 (12684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:01 (12684): No heartbeat from core client for 30 sec - exiting 19:41:05 (12684): No heartbeat from core client for 30 sec - exiting 19:41:06 (12684): No heartbeat from core client for 30 sec - exiting 19:41:07 (12684): No heartbeat from core client for 30 sec - exiting 19:41:08 (12684): No heartbeat from core client for 30 sec - exiting 19:41:09 (12684): No heartbeat from core client for 30 sec - exiting 19:41:10 (12684): No heartbeat from core client for 30 sec - exiting 19:41:11 (12684): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7084, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 16:29:28 (7180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:49:37 (5688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Jan 2014 20:00:44	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	596,160	2,151,658	3.6092
19 Jan 2014 18:06:45	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	570,240	2,075,441	3.6396
08 Jan 2014 02:49:04	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	544,320	1,990,122	3.6562
04 Jan 2014 21:52:24	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	518,400	1,878,539	3.6237
26 Dec 2013 22:54:30	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	492,480	1,775,492	3.6052
21 Dec 2013 15:54:15	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	466,560	1,689,401	3.6210
15 Dec 2013 04:20:30	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	440,640	1,616,893	3.6694
07 Dec 2013 22:19:30	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	414,720	1,546,143	3.7282
27 Nov 2013 16:27:37	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	388,800	1,465,667	3.7697
18 Nov 2013 20:41:27	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	362,880	1,373,125	3.7840
09 Nov 2013 12:32:08	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	336,960	1,264,281	3.7520
29 Oct 2013 02:40:14	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	311,040	1,156,842	3.7193
22 Oct 2013 03:00:55	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	285,120	1,052,735	3.6923
12 Oct 2013 15:40:20	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	259,200	946,184	3.6504
14 Sep 2013 02:40:33	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	233,280	849,307	3.6407
07 Sep 2013 00:16:57	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	207,360	756,105	3.6463
18 Aug 2013 12:50:39	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	181,440	675,020	3.7203
25 Jun 2013 21:31:38	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	155,520	584,457	3.7581
15 May 2013 15:22:50	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	129,600	497,068	3.8354
12 May 2013 09:26:18	1070147	15690315	hadcm3n_4byd_1940_40_008310235_4	103,680	410,889	3.9630