Task 13572120

Name	hadcm3n_o39f_1940_40_007444844_4
Workunit	7642347
Created	31 Oct 2011, 8:16:09 UTC
Sent	31 Oct 2011, 8:19:48 UTC
Report deadline	30 Jan 2012, 15:46:59 UTC
Received	21 Dec 2011, 13:10:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1177478
Run time	44 days 1 hours 33 min 3 sec
CPU time	34 days 8 hours 41 min 58 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2212, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2212, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2212, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2212, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2284, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2284, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish 23:05:34 (2640): No heartbeat from core client for 30 sec - exiting 23:05:35 (2640): No heartbeat from core client for 30 sec - exiting 23:05:36 (2640): No heartbeat from core client for 30 sec - exiting 23:05:37 (2640): No heartbeat from core client for 30 sec - exiting 23:05:38 (2640): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o39fko.pjf1c10 Error converting file to netcdf: dataout/o39fko.pif1c10 Error converting file to netcdf: dataout/o39fko.pff1c10 Error converting file to netcdf: dataout/o39fka.phf1c10 Error converting file to netcdf: dataout/o39fka.pgf1c10 Error converting file to netcdf: dataout/o39fka.pef1c10 Error converting file to netcdf: dataout/o39fka.pdf1c10 13:32:03 (1756): No heartbeat from core client for 30 sec - exiting 13:32:04 (1756): No heartbeat from core client for 30 sec - exiting 13:32:05 (1756): No heartbeat from core client for 30 sec - exiting 13:32:06 (1756): No heartbeat from core client for 30 sec - exiting 13:32:07 (1756): No heartbeat from core client for 30 sec - exiting 13:32:08 (1756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:32:09 (1756): No heartbeat from core client for 30 sec - exiting 13:32:10 (1756): No heartbeat from core client for 30 sec - exiting 13:32:11 (1756): No heartbeat from core client for 30 sec - exiting 13:32:12 (1756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:05:23 (3508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:05:25 (3508): No heartbeat from core client for 30 sec - exiting 00:07:03 (4924): No heartbeat from core client for 30 sec - exiting 00:07:04 (4924): No heartbeat from core client for 30 sec - exiting 00:07:05 (4924): No heartbeat from core client for 30 sec - exiting 00:07:06 (4924): No heartbeat from core client for 30 sec - exiting 00:07:07 (4924): No heartbeat from core client for 30 sec - exiting 00:07:08 (4924): No heartbeat from core client for 30 sec - exiting 00:07:09 (4924): No heartbeat from core client for 30 sec - exiting 00:07:10 (4924): No heartbeat from core client for 30 sec - exiting 00:07:11 (4924): No heartbeat from core client for 30 sec - exiting 00:07:12 (4924): No heartbeat from core client for 30 sec - exiting 00:07:13 (4924): No heartbeat from core client for 30 sec - exiting 00:07:14 (4924): No heartbeat from core client for 30 sec - exiting 00:07:15 (4924): No heartbeat from core client for 30 sec - exiting 00:07:16 (4924): No heartbeat from core client for 30 sec - exiting 00:07:17 (4924): No heartbeat from core client for 30 sec - exiting 00:07:18 (4924): No heartbeat from core client for 30 sec - exiting 00:07:19 (4924): No heartbeat from core client for 30 sec - exiting 00:07:20 (4924): No heartbeat from core client for 30 sec - exiting 00:07:21 (4924): No heartbeat from core client for 30 sec - exiting 00:07:22 (4924): No heartbeat from core client for 30 sec - exiting 00:07:23 (4924): No heartbeat from core client for 30 sec - exiting 00:07:24 (4924): No heartbeat from core client for 30 sec - exiting 00:07:25 (4924): No heartbeat from core client for 30 sec - exiting 00:07:27 (4924): No heartbeat from core client for 30 sec - exiting 00:07:28 (4924): No heartbeat from core client for 30 sec - exiting 00:07:29 (4924): No heartbeat from core client for 30 sec - exiting 00:07:30 (4924): No heartbeat from core client for 30 sec - exiting 00:07:31 (4924): No heartbeat from core client for 30 sec - exiting 00:07:32 (4924): No heartbeat from core client for 30 sec - exiting 00:07:33 (4924): No heartbeat from core client for 30 sec - exiting 00:07:34 (4924): No heartbeat from core client for 30 sec - exiting 00:07:35 (4924): No heartbeat from core client for 30 sec - exiting 00:07:36 (4924): No heartbeat from core client for 30 sec - exiting 00:07:37 (4924): No heartbeat from core client for 30 sec - exiting 00:07:38 (4924): No heartbeat from core client for 30 sec - exiting 00:07:39 (4924): No heartbeat from core client for 30 sec - exiting 00:07:40 (4924): No heartbeat from core client for 30 sec - exiting 00:07:41 (4924): No heartbeat from core client for 30 sec - exiting 00:07:42 (4924): No heartbeat from core client for 30 sec - exiting 00:07:43 (4924): No heartbeat from core client for 30 sec - exiting 00:07:44 (4924): No heartbeat from core client for 30 sec - exiting 00:07:45 (4924): No heartbeat from core client for 30 sec - exiting 00:07:46 (4924): No heartbeat from core client for 30 sec - exiting 00:07:47 (4924): No heartbeat from core client for 30 sec - exiting 00:07:48 (4924): No heartbeat from core client for 30 sec - exiting 00:07:49 (4924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:07:50 (4924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:21:42 (5696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C931780 read attempt to address 0x3FF52B85 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Dec 2011 13:15:24	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	518,400	2,968,889	5.7270
18 Dec 2011 16:44:08	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	492,480	2,841,532	5.7698
16 Dec 2011 19:44:22	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	466,560	2,716,283	5.8219
14 Dec 2011 18:42:49	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	440,640	2,592,386	5.8832
13 Dec 2011 04:04:19	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	414,720	2,469,123	5.9537
11 Dec 2011 11:47:31	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	388,800	2,343,964	6.0287
09 Dec 2011 18:10:27	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	362,880	2,220,664	6.1196
07 Dec 2011 19:17:16	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	336,960	2,096,146	6.2208
04 Dec 2011 18:44:59	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	311,040	1,951,416	6.2738
02 Dec 2011 10:32:25	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	285,120	1,789,337	6.2757
30 Nov 2011 04:05:07	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	259,200	1,641,106	6.3314
27 Nov 2011 15:07:23	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	233,280	1,478,064	6.3360
25 Nov 2011 08:35:56	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	207,360	1,322,047	6.3756
22 Nov 2011 21:49:35	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	181,440	1,162,742	6.4084
20 Nov 2011 18:10:09	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	155,520	1,011,916	6.5067
18 Nov 2011 12:55:18	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	129,600	859,744	6.6338
16 Nov 2011 05:53:27	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	103,680	707,976	6.8285
15 Nov 2011 18:26:39	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	77,760	555,267	7.1408
15 Nov 2011 18:26:39	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	51,840	396,169	7.6421
07 Nov 2011 03:26:49	1177478	13572120	hadcm3n_o39f_1940_40_007444844_4	25,920	246,001	9.4908