Task 16290763

Name	hadcm3n_7ijy_1980_40_008435329_3
Workunit	8586185
Created	13 Feb 2014, 20:10:24 UTC
Sent	13 Feb 2014, 20:10:36 UTC
Report deadline	18 Aug 2023, 1:30:36 UTC
Received	20 Aug 2014, 12:38:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1262606
Run time	22 days 11 hours 1 min 31 sec
CPU time	22 days 3 hours 24 min 23 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.80 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> 17:50:08 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:59:30 (4124): Can't acquire lockfile (32) - waiting 35s 16:59:42 (4916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:25:41 (4908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:27:35 (2880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:20:50 (3196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:21:24 (4844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:22:19 (6092): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:23:25 (3220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:27:59 (1708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:28:00 (1708): No heartbeat from core client for 30 sec - exiting 12:28:01 (1708): No heartbeat from core client for 30 sec - exiting 12:28:02 (1708): No heartbeat from core client for 30 sec - exiting 12:28:03 (1708): No heartbeat from core client for 30 sec - exiting 12:28:05 (1708): No heartbeat from core client for 30 sec - exiting 12:28:06 (1708): No heartbeat from core client for 30 sec - exiting 12:28:07 (1708): No heartbeat from core client for 30 sec - exiting 12:28:08 (1708): No heartbeat from core client for 30 sec - exiting 12:28:09 (1708): No heartbeat from core client for 30 sec - exiting 12:28:10 (1708): No heartbeat from core client for 30 sec - exiting 12:30:07 (3996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 09:52:32 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:33 (5804): No heartbeat from core client for 30 sec - exiting 09:52:34 (5804): No heartbeat from core client for 30 sec - exiting 09:52:35 (5804): No heartbeat from core client for 30 sec - exiting 09:52:36 (5804): No heartbeat from core client for 30 sec - exiting 09:52:37 (5804): No heartbeat from core client for 30 sec - exiting 09:52:39 (5804): No heartbeat from core client for 30 sec - exiting 09:52:40 (5804): No heartbeat from core client for 30 sec - exiting 09:52:41 (5804): No heartbeat from core client for 30 sec - exiting 09:52:42 (5804): No heartbeat from core client for 30 sec - exiting 09:52:43 (5804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:22:50 (5964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:42:56 (6076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:23:13 (4496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:32:08 (1564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:37:18 (4764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:42:07 (5316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:28:03 (5212): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:30 (3500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:31 (3500): No heartbeat from core client for 30 sec - exiting 13:29:32 (3500): No heartbeat from core client for 30 sec - exiting 13:29:33 (3500): No heartbeat from core client for 30 sec - exiting 13:29:34 (3500): No heartbeat from core client for 30 sec - exiting 13:29:35 (3500): No heartbeat from core client for 30 sec - exiting 13:29:36 (3500): No heartbeat from core client for 30 sec - exiting 13:29:37 (3500): No heartbeat from core client for 30 sec - exiting 13:29:38 (3500): No heartbeat from core client for 30 sec - exiting 13:29:39 (3500): No heartbeat from core client for 30 sec - exiting 13:29:40 (3500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 14:17:29 (284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:21:44 (1408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:21:45 (1408): No heartbeat from core client for 30 sec - exiting 14:21:46 (1408): No heartbeat from core client for 30 sec - exiting 14:21:47 (1408): No heartbeat from core client for 30 sec - exiting 14:21:48 (1408): No heartbeat from core client for 30 sec - exiting 14:21:49 (1408): No heartbeat from core client for 30 sec - exiting 14:21:50 (1408): No heartbeat from core client for 30 sec - exiting 14:21:51 (1408): No heartbeat from core client for 30 sec - exiting 14:21:52 (1408): No heartbeat from core client for 30 sec - exiting 14:21:53 (1408): No heartbeat from core client for 30 sec - exiting 14:21:54 (1408): No heartbeat from core client for 30 sec - exiting 14:23:35 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:27:08 (5972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:51:37 (5888): No heartbeat from core client for 30 sec - exiting 13:51:38 (5888): No heartbeat from core client for 30 sec - exiting 13:51:39 (5888): No heartbeat from core client for 30 sec - exiting 13:51:40 (5888): No heartbeat from core client for 30 sec - exiting 13:51:41 (5888): No heartbeat from core client for 30 sec - exiting 13:51:42 (5888): No heartbeat from core client for 30 sec - exiting 13:51:44 (5888): No heartbeat from core client for 30 sec - exiting 13:51:45 (5888): No heartbeat from core client for 30 sec - exiting 13:51:46 (5888): No heartbeat from core client for 30 sec - exiting 13:51:47 (5888): No heartbeat from core client for 30 sec - exiting 13:51:48 (5888): No heartbeat from core client for 30 sec - exiting 13:51:49 (5888): No heartbeat from core client for 30 sec - exiting 13:51:50 (5888): No heartbeat from core client for 30 sec - exiting 13:51:51 (5888): No heartbeat from core client for 30 sec - exiting 13:51:52 (5888): No heartbeat from core client for 30 sec - exiting 13:51:53 (5888): No heartbeat from core client for 30 sec - exiting 13:51:54 (5888): No heartbeat from core client for 30 sec - exiting 13:51:56 (5888): No heartbeat from core client for 30 sec - exiting 13:51:57 (5888): No heartbeat from core client for 30 sec - exiting 13:51:58 (5888): No heartbeat from core client for 30 sec - exiting 13:51:59 (5888): No heartbeat from core client for 30 sec - exiting 13:52:00 (5888): No heartbeat from core client for 30 sec - exiting 13:52:01 (5888): No heartbeat from core client for 30 sec - exiting 13:52:02 (5888): No heartbeat from core client for 30 sec - exiting 13:52:03 (5888): No heartbeat from core client for 30 sec - exiting 13:52:04 (5888): No heartbeat from core client for 30 sec - exiting 13:52:05 (5888): No heartbeat from core client for 30 sec - exiting 13:52:06 (5888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:52:08 (5888): No heartbeat from core client for 30 sec - exiting 13:52:09 (5888): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:24:39 (5760): No heartbeat from core client for 30 sec - exiting 08:24:40 (5760): No heartbeat from core client for 30 sec - exiting 08:24:42 (5760): No heartbeat from core client for 30 sec - exiting 08:24:43 (5760): No heartbeat from core client for 30 sec - exiting 08:24:44 (5760): No heartbeat from core client for 30 sec - exiting 08:24:45 (5760): No heartbeat from core client for 30 sec - exiting 08:24:46 (5760): No heartbeat from core client for 30 sec - exiting 08:24:47 (5760): No heartbeat from core client for 30 sec - exiting 08:24:48 (5760): No heartbeat from core client for 30 sec - exiting 08:24:49 (5760): No heartbeat from core client for 30 sec - exiting 08:24:50 (5760): No heartbeat from core client for 30 sec - exiting 08:24:51 (5760): No heartbeat from core client for 30 sec - exiting 08:24:52 (5760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:26:26 (952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:28:10 (5720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:32:18 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:33:13 (4220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:00:07 (4112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4424, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/7ijyko.pjk8c10 Error converting file to netcdf: dataout/7ijyko.pik8c10 Error converting file to netcdf: dataout/7ijyko.pfk8c10 Error converting file to netcdf: dataout/7ijyka.phk8c10 Error converting file to netcdf: dataout/7ijyka.pgk8c10 Error converting file to netcdf: dataout/7ijyka.pek8c10 Error converting file to netcdf: dataout/7ijyka.pdk8c10 11:14:09 (1864): No heartbeat from core client for 30 sec - exiting 11:14:10 (1864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/7ijyko.pjk8c10 Error converting file to netcdf: dataout/7ijyko.pik8c10 Error converting file to netcdf: dataout/7ijyko.pfk8c10 Error converting file to netcdf: dataout/7ijyka.phk8c10 Error converting file to netcdf: dataout/7ijyka.pgk8c10 Error converting file to netcdf: dataout/7ijyka.pek8c10 Error converting file to netcdf: dataout/7ijyka.pdk8c10 09:32:30 (4880): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:01 (3468): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:34:39 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FC7383 read attempt to address 0x4090BDC4 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76FC3AC3 read attempt to address 0x4090BDCC Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Aug 2014 08:40:01	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	1,036,800	1,912,557	1.8447
18 Aug 2014 08:40:01	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	1,010,880	1,878,302	1.8581
18 Aug 2014 08:40:01	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	984,960	1,843,473	1.8716
18 Aug 2014 08:40:01	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	959,040	1,807,564	1.8848
14 Aug 2014 16:05:16	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	933,120	1,761,290	1.8875
14 Aug 2014 16:05:16	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	907,200	1,707,142	1.8818
14 Aug 2014 16:05:16	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	881,280	1,647,686	1.8697
08 Aug 2014 13:16:55	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	855,360	1,595,136	1.8649
08 Aug 2014 13:16:55	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	829,440	1,530,725	1.8455
08 Aug 2014 13:16:55	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	803,520	1,460,366	1.8175
06 Aug 2014 12:58:58	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	777,600	1,408,065	1.8108
06 Aug 2014 12:58:58	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	751,680	1,347,410	1.7925
06 Aug 2014 12:58:58	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	725,760	1,282,337	1.7669
13 Jun 2014 12:52:13	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	699,840	1,223,386	1.7481
12 Jun 2014 07:59:18	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	673,920	1,187,381	1.7619
12 Jun 2014 07:59:18	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	648,000	1,109,097	1.7116
12 Jun 2014 07:59:18	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	622,080	1,045,702	1.6810
28 May 2014 15:18:01	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	596,160	977,897	1.6403
21 May 2014 08:41:04	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	570,240	907,505	1.5914
21 May 2014 08:41:04	1262606	16290763	hadcm3n_7ijy_1980_40_008435329_3	544,320	836,699	1.5371