Task 15997213

Name	hadcm3n_841h_1980_40_008463176_0
Workunit	8614032
Created	30 Aug 2013, 23:11:47 UTC
Sent	31 Aug 2013, 3:08:52 UTC
Report deadline	30 Nov 2013, 10:36:03 UTC
Received	12 Oct 2013, 2:23:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	934531
Run time	17 days 19 hours 11 min 59 sec
CPU time	16 days 7 hours 23 min 2 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.54 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7884, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 01:42:44 (4624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:33:55 (5952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:02:02 (4392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:14:21 (1156): No heartbeat from core client for 30 sec - exiting 00:14:22 (1156): No heartbeat from core client for 30 sec - exiting 00:14:23 (1156): No heartbeat from core client for 30 sec - exiting 00:14:24 (1156): No heartbeat from core client for 30 sec - exiting 00:14:25 (1156): No heartbeat from core client for 30 sec - exiting 00:14:26 (1156): No heartbeat from core client for 30 sec - exiting 00:14:27 (1156): No heartbeat from core client for 30 sec - exiting 00:14:28 (1156): No heartbeat from core client for 30 sec - exiting 00:14:29 (1156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/841hko.pji2c10 Error converting file to netcdf: dataout/841hko.pii2c10 Error converting file to netcdf: dataout/841hko.pfi2c10 Error converting file to netcdf: dataout/841hka.phi2c10 Error converting file to netcdf: dataout/841hka.pgi2c10 Error converting file to netcdf: dataout/841hka.pei2c10 Error converting file to netcdf: dataout/841hka.pdi2c10 Suspended CPDN Monitor - Suspend request from BOINC... 21:05:35 (4744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:17:09 (1936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:52 (4460): No heartbeat from core client for 30 sec - exiting 22:24:53 (4460): No heartbeat from core client for 30 sec - exiting 22:24:54 (4460): No heartbeat from core client for 30 sec - exiting 22:24:55 (4460): No heartbeat from core client for 30 sec - exiting 22:24:56 (4460): No heartbeat from core client for 30 sec - exiting 22:24:57 (4460): No heartbeat from core client for 30 sec - exiting 22:24:58 (4460): No heartbeat from core client for 30 sec - exiting 22:24:59 (4460): No heartbeat from core client for 30 sec - exiting 22:25:00 (4460): No heartbeat from core client for 30 sec - exiting 22:25:01 (4460): No heartbeat from core client for 30 sec - exiting 22:25:02 (4460): No heartbeat from core client for 30 sec - exiting 22:25:03 (4460): No heartbeat from core client for 30 sec - exiting 22:25:04 (4460): No heartbeat from core client for 30 sec - exiting 22:25:05 (4460): No heartbeat from core client for 30 sec - exiting 22:25:06 (4460): No heartbeat from core client for 30 sec - exiting 22:25:07 (4460): No heartbeat from core client for 30 sec - exiting 22:25:08 (4460): No heartbeat from core client for 30 sec - exiting 22:25:09 (4460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:25:10 (4460): No heartbeat from core client for 30 sec - exiting 21:43:45 (5652): No heartbeat from core client for 30 sec - exiting 21:43:46 (5652): No heartbeat from core client for 30 sec - exiting 21:43:47 (5652): No heartbeat from core client for 30 sec - exiting 21:43:48 (5652): No heartbeat from core client for 30 sec - exiting 21:43:49 (5652): No heartbeat from core client for 30 sec - exiting 21:43:50 (5652): No heartbeat from core client for 30 sec - exiting 21:43:51 (5652): No heartbeat from core client for 30 sec - exiting 21:43:52 (5652): No heartbeat from core client for 30 sec - exiting 21:43:53 (5652): No heartbeat from core client for 30 sec - exiting 21:43:54 (5652): No heartbeat from core client for 30 sec - exiting 21:43:55 (5652): No heartbeat from core client for 30 sec - exiting 21:43:56 (5652): No heartbeat from core client for 30 sec - exiting 21:43:57 (5652): No heartbeat from core client for 30 sec - exiting 21:43:58 (5652): No heartbeat from core client for 30 sec - exiting 21:43:59 (5652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:00 (5652): No heartbeat from core client for 30 sec - exiting 21:44:01 (5652): No heartbeat from core client for 30 sec - exiting 21:44:02 (5652): No heartbeat from core client for 30 sec - exiting 21:44:03 (5652): No heartbeat from core client for 30 sec - exiting 21:44:04 (5652): No heartbeat from core client for 30 sec - exiting 21:44:05 (5652): No heartbeat from core client for 30 sec - exiting 21:44:06 (5652): No heartbeat from core client for 30 sec - exiting 21:44:07 (5652): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5696, iMonCtr=1 Model crash detected, will try to restart... 21:18:58 (4760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:39:23 (5392): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 00:39:24 (5392): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:41:12 (2948): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:48:38 (1128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:48:39 (1128): No heartbeat from core client for 30 sec - exiting 02:48:40 (1128): No heartbeat from core client for 30 sec - exiting 02:48:41 (1128): No heartbeat from core client for 30 sec - exiting 02:48:42 (1128): No heartbeat from core client for 30 sec - exiting 02:48:43 (1128): No heartbeat from core client for 30 sec - exiting 02:48:44 (1128): No heartbeat from core client for 30 sec - exiting 02:48:45 (1128): No heartbeat from core client for 30 sec - exiting 02:48:46 (1128): No heartbeat from core client for 30 sec - exiting 02:48:47 (1128): No heartbeat from core client for 30 sec - exiting 02:48:48 (1128): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 00:21:20 (4252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:21:21 (4252): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=14344, iMonCtr=1 Model crash detected, will try to restart... 21:02:36 (2220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:57:56 (4812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:44:43 (3168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:08:17 (5180): No heartbeat from core client for 30 sec - exiting 21:08:18 (5180): No heartbeat from core client for 30 sec - exiting 21:08:19 (5180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CF7361 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... 21:53:57 (5572): No heartbeat from core client for 30 sec - exiting 21:53:58 (5572): No heartbeat from core client for 30 sec - exiting 21:53:59 (5572): No heartbeat from core client for 30 sec - exiting 21:54:00 (5572): No heartbeat from core client for 30 sec - exiting 21:54:01 (5572): No heartbeat from core client for 30 sec - exiting 21:54:02 (5572): No heartbeat from core client for 30 sec - exiting 21:54:03 (5572): No heartbeat from core client for 30 sec - exiting 21:54:04 (5572): No heartbeat from core client for 30 sec - exiting 21:54:05 (5572): No heartbeat from core client for 30 sec - exiting 21:54:06 (5572): No heartbeat from core client for 30 sec - exiting 21:54:07 (5572): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x770D7036 read attempt to address 0x00000004 Engaging BOINC Windows Runtime Debugger... 21:54:08 (5572): No heartbeat from core client for 30 sec - exiting 21:54:09 (5572): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Oct 2013 21:04:08	934531	15997213	hadcm3n_841h_1980_40_008463176_0	259,200	1,371,016	5.2894
07 Oct 2013 05:02:27	934531	15997213	hadcm3n_841h_1980_40_008463176_0	233,280	1,240,482	5.3176
05 Oct 2013 10:29:34	934531	15997213	hadcm3n_841h_1980_40_008463176_0	207,360	1,102,713	5.3179
20 Sep 2013 16:49:37	934531	15997213	hadcm3n_841h_1980_40_008463176_0	181,440	965,715	5.3225
18 Sep 2013 23:39:50	934531	15997213	hadcm3n_841h_1980_40_008463176_0	155,520	825,963	5.3110
15 Sep 2013 18:46:53	934531	15997213	hadcm3n_841h_1980_40_008463176_0	129,600	689,925	5.3235
11 Sep 2013 23:45:53	934531	15997213	hadcm3n_841h_1980_40_008463176_0	103,680	558,980	5.3914
08 Sep 2013 18:54:48	934531	15997213	hadcm3n_841h_1980_40_008463176_0	77,760	427,700	5.5003
06 Sep 2013 23:21:45	934531	15997213	hadcm3n_841h_1980_40_008463176_0	51,840	284,778	5.4934
01 Sep 2013 23:45:27	934531	15997213	hadcm3n_841h_1980_40_008463176_0	25,920	145,643	5.6189