Task 15993147

Name	hadcm3n_8103_1980_40_008459238_0
Workunit	8610094
Created	30 Aug 2013, 20:44:59 UTC
Sent	6 Sep 2013, 9:29:40 UTC
Report deadline	6 Dec 2013, 16:56:51 UTC
Received	19 Sep 2013, 6:52:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1158872
Run time	9 days 20 hours 48 min 30 sec
CPU time	9 days 20 hours 4 min 5 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.39 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> 20:06:10 (9384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:47:44 (672): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 07:47:45 (672): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 10:33:37 (11296): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:48:43 (11188): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 17:48:45 (11188): No heartbeat from core client for 30 sec - exiting 05:16:13 (1188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:32:38 (13072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:36:59 (7440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:52 (9336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:39:53 (9336): No heartbeat from core client for 30 sec - exiting 05:39:54 (9336): No heartbeat from core client for 30 sec - exiting 05:39:55 (9336): No heartbeat from core client for 30 sec - exiting 05:39:56 (9336): No heartbeat from core client for 30 sec - exiting 05:39:57 (9336): No heartbeat from core client for 30 sec - exiting 05:39:58 (9336): No heartbeat from core client for 30 sec - exiting 05:39:59 (9336): No heartbeat from core client for 30 sec - exiting 05:40:01 (9336): No heartbeat from core client for 30 sec - exiting 05:40:02 (9336): No heartbeat from core client for 30 sec - exiting 05:40:03 (9336): No heartbeat from core client for 30 sec - exiting 16:21:55 (10716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:26:12 (7568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:16 (12268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:40:23 (7472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:46:10 (8244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:53:06 (12580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:55:42 (2848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:29:37 (13056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:40:33 (7544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:55:47 (13120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:59:37 (10264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:05:41 (9020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:09:31 (10924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:49 (7464): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:13:50 (7464): No heartbeat from core client for 30 sec - exiting 08:13:51 (7464): No heartbeat from core client for 30 sec - exiting 08:13:52 (7464): No heartbeat from core client for 30 sec - exiting 08:13:53 (7464): No heartbeat from core client for 30 sec - exiting 08:13:54 (7464): No heartbeat from core client for 30 sec - exiting 08:13:55 (7464): No heartbeat from core client for 30 sec - exiting 08:13:56 (7464): No heartbeat from core client for 30 sec - exiting 08:13:57 (7464): No heartbeat from core client for 30 sec - exiting 08:13:58 (7464): No heartbeat from core client for 30 sec - exiting 08:13:59 (7464): No heartbeat from core client for 30 sec - exiting 08:20:07 (9356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:08:05 (7160): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:35:29 (9828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:05:31 (12008): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:55:40 (13940): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:15:48 (13240): No heartbeat from core client for 30 sec - exiting 23:15:49 (13240): No heartbeat from core client for 30 sec - exiting 23:15:50 (13240): No heartbeat from core client for 30 sec - exiting 23:15:51 (13240): No heartbeat from core client for 30 sec - exiting 23:15:52 (13240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/8103ko.pji3c10 Error converting file to netcdf: dataout/8103ko.pii3c10 Error converting file to netcdf: dataout/8103ko.pfi3c10 Error converting file to netcdf: dataout/8103ka.phi3c10 Error converting file to netcdf: dataout/8103ka.pgi3c10 Error converting file to netcdf: dataout/8103ka.pei3c10 Error converting file to netcdf: dataout/8103ka.pdi3c10 03:06:00 (7064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 03:16:00 (13776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:26:01 (9128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:36:02 (13524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:06:04 (13012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:36:06 (13776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:06:07 (2252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:26:20 (6312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:59:27 (6816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:24:41 (6564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:28:18 (6564): No heartbeat from core client for 30 sec - exiting 22:28:19 (6564): No heartbeat from core client for 30 sec - exiting 22:28:20 (6564): No heartbeat from core client for 30 sec - exiting 22:28:21 (6564): No heartbeat from core client for 30 sec - exiting 22:33:10 (9108): No heartbeat from core client for 30 sec - exiting 22:33:11 (9108): No heartbeat from core client for 30 sec - exiting 22:33:12 (9108): No heartbeat from core client for 30 sec - exiting 22:33:13 (9108): No heartbeat from core client for 30 sec - exiting 22:33:14 (9108): No heartbeat from core client for 30 sec - exiting 22:33:15 (9108): No heartbeat from core client for 30 sec - exiting 22:33:16 (9108): No heartbeat from core client for 30 sec - exiting 22:33:17 (9108): No heartbeat from core client for 30 sec - exiting 22:33:18 (9108): No heartbeat from core client for 30 sec - exiting 22:33:19 (9108): No heartbeat from core client for 30 sec - exiting 22:33:20 (9108): No heartbeat from core client for 30 sec - exiting 22:33:21 (9108): No heartbeat from core client for 30 sec - exiting 22:33:22 (9108): No heartbeat from core client for 30 sec - exiting 22:33:23 (9108): No heartbeat from core client for 30 sec - exiting 22:33:24 (9108): No heartbeat from core client for 30 sec - exiting 22:33:25 (9108): No heartbeat from core client for 30 sec - exiting 22:33:26 (9108): No heartbeat from core client for 30 sec - exiting 22:33:27 (9108): No heartbeat from core client for 30 sec - exiting 22:33:28 (9108): No heartbeat from core client for 30 sec - exiting 22:33:29 (9108): No heartbeat from core client for 30 sec - exiting 22:33:30 (9108): No heartbeat from core client for 30 sec - exiting 22:33:31 (9108): No heartbeat from core client for 30 sec - exiting 22:33:32 (9108): No heartbeat from core client for 30 sec - exiting 22:33:33 (9108): No heartbeat from core client for 30 sec - exiting 22:33:34 (9108): No heartbeat from core client for 30 sec - exiting 22:33:35 (9108): No heartbeat from core client for 30 sec - exiting 22:33:36 (9108): No heartbeat from core client for 30 sec - exiting 22:33:37 (9108): No heartbeat from core client for 30 sec - exiting 22:33:38 (9108): No heartbeat from core client for 30 sec - exiting 22:33:39 (9108): No heartbeat from core client for 30 sec - exiting 22:33:40 (9108): No heartbeat from core client for 30 sec - exiting 22:33:41 (9108): No heartbeat from core client for 30 sec - exiting 22:33:42 (9108): No heartbeat from core client for 30 sec - exiting 22:33:43 (9108): No heartbeat from core client for 30 sec - exiting 22:33:44 (9108): No heartbeat from core client for 30 sec - exiting 22:33:45 (9108): No heartbeat from core client for 30 sec - exiting 22:33:46 (9108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:38:50 (7500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:07 (3540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:09:08 (3540): No heartbeat from core client for 30 sec - exiting 03:09:09 (3540): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12692, iMonCtr=1 Model crash detected, will try to restart... 04:14:00 (1104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:24:02 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:28:29 (1336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:31:12 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:47 (9208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:04:49 (9208): No heartbeat from core client for 30 sec - exiting 21:04:50 (9208): No heartbeat from core client for 30 sec - exiting 21:04:51 (9208): No heartbeat from core client for 30 sec - exiting 21:04:52 (9208): No heartbeat from core client for 30 sec - exiting 21:04:53 (9208): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:41:09 (9788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:49 (9572): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:47:50 (9572): No heartbeat from core client for 30 sec - exiting 20:23:09 (8144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:19 (6452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:05:57 (9164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:03 (7732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:38:04 (7732): No heartbeat from core client for 30 sec - exiting 10:44:11 (5796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:09:43 (6432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:19:40 (11140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:51:12 (9244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:11 (6256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:07:15 (6256): No heartbeat from core client for 30 sec - exiting 01:07:16 (6256): No heartbeat from core client for 30 sec - exiting 01:07:17 (6256): No heartbeat from core client for 30 sec - exiting 01:07:18 (6256): No heartbeat from core client for 30 sec - exiting 01:07:19 (6256): No heartbeat from core client for 30 sec - exiting 01:07:20 (6256): No heartbeat from core client for 30 sec - exiting 01:07:21 (6256): No heartbeat from core client for 30 sec - exiting 01:07:23 (6256): No heartbeat from core client for 30 sec - exiting 01:07:24 (6256): No heartbeat from core client for 30 sec - exiting 01:07:25 (6256): No heartbeat from core client for 30 sec - exiting 01:07:26 (6256): No heartbeat from core client for 30 sec - exiting 01:07:27 (6256): No heartbeat from core client for 30 sec - exiting 01:07:28 (6256): No heartbeat from core client for 30 sec - exiting 01:07:29 (6256): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 08:17:27 (9320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:58:51 (5376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:45 (10772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:07:46 (10772): No heartbeat from core client for 30 sec - exiting 15:27:46 (276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:01:30 (292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:38:33 (4052): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:18:41 (10956): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:38:47 (284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:49:33 (7056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:49:34 (7056): No heartbeat from core client for 30 sec - exiting 04:39:46 (6148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:39:47 (6148): No heartbeat from core client for 30 sec - exiting 04:39:48 (6148): No heartbeat from core client for 30 sec - exiting 04:39:49 (6148): No heartbeat from core client for 30 sec - exiting 04:39:50 (6148): No heartbeat from core client for 30 sec - exiting 04:39:51 (6148): No heartbeat from core client for 30 sec - exiting 04:39:52 (6148): No heartbeat from core client for 30 sec - exiting 04:39:53 (6148): No heartbeat from core client for 30 sec - exiting 04:39:54 (6148): No heartbeat from core client for 30 sec - exiting 04:39:55 (6148): No heartbeat from core client for 30 sec - exiting 04:39:56 (6148): No heartbeat from core client for 30 sec - exiting 05:51:44 (3668): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:20:01 (6928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:20:02 (6928): No heartbeat from core client for 30 sec - exiting 10:20:03 (6928): No heartbeat from core client for 30 sec - exiting 10:20:04 (6928): No heartbeat from core client for 30 sec - exiting 10:20:05 (6928): No heartbeat from core client for 30 sec - exiting 12:00:05 (8596): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77E83AC3 read attempt to address 0x403C47B9 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77E83AC3 read attempt to address 0x403C47B9 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
18 Sep 2013 16:51:37	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	259,200	800,424	3.0881
17 Sep 2013 16:07:30	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	233,280	720,262	3.0875
16 Sep 2013 17:19:15	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	207,360	640,288	3.0878
15 Sep 2013 17:56:41	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	181,440	559,941	3.0861
14 Sep 2013 19:10:32	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	155,520	479,696	3.0845
13 Sep 2013 20:38:43	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	129,600	399,745	3.0845
12 Sep 2013 22:08:34	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	103,680	320,817	3.0943
11 Sep 2013 22:30:34	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	77,760	241,441	3.1050
11 Sep 2013 06:50:39	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	51,840	161,462	3.1146
10 Sep 2013 00:45:57	1158872	15993147	hadcm3n_8103_1980_40_008459238_0	25,920	81,049	3.1269