Task 15798002

Name	hadcm3n_38q0_1980_40_008320767_1
Workunit	8471902
Created	26 May 2013, 23:16:59 UTC
Sent	26 May 2013, 23:17:20 UTC
Report deadline	26 Aug 2013, 6:44:31 UTC
Received	27 Oct 2013, 1:41:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1281494
Run time	15 days 23 hours 51 min 12 sec
CPU time	15 days 10 hours 14 min 20 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.11 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4508, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:10:35 (6920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:10:36 (6920): No heartbeat from core client for 30 sec - exiting 21:10:37 (6920): No heartbeat from core client for 30 sec - exiting 21:10:38 (6920): No heartbeat from core client for 30 sec - exiting 21:10:39 (6920): No heartbeat from core client for 30 sec - exiting 21:10:40 (6920): No heartbeat from core client for 30 sec - exiting 21:10:41 (6920): No heartbeat from core client for 30 sec - exiting 21:10:42 (6920): No heartbeat from core client for 30 sec - exiting 21:10:43 (6920): No heartbeat from core client for 30 sec - exiting 21:10:44 (6920): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5936, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:35:37 (4196): No heartbeat from core client for 30 sec - exiting 11:35:39 (4196): No heartbeat from core client for 30 sec - exiting 11:35:40 (4196): No heartbeat from core client for 30 sec - exiting 11:35:41 (4196): No heartbeat from core client for 30 sec - exiting 11:35:42 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7216, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8664, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3392, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5696, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6864, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6864, iMonCtr=1 Model crash detected, will try to restart... 15:45:21 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1716, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2000, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1016, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1016, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 17:23:36 (5224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5148, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5656, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5280, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5316, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5752, iMonCtr=1 Model crash detected, will try to restart... 17:24:16 (5756): No heartbeat from core client for 30 sec - exiting 17:24:17 (5756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1188, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... 13:49:42 (6068): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1520, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5836, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5968, iMonCtr=1 Model crash detected, will try to restart... 15:20:29 (5300): No heartbeat from core client for 30 sec - exiting 15:20:30 (5300): No heartbeat from core client for 30 sec - exiting 15:20:31 (5300): No heartbeat from core client for 30 sec - exiting 15:20:32 (5300): No heartbeat from core client for 30 sec - exiting 15:20:33 (5300): No heartbeat from core client for 30 sec - exiting 15:20:34 (5300): No heartbeat from core client for 30 sec - exiting 15:20:36 (5300): No heartbeat from core client for 30 sec - exiting 15:20:37 (5300): No heartbeat from core client for 30 sec - exiting 15:20:38 (5300): No heartbeat from core client for 30 sec - exiting 15:20:39 (5300): No heartbeat from core client for 30 sec - exiting 15:20:40 (5300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6556, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4244, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6776, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:44:49 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:44:50 (5436): No heartbeat from core client for 30 sec - exiting 17:44:51 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6000, iMonCtr=1 Model crash detected, will try to restart... 13:05:17 (4732): No heartbeat from core client for 30 sec - exiting 13:05:18 (4732): No heartbeat from core client for 30 sec - exiting 13:05:19 (4732): No heartbeat from core client for 30 sec - exiting 13:05:20 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:41 (5076): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771D3AC3 read attempt to address 0x40C7FF07 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771D3AC3 read attempt to address 0x40C7FF07 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Oct 2013 21:44:01	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	777,600	1,329,306	1.7095
15 Oct 2013 02:32:33	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	751,680	1,287,416	1.7127
11 Oct 2013 21:44:07	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	725,760	1,244,667	1.7150
01 Oct 2013 02:12:50	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	699,840	1,198,260	1.7122
27 Sep 2013 01:34:57	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	673,920	1,149,204	1.7053
21 Sep 2013 03:49:02	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	648,000	1,100,352	1.6981
15 Sep 2013 17:26:33	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	622,080	1,051,573	1.6904
14 Sep 2013 01:45:17	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	596,160	1,002,865	1.6822
09 Sep 2013 21:44:40	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	570,240	954,488	1.6738
02 Sep 2013 01:56:30	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	544,320	907,088	1.6665
31 Aug 2013 19:04:35	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	518,400	867,661	1.6737
29 Aug 2013 02:27:49	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	492,480	830,625	1.6866
26 Aug 2013 22:07:32	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	466,560	794,591	1.7031
25 Aug 2013 06:22:36	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	440,640	757,875	1.7199
24 Aug 2013 01:56:23	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	414,720	721,322	1.7393
20 Aug 2013 02:26:27	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	388,800	683,152	1.7571
14 Aug 2013 21:30:53	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	362,880	640,035	1.7638
14 Aug 2013 21:30:53	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	336,960	595,823	1.7682
14 Aug 2013 21:30:53	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	311,040	552,269	1.7756
14 Aug 2013 21:30:53	1281494	15798002	hadcm3n_38q0_1980_40_008320767_1	285,120	508,306	1.7828