Task 15908144

Name	hadcm3n_n256_1920_40_008377719_3
Workunit	8528578
Created	25 Jul 2013, 20:04:22 UTC
Sent	25 Jul 2013, 20:04:48 UTC
Report deadline	25 Oct 2013, 3:31:59 UTC
Received	27 Sep 2013, 8:04:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1229098
Run time	15 days 17 hours 10 min 24 sec
CPU time	14 days 21 hours 7 min 14 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.17 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5676, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5060, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Hold Restart file rename failed on atmos_restart.hold 09:44:41 (4592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:34:55 (5148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:56:53 (5520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:58:31 (3192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:00:08 (2928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:15 (6632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:02:20 (7092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:05:12 (5320): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:16:15 (5564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:39:19 (4800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:25 (4408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:13:56 (5560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold 16:27:08 (6516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:27:57 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4276, iMonCtr=1 Model crash detected, will try to restart... 08:20:38 (4288): Can't acquire lockfile (32) - waiting 35s 08:21:13 (4288): Can't acquire lockfile (32) - exiting 08:21:13 (4288): Error: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird. (0x20) Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:03:24 (4672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:07:24 (6260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:14:31 (6284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:14:32 (6284): No heartbeat from core client for 30 sec - exiting 22:14:33 (6284): No heartbeat from core client for 30 sec - exiting 22:14:34 (6284): No heartbeat from core client for 30 sec - exiting 22:14:35 (6284): No heartbeat from core client for 30 sec - exiting 22:14:36 (6284): No heartbeat from core client for 30 sec - exiting 22:15:37 (3756): No heartbeat from core client for 30 sec - exiting 22:15:38 (3756): No heartbeat from core client for 30 sec - exiting 22:15:40 (3756): No heartbeat from core client for 30 sec - exiting 22:15:41 (3756): No heartbeat from core client for 30 sec - exiting 22:15:42 (3756): No heartbeat from core client for 30 sec - exiting 22:15:44 (3756): No heartbeat from core client for 30 sec - exiting 22:15:45 (3756): No heartbeat from core client for 30 sec - exiting 22:15:47 (3756): No heartbeat from core client for 30 sec - exiting 22:15:48 (3756): No heartbeat from core client for 30 sec - exiting 22:15:49 (3756): No heartbeat from core client for 30 sec - exiting 22:15:51 (3756): No heartbeat from core client for 30 sec - exiting 22:15:52 (3756): No heartbeat from core client for 30 sec - exiting 22:15:53 (3756): No heartbeat from core client for 30 sec - exiting 22:15:55 (3756): No heartbeat from core client for 30 sec - exiting 22:15:56 (3756): No heartbeat from core client for 30 sec - exiting 22:15:57 (3756): No heartbeat from core client for 30 sec - exiting 22:15:59 (3756): No heartbeat from core client for 30 sec - exiting 22:16:00 (3756): No heartbeat from core client for 30 sec - exiting 22:16:01 (3756): No heartbeat from core client for 30 sec - exiting 22:16:03 (3756): No heartbeat from core client for 30 sec - exiting 22:16:04 (3756): No heartbeat from core client for 30 sec - exiting 22:16:06 (3756): No heartbeat from core client for 30 sec - exiting 22:16:07 (3756): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:16:08 (3756): No heartbeat from core client for 30 sec - exiting 22:16:10 (3756): No heartbeat from core client for 30 sec - exiting 22:16:11 (3756): No heartbeat from core client for 30 sec - exiting 22:16:12 (3756): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:16:55 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Atmos Restart file copy failed on n256ka.dac87r0 Atmos Hold Restart file rename failed on atmos_restart.hold 11:07:39 (3564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76ED3AC3 read attempt to address 0x403E6775 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77523AC3 read attempt to address 0x403E6775 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Sep 2013 06:21:10	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	259,200	1,223,406	4.7199
18 Sep 2013 10:03:42	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	233,280	1,150,623	4.9324
18 Sep 2013 10:03:42	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	233,280	1,150,623	4.9324
17 Sep 2013 06:37:25	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	207,360	1,064,347	5.1328
14 Sep 2013 06:16:59	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	181,440	906,358	4.9954
11 Sep 2013 06:25:33	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	155,520	806,260	5.1843
02 Sep 2013 07:08:27	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	129,600	675,231	5.2101
29 Aug 2013 08:03:51	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	103,680	479,380	4.6236
14 Aug 2013 16:08:57	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	77,760	317,498	4.0831
14 Aug 2013 16:08:57	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	51,840	143,052	2.7595
14 Aug 2013 16:08:57	1229098	15908144	hadcm3n_n256_1920_40_008377719_3	25,920	82,106	3.1677