Task 13543565

Name	hadcm3n_yd3e_1900_40_007518942_0
Workunit	7716417
Created	28 Oct 2011, 13:00:31 UTC
Sent	20 Nov 2011, 0:24:56 UTC
Report deadline	19 Feb 2012, 7:52:07 UTC
Received	27 Dec 2011, 18:39:38 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1075370
Run time	8 days 3 hours 2 min 21 sec
CPU time	5 days 9 hours 39 min 44 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.48 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.18</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=404, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2496, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3928, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=984, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:22:17 (2124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:24:14 (376): No heartbeat from core client for 30 sec - exiting 10:24:15 (376): No heartbeat from core client for 30 sec - exiting 10:24:16 (376): No heartbeat from core client for 30 sec - exiting 10:24:17 (376): No heartbeat from core client for 30 sec - exiting 10:24:18 (376): No heartbeat from core client for 30 sec - exiting 10:24:19 (376): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:37 (5540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:26:38 (5540): No heartbeat from core client for 30 sec - exiting 10:26:39 (5540): No heartbeat from core client for 30 sec - exiting 10:26:40 (5540): No heartbeat from core client for 30 sec - exiting 10:26:41 (5540): No heartbeat from core client for 30 sec - exiting 10:29:31 (3356): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:29:32 (3356): No heartbeat from core client for 30 sec - exiting 10:29:33 (3356): No heartbeat from core client for 30 sec - exiting 10:29:34 (3356): No heartbeat from core client for 30 sec - exiting 10:29:35 (3356): No heartbeat from core client for 30 sec - exiting 10:29:36 (3356): No heartbeat from core client for 30 sec - exiting 10:33:14 (5696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:33:15 (5696): No heartbeat from core client for 30 sec - exiting 10:33:16 (5696): No heartbeat from core client for 30 sec - exiting 10:33:17 (5696): No heartbeat from core client for 30 sec - exiting 10:33:18 (5696): No heartbeat from core client for 30 sec - exiting 10:36:46 (3444): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:36:47 (3444): No heartbeat from core client for 30 sec - exiting 10:36:48 (3444): No heartbeat from core client for 30 sec - exiting 10:36:49 (3444): No heartbeat from core client for 30 sec - exiting 10:36:50 (3444): No heartbeat from core client for 30 sec - exiting 10:36:51 (3444): No heartbeat from core client for 30 sec - exiting 10:39:53 (5560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:39:54 (5560): No heartbeat from core client for 30 sec - exiting 10:39:55 (5560): No heartbeat from core client for 30 sec - exiting 10:39:56 (5560): No heartbeat from core client for 30 sec - exiting 14:59:37 (5436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:59:38 (5436): No heartbeat from core client for 30 sec - exiting 14:59:39 (5436): No heartbeat from core client for 30 sec - exiting 14:59:40 (5436): No heartbeat from core client for 30 sec - exiting 15:00:55 (2708): No heartbeat from core client for 30 sec - exiting 15:00:56 (2708): No heartbeat from core client for 30 sec - exiting 15:00:57 (2708): No heartbeat from core client for 30 sec - exiting 15:00:58 (2708): No heartbeat from core client for 30 sec - exiting 15:00:59 (2708): No heartbeat from core client for 30 sec - exiting 15:01:00 (2708): No heartbeat from core client for 30 sec - exiting 15:01:01 (2708): No heartbeat from core client for 30 sec - exiting 15:01:02 (2708): No heartbeat from core client for 30 sec - exiting 15:01:03 (2708): No heartbeat from core client for 30 sec - exiting 15:01:04 (2708): No heartbeat from core client for 30 sec - exiting 15:01:05 (2708): No heartbeat from core client for 30 sec - exiting 15:01:06 (2708): No heartbeat from core client for 30 sec - exiting 15:01:07 (2708): No heartbeat from core client for 30 sec - exiting 15:01:08 (2708): No heartbeat from core client for 30 sec - exiting 15:01:09 (2708): No heartbeat from core client for 30 sec - exiting 15:01:10 (2708): No heartbeat from core client for 30 sec - exiting 15:01:11 (2708): No heartbeat from core client for 30 sec - exiting 15:01:12 (2708): No heartbeat from core client for 30 sec - exiting 15:01:13 (2708): No heartbeat from core client for 30 sec - exiting 15:01:14 (2708): No heartbeat from core client for 30 sec - exiting 15:01:15 (2708): No heartbeat from core client for 30 sec - exiting 15:01:16 (2708): No heartbeat from core client for 30 sec - exiting 15:01:17 (2708): No heartbeat from core client for 30 sec - exiting 15:01:18 (2708): No heartbeat from core client for 30 sec - exiting 15:01:19 (2708): No heartbeat from core client for 30 sec - exiting 15:01:20 (2708): No heartbeat from core client for 30 sec - exiting 15:01:21 (2708): No heartbeat from core client for 30 sec - exiting 15:01:22 (2708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:23 (2708): No heartbeat from core client for 30 sec - exiting 15:01:24 (2708): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2948, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3016, iMonCtr=1 Model crash detected, will try to restart... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3540, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2664, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3864, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3544, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5024, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7084, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5260, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5436, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CFC3CB write attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77CFE722 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Dec 2011 14:31:13	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	259,200	465,733	1.7968
22 Dec 2011 21:12:39	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	233,280	419,441	1.7980
20 Dec 2011 00:50:55	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	207,360	373,864	1.8030
17 Dec 2011 15:05:18	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	181,440	328,017	1.8079
13 Dec 2011 12:17:28	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	155,520	282,721	1.8179
10 Dec 2011 13:42:09	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	129,600	234,969	1.8130
07 Dec 2011 14:43:53	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	103,680	189,802	1.8307
04 Dec 2011 15:21:34	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	77,760	142,792	1.8363
26 Nov 2011 14:56:04	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	51,840	94,479	1.8225
23 Nov 2011 02:00:59	1075370	13543565	hadcm3n_yd3e_1900_40_007518942_0	25,920	46,386	1.7896