Task 13370614

Name	hadcm3n_o3io_1940_40_007452024_1
Workunit	7649527
Created	10 Sep 2011, 12:27:08 UTC
Sent	11 Sep 2011, 17:25:34 UTC
Report deadline	12 Dec 2011, 0:52:45 UTC
Received	28 Oct 2011, 21:56:58 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1161753
Run time	10 days 9 hours 0 min 37 sec
CPU time	9 days 2 hours 31 min 14 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.50 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... 10:38:44 (3452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3244, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3244, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 18:15:19 (4216): No heartbeat from core client for 30 sec - exiting 18:15:20 (4216): No heartbeat from core client for 30 sec - exiting 18:15:21 (4216): No heartbeat from core client for 30 sec - exiting 18:15:22 (4216): No heartbeat from core client for 30 sec - exiting 18:15:23 (4216): No heartbeat from core client for 30 sec - exiting 18:15:24 (4216): No heartbeat from core client for 30 sec - exiting 18:15:25 (4216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4396, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4208, iMonCtr=1 Model crash detected, will try to restart... 23:23:35 (3584): No heartbeat from core client for 30 sec - exiting 23:23:36 (3584): No heartbeat from core client for 30 sec - exiting 23:23:37 (3584): No heartbeat from core client for 30 sec - exiting 23:23:38 (3584): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:35:16 (2404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:35:19 (2404): No heartbeat from core client for 30 sec - exiting 22:35:20 (2404): No heartbeat from core client for 30 sec - exiting 22:35:21 (2404): No heartbeat from core client for 30 sec - exiting 22:35:22 (2404): No heartbeat from core client for 30 sec - exiting 22:35:23 (2404): No heartbeat from core client for 30 sec - exiting 22:35:24 (2404): No heartbeat from core client for 30 sec - exiting 22:35:25 (2404): No heartbeat from core client for 30 sec - exiting 22:35:26 (2404): No heartbeat from core client for 30 sec - exiting 23:38:56 (3000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:38:58 (3000): No heartbeat from core client for 30 sec - exiting 23:38:59 (3000): No heartbeat from core client for 30 sec - exiting 23:39:00 (3000): No heartbeat from core client for 30 sec - exiting 23:41:18 (4608): No heartbeat from core client for 30 sec - exiting 23:41:19 (4608): No heartbeat from core client for 30 sec - exiting 23:41:20 (4608): No heartbeat from core client for 30 sec - exiting 23:41:21 (4608): No heartbeat from core client for 30 sec - exiting 23:41:22 (4608): No heartbeat from core client for 30 sec - exiting 23:41:23 (4608): No heartbeat from core client for 30 sec - exiting 23:41:24 (4608): No heartbeat from core client for 30 sec - exiting 23:41:25 (4608): No heartbeat from core client for 30 sec - exiting 23:41:26 (4608): No heartbeat from core client for 30 sec - exiting 23:41:27 (4608): No heartbeat from core client for 30 sec - exiting 23:41:29 (4608): No heartbeat from core client for 30 sec - exiting 23:41:30 (4608): No heartbeat from core client for 30 sec - exiting 23:41:31 (4608): No heartbeat from core client for 30 sec - exiting 23:41:32 (4608): No heartbeat from core client for 30 sec - exiting 23:41:33 (4608): No heartbeat from core client for 30 sec - exiting 23:41:34 (4608): No heartbeat from core client for 30 sec - exiting 23:41:35 (4608): No heartbeat from core client for 30 sec - exiting 23:41:36 (4608): No heartbeat from core client for 30 sec - exiting 23:41:37 (4608): No heartbeat from core client for 30 sec - exiting 23:41:38 (4608): No heartbeat from core client for 30 sec - exiting 23:41:39 (4608): No heartbeat from core client for 30 sec - exiting 23:41:40 (4608): No heartbeat from core client for 30 sec - exiting 23:41:41 (4608): No heartbeat from core client for 30 sec - exiting 23:41:42 (4608): No heartbeat from core client for 30 sec - exiting 23:41:43 (4608): No heartbeat from core client for 30 sec - exiting 23:41:44 (4608): No heartbeat from core client for 30 sec - exiting 23:41:45 (4608): No heartbeat from core client for 30 sec - exiting 23:41:46 (4608): No heartbeat from core client for 30 sec - exiting 23:41:47 (4608): No heartbeat from core client for 30 sec - exiting 23:41:48 (4608): No heartbeat from core client for 30 sec - exiting 23:41:49 (4608): No heartbeat from core client for 30 sec - exiting 23:41:50 (4608): No heartbeat from core client for 30 sec - exiting 23:41:51 (4608): No heartbeat from core client for 30 sec - exiting 23:41:52 (4608): No heartbeat from core client for 30 sec - exiting 23:41:53 (4608): No heartbeat from core client for 30 sec - exiting 23:41:54 (4608): No heartbeat from core client for 30 sec - exiting 23:41:55 (4608): No heartbeat from core client for 30 sec - exiting 23:41:56 (4608): No heartbeat from core client for 30 sec - exiting 23:41:57 (4608): No heartbeat from core client for 30 sec - exiting 23:41:58 (4608): No heartbeat from core client for 30 sec - exiting 23:41:59 (4608): No heartbeat from core client for 30 sec - exiting 23:42:00 (4608): No heartbeat from core client for 30 sec - exiting 23:42:01 (4608): No heartbeat from core client for 30 sec - exiting 23:42:02 (4608): No heartbeat from core client for 30 sec - exiting 23:42:03 (4608): No heartbeat from core client for 30 sec - exiting 23:42:04 (4608): No heartbeat from core client for 30 sec - exiting 23:42:05 (4608): No heartbeat from core client for 30 sec - exiting 23:42:06 (4608): No heartbeat from core client for 30 sec - exiting 23:42:07 (4608): No heartbeat from core client for 30 sec - exiting 23:42:08 (4608): No heartbeat from core client for 30 sec - exiting 23:42:09 (4608): No heartbeat from core client for 30 sec - exiting 23:42:10 (4608): No heartbeat from core client for 30 sec - exiting 23:42:11 (4608): No heartbeat from core client for 30 sec - exiting 23:42:12 (4608): No heartbeat from core client for 30 sec - exiting 23:42:13 (4608): No heartbeat from core client for 30 sec - exiting 23:42:14 (4608): No heartbeat from core client for 30 sec - exiting 23:42:15 (4608): No heartbeat from core client for 30 sec - exiting 23:42:16 (4608): No heartbeat from core client for 30 sec - exiting 23:42:17 (4608): No heartbeat from core client for 30 sec - exiting 23:42:18 (4608): No heartbeat from core client for 30 sec - exiting 23:42:19 (4608): No heartbeat from core client for 30 sec - exiting 23:42:20 (4608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:43:56 (1820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3340, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3576, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3252, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 01:12:49 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2268, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2268, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7755B84B write attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77923A93 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Oct 2011 17:18:41	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	259,200	782,564	3.0192
17 Oct 2011 19:29:29	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	233,280	702,363	3.0108
14 Oct 2011 22:13:13	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	207,360	623,811	3.0083
09 Oct 2011 11:49:21	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	181,440	545,945	3.0090
05 Oct 2011 21:57:59	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	155,520	470,222	3.0235
02 Oct 2011 22:27:55	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	129,600	403,828	3.1160
30 Sep 2011 23:24:24	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	103,680	329,094	3.1741
24 Sep 2011 22:32:17	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	77,760	246,392	3.1686
18 Sep 2011 14:32:29	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	51,840	163,567	3.1552
15 Sep 2011 23:10:19	1161753	13370614	hadcm3n_o3io_1940_40_007452024_1	25,920	81,286	3.1360