Task 15773223

Name	hadcm3n_3k8w_1980_40_008363921_0
Workunit	8514780
Created	10 May 2013, 18:50:03 UTC
Sent	10 May 2013, 18:51:04 UTC
Report deadline	10 Aug 2013, 2:18:15 UTC
Received	4 Jun 2013, 8:43:13 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1245161
Run time	6 days 1 hours 41 min 50 sec
CPU time	6 days 0 hours 4 min 40 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.53 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3912, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1756, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5156, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3968, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5684, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4324, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4032, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4064, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 10:02:19 (6316): No heartbeat from core client for 30 sec - exiting 10:02:20 (6316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:39:48 (2268): No heartbeat from core client for 30 sec - exiting 19:39:49 (2268): No heartbeat from core client for 30 sec - exiting 19:39:50 (2268): No heartbeat from core client for 30 sec - exiting 19:39:51 (2268): No heartbeat from core client for 30 sec - exiting 19:39:52 (2268): No heartbeat from core client for 30 sec - exiting 19:39:53 (2268): No heartbeat from core client for 30 sec - exiting 19:39:54 (2268): No heartbeat from core client for 30 sec - exiting 19:39:55 (2268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:02:21 (5576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:00:31 (5340): No heartbeat from core client for 30 sec - exiting 10:00:33 (5340): No heartbeat from core client for 30 sec - exiting 10:00:34 (5340): No heartbeat from core client for 30 sec - exiting 10:00:35 (5340): No heartbeat from core client for 30 sec - exiting 10:00:36 (5340): No heartbeat from core client for 30 sec - exiting 10:00:37 (5340): No heartbeat from core client for 30 sec - exiting 10:00:38 (5340): No heartbeat from core client for 30 sec - exiting 10:00:39 (5340): No heartbeat from core client for 30 sec - exiting 10:00:40 (5340): No heartbeat from core client for 30 sec - exiting 10:00:41 (5340): No heartbeat from core client for 30 sec - exiting 10:00:42 (5340): No heartbeat from core client for 30 sec - exiting 10:00:43 (5340): No heartbeat from core client for 30 sec - exiting 10:00:44 (5340): No heartbeat from core client for 30 sec - exiting 10:00:45 (5340): No heartbeat from core client for 30 sec - exiting 10:00:46 (5340): No heartbeat from core client for 30 sec - exiting 10:00:47 (5340): No heartbeat from core client for 30 sec - exiting 10:00:48 (5340): No heartbeat from core client for 30 sec - exiting 10:00:49 (5340): No heartbeat from core client for 30 sec - exiting 10:00:50 (5340): No heartbeat from core client for 30 sec - exiting 10:00:51 (5340): No heartbeat from core client for 30 sec - exiting 10:00:52 (5340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:26:23 (1460): No heartbeat from core client for 30 sec - exiting 18:26:25 (1460): No heartbeat from core client for 30 sec - exiting 18:26:26 (1460): No heartbeat from core client for 30 sec - exiting 18:26:27 (1460): No heartbeat from core client for 30 sec - exiting 18:26:28 (1460): No heartbeat from core client for 30 sec - exiting 18:26:29 (1460): No heartbeat from core client for 30 sec - exiting 18:26:30 (1460): No heartbeat from core client for 30 sec - exiting 18:26:31 (1460): No heartbeat from core client for 30 sec - exiting 18:26:32 (1460): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:49:29 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:39:27 (2816): No heartbeat from core client for 30 sec - exiting 16:39:28 (2816): No heartbeat from core client for 30 sec - exiting 16:39:29 (2816): No heartbeat from core client for 30 sec - exiting 16:39:30 (2816): No heartbeat from core client for 30 sec - exiting 16:39:31 (2816): No heartbeat from core client for 30 sec - exiting 16:39:32 (2816): No heartbeat from core client for 30 sec - exiting 16:39:33 (2816): No heartbeat from core client for 30 sec - exiting 16:39:34 (2816): No heartbeat from core client for 30 sec - exiting 16:39:35 (2816): No heartbeat from core client for 30 sec - exiting 16:39:36 (2816): No heartbeat from core client for 30 sec - exiting 16:39:37 (2816): No heartbeat from core client for 30 sec - exiting 16:39:38 (2816): No heartbeat from core client for 30 sec - exiting 16:39:39 (2816): No heartbeat from core client for 30 sec - exiting 16:39:40 (2816): No heartbeat from core client for 30 sec - exiting 16:39:41 (2816): No heartbeat from core client for 30 sec - exiting 16:39:42 (2816): No heartbeat from core client for 30 sec - exiting 16:39:43 (2816): No heartbeat from core client for 30 sec - exiting 16:39:44 (2816): No heartbeat from core client for 30 sec - exiting 16:39:45 (2816): No heartbeat from core client for 30 sec - exiting 16:39:46 (2816): No heartbeat from core client for 30 sec - exiting 16:39:47 (2816): No heartbeat from core client for 30 sec - exiting 16:39:48 (2816): No heartbeat from core client for 30 sec - exiting 16:39:49 (2816): No heartbeat from core client for 30 sec - exiting 16:39:50 (2816): No heartbeat from core client for 30 sec - exiting 16:39:51 (2816): No heartbeat from core client for 30 sec - exiting 16:39:52 (2816): No heartbeat from core client for 30 sec - exiting 16:39:53 (2816): No heartbeat from core client for 30 sec - exiting 16:39:54 (2816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:58:15 (2480): No heartbeat from core client for 30 sec - exiting 16:58:16 (2480): No heartbeat from core client for 30 sec - exiting 16:58:17 (2480): No heartbeat from core client for 30 sec - exiting 16:58:18 (2480): No heartbeat from core client for 30 sec - exiting 16:58:19 (2480): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:05 (4904): No heartbeat from core client for 30 sec - exiting 09:40:06 (4904): No heartbeat from core client for 30 sec - exiting 09:40:07 (4904): No heartbeat from core client for 30 sec - exiting 09:40:08 (4904): No heartbeat from core client for 30 sec - exiting 09:40:09 (4904): No heartbeat from core client for 30 sec - exiting 09:40:10 (4904): No heartbeat from core client for 30 sec - exiting 09:40:11 (4904): No heartbeat from core client for 30 sec - exiting 09:40:12 (4904): No heartbeat from core client for 30 sec - exiting 09:40:13 (4904): No heartbeat from core client for 30 sec - exiting 09:40:14 (4904): No heartbeat from core client for 30 sec - exiting 09:40:15 (4904): No heartbeat from core client for 30 sec - exiting 09:40:16 (4904): No heartbeat from core client for 30 sec - exiting 09:40:17 (4904): No heartbeat from core client for 30 sec - exiting 09:40:18 (4904): No heartbeat from core client for 30 sec - exiting 09:40:19 (4904): No heartbeat from core client for 30 sec - exiting 09:40:20 (4904): No heartbeat from core client for 30 sec - exiting 09:40:21 (4904): No heartbeat from core client for 30 sec - exiting 09:40:22 (4904): No heartbeat from core client for 30 sec - exiting 09:40:23 (4904): No heartbeat from core client for 30 sec - exiting 09:40:24 (4904): No heartbeat from core client for 30 sec - exiting 09:40:25 (4904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:20:26 (412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:01:19 (6432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:03:29 (5800): No heartbeat from core client for 30 sec - exiting 10:03:30 (5800): No heartbeat from core client for 30 sec - exiting 10:03:31 (5800): No heartbeat from core client for 30 sec - exiting 10:03:32 (5800): No heartbeat from core client for 30 sec - exiting 10:03:33 (5800): No heartbeat from core client for 30 sec - exiting 10:03:34 (5800): No heartbeat from core client for 30 sec - exiting 10:03:35 (5800): No heartbeat from core client for 30 sec - exiting 10:03:36 (5800): No heartbeat from core client for 30 sec - exiting 10:03:37 (5800): No heartbeat from core client for 30 sec - exiting 10:03:38 (5800): No heartbeat from core client for 30 sec - exiting 10:03:39 (5800): No heartbeat from core client for 30 sec - exiting 10:03:40 (5800): No heartbeat from core client for 30 sec - exiting 10:03:41 (5800): No heartbeat from core client for 30 sec - exiting 10:03:42 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:27:05 (5904): No heartbeat from core client for 30 sec - exiting 19:27:06 (5904): No heartbeat from core client for 30 sec - exiting 19:27:07 (5904): No heartbeat from core client for 30 sec - exiting 19:27:08 (5904): No heartbeat from core client for 30 sec - exiting 19:27:09 (5904): No heartbeat from core client for 30 sec - exiting 19:27:10 (5904): No heartbeat from core client for 30 sec - exiting 19:27:11 (5904): No heartbeat from core client for 30 sec - exiting 19:27:12 (5904): No heartbeat from core client for 30 sec - exiting 19:27:13 (5904): No heartbeat from core client for 30 sec - exiting 19:27:14 (5904): No heartbeat from core client for 30 sec - exiting 19:27:15 (5904): No heartbeat from core client for 30 sec - exiting 19:27:16 (5904): No heartbeat from core client for 30 sec - exiting 19:27:17 (5904): No heartbeat from core client for 30 sec - exiting 19:27:18 (5904): No heartbeat from core client for 30 sec - exiting 19:27:19 (5904): No heartbeat from core client for 30 sec - exiting 19:27:20 (5904): No heartbeat from core client for 30 sec - exiting 19:27:21 (5904): No heartbeat from core client for 30 sec - exiting 19:27:22 (5904): No heartbeat from core client for 30 sec - exiting 19:27:23 (5904): No heartbeat from core client for 30 sec - exiting 19:27:24 (5904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:26:55 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x770B7373 read attempt to address 0x40513CE0 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77253AB3 read attempt to address 0x40513CE8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Jun 2013 09:29:12	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	518,400	509,110	0.9821
31 May 2013 16:41:28	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	492,480	482,208	0.9791
30 May 2013 17:59:36	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	466,560	456,167	0.9777
29 May 2013 17:49:46	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	440,640	432,094	0.9806
28 May 2013 18:39:49	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	414,720	406,571	0.9804
28 May 2013 05:52:38	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	388,800	380,784	0.9794
27 May 2013 22:59:49	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	362,880	356,141	0.9814
27 May 2013 16:06:47	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	336,960	331,081	0.9826
25 May 2013 11:07:53	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	311,040	305,354	0.9817
24 May 2013 14:29:45	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	285,120	278,791	0.9778
22 May 2013 21:18:09	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	259,200	252,399	0.9738
22 May 2013 09:08:30	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	233,280	226,417	0.9706
21 May 2013 07:47:37	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	207,360	199,751	0.9633
20 May 2013 08:35:47	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	181,440	173,856	0.9582
17 May 2013 18:49:45	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	155,520	147,668	0.9495
17 May 2013 08:40:24	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	129,600	122,427	0.9447
16 May 2013 08:47:19	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	103,680	96,417	0.9299
14 May 2013 18:34:28	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	77,760	71,201	0.9157
14 May 2013 07:43:19	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	51,840	46,219	0.8916
13 May 2013 10:04:00	1245161	15773223	hadcm3n_3k8w_1980_40_008363921_0	25,920	23,019	0.8881