Task 15600960

Name	hadcm3n_4mnw_1940_40_008312131_1
Workunit	8463266
Created	8 Feb 2013, 11:29:53 UTC
Sent	8 Feb 2013, 11:40:01 UTC
Report deadline	10 May 2013, 19:07:12 UTC
Received	2 Mar 2013, 21:55:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1233009
Run time	13 days 4 hours 1 min 25 sec
CPU time	12 days 17 hours 51 min 59 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.66 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... 07:02:47 (12792): No heartbeat from core client for 30 sec - exiting 07:02:49 (12792): No heartbeat from core client for 30 sec - exiting 07:02:50 (12792): No heartbeat from core client for 30 sec - exiting 07:02:51 (12792): No heartbeat from core client for 30 sec - exiting 07:02:52 (12792): No heartbeat from core client for 30 sec - exiting 07:02:53 (12792): No heartbeat from core client for 30 sec - exiting 07:02:54 (12792): No heartbeat from core client for 30 sec - exiting 07:02:55 (12792): No heartbeat from core client for 30 sec - exiting 07:02:56 (12792): No heartbeat from core client for 30 sec - exiting 07:02:57 (12792): No heartbeat from core client for 30 sec - exiting 07:02:58 (12792): No heartbeat from core client for 30 sec - exiting 07:02:59 (12792): No heartbeat from core client for 30 sec - exiting 07:03:01 (12792): No heartbeat from core client for 30 sec - exiting 07:03:02 (12792): No heartbeat from core client for 30 sec - exiting 07:03:03 (12792): No heartbeat from core client for 30 sec - exiting 07:03:04 (12792): No heartbeat from core client for 30 sec - exiting 07:03:05 (12792): No heartbeat from core client for 30 sec - exiting 07:03:06 (12792): No heartbeat from core client for 30 sec - exiting 07:03:07 (12792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:27:07 (14128): No heartbeat from core client for 30 sec - exiting 03:27:10 (14128): No heartbeat from core client for 30 sec - exiting 03:27:12 (14128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:27:13 (14128): No heartbeat from core client for 30 sec - exiting 03:35:47 (5316): No heartbeat from core client for 30 sec - exiting 03:35:48 (5316): No heartbeat from core client for 30 sec - exiting 03:35:49 (5316): No heartbeat from core client for 30 sec - exiting 03:35:50 (5316): No heartbeat from core client for 30 sec - exiting 03:35:51 (5316): No heartbeat from core client for 30 sec - exiting 03:35:52 (5316): No heartbeat from core client for 30 sec - exiting 03:35:53 (5316): No heartbeat from core client for 30 sec - exiting 03:35:54 (5316): No heartbeat from core client for 30 sec - exiting 03:35:55 (5316): No heartbeat from core client for 30 sec - exiting 03:35:57 (5316): No heartbeat from core client for 30 sec - exiting 03:35:58 (5316): No heartbeat from core client for 30 sec - exiting 03:35:59 (5316): No heartbeat from core client for 30 sec - exiting 03:36:00 (5316): No heartbeat from core client for 30 sec - exiting 03:36:01 (5316): No heartbeat from core client for 30 sec - exiting 03:36:02 (5316): No heartbeat from core client for 30 sec - exiting 03:36:03 (5316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:59:41 (5684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6892, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 07:10:15 (6492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:22:54 (5328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4016, iMonCtr=1 Model crash detected, will try to restart... 07:58:13 (5208): No heartbeat from core client for 30 sec - exiting 07:58:14 (5208): No heartbeat from core client for 30 sec - exiting 07:58:15 (5208): No heartbeat from core client for 30 sec - exiting 07:58:16 (5208): No heartbeat from core client for 30 sec - exiting 07:58:17 (5208): No heartbeat from core client for 30 sec - exiting 07:58:18 (5208): No heartbeat from core client for 30 sec - exiting 07:58:19 (5208): No heartbeat from core client for 30 sec - exiting 07:58:20 (5208): No heartbeat from core client for 30 sec - exiting 07:58:21 (5208): No heartbeat from core client for 30 sec - exiting 07:58:22 (5208): No heartbeat from core client for 30 sec - exiting 07:58:23 (5208): No heartbeat from core client for 30 sec - exiting 07:58:24 (5208): No heartbeat from core client for 30 sec - exiting 07:58:25 (5208): No heartbeat from core client for 30 sec - exiting 07:58:26 (5208): No heartbeat from core client for 30 sec - exiting 07:58:27 (5208): No heartbeat from core client for 30 sec - exiting 07:58:28 (5208): No heartbeat from core client for 30 sec - exiting 07:58:29 (5208): No heartbeat from core client for 30 sec - exiting 07:58:30 (5208): No heartbeat from core client for 30 sec - exiting 07:58:31 (5208): No heartbeat from core client for 30 sec - exiting 07:58:32 (5208): No heartbeat from core client for 30 sec - exiting 07:58:33 (5208): No heartbeat from core client for 30 sec - exiting 07:58:34 (5208): No heartbeat from core client for 30 sec - exiting 07:58:35 (5208): No heartbeat from core client for 30 sec - exiting 07:58:36 (5208): No heartbeat from core client for 30 sec - exiting 07:58:37 (5208): No heartbeat from core client for 30 sec - exiting 07:58:38 (5208): No heartbeat from core client for 30 sec - exiting 07:58:40 (5208): No heartbeat from core client for 30 sec - exiting 07:58:41 (5208): No heartbeat from core client for 30 sec - exiting 07:58:42 (5208): No heartbeat from core client for 30 sec - exiting 07:58:43 (5208): No heartbeat from core client for 30 sec - exiting 07:58:44 (5208): No heartbeat from core client for 30 sec - exiting 07:58:45 (5208): No heartbeat from core client for 30 sec - exiting 07:58:46 (5208): No heartbeat from core client for 30 sec - exiting 07:58:47 (5208): No heartbeat from core client for 30 sec - exiting 07:58:48 (5208): No heartbeat from core client for 30 sec - exiting 07:58:49 (5208): No heartbeat from core client for 30 sec - exiting 07:58:50 (5208): No heartbeat from core client for 30 sec - exiting 07:58:51 (5208): No heartbeat from core client for 30 sec - exiting 07:58:52 (5208): No heartbeat from core client for 30 sec - exiting 07:58:53 (5208): No heartbeat from core client for 30 sec - exiting 07:58:54 (5208): No heartbeat from core client for 30 sec - exiting 07:58:55 (5208): No heartbeat from core client for 30 sec - exiting 07:58:56 (5208): No heartbeat from core client for 30 sec - exiting 07:58:57 (5208): No heartbeat from core client for 30 sec - exiting 07:58:58 (5208): No heartbeat from core client for 30 sec - exiting 07:58:59 (5208): No heartbeat from core client for 30 sec - exiting 07:59:00 (5208): No heartbeat from core client for 30 sec - exiting 07:59:01 (5208): No heartbeat from core client for 30 sec - exiting 07:59:02 (5208): No heartbeat from core client for 30 sec - exiting 07:59:03 (5208): No heartbeat from core client for 30 sec - exiting 07:59:04 (5208): No heartbeat from core client for 30 sec - exiting 07:59:05 (5208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:21:43 (856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 05:30:51 (7348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:19:01 (7360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:55:52 (4248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:03:57 (7072): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:22:04 (11888): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:24:34 (9740): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:10:07 (12856): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:43:11 (5792): No heartbeat from core client for 30 sec - exiting 20:43:12 (5792): No heartbeat from core client for 30 sec - exiting 20:43:14 (5792): No heartbeat from core client for 30 sec - exiting 20:43:15 (5792): No heartbeat from core client for 30 sec - exiting 20:43:16 (5792): No heartbeat from core client for 30 sec - exiting 20:43:17 (5792): No heartbeat from core client for 30 sec - exiting 20:43:18 (5792): No heartbeat from core client for 30 sec - exiting 20:43:19 (5792): No heartbeat from core client for 30 sec - exiting 20:43:20 (5792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:05:53 (3592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:08:17 (1564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:11:32 (5140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:44:40 (4172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:50:43 (10724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:28:55 (7908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... C18:04:02 (5612): No heartbeat from core client for 30 sec - exiting 18:04:03 (5612): No heartbeat from core client for 30 sec - exiting 18:04:04 (5612): No heartbeat from core client for 30 sec - exiting 18:04:05 (5612): No heartbeat from core client for 30 sec - exiting 18:04:06 (5612): No heartbeat from core client for 30 sec - exiting 18:04:07 (5612): No heartbeat from core client for 30 sec - exiting 18:04:08 (5612): No heartbeat from core client for 30 sec - exiting 18:04:10 (5612): No heartbeat from core client for 30 sec - exiting 18:04:11 (5612): No heartbeat from core client for 30 sec - exiting 18:04:12 (5612): No heartbeat from core client for 30 sec - exiting 18:04:13 (5612): No heartbeat from core client for 30 sec - exiting 18:04:14 (5612): No heartbeat from core client for 30 sec - exiting 18:04:15 (5612): No heartbeat from core client for 30 sec - exiting 18:04:16 (5612): No heartbeat from core client for 30 sec - exiting 18:04:17 (5612): No heartbeat from core client for 30 sec - exiting 18:04:18 (5612): No heartbeat from core client for 30 sec - exiting 18:04:19 (5612): No heartbeat from core client for 30 sec - exiting 18:04:20 (5612): No heartbeat from core client for 30 sec - exiting 18:04:22 (5612): No heartbeat from core client for 30 sec - exiting 18:04:23 (5612): No heartbeat from core client for 30 sec - exiting 18:04:24 (5612): No heartbeat from core client for 30 sec - exiting 18:04:25 (5612): No heartbeat from core client for 30 sec - exiting 18:04:26 (5612): No heartbeat from core client for 30 sec - exiting 18:04:27 (5612): No heartbeat from core client for 30 sec - exiting 18:04:28 (5612): No heartbeat from core client for 30 sec - exiting 18:04:29 (5612): No heartbeat from core client for 30 sec - exiting 18:04:30 (5612): No heartbeat from core client for 30 sec - exiting 18:04:31 (5612): No heartbeat from core client for 30 sec - exiting 18:04:32 (5612): No heartbeat from core client for 30 sec - exiting 18:04:34 (5612): No heartbeat from core client for 30 sec - exiting 18:04:35 (5612): No heartbeat from core client for 30 sec - exiting 18:04:36 (5612): No heartbeat from core client for 30 sec - exiting 18:04:37 (5612): No heartbeat from core client for 30 sec - exiting 18:04:38 (5612): No heartbeat from core client for 30 sec - exiting 18:04:39 (5612): No heartbeat from core client for 30 sec - exiting 18:04:40 (5612): No heartbeat from core client for 30 sec - exiting 18:04:41 (5612): No heartbeat from core client for 30 sec - exiting 18:04:42 (5612): No heartbeat from core client for 30 sec - exiting 18:04:43 (5612): No heartbeat from core client for 30 sec - exiting 18:04:44 (5612): No heartbeat from core client for 30 sec - exiting 18:04:46 (5612): No heartbeat from core client for 30 sec - exiting 18:04:47 (5612): No heartbeat from core client for 30 sec - exiting 18:04:48 (5612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:22:37 (5864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6396, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5872, iMonCtr=1 Model crash detected, will try to restart... 09:45:52 (5332): No heartbeat from core client for 30 sec - exiting 09:45:53 (5332): No heartbeat from core client for 30 sec - exiting 09:45:54 (5332): No heartbeat from core client for 30 sec - exiting 09:46:50 (5332): No heartbeat from core client for 30 sec - exiting 09:46:52 (5332): No heartbeat from core client for 30 sec - exiting 09:46:53 (5332): No heartbeat from core client for 30 sec - exiting 09:46:54 (5332): No heartbeat from core client for 30 sec - exiting 09:46:55 (5332): No heartbeat from core client for 30 sec - exiting 09:46:56 (5332): No heartbeat from core client for 30 sec - exiting 09:46:57 (5332): No heartbeat from core client for 30 sec - exiting 09:46:58 (5332): No heartbeat from core client for 30 sec - exiting 09:46:59 (5332): No heartbeat from core client for 30 sec - exiting 09:47:00 (5332): No heartbeat from core client for 30 sec - exiting 09:47:01 (5332): No heartbeat from core client for 30 sec - exiting 09:47:03 (5332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:46:10 (6472): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:06:53 (7300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:07:49 (6164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:14:58 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:11:28 (6912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x776BFF2B write attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Mar 2013 20:56:07	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	518,400	1,101,074	2.1240
27 Feb 2013 17:36:20	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	492,480	969,724	1.9691
25 Feb 2013 08:59:48	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	466,560	838,134	1.7964
22 Feb 2013 17:21:06	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	440,640	709,673	1.6106
21 Feb 2013 18:54:24	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	414,720	663,815	1.6006
20 Feb 2013 20:56:10	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	388,800	618,333	1.5904
19 Feb 2013 19:03:20	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	362,880	572,411	1.5774
18 Feb 2013 19:53:04	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	336,960	529,090	1.5702
18 Feb 2013 07:17:59	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	311,040	484,736	1.5584
17 Feb 2013 07:31:44	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	285,120	440,530	1.5451
16 Feb 2013 08:13:48	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	259,200	394,391	1.5216
15 Feb 2013 11:12:36	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	233,280	348,084	1.4921
14 Feb 2013 12:50:08	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	207,360	302,542	1.4590
13 Feb 2013 11:46:58	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	181,440	254,460	1.4024
12 Feb 2013 13:10:24	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	155,520	210,656	1.3545
12 Feb 2013 09:14:43	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	129,600	178,344	1.3761
12 Feb 2013 09:14:43	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	103,680	151,257	1.4589
12 Feb 2013 09:14:43	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	77,760	124,072	1.5956
12 Feb 2013 09:14:43	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	51,840	91,823	1.7713
09 Feb 2013 10:57:21	1233009	15600960	hadcm3n_4mnw_1940_40_008312131_1	25,920	45,012	1.7366