Task 13375401

Name	hadcm3n_o1gx_1940_40_007452107_2
Workunit	7649610
Created	11 Sep 2011, 15:50:50 UTC
Sent	11 Sep 2011, 16:07:41 UTC
Report deadline	11 Dec 2011, 23:34:52 UTC
Received	13 Oct 2011, 1:06:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1159511
Run time	9 days 11 hours 13 min 37 sec
CPU time	9 days 6 hours 35 min 55 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4484, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 09:52:40 (8220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:52:41 (8220): No heartbeat from core client for 30 sec - exiting 09:52:42 (8220): No heartbeat from core client for 30 sec - exiting 09:52:43 (8220): No heartbeat from core client for 30 sec - exiting 09:52:44 (8220): No heartbeat from core client for 30 sec - exiting 09:52:45 (8220): No heartbeat from core client for 30 sec - exiting 09:52:47 (8220): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 11:53:02 (4492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:53:03 (4492): No heartbeat from core client for 30 sec - exiting 11:53:04 (4492): No heartbeat from core client for 30 sec - exiting 11:53:05 (4492): No heartbeat from core client for 30 sec - exiting 11:53:06 (4492): No heartbeat from core client for 30 sec - exiting 11:53:07 (4492): No heartbeat from core client for 30 sec - exiting 11:53:08 (4492): No heartbeat from core client for 30 sec - exiting 11:53:09 (4492): No heartbeat from core client for 30 sec - exiting 11:53:10 (4492): No heartbeat from core client for 30 sec - exiting 11:53:11 (4492): No heartbeat from core client for 30 sec - exiting 11:53:12 (4492): No heartbeat from core client for 30 sec - exiting 11:53:13 (4492): No heartbeat from core client for 30 sec - exiting 11:53:14 (4492): No heartbeat from core client for 30 sec - exiting 11:53:15 (4492): No heartbeat from core client for 30 sec - exiting 11:53:16 (4492): No heartbeat from core client for 30 sec - exiting 11:53:17 (4492): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3392, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:44:43 (3628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:44:44 (3628): No heartbeat from core client for 30 sec - exiting 12:44:45 (3628): No heartbeat from core client for 30 sec - exiting 12:44:46 (3628): No heartbeat from core client for 30 sec - exiting 12:44:47 (3628): No heartbeat from core client for 30 sec - exiting 12:44:48 (3628): No heartbeat from core client for 30 sec - exiting 12:44:49 (3628): No heartbeat from core client for 30 sec - exiting 12:44:50 (3628): No heartbeat from core client for 30 sec - exiting 12:44:51 (3628): No heartbeat from core client for 30 sec - exiting 12:44:52 (3628): No heartbeat from core client for 30 sec - exiting 12:44:53 (3628): No heartbeat from core client for 30 sec - exiting 12:44:54 (3628): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:04:44 (7884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:04:46 (7884): No heartbeat from core client for 30 sec - exiting 13:04:47 (7884): No heartbeat from core client for 30 sec - exiting 13:04:48 (7884): No heartbeat from core client for 30 sec - exiting 13:04:49 (7884): No heartbeat from core client for 30 sec - exiting 13:04:50 (7884): No heartbeat from core client for 30 sec - exiting 13:04:51 (7884): No heartbeat from core client for 30 sec - exiting 13:04:52 (7884): No heartbeat from core client for 30 sec - exiting 13:04:53 (7884): No heartbeat from core client for 30 sec - exiting 13:04:54 (7884): No heartbeat from core client for 30 sec - exiting 13:04:55 (7884): No heartbeat from core client for 30 sec - exiting 13:04:56 (7884): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:04:57 (5680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:05:02 (5680): No heartbeat from core client for 30 sec - exiting 16:05:03 (5680): No heartbeat from core client for 30 sec - exiting 16:05:04 (5680): No heartbeat from core client for 30 sec - exiting 16:05:05 (5680): No heartbeat from core client for 30 sec - exiting 16:05:06 (5680): No heartbeat from core client for 30 sec - exiting 16:05:07 (5680): No heartbeat from core client for 30 sec - exiting 16:05:08 (5680): No heartbeat from core client for 30 sec - exiting 16:05:09 (5680): No heartbeat from core client for 30 sec - exiting 16:05:10 (5680): No heartbeat from core client for 30 sec - exiting 16:05:11 (5680): No heartbeat from core client for 30 sec - exiting 18:20:03 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:20:05 (3156): No heartbeat from core client for 30 sec - exiting 18:20:06 (3156): No heartbeat from core client for 30 sec - exiting 18:20:07 (3156): No heartbeat from core client for 30 sec - exiting 18:20:08 (3156): No heartbeat from core client for 30 sec - exiting 18:20:09 (3156): No heartbeat from core client for 30 sec - exiting 18:20:10 (3156): No heartbeat from core client for 30 sec - exiting 18:20:11 (3156): No heartbeat from core client for 30 sec - exiting 18:20:12 (3156): No heartbeat from core client for 30 sec - exiting 18:20:13 (3156): No heartbeat from core client for 30 sec - exiting 18:20:14 (3156): No heartbeat from core client for 30 sec - exiting 18:20:15 (3156): No heartbeat from core client for 30 sec - exiting 18:20:16 (3156): No heartbeat from core client for 30 sec - exiting 18:20:17 (3156): No heartbeat from core client for 30 sec - exiting 18:20:18 (3156): No heartbeat from core client for 30 sec - exiting 18:20:19 (3156): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 22:10:40 (8872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:10:41 (8872): No heartbeat from core client for 30 sec - exiting 22:10:42 (8872): No heartbeat from core client for 30 sec - exiting 22:10:43 (8872): No heartbeat from core client for 30 sec - exiting 22:10:44 (8872): No heartbeat from core client for 30 sec - exiting 22:10:45 (8872): No heartbeat from core client for 30 sec - exiting 22:10:46 (8872): No heartbeat from core client for 30 sec - exiting 22:10:47 (8872): No heartbeat from core client for 30 sec - exiting 22:10:48 (8872): No heartbeat from core client for 30 sec - exiting 22:10:49 (8872): No heartbeat from core client for 30 sec - exiting 22:10:50 (8872): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7696, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:53:50 (10624): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:53:51 (10624): No heartbeat from core client for 30 sec - exiting 16:53:52 (10624): No heartbeat from core client for 30 sec - exiting 16:53:53 (10624): No heartbeat from core client for 30 sec - exiting 16:53:54 (10624): No heartbeat from core client for 30 sec - exiting 16:53:55 (10624): No heartbeat from core client for 30 sec - exiting 16:53:56 (10624): No heartbeat from core client for 30 sec - exiting 16:53:57 (10624): No heartbeat from core client for 30 sec - exiting 16:53:58 (10624): No heartbeat from core client for 30 sec - exiting 16:53:59 (10624): No heartbeat from core client for 30 sec - exiting 16:54:00 (10624): No heartbeat from core client for 30 sec - exiting 17:58:55 (12028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:58:57 (12028): No heartbeat from core client for 30 sec - exiting 17:58:58 (12028): No heartbeat from core client for 30 sec - exiting 17:58:59 (12028): No heartbeat from core client for 30 sec - exiting 17:59:00 (12028): No heartbeat from core client for 30 sec - exiting 17:59:01 (12028): No heartbeat from core client for 30 sec - exiting 17:59:02 (12028): No heartbeat from core client for 30 sec - exiting 17:59:03 (12028): No heartbeat from core client for 30 sec - exiting 17:59:04 (12028): No heartbeat from core client for 30 sec - exiting 17:59:05 (12028): No heartbeat from core client for 30 sec - exiting 17:59:06 (12028): No heartbeat from core client for 30 sec - exiting 18:34:13 (11728): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:19:07 (12000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:19:08 (12000): No heartbeat from core client for 30 sec - exiting 22:19:09 (12000): No heartbeat from core client for 30 sec - exiting 22:19:10 (12000): No heartbeat from core client for 30 sec - exiting 22:19:11 (12000): No heartbeat from core client for 30 sec - exiting 22:19:12 (12000): No heartbeat from core client for 30 sec - exiting 22:19:13 (12000): No heartbeat from core client for 30 sec - exiting 22:19:14 (12000): No heartbeat from core client for 30 sec - exiting 22:19:15 (12000): No heartbeat from core client for 30 sec - exiting 22:19:16 (12000): No heartbeat from core client for 30 sec - exiting 22:19:17 (12000): No heartbeat from core client for 30 sec - exiting 22:19:18 (12000): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x777E3A93 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13092, selfPID=13092, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77447353 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778F3A93 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Oct 2011 00:58:14	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	518,400	788,526	1.5211
11 Oct 2011 10:39:41	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	492,480	756,646	1.5364
10 Oct 2011 19:12:15	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	466,560	717,917	1.5387
10 Oct 2011 02:58:15	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	440,640	673,427	1.5283
07 Oct 2011 14:07:05	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	414,720	631,417	1.5225
06 Oct 2011 23:35:22	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	388,800	592,917	1.5250
06 Oct 2011 09:08:38	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	362,880	552,261	1.5219
04 Oct 2011 15:07:44	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	336,960	518,999	1.5402
04 Oct 2011 04:47:58	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	311,040	485,095	1.5596
03 Oct 2011 16:07:12	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	285,120	440,372	1.5445
30 Sep 2011 12:04:06	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	259,200	404,861	1.5620
29 Sep 2011 19:18:06	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	233,280	374,562	1.6056
26 Sep 2011 19:42:23	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	207,360	340,167	1.6405
26 Sep 2011 02:50:52	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	181,440	305,877	1.6858
25 Sep 2011 13:58:50	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	155,520	267,332	1.7190
25 Sep 2011 02:56:47	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	129,600	226,940	1.7511
24 Sep 2011 13:35:03	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	103,680	182,528	1.7605
24 Sep 2011 00:30:59	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	77,760	134,945	1.7354
19 Sep 2011 17:46:47	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	51,840	86,709	1.6726
14 Sep 2011 21:25:45	1159511	13375401	hadcm3n_o1gx_1940_40_007452107_2	25,920	40,999	1.5818