Task 12822573

Name	hadcm3n_p2vx_1900_40_007221325_0
Workunit	7419565
Created	26 Apr 2011, 15:22:37 UTC
Sent	1 May 2011, 21:38:04 UTC
Report deadline	1 Aug 2011, 5:05:15 UTC
Received	3 Jul 2011, 0:40:35 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1378261
Run time	26 days 22 hours 6 min 24 sec
CPU time	24 days 21 hours 9 min 8 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4312, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4236, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:22:55 (2588): No heartbeat from core client for 30 sec - exiting 11:22:56 (2588): No heartbeat from core client for 30 sec - exiting 11:22:57 (2588): No heartbeat from core client for 30 sec - exiting 11:22:58 (2588): No heartbeat from core client for 30 sec - exiting 11:22:59 (2588): No heartbeat from core client for 30 sec - exiting 11:23:00 (2588): No heartbeat from core client for 30 sec - exiting 11:23:01 (2588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3708, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5004, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4692, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... 21:47:41 (3172): No heartbeat from core client for 30 sec - exiting 21:47:42 (3172): No heartbeat from core client for 30 sec - exiting 21:47:43 (3172): No heartbeat from core client for 30 sec - exiting 21:47:44 (3172): No heartbeat from core client for 30 sec - exiting 21:47:45 (3172): No heartbeat from core client for 30 sec - exiting 21:47:46 (3172): No heartbeat from core client for 30 sec - exiting 21:47:47 (3172): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:19:35 (2312): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:46:53 (3108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:47:41 (5628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4908, iMonCtr=1 Model crash detected, will try to restart... 05:55:13 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:57:17 (2904): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:06:31 (2696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:45 (3096): No heartbeat from core client for 30 sec - exiting 13:29:46 (3096): No heartbeat from core client for 30 sec - exiting 13:29:47 (3096): No heartbeat from core client for 30 sec - exiting 13:29:48 (3096): No heartbeat from core client for 30 sec - exiting 13:29:50 (3096): No heartbeat from core client for 30 sec - exiting 13:29:51 (3096): No heartbeat from core client for 30 sec - exiting 13:29:52 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:29:53 (3096): No heartbeat from core client for 30 sec - exiting 13:29:54 (3096): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4752, iMonCtr=1 Model crash detected, will try to restart... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2532, selfPID=2532, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/p2vxko.pjd5c10 Error converting file to netcdf: dataout/p2vxko.pid5c10 Error converting file to netcdf: dataout/p2vxko.pfd5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771FB84B write attempt to address 0xC235CE2E Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x771E71D3 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Jul 2011 19:37:14	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	1,036,800	2,148,992	2.0727
29 Jun 2011 01:35:24	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	1,010,880	2,096,749	2.0742
27 Jun 2011 05:48:00	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	984,960	2,044,293	2.0755
25 Jun 2011 05:21:10	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	959,040	1,991,752	2.0768
23 Jun 2011 01:34:31	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	933,120	1,939,893	2.0789
19 Jun 2011 22:17:36	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	907,200	1,887,456	2.0805
15 Jun 2011 04:08:06	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	881,280	1,835,037	2.0822
13 Jun 2011 22:12:49	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	855,360	1,782,722	2.0842
12 Jun 2011 01:00:01	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	829,440	1,730,932	2.0869
11 Jun 2011 06:29:40	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	803,520	1,677,878	2.0882
09 Jun 2011 17:35:57	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	777,600	1,624,774	2.0895
08 Jun 2011 00:59:14	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	751,680	1,570,226	2.0890
04 Jun 2011 23:26:46	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	725,760	1,515,477	2.0881
02 Jun 2011 02:29:23	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	699,840	1,460,920	2.0875
29 May 2011 22:30:37	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	673,920	1,406,766	2.0874
28 May 2011 20:43:36	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	648,000	1,352,703	2.0875
26 May 2011 07:10:45	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	622,080	1,298,433	2.0872
25 May 2011 15:01:31	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	596,160	1,244,411	2.0874
24 May 2011 18:14:58	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	570,240	1,191,176	2.0889
22 May 2011 16:58:45	1071755	12822573	hadcm3n_p2vx_1900_40_007221325_0	544,320	1,137,071	2.0890