Task 13292066

Name	hadcm3n_p7k5_1940_40_007421794_0
Workunit	7619429
Created	25 Aug 2011, 3:30:56 UTC
Sent	25 Aug 2011, 3:35:33 UTC
Report deadline	24 Nov 2011, 11:02:44 UTC
Received	7 Sep 2011, 10:29:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1160470
Run time	6 days 19 hours 55 min 33 sec
CPU time	6 days 3 hours 41 min 37 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.86 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:41:48 (12292): No heartbeat from core client for 30 sec - exiting 22:41:50 (12292): No heartbeat from core client for 30 sec - exiting 22:41:51 (12292): No heartbeat from core client for 30 sec - exiting 22:41:52 (12292): No heartbeat from core client for 30 sec - exiting 22:41:53 (12292): No heartbeat from core client for 30 sec - exiting 22:41:54 (12292): No heartbeat from core client for 30 sec - exiting 22:41:55 (12292): No heartbeat from core client for 30 sec - exiting 22:41:56 (12292): No heartbeat from core client for 30 sec - exiting 22:41:57 (12292): No heartbeat from core client for 30 sec - exiting 22:41:58 (12292): No heartbeat from core client for 30 sec - exiting 22:41:59 (12292): No heartbeat from core client for 30 sec - exiting 22:42:00 (12292): No heartbeat from core client for 30 sec - exiting 22:42:01 (12292): No heartbeat from core client for 30 sec - exiting 22:42:02 (12292): No heartbeat from core client for 30 sec - exiting 22:42:03 (12292): No heartbeat from core client for 30 sec - exiting 22:42:04 (12292): No heartbeat from core client for 30 sec - exiting 22:42:05 (12292): No heartbeat from core client for 30 sec - exiting 22:42:06 (12292): No heartbeat from core client for 30 sec - exiting 22:42:07 (12292): No heartbeat from core client for 30 sec - exiting 22:42:08 (12292): No heartbeat from core client for 30 sec - exiting 22:42:09 (12292): No heartbeat from core client for 30 sec - exiting 22:42:10 (12292): No heartbeat from core client for 30 sec - exiting 22:42:11 (12292): No heartbeat from core client for 30 sec - exiting 22:42:12 (12292): No heartbeat from core client for 30 sec - exiting 22:42:13 (12292): No heartbeat from core client for 30 sec - exiting 22:42:14 (12292): No heartbeat from core client for 30 sec - exiting 22:42:15 (12292): No heartbeat from core client for 30 sec - exiting 22:42:16 (12292): No heartbeat from core client for 30 sec - exiting 22:42:17 (12292): No heartbeat from core client for 30 sec - exiting 22:42:18 (12292): No heartbeat from core client for 30 sec - exiting 22:42:19 (12292): No heartbeat from core client for 30 sec - exiting 22:42:20 (12292): No heartbeat from core client for 30 sec - exiting 22:42:21 (12292): No heartbeat from core client for 30 sec - exiting 22:42:22 (12292): No heartbeat from core client for 30 sec - exiting 22:42:23 (12292): No heartbeat from core client for 30 sec - exiting 22:42:24 (12292): No heartbeat from core client for 30 sec - exiting 22:42:25 (12292): No heartbeat from core client for 30 sec - exiting 22:42:26 (12292): No heartbeat from core client for 30 sec - exiting 22:42:27 (12292): No heartbeat from core client for 30 sec - exiting 22:42:28 (12292): No heartbeat from core client for 30 sec - exiting 22:42:29 (12292): No heartbeat from core client for 30 sec - exiting 22:42:30 (12292): No heartbeat from core client for 30 sec - exiting 22:42:31 (12292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 09:54:31 (1088): No heartbeat from core client for 30 sec - exiting 09:54:32 (1088): No heartbeat from core client for 30 sec - exiting 09:54:33 (1088): No heartbeat from core client for 30 sec - exiting 09:54:34 (1088): No heartbeat from core client for 30 sec - exiting 09:54:35 (1088): No heartbeat from core client for 30 sec - exiting 09:54:36 (1088): No heartbeat from core client for 30 sec - exiting 09:54:37 (1088): No heartbeat from core client for 30 sec - exiting 09:54:38 (1088): No heartbeat from core client for 30 sec - exiting 09:54:39 (1088): No heartbeat from core client for 30 sec - exiting 09:54:40 (1088): No heartbeat from core client for 30 sec - exiting 09:54:41 (1088): No heartbeat from core client for 30 sec - exiting 09:54:42 (1088): No heartbeat from core client for 30 sec - exiting 09:54:43 (1088): No heartbeat from core client for 30 sec - exiting 09:54:44 (1088): No heartbeat from core client for 30 sec - exiting 09:54:45 (1088): No heartbeat from core client for 30 sec - exiting 09:54:46 (1088): No heartbeat from core client for 30 sec - exiting 09:54:47 (1088): No heartbeat from core client for 30 sec - exiting 09:54:48 (1088): No heartbeat from core client for 30 sec - exiting 09:54:49 (1088): No heartbeat from core client for 30 sec - exiting 09:54:50 (1088): No heartbeat from core client for 30 sec - exiting 09:54:51 (1088): No heartbeat from core client for 30 sec - exiting 09:54:52 (1088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:15:22 (13936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:03:24 (13716): No heartbeat from core client for 30 sec - exiting 12:03:25 (13716): No heartbeat from core client for 30 sec - exiting 12:03:26 (13716): No heartbeat from core client for 30 sec - exiting 12:03:27 (13716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:26:04 (7520): No heartbeat from core client for 30 sec - exiting 16:26:05 (7520): No heartbeat from core client for 30 sec - exiting 16:26:06 (7520): No heartbeat from core client for 30 sec - exiting 16:26:07 (7520): No heartbeat from core client for 30 sec - exiting 16:26:08 (7520): No heartbeat from core client for 30 sec - exiting 16:26:09 (7520): No heartbeat from core client for 30 sec - exiting 16:26:10 (7520): No heartbeat from core client for 30 sec - exiting 16:26:11 (7520): No heartbeat from core client for 30 sec - exiting 16:26:12 (7520): No heartbeat from core client for 30 sec - exiting 16:26:13 (7520): No heartbeat from core client for 30 sec - exiting 16:26:14 (7520): No heartbeat from core client for 30 sec - exiting 16:26:15 (7520): No heartbeat from core client for 30 sec - exiting 16:26:16 (7520): No heartbeat from core client for 30 sec - exiting 16:26:17 (7520): No heartbeat from core client for 30 sec - exiting 16:26:18 (7520): No heartbeat from core client for 30 sec - exiting 16:26:19 (7520): No heartbeat from core client for 30 sec - exiting 16:26:20 (7520): No heartbeat from core client for 30 sec - exiting 16:26:21 (7520): No heartbeat from core client for 30 sec - exiting 16:26:22 (7520): No heartbeat from core client for 30 sec - exiting 16:26:23 (7520): No heartbeat from core client for 30 sec - exiting 16:26:24 (7520): No heartbeat from core client for 30 sec - exiting 16:26:25 (7520): No heartbeat from core client for 30 sec - exiting 16:26:26 (7520): No heartbeat from core client for 30 sec - exiting 16:26:27 (7520): No heartbeat from core client for 30 sec - exiting 16:26:28 (7520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:50:35 (7868): No heartbeat from core client for 30 sec - exiting 23:50:36 (7868): No heartbeat from core client for 30 sec - exiting 23:50:37 (7868): No heartbeat from core client for 30 sec - exiting 23:50:38 (7868): No heartbeat from core client for 30 sec - exiting 23:50:39 (7868): No heartbeat from core client for 30 sec - exiting 23:50:40 (7868): No heartbeat from core client for 30 sec - exiting 23:50:41 (7868): No heartbeat from core client for 30 sec - exiting 23:50:42 (7868): No heartbeat from core client for 30 sec - exiting 23:50:43 (7868): No heartbeat from core client for 30 sec - exiting 23:50:44 (7868): No heartbeat from core client for 30 sec - exiting 23:50:45 (7868): No heartbeat from core client for 30 sec - exiting 23:50:46 (7868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/p7k5ko.pje6c10 Error converting file to netcdf: dataout/p7k5ko.pie6c10 Error converting file to netcdf: dataout/p7k5ko.pfe6c10 Error converting file to netcdf: dataout/p7k5ka.phe6c10 Error converting file to netcdf: dataout/p7k5ka.pge6c10 Error converting file to netcdf: dataout/p7k5ka.pee6c10 Error converting file to netcdf: dataout/p7k5ka.pde6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:00:19 (4788): No heartbeat from core client for 30 sec - exiting 10:00:20 (4788): No heartbeat from core client for 30 sec - exiting 10:00:21 (4788): No heartbeat from core client for 30 sec - exiting 10:00:22 (4788): No heartbeat from core client for 30 sec - exiting 10:00:23 (4788): No heartbeat from core client for 30 sec - exiting 10:00:24 (4788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 10:37:57 (6500): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:14:32 (10560): No heartbeat from core client for 30 sec - exiting 17:14:33 (10560): No heartbeat from core client for 30 sec - exiting 17:14:34 (10560): No heartbeat from core client for 30 sec - exiting 17:14:35 (10560): No heartbeat from core client for 30 sec - exiting 17:14:36 (10560): No heartbeat from core client for 30 sec - exiting 17:14:37 (10560): No heartbeat from core client for 30 sec - exiting 17:14:38 (10560): No heartbeat from core client for 30 sec - exiting 17:14:39 (10560): No heartbeat from core client for 30 sec - exiting 17:14:40 (10560): No heartbeat from core client for 30 sec - exiting 17:14:41 (10560): No heartbeat from core client for 30 sec - exiting 17:14:42 (10560): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:47 (8272): No heartbeat from core client for 30 sec - exiting 21:33:48 (8272): No heartbeat from core client for 30 sec - exiting 21:33:49 (8272): No heartbeat from core client for 30 sec - exiting 21:33:50 (8272): No heartbeat from core client for 30 sec - exiting 21:33:51 (8272): No heartbeat from core client for 30 sec - exiting 21:33:52 (8272): No heartbeat from core client for 30 sec - exiting 21:33:53 (8272): No heartbeat from core client for 30 sec - exiting 21:33:54 (8272): No heartbeat from core client for 30 sec - exiting 21:33:55 (8272): No heartbeat from core client for 30 sec - exiting 21:33:56 (8272): No heartbeat from core client for 30 sec - exiting 21:33:57 (8272): No heartbeat from core client for 30 sec - exiting 21:33:58 (8272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:33:59 (8272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 10:19:49 (6800): No heartbeat from core client for 30 sec - exiting 10:19:50 (6800): No heartbeat from core client for 30 sec - exiting 10:19:51 (6800): No heartbeat from core client for 30 sec - exiting 10:19:52 (6800): No heartbeat from core client for 30 sec - exiting 10:19:53 (6800): No heartbeat from core client for 30 sec - exiting 10:19:54 (6800): No heartbeat from core client for 30 sec - exiting 10:19:55 (6800): No heartbeat from core client for 30 sec - exiting 10:19:56 (6800): No heartbeat from core client for 30 sec - exiting 10:19:57 (6800): No heartbeat from core client for 30 sec - exiting 10:19:58 (6800): No heartbeat from core client for 30 sec - exiting 10:19:59 (6800): No heartbeat from core client for 30 sec - exiting 10:20:00 (6800): No heartbeat from core client for 30 sec - exiting 10:20:01 (6800): No heartbeat from core client for 30 sec - exiting 10:20:02 (6800): No heartbeat from core client for 30 sec - exiting 10:20:03 (6800): No heartbeat from core client for 30 sec - exiting 10:20:04 (6800): No heartbeat from core client for 30 sec - exiting 10:20:05 (6800): No heartbeat from core client for 30 sec - exiting 10:20:06 (6800): No heartbeat from core client for 30 sec - exiting 10:20:07 (6800): No heartbeat from core client for 30 sec - exiting 10:20:08 (6800): No heartbeat from core client for 30 sec - exiting 10:20:09 (6800): No heartbeat from core client for 30 sec - exiting 10:20:10 (6800): No heartbeat from core client for 30 sec - exiting 10:20:11 (6800): No heartbeat from core client for 30 sec - exiting 10:20:12 (6800): No heartbeat from core client for 30 sec - exiting 10:20:13 (6800): No heartbeat from core client for 30 sec - exiting 10:20:14 (6800): No heartbeat from core client for 30 sec - exiting 10:20:15 (6800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:20:16 (6800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7484, iMonCtr=1 Model crash detected, will try to restart... 09:40:55 (6272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778E32FF read attempt to address 0x00000004 Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8448, selfPID=8448, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x778E3A93 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Sep 2011 09:01:08	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	259,200	527,497	2.0351
07 Sep 2011 01:41:22	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	233,280	487,576	2.0901
06 Sep 2011 05:45:06	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	207,360	449,680	2.1686
04 Sep 2011 18:45:07	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	181,440	384,194	2.1175
01 Sep 2011 22:59:13	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	155,520	329,438	2.1183
01 Sep 2011 05:56:31	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	129,600	274,887	2.1210
31 Aug 2011 03:18:34	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	103,680	220,550	2.1272
28 Aug 2011 20:58:18	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	77,760	166,328	2.1390
28 Aug 2011 01:47:08	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	51,840	110,518	2.1319
26 Aug 2011 04:21:51	1160470	13292066	hadcm3n_p7k5_1940_40_007421794_0	25,920	54,795	2.1140