Task 15789747

Name	hadcm3n_4joo_1940_40_008303650_3
Workunit	8454785
Created	19 May 2013, 3:23:33 UTC
Sent	19 May 2013, 3:23:36 UTC
Report deadline	18 Aug 2013, 10:50:47 UTC
Received	28 Oct 2013, 19:08:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1217831
Run time	6 days 1 hours 20 min 28 sec
CPU time	4 days 11 hours 51 min 45 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.41 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:40:52 (161848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:46:53 (129608): No heartbeat from core client for 30 sec - exiting 16:46:54 (129608): No heartbeat from core client for 30 sec - exiting 16:46:55 (129608): No heartbeat from core client for 30 sec - exiting 16:46:56 (129608): No heartbeat from core client for 30 sec - exiting 16:46:57 (129608): No heartbeat from core client for 30 sec - exiting 16:46:58 (129608): No heartbeat from core client for 30 sec - exiting 16:46:59 (129608): No heartbeat from core client for 30 sec - exiting 16:47:00 (129608): No heartbeat from core client for 30 sec - exiting 16:47:01 (129608): No heartbeat from core client for 30 sec - exiting 16:47:02 (129608): No heartbeat from core client for 30 sec - exiting 16:47:03 (129608): No heartbeat from core client for 30 sec - exiting 16:47:04 (129608): No heartbeat from core client for 30 sec - exiting 16:47:05 (129608): No heartbeat from core client for 30 sec - exiting 16:47:06 (129608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:35:39 (9724): No heartbeat from core client for 30 sec - exiting 01:35:40 (9724): No heartbeat from core client for 30 sec - exiting 01:35:41 (9724): No heartbeat from core client for 30 sec - exiting 01:35:42 (9724): No heartbeat from core client for 30 sec - exiting 01:35:43 (9724): No heartbeat from core client for 30 sec - exiting 01:35:44 (9724): No heartbeat from core client for 30 sec - exiting 01:35:45 (9724): No heartbeat from core client for 30 sec - exiting 01:35:46 (9724): No heartbeat from core client for 30 sec - exiting 01:35:47 (9724): No heartbeat from core client for 30 sec - exiting 01:35:48 (9724): No heartbeat from core client for 30 sec - exiting 01:35:49 (9724): No heartbeat from core client for 30 sec - exiting 01:35:50 (9724): No heartbeat from core client for 30 sec - exiting 01:35:51 (9724): No heartbeat from core client for 30 sec - exiting 01:35:52 (9724): No heartbeat from core client for 30 sec - exiting 01:35:53 (9724): No heartbeat from core client for 30 sec - exiting 01:35:54 (9724): No heartbeat from core client for 30 sec - exiting 01:35:55 (9724): No heartbeat from core client for 30 sec - exiting 01:35:56 (9724): No heartbeat from core client for 30 sec - exiting 01:35:57 (9724): No heartbeat from core client for 30 sec - exiting 01:35:58 (9724): No heartbeat from core client for 30 sec - exiting 01:35:59 (9724): No heartbeat from core client for 30 sec - exiting 01:36:00 (9724): No heartbeat from core client for 30 sec - exiting 01:36:01 (9724): No heartbeat from core client for 30 sec - exiting 01:36:02 (9724): No heartbeat from core client for 30 sec - exiting 01:36:03 (9724): No heartbeat from core client for 30 sec - exiting 01:36:04 (9724): No heartbeat from core client for 30 sec - exiting 01:36:05 (9724): No heartbeat from core client for 30 sec - exiting 01:36:06 (9724): No heartbeat from core client for 30 sec - exiting 01:36:07 (9724): No heartbeat from core client for 30 sec - exiting 01:36:08 (9724): No heartbeat from core client for 30 sec - exiting 01:36:09 (9724): No heartbeat from core client for 30 sec - exiting 01:36:10 (9724): No heartbeat from core client for 30 sec - exiting 01:36:11 (9724): No heartbeat from core client for 30 sec - exiting 01:36:12 (9724): No heartbeat from core client for 30 sec - exiting 01:36:13 (9724): No heartbeat from core client for 30 sec - exiting 01:36:14 (9724): No heartbeat from core client for 30 sec - exiting 01:36:15 (9724): No heartbeat from core client for 30 sec - exiting 01:36:16 (9724): No heartbeat from core client for 30 sec - exiting 01:36:17 (9724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:37:28 (12852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=12008, iMonCtr=1 Model crash detected, will try to restart... 03:44:00 (6928): No heartbeat from core client for 30 sec - exiting 03:44:01 (6928): No heartbeat from core client for 30 sec - exiting 03:44:02 (6928): No heartbeat from core client for 30 sec - exiting 03:44:03 (6928): No heartbeat from core client for 30 sec - exiting 03:44:04 (6928): No heartbeat from core client for 30 sec - exiting 03:44:05 (6928): No heartbeat from core client for 30 sec - exiting 03:44:06 (6928): No heartbeat from core client for 30 sec - exiting 03:44:07 (6928): No heartbeat from core client for 30 sec - exiting 03:44:08 (6928): No heartbeat from core client for 30 sec - exiting 03:44:09 (6928): No heartbeat from core client for 30 sec - exiting 03:44:10 (6928): No heartbeat from core client for 30 sec - exiting 03:44:11 (6928): No heartbeat from core client for 30 sec - exiting 03:44:12 (6928): No heartbeat from core client for 30 sec - exiting 03:44:13 (6928): No heartbeat from core client for 30 sec - exiting 03:44:14 (6928): No heartbeat from core client for 30 sec - exiting 03:44:15 (6928): No heartbeat from core client for 30 sec - exiting 03:44:16 (6928): No heartbeat from core client for 30 sec - exiting 03:44:17 (6928): No heartbeat from core client for 30 sec - exiting 03:44:18 (6928): No heartbeat from core client for 30 sec - exiting 03:44:19 (6928): No heartbeat from core client for 30 sec - exiting 03:44:20 (6928): No heartbeat from core client for 30 sec - exiting 03:44:21 (6928): No heartbeat from core client for 30 sec - exiting 03:44:22 (6928): No heartbeat from core client for 30 sec - exiting 03:44:23 (6928): No heartbeat from core client for 30 sec - exiting 03:44:24 (6928): No heartbeat from core client for 30 sec - exiting 03:44:25 (6928): No heartbeat from core client for 30 sec - exiting 03:44:26 (6928): No heartbeat from core client for 30 sec - exiting 03:44:27 (6928): No heartbeat from core client for 30 sec - exiting 03:44:28 (6928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:45:53 (13432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:48:37 (53100): No heartbeat from core client for 30 sec - exiting 16:48:38 (53100): No heartbeat from core client for 30 sec - exiting 16:48:39 (53100): No heartbeat from core client for 30 sec - exiting 16:48:40 (53100): No heartbeat from core client for 30 sec - exiting 16:48:41 (53100): No heartbeat from core client for 30 sec - exiting 16:48:42 (53100): No heartbeat from core client for 30 sec - exiting 16:48:43 (53100): No heartbeat from core client for 30 sec - exiting 16:48:44 (53100): No heartbeat from core client for 30 sec - exiting 16:48:45 (53100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=56512, iMonCtr=1 Model crash detected, will try to restart... 00:59:55 (6944): No heartbeat from core client for 30 sec - exiting 00:59:56 (6944): No heartbeat from core client for 30 sec - exiting 00:59:57 (6944): No heartbeat from core client for 30 sec - exiting 00:59:58 (6944): No heartbeat from core client for 30 sec - exiting 00:59:59 (6944): No heartbeat from core client for 30 sec - exiting 01:00:00 (6944): No heartbeat from core client for 30 sec - exiting 01:00:01 (6944): No heartbeat from core client for 30 sec - exiting 01:00:02 (6944): No heartbeat from core client for 30 sec - exiting 01:00:03 (6944): No heartbeat from core client for 30 sec - exiting 01:00:04 (6944): No heartbeat from core client for 30 sec - exiting 01:00:05 (6944): No heartbeat from core client for 30 sec - exiting 01:00:06 (6944): No heartbeat from core client for 30 sec - exiting 01:00:07 (6944): No heartbeat from core client for 30 sec - exiting 01:00:08 (6944): No heartbeat from core client for 30 sec - exiting 01:00:09 (6944): No heartbeat from core client for 30 sec - exiting 01:00:10 (6944): No heartbeat from core client for 30 sec - exiting 01:00:11 (6944): No heartbeat from core client for 30 sec - exiting 01:00:12 (6944): No heartbeat from core client for 30 sec - exiting 01:00:13 (6944): No heartbeat from core client for 30 sec - exiting 01:00:14 (6944): No heartbeat from core client for 30 sec - exiting 01:00:15 (6944): No heartbeat from core client for 30 sec - exiting 01:00:16 (6944): No heartbeat from core client for 30 sec - exiting 01:00:17 (6944): No heartbeat from core client for 30 sec - exiting 01:00:18 (6944): No heartbeat from core client for 30 sec - exiting 01:00:19 (6944): No heartbeat from core client for 30 sec - exiting 01:00:20 (6944): No heartbeat from core client for 30 sec - exiting 01:00:21 (6944): No heartbeat from core client for 30 sec - exiting 01:00:22 (6944): No heartbeat from core client for 30 sec - exiting 01:00:23 (6944): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 02:11:09 (7408): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/4jooko.pje5c10 Error converting file to netcdf: dataout/4jooko.pie5c10 Error converting file to netcdf: dataout/4jooko.pfe5c10 Error converting file to netcdf: dataout/4jooka.phe5c10 Error converting file to netcdf: dataout/4jooka.pge5c10 Error converting file to netcdf: dataout/4jooka.pee5c10 Error converting file to netcdf: dataout/4jooka.pde5c10 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8320, iMonCtr=1 Model crash detected, will try to restart... 02:17:02 (7996): No heartbeat from core client for 30 sec - exiting 02:17:03 (7996): No heartbeat from core client for 30 sec - exiting 02:17:04 (7996): No heartbeat from core client for 30 sec - exiting 02:17:05 (7996): No heartbeat from core client for 30 sec - exiting 02:17:06 (7996): No heartbeat from core client for 30 sec - exiting 02:17:07 (7996): No heartbeat from core client for 30 sec - exiting 02:17:08 (7996): No heartbeat from core client for 30 sec - exiting 02:17:09 (7996): No heartbeat from core client for 30 sec - exiting 02:17:10 (7996): No heartbeat from core client for 30 sec - exiting 02:17:11 (7996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:17:12 (7996): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:20:47 (7328): No heartbeat from core client for 30 sec - exiting 22:20:48 (7328): No heartbeat from core client for 30 sec - exiting 22:20:49 (7328): No heartbeat from core client for 30 sec - exiting 22:20:50 (7328): No heartbeat from core client for 30 sec - exiting 22:20:51 (7328): No heartbeat from core client for 30 sec - exiting 22:20:52 (7328): No heartbeat from core client for 30 sec - exiting 22:20:53 (7328): No heartbeat from core client for 30 sec - exiting 22:20:54 (7328): No heartbeat from core client for 30 sec - exiting 22:20:55 (7328): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:39:21 (9144): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:53:49 (4796): No heartbeat from core client for 30 sec - exiting 03:53:50 (4796): No heartbeat from core client for 30 sec - exiting 03:53:51 (4796): No heartbeat from core client for 30 sec - exiting 03:53:52 (4796): No heartbeat from core client for 30 sec - exiting 03:53:53 (4796): No heartbeat from core client for 30 sec - exiting 03:53:54 (4796): No heartbeat from core client for 30 sec - exiting 03:53:55 (4796): No heartbeat from core client for 30 sec - exiting 03:53:56 (4796): No heartbeat from core client for 30 sec - exiting 03:53:57 (4796): No heartbeat from core client for 30 sec - exiting 03:53:58 (4796): No heartbeat from core client for 30 sec - exiting 03:53:59 (4796): No heartbeat from core client for 30 sec - exiting 03:54:00 (4796): No heartbeat from core client for 30 sec - exiting 03:54:01 (4796): No heartbeat from core client for 30 sec - exiting 03:54:02 (4796): No heartbeat from core client for 30 sec - exiting 03:54:03 (4796): No heartbeat from core client for 30 sec - exiting 03:54:04 (4796): No heartbeat from core client for 30 sec - exiting 03:54:05 (4796): No heartbeat from core client for 30 sec - exiting 03:54:06 (4796): No heartbeat from core client for 30 sec - exiting 03:54:07 (4796): No heartbeat from core client for 30 sec - exiting 03:54:08 (4796): No heartbeat from core client for 30 sec - exiting 03:54:09 (4796): No heartbeat from core client for 30 sec - exiting 03:54:10 (4796): No heartbeat from core client for 30 sec - exiting 03:54:11 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 05:09:50 (9244): No heartbeat from core client for 30 sec - exiting 05:09:51 (9244): No heartbeat from core client for 30 sec - exiting 05:09:52 (9244): No heartbeat from core client for 30 sec - exiting 05:09:53 (9244): No heartbeat from core client for 30 sec - exiting 05:09:54 (9244): No heartbeat from core client for 30 sec - exiting 05:09:55 (9244): No heartbeat from core client for 30 sec - exiting 05:09:56 (9244): No heartbeat from core client for 30 sec - exiting 05:09:57 (9244): No heartbeat from core client for 30 sec - exiting 05:09:58 (9244): No heartbeat from core client for 30 sec - exiting 05:09:59 (9244): No heartbeat from core client for 30 sec - exiting 05:10:00 (9244): No heartbeat from core client for 30 sec - exiting 05:10:01 (9244): No heartbeat from core client for 30 sec - exiting 05:10:02 (9244): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:27:30 (11524): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:39:08 (7300): No heartbeat from core client for 30 sec - exiting 19:39:09 (7300): No heartbeat from core client for 30 sec - exiting 19:39:10 (7300): No heartbeat from core client for 30 sec - exiting 19:39:11 (7300): No heartbeat from core client for 30 sec - exiting 19:39:12 (7300): No heartbeat from core client for 30 sec - exiting 19:39:13 (7300): No heartbeat from core client for 30 sec - exiting 19:39:14 (7300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7777332F read attempt to address 0x00000004 Engaging BOINC Windows Runtime Debugger... 15:05:04 (7712): No heartbeat from core client for 30 sec - exiting 15:05:05 (7712): No heartbeat from core client for 30 sec - exiting 15:05:06 (7712): No heartbeat from core client for 30 sec - exiting 15:05:07 (7712): No heartbeat from core client for 30 sec - exiting 15:05:08 (7712): No heartbeat from core client for 30 sec - exiting 15:05:09 (7712): No heartbeat from core client for 30 sec - exiting 15:05:10 (7712): No heartbeat from core client for 30 sec - exiting 15:05:11 (7712): No heartbeat from core client for 30 sec - exiting 15:05:12 (7712): No heartbeat from core client for 30 sec - exiting 15:05:13 (7712): No heartbeat from core client for 30 sec - exiting 15:05:14 (7712): No heartbeat from core client for 30 sec - exiting 15:05:15 (7712): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77363AC3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Oct 2013 06:29:28	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	259,200	353,849	1.3652
27 Oct 2013 14:21:12	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	233,280	317,398	1.3606
26 Oct 2013 18:17:16	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	207,360	280,073	1.3507
26 Oct 2013 05:35:32	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	181,440	242,846	1.3384
25 Oct 2013 18:34:51	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	155,520	205,738	1.3229
25 Oct 2013 06:16:59	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	129,600	167,871	1.2953
25 May 2013 10:22:38	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	103,680	134,958	1.3017
24 May 2013 18:59:18	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	77,760	101,026	1.2992
23 May 2013 19:10:48	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	51,840	67,225	1.2968
22 May 2013 18:59:57	1217831	15789747	hadcm3n_4joo_1940_40_008303650_3	25,920	33,399	1.2885