Task 15560575

Name	hadcm3n_3d7m_1940_40_008265831_1
Workunit	8420955
Created	29 Jan 2013, 16:04:25 UTC
Sent	29 Jan 2013, 16:04:31 UTC
Report deadline	30 Apr 2013, 23:31:42 UTC
Received	11 Apr 2013, 11:18:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1241760
Run time	7 days 14 hours 26 min 30 sec
CPU time	7 days 13 hours 4 min 53 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.57 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Ocean Restart file copy failed on 3d7mko.dae1620 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dae34m0 Ocean Restart file copy failed on 3d7mko.dae34p0 Ocean Restart file copy failed on 3d7mko.dae34r0 Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3d7mko.dae3520 Ocean Restart file copy failed on 3d7mko.dae3570 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3d7mko.daf0a60 Ocean Restart file copy failed on 3d7mko.daf0a80 Ocean Restart file copy failed on 3d7mko.daf0aa0 Ocean Restart file copy failed on 3d7mko.daf0ak0 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3d7mko.daf0au0 Ocean Restart file copy failed on 3d7mko.daf0b30 Ocean Restart file copy failed on 3d7mko.daf0b60 Ocean Restart file copy failed on 3d7mko.daf0b80 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.daf0bo0 Ocean Restart file copy failed on 3d7mko.daf0c70 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.daf45n0 Ocean Restart file copy failed on 3d7mko.daf45q0 Ocean Restart file copy failed on 3d7mko.daf4640 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.daf46l0 Ocean Restart file copy failed on 3d7mko.daf46n0 Ocean Restart file copy failed on 3d7mko.daf4750 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.daf47j0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:33:08 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:01:45 (4360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3d7mko.daf6890 Ocean Restart file copy failed on 3d7mko.daf68d0 Ocean Restart file copy failed on 3d7mko.daf68g0 Ocean Restart file copy failed on 3d7mko.daf68k0 Ocean Restart file copy failed on 3d7mko.daf68u0 Ocean Restart file copy failed on 3d7mko.daf6970 Ocean Restart file copy failed on 3d7mko.daf6990 Ocean Restart file copy failed on 3d7mko.daf69c0 12:40:01 (5516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 09:02:21 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 11:29:34 (4304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:48:34 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:20:12 (2120): No heartbeat from core client for 30 sec - exiting 10:20:13 (2120): No heartbeat from core client for 30 sec - exiting 10:20:14 (2120): No heartbeat from core client for 30 sec - exiting 10:20:16 (2120): No heartbeat from core client for 30 sec - exiting 10:20:17 (2120): No heartbeat from core client for 30 sec - exiting 10:20:18 (2120): No heartbeat from core client for 30 sec - exiting 10:20:19 (2120): No heartbeat from core client for 30 sec - exiting 10:20:20 (2120): No heartbeat from core client for 30 sec - exiting 10:20:21 (2120): No heartbeat from core client for 30 sec - exiting 10:20:22 (2120): No heartbeat from core client for 30 sec - exiting 10:20:23 (2120): No heartbeat from core client for 30 sec - exiting 10:20:24 (2120): No heartbeat from core client for 30 sec - exiting 10:20:25 (2120): No heartbeat from core client for 30 sec - exiting 10:20:26 (2120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag2120 Ocean Restart file copy failed on 3d7mko.dag2140 Ocean Restart file copy failed on 3d7mko.dag2170 Ocean Restart file copy failed on 3d7mko.dag2190 Ocean Restart file copy failed on 3d7mko.dag21b0 Ocean Restart file copy failed on 3d7mko.dag21f0 Ocean Restart file copy failed on 3d7mko.dag21g0 Ocean Restart file copy failed on 3d7mko.dag21h0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag21m0 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag21n0 Ocean Restart file copy failed on 3d7mko.dag21o0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag21r0 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag2220 Ocean Restart file copy failed on 3d7mko.dag2240 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag22b0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag22g0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag22p0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=872, iMonCtr=1 Model crash detected, will try to restart... 08:03:54 (5412): No heartbeat from core client for 30 sec - exiting 08:03:55 (5412): No heartbeat from core client for 30 sec - exiting 08:03:56 (5412): No heartbeat from core client for 30 sec - exiting 08:03:57 (5412): No heartbeat from core client for 30 sec - exiting 08:03:58 (5412): No heartbeat from core client for 30 sec - exiting 08:03:59 (5412): No heartbeat from core client for 30 sec - exiting 08:04:00 (5412): No heartbeat from core client for 30 sec - exiting 08:04:02 (5412): No heartbeat from core client for 30 sec - exiting 08:04:03 (5412): No heartbeat from core client for 30 sec - exiting 08:04:04 (5412): No heartbeat from core client for 30 sec - exiting 08:04:05 (5412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Ocean Restart file copy failed on 3d7mko.dag43j0 Ocean Restart file copy failed on 3d7mko.dag43r0 Ocean Restart file copy failed on 3d7mko.dag43t0 Ocean Restart file copy failed on 3d7mko.dag4410 Ocean Restart file copy failed on 3d7mko.dag44b0 Ocean Restart file copy failed on 3d7mko.dag44d0 Ocean Restart file copy failed on 3d7mko.dag44k0 Ocean Restart file copy failed on 3d7mko.dag44n0 Ocean Restart file copy failed on 3d7mko.dag44u0 Ocean Restart file copy failed on 3d7mko.dag4540 Ocean Restart file copy failed on 3d7mko.dag45c0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dag62r0 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:05:03 (4104): No heartbeat from core client for 30 sec - exiting 17:05:04 (4104): No heartbeat from core client for 30 sec - exiting 17:05:05 (4104): No heartbeat from core client for 30 sec - exiting 17:05:06 (4104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:31:12 (1032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:20:47 (2976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:04:04 (404): No heartbeat from core client for 30 sec - exiting 12:04:05 (404): No heartbeat from core client for 30 sec - exiting 12:04:07 (404): No heartbeat from core client for 30 sec - exiting 12:04:08 (404): No heartbeat from core client for 30 sec - exiting 12:04:09 (404): No heartbeat from core client for 30 sec - exiting 12:04:10 (404): No heartbeat from core client for 30 sec - exiting 12:04:11 (404): No heartbeat from core client for 30 sec - exiting 12:04:12 (404): No heartbeat from core client for 30 sec - exiting 12:04:13 (404): No heartbeat from core client for 30 sec - exiting 12:04:14 (404): No heartbeat from core client for 30 sec - exiting 12:04:15 (404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 20:12:27 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 3d7mko.dah08s0 CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 3d7mko.dah09i0 18:57:17 (5032): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Ocean Restart file copy failed on 3d7mko.dah1280 CPDN Monitor - Quit request from BOINC... 09:58:12 (4840): No heartbeat from core client for 30 sec - exiting 09:58:13 (4840): No heartbeat from core client for 30 sec - exiting 09:58:14 (4840): No heartbeat from core client for 30 sec - exiting 09:58:15 (4840): No heartbeat from core client for 30 sec - exiting 09:58:16 (4840): No heartbeat from core client for 30 sec - exiting 09:58:17 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:58:18 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:01:42 (4332): No heartbeat from core client for 30 sec - exiting 08:01:43 (4332): No heartbeat from core client for 30 sec - exiting 08:01:44 (4332): No heartbeat from core client for 30 sec - exiting 08:01:45 (4332): No heartbeat from core client for 30 sec - exiting 08:01:46 (4332): No heartbeat from core client for 30 sec - exiting 08:01:47 (4332): No heartbeat from core client for 30 sec - exiting 08:01:48 (4332): No heartbeat from core client for 30 sec - exiting 08:01:49 (4332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:19:16 (5412): No heartbeat from core client for 30 sec - exiting 18:19:17 (5412): No heartbeat from core client for 30 sec - exiting 18:19:18 (5412): No heartbeat from core client for 30 sec - exiting 18:19:19 (5412): No heartbeat from core client for 30 sec - exiting 18:19:20 (5412): No heartbeat from core client for 30 sec - exiting 18:19:21 (5412): No heartbeat from core client for 30 sec - exiting 18:19:23 (5412): No heartbeat from core client for 30 sec - exiting 18:19:24 (5412): No heartbeat from core client for 30 sec - exiting 18:19:25 (5412): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dah86t0 Ocean Restart file copy failed on 3d7mko.dah8710 Ocean Restart file copy failed on 3d7mko.dah87b0 Ocean Restart file copy failed on 3d7mko.dah87d0 Ocean Restart file copy failed on 3d7mko.dah87k0 Suspended CPDN Monitor - Suspend request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold Ocean Restart file copy failed on 3d7mko.dah87u0 Ocean Restart file copy failed on 3d7mko.dah8840 Ocean Restart file copy failed on 3d7mko.dah8870 Ocean Restart file copy failed on 3d7mko.dah89n0 Suspended CPDN Monitor - Suspend request from BOINC... Ocean Restart file copy failed on 3d7mko.dah94e0 Ocean Restart file copy failed on 3d7mko.dah94g0 Ocean Restart file copy failed on 3d7mko.dah94l0 12:00:34 (5672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Ocean Restart file copy failed on 3d7mko.dai0b90 Ocean Restart file copy failed on 3d7mko.dai0c30 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x76F13AB3 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6000, selfPID=6000, iMonCtr=1 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Apr 2013 11:19:51	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	1,036,800	651,879	0.6287
10 Apr 2013 18:07:37	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	1,010,880	635,579	0.6287
10 Apr 2013 12:34:32	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	984,960	619,377	0.6288
10 Apr 2013 07:52:53	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	959,040	603,313	0.6291
10 Apr 2013 03:01:02	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	933,120	587,137	0.6292
09 Apr 2013 22:29:07	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	907,200	570,978	0.6294
09 Apr 2013 19:00:21	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	881,280	554,986	0.6297
09 Apr 2013 07:24:31	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	855,360	538,620	0.6297
08 Apr 2013 14:25:20	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	829,440	522,695	0.6302
07 Apr 2013 16:36:09	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	803,520	506,431	0.6303
05 Apr 2013 13:40:09	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	777,600	489,584	0.6296
28 Mar 2013 14:40:39	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	751,680	473,992	0.6306
28 Mar 2013 03:57:04	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	725,760	458,117	0.6312
22 Mar 2013 18:06:14	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	699,840	442,251	0.6319
20 Mar 2013 16:20:47	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	673,920	426,045	0.6322
17 Mar 2013 13:45:16	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	648,000	409,841	0.6325
16 Mar 2013 13:25:07	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	622,080	393,192	0.6321
16 Mar 2013 09:03:15	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	596,160	376,802	0.6320
14 Mar 2013 16:45:52	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	570,240	360,374	0.6320
14 Mar 2013 13:25:08	1241760	15560575	hadcm3n_3d7m_1940_40_008265831_1	544,320	343,239	0.6306