Task 13266689

Name	hadcm3n_t2ry_1980_40_007412604_1
Workunit	7610234
Created	17 Aug 2011, 2:32:10 UTC
Sent	17 Aug 2011, 2:45:47 UTC
Report deadline	16 Nov 2011, 10:12:58 UTC
Received	1 Feb 2012, 9:07:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1157839
Run time	13 days 9 hours 43 min 53 sec
CPU time	12 days 22 hours 32 min 2 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.16 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2684, selfPID=2684, iMonCtr=1 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t2ryko.pji4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910717 read attempt to address 0x00F0310D Engaging BOINC Windows Runtime Debugger... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3524, selfPID=3524, iMonCtr=1 Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... 13:48:35 (3472): No heartbeat from core client for 30 sec - exiting 13:48:37 (3472): No heartbeat from core client for 30 sec - exiting 13:48:38 (3472): No heartbeat from core client for 30 sec - exiting 13:48:39 (3472): No heartbeat from core client for 30 sec - exiting 13:48:40 (3472): No heartbeat from core client for 30 sec - exiting 13:48:41 (3472): No heartbeat from core client for 30 sec - exiting 13:48:42 (3472): No heartbeat from core client for 30 sec - exiting 13:48:43 (3472): No heartbeat from core client for 30 sec - exiting 13:48:44 (3472): No heartbeat from core client for 30 sec - exiting 13:48:45 (3472): No heartbeat from core client for 30 sec - exiting 13:48:47 (3472): No heartbeat from core client for 30 sec - exiting 13:48:48 (3472): No heartbeat from core client for 30 sec - exiting 13:48:49 (3472): No heartbeat from core client for 30 sec - exiting 13:07:33 (3472): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... 13:04:49 (972): No heartbeat from core client for 30 sec - exiting 13:04:51 (972): No heartbeat from core client for 30 sec - exiting 13:04:52 (972): No heartbeat from core client for 30 sec - exiting 13:04:53 (972): No heartbeat from core client for 30 sec - exiting 13:04:54 (972): No heartbeat from core client for 30 sec - exiting 13:04:55 (972): No heartbeat from core client for 30 sec - exiting 13:04:56 (972): No heartbeat from core client for 30 sec - exiting 13:04:57 (972): No heartbeat from core client for 30 sec - exiting 13:04:58 (972): No heartbeat from core client for 30 sec - exiting 13:04:59 (972): No heartbeat from core client for 30 sec - exiting 13:05:00 (972): No heartbeat from core client for 30 sec - exiting 13:05:02 (972): No heartbeat from core client for 30 sec - exiting 13:20:26 (972): No heartbeat from core client for 30 sec - exiting 13:20:28 (972): No heartbeat from core client for 30 sec - exiting 13:20:29 (972): No heartbeat from core client for 30 sec - exiting 13:20:30 (972): No heartbeat from core client for 30 sec - exiting 13:20:31 (972): No heartbeat from core client for 30 sec - exiting 13:20:32 (972): No heartbeat from core client for 30 sec - exiting 13:20:33 (972): No heartbeat from core client for 30 sec - exiting 13:20:34 (972): No heartbeat from core client for 30 sec - exiting 13:20:36 (972): No heartbeat from core client for 30 sec - exiting 13:20:37 (972): No heartbeat from core client for 30 sec - exiting 13:20:38 (972): No heartbeat from core client for 30 sec - exiting 13:20:39 (972): No heartbeat from core client for 30 sec - exiting 13:20:40 (972): No heartbeat from core client for 30 sec - exiting 13:20:41 (972): No heartbeat from core client for 30 sec - exiting 13:20:42 (972): No heartbeat from core client for 30 sec - exiting 13:20:43 (972): No heartbeat from core client for 30 sec - exiting 13:20:44 (972): No heartbeat from core client for 30 sec - exiting 13:20:45 (972): No heartbeat from core client for 30 sec - exiting 13:20:46 (972): No heartbeat from core client for 30 sec - exiting 13:20:48 (972): No heartbeat from core client for 30 sec - exiting 13:20:49 (972): No heartbeat from core client for 30 sec - exiting 13:20:50 (972): No heartbeat from core client for 30 sec - exiting 13:20:51 (972): No heartbeat from core client for 30 sec - exiting 13:20:52 (972): No heartbeat from core client for 30 sec - exiting 13:20:53 (972): No heartbeat from core client for 30 sec - exiting 13:20:54 (972): No heartbeat from core client for 30 sec - exiting 13:20:55 (972): No heartbeat from core client for 30 sec - exiting 13:20:56 (972): No heartbeat from core client for 30 sec - exiting 13:20:57 (972): No heartbeat from core client for 30 sec - exiting 13:20:58 (972): No heartbeat from core client for 30 sec - exiting 13:21:00 (972): No heartbeat from core client for 30 sec - exiting 13:21:01 (972): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... 13:36:26 (1200): No heartbeat from core client for 30 sec - exiting 13:36:28 (1200): No heartbeat from core client for 30 sec - exiting 13:36:29 (1200): No heartbeat from core client for 30 sec - exiting 13:36:30 (1200): No heartbeat from core client for 30 sec - exiting 13:36:31 (1200): No heartbeat from core client for 30 sec - exiting 13:36:32 (1200): No heartbeat from core client for 30 sec - exiting 13:36:33 (1200): No heartbeat from core client for 30 sec - exiting 13:36:34 (1200): No heartbeat from core client for 30 sec - exiting 13:36:35 (1200): No heartbeat from core client for 30 sec - exiting 13:36:36 (1200): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... 05:25:12 (792): No heartbeat from core client for 30 sec - exiting 05:25:13 (792): No heartbeat from core client for 30 sec - exiting 05:25:14 (792): No heartbeat from core client for 30 sec - exiting 05:25:16 (792): No heartbeat from core client for 30 sec - exiting 05:25:17 (792): No heartbeat from core client for 30 sec - exiting 05:25:18 (792): No heartbeat from core client for 30 sec - exiting 05:25:19 (792): No heartbeat from core client for 30 sec - exiting 05:25:20 (792): No heartbeat from core client for 30 sec - exiting 05:25:21 (792): No heartbeat from core client for 30 sec - exiting 05:25:22 (792): No heartbeat from core client for 30 sec - exiting 05:25:23 (792): No heartbeat from core client for 30 sec - exiting 05:25:24 (792): No heartbeat from core client for 30 sec - exiting 05:25:25 (792): No heartbeat from core client for 30 sec - exiting 05:25:27 (792): No heartbeat from core client for 30 sec - exiting 05:25:28 (792): No heartbeat from core client for 30 sec - exiting 05:25:29 (792): No heartbeat from core client for 30 sec - exiting 05:25:30 (792): No heartbeat from core client for 30 sec - exiting 05:25:31 (792): No heartbeat from core client for 30 sec - exiting 05:25:32 (792): No heartbeat from core client for 30 sec - exiting 05:25:33 (792): No heartbeat from core client for 30 sec - exiting 05:25:34 (792): No heartbeat from core client for 30 sec - exiting 05:25:35 (792): No heartbeat from core client for 30 sec - exiting 05:25:36 (792): No heartbeat from core client for 30 sec - exiting 05:25:38 (792): No heartbeat from core client for 30 sec - exiting 05:25:39 (792): No heartbeat from core client for 30 sec - exiting 05:25:40 (792): No heartbeat from core client for 30 sec - exiting 13:50:16 (792): No heartbeat from core client for 30 sec - exiting 13:50:17 (792): No heartbeat from core client for 30 sec - exiting 13:50:18 (792): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... 14:31:18 (740): No heartbeat from core client for 30 sec - exiting 14:31:19 (740): No heartbeat from core client for 30 sec - exiting 14:31:20 (740): No heartbeat from core client for 30 sec - exiting 14:31:21 (740): No heartbeat from core client for 30 sec - exiting 14:31:22 (740): No heartbeat from core client for 30 sec - exiting 15:35:28 (740): No heartbeat from core client for 30 sec - exiting 15:35:30 (740): No heartbeat from core client for 30 sec - exiting 15:35:31 (740): No heartbeat from core client for 30 sec - exiting 15:35:32 (740): No heartbeat from core client for 30 sec - exiting 15:35:33 (740): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... 12:49:06 (3644): No heartbeat from core client for 30 sec - exiting 12:49:07 (3644): No heartbeat from core client for 30 sec - exiting 12:49:08 (3644): No heartbeat from core client for 30 sec - exiting 12:49:09 (3644): No heartbeat from core client for 30 sec - exiting 12:49:10 (3644): No heartbeat from core client for 30 sec - exiting 13:40:57 (3644): No heartbeat from core client for 30 sec - exiting 13:40:58 (3644): No heartbeat from core client for 30 sec - exiting 18:20:00 (3644): No heartbeat from core client for 30 sec - exiting 18:20:02 (3644): No heartbeat from core client for 30 sec - exiting 18:20:03 (3644): No heartbeat from core client for 30 sec - exiting 18:20:04 (3644): No heartbeat from core client for 30 sec - exiting 18:20:05 (3644): No heartbeat from core client for 30 sec - exiting 18:20:06 (3644): No heartbeat from core client for 30 sec - exiting 18:20:07 (3644): No heartbeat from core client for 30 sec - exiting 18:20:09 (3644): No heartbeat from core client for 30 sec - exiting 18:20:10 (3644): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... 07:24:39 (244): No heartbeat from core client for 30 sec - exiting 07:24:40 (244): No heartbeat from core client for 30 sec - exiting 07:24:41 (244): No heartbeat from core client for 30 sec - exiting 07:24:42 (244): No heartbeat from core client for 30 sec - exiting 07:24:44 (244): No heartbeat from core client for 30 sec - exiting 12:45:49 (244): No heartbeat from core client for 30 sec - exiting 13:34:47 (244): No heartbeat from core client for 30 sec - exiting 13:34:48 (244): No heartbeat from core client for 30 sec - exiting 13:34:50 (244): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7C910F1E read attempt to address 0x408D9DDF Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Sep 2011 04:59:48	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	777,600	1,098,303	1.4124
01 Sep 2011 13:46:37	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	751,680	1,062,584	1.4136
01 Sep 2011 02:47:45	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	725,760	1,024,354	1.4114
31 Aug 2011 15:45:30	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	699,840	986,168	1.4091
31 Aug 2011 01:05:13	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	673,920	951,024	1.4112
30 Aug 2011 14:20:07	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	648,000	913,585	1.4099
30 Aug 2011 03:48:04	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	622,080	876,819	1.4095
29 Aug 2011 16:47:08	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	596,160	838,647	1.4067
29 Aug 2011 05:51:59	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	570,240	800,565	1.4039
28 Aug 2011 18:24:48	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	544,320	762,187	1.4003
28 Aug 2011 07:24:50	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	518,400	724,129	1.3969
27 Aug 2011 20:29:41	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	492,480	686,085	1.3931
27 Aug 2011 09:34:07	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	466,560	648,142	1.3892
26 Aug 2011 22:39:43	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	440,640	610,197	1.3848
26 Aug 2011 11:44:53	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	414,720	572,214	1.3798
26 Aug 2011 00:52:22	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	388,800	534,263	1.3741
25 Aug 2011 08:52:07	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	362,880	495,880	1.3665
24 Aug 2011 21:51:55	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	336,960	457,628	1.3581
24 Aug 2011 10:51:17	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	311,040	419,331	1.3482
23 Aug 2011 23:50:23	1157839	13266689	hadcm3n_t2ry_1980_40_007412604_1	285,120	381,065	1.3365