Task 15280832

Name	hadcm3n_zcfv_1880_40_008198832_2
Workunit	8353956
Created	13 Sep 2012, 10:06:39 UTC
Sent	13 Sep 2012, 20:19:31 UTC
Report deadline	14 Dec 2012, 3:46:42 UTC
Received	3 Nov 2012, 22:48:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1175227
Run time	4 days 23 hours 47 min 44 sec
CPU time	3 days 20 hours 5 min 49 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.81 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:08:07 (3972): No heartbeat from core client for 30 sec - exiting 16:08:08 (3972): No heartbeat from core client for 30 sec - exiting 16:08:09 (3972): No heartbeat from core client for 30 sec - exiting 16:08:10 (3972): No heartbeat from core client for 30 sec - exiting 16:08:11 (3972): No heartbeat from core client for 30 sec - exiting 16:08:12 (3972): No heartbeat from core client for 30 sec - exiting 16:08:13 (3972): No heartbeat from core client for 30 sec - exiting 16:08:14 (3972): No heartbeat from core client for 30 sec - exiting 16:08:15 (3972): No heartbeat from core client for 30 sec - exiting 16:08:16 (3972): No heartbeat from core client for 30 sec - exiting 16:08:17 (3972): No heartbeat from core client for 30 sec - exiting 16:08:18 (3972): No heartbeat from core client for 30 sec - exiting 16:08:19 (3972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:12:23 (6684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5804, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CCPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:22:33 (4776): No heartbeat from core client for 30 sec - exiting 05:22:34 (4776): No heartbeat from core client for 30 sec - exiting 05:22:35 (4776): No heartbeat from core client for 30 sec - exiting 05:22:36 (4776): No heartbeat from core client for 30 sec - exiting 05:22:37 (4776): No heartbeat from core client for 30 sec - exiting 05:22:38 (4776): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3852, iMonCtr=1 Model crash detected, will try to restart... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4284, selfPID=4284, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:22:22 (2220): No heartbeat from core client for 30 sec - exiting 06:22:23 (2220): No heartbeat from core client for 30 sec - exiting 06:22:24 (2220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:49:12 (712): No heartbeat from core client for 30 sec - exiting 17:49:13 (712): No heartbeat from core client for 30 sec - exiting 17:49:14 (712): No heartbeat from core client for 30 sec - exiting 17:49:15 (712): No heartbeat from core client for 30 sec - exiting 17:49:16 (712): No heartbeat from core client for 30 sec - exiting 17:49:17 (712): No heartbeat from core client for 30 sec - exiting 17:49:18 (712): No heartbeat from core client for 30 sec - exiting 17:49:19 (712): No heartbeat from core client for 30 sec - exiting 17:49:20 (712): No heartbeat from core client for 30 sec - exiting 17:49:21 (712): No heartbeat from core client for 30 sec - exiting 17:49:22 (712): No heartbeat from core client for 30 sec - exiting 17:49:23 (712): No heartbeat from core client for 30 sec - exiting 17:49:24 (712): No heartbeat from core client for 30 sec - exiting 17:49:25 (712): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23248, iMonCtr=1 Model crash detected, will try to restart... 08:20:35 (4760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:11:00 (5316): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:36:27 (3616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:47:16 (6672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... 19:41:03 (4632): No heartbeat from core client for 30 sec - exiting 19:41:04 (4632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:41:05 (4632): No heartbeat from core client for 30 sec - exiting 15:03:29 (5176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:13:35 (4364): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:25:34 (7496): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:38:14 (4992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:00:43 (344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:23:18 (4352): No heartbeat from core client for 30 sec - exiting 19:23:19 (4352): No heartbeat from core client for 30 sec - exiting 19:23:20 (4352): No heartbeat from core client for 30 sec - exiting 19:23:21 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 00:07:42 (1752): No heartbeat from core client for 30 sec - exiting 00:07:43 (1752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:20:18 (4832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6044, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6044, iMonCtr=1 Model crash detected, will try to restart... 07:51:00 (6044): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:39:33 (380): No heartbeat from core client for 30 sec - exiting 15:39:35 (380): No heartbeat from core client for 30 sec - exiting 15:39:36 (380): No heartbeat from core client for 30 sec - exiting 15:39:37 (380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:19:48 (4236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 21:49:54 (5492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:05:23 (5988): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:36:24 (3272): No heartbeat from core client for 30 sec - exiting 19:36:25 (3272): No heartbeat from core client for 30 sec - exiting 19:36:26 (3272): No heartbeat from core client for 30 sec - exiting 19:36:27 (3272): No heartbeat from core client for 30 sec - exiting 19:36:28 (3272): No heartbeat from core client for 30 sec - exiting 19:36:29 (3272): No heartbeat from core client for 30 sec - exiting 19:36:30 (3272): No heartbeat from core client for 30 sec - exiting 19:36:31 (3272): No heartbeat from core client for 30 sec - exiting 19:36:32 (3272): No heartbeat from core client for 30 sec - exiting 19:36:33 (3272): No heartbeat from core client for 30 sec - exiting 19:36:34 (3272): No heartbeat from core client for 30 sec - exiting 19:36:35 (3272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5576, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:34 (4852): No heartbeat from core client for 30 sec - exiting 17:21:35 (4852): No heartbeat from core client for 30 sec - exiting 17:21:36 (4852): No heartbeat from core client for 30 sec - exiting 17:21:37 (4852): No heartbeat from core client for 30 sec - exiting 17:21:38 (4852): No heartbeat from core client for 30 sec - exiting 17:21:39 (4852): No heartbeat from core client for 30 sec - exiting 17:21:40 (4852): No heartbeat from core client for 30 sec - exiting 17:21:41 (4852): No heartbeat from core client for 30 sec - exiting 17:21:42 (4852): No heartbeat from core client for 30 sec - exiting 17:21:43 (4852): No heartbeat from core client for 30 sec - exiting 17:21:44 (4852): No heartbeat from core client for 30 sec - exiting 17:21:45 (4852): No heartbeat from core client for 30 sec - exiting 17:21:46 (4852): No heartbeat from core client for 30 sec - exiting 17:21:47 (4852): No heartbeat from core client for 30 sec - exiting 17:21:48 (4852): No heartbeat from core client for 30 sec - exiting 17:21:49 (4852): No heartbeat from core client for 30 sec - exiting 17:21:50 (4852): No heartbeat from core client for 30 sec - exiting 17:21:51 (4852): No heartbeat from core client for 30 sec - exiting 17:21:52 (4852): No heartbeat from core client for 30 sec - exiting 17:21:53 (4852): No heartbeat from core client for 30 sec - exiting 17:21:54 (4852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:48:40 (4196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:12:19 (4660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5740, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 22:43:57 (6084): No heartbeat from core client for 30 sec - exiting 22:43:58 (6084): No heartbeat from core client for 30 sec - exiting 22:43:59 (6084): No heartbeat from core client for 30 sec - exiting 22:44:00 (6084): No heartbeat from core client for 30 sec - exiting 22:44:01 (6084): No heartbeat from core client for 30 sec - exiting 22:44:02 (6084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:21:54 (5656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:43:01 (3716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:26:02 (4280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 20:36:29 (6504): No heartbeat from core client for 30 sec - exiting 20:36:30 (6504): No heartbeat from core client for 30 sec - exiting 20:36:31 (6504): No heartbeat from core client for 30 sec - exiting 20:36:32 (6504): No heartbeat from core client for 30 sec - exiting 20:36:33 (6504): No heartbeat from core client for 30 sec - exiting 20:36:34 (6504): No heartbeat from core client for 30 sec - exiting 20:36:35 (6504): No heartbeat from core client for 30 sec - exiting 20:36:36 (6504): No heartbeat from core client for 30 sec - exiting 20:36:37 (6504): No heartbeat from core client for 30 sec - exiting 20:36:38 (6504): No heartbeat from core client for 30 sec - exiting 20:36:39 (6504): No heartbeat from core client for 30 sec - exiting 20:36:40 (6504): No heartbeat from core client for 30 sec - exiting 20:36:41 (6504): No heartbeat from core client for 30 sec - exiting 20:36:42 (6504): No heartbeat from core client for 30 sec - exiting 20:36:43 (6504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:59:58 (4700): No heartbeat from core client for 30 sec - exiting 12:59:59 (4700): No heartbeat from core client for 30 sec - exiting 13:00:00 (4700): No heartbeat from core client for 30 sec - exiting 13:00:01 (4700): No heartbeat from core client for 30 sec - exiting 13:00:02 (4700): No heartbeat from core client for 30 sec - exiting 13:00:03 (4700): No heartbeat from core client for 30 sec - exiting 13:00:04 (4700): No heartbeat from core client for 30 sec - exiting 13:00:05 (4700): No heartbeat from core client for 30 sec - exiting 13:00:06 (4700): No heartbeat from core client for 30 sec - exiting 13:00:07 (4700): No heartbeat from core client for 30 sec - exiting 13:00:08 (4700): No heartbeat from core client for 30 sec - exiting 13:00:09 (4700): No heartbeat from core client for 30 sec - exiting 13:00:10 (4700): No heartbeat from core client for 30 sec - exiting 13:00:11 (4700): No heartbeat from core client for 30 sec - exiting 13:00:12 (4700): No heartbeat from core client for 30 sec - exiting 13:00:13 (4700): No heartbeat from core client for 30 sec - exiting 13:00:14 (4700): No heartbeat from core client for 30 sec - exiting 13:00:15 (4700): No heartbeat from core client for 30 sec - exiting 13:00:16 (4700): No heartbeat from core client for 30 sec - exiting 13:00:17 (4700): No heartbeat from core client for 30 sec - exiting 13:00:18 (4700): No heartbeat from core client for 30 sec - exiting 13:00:19 (4700): No heartbeat from core client for 30 sec - exiting 13:00:20 (4700): No heartbeat from core client for 30 sec - exiting 13:00:21 (4700): No heartbeat from core client for 30 sec - exiting 13:00:22 (4700): No heartbeat from core client for 30 sec - exiting 13:00:23 (4700): No heartbeat from core client for 30 sec - exiting 13:00:24 (4700): No heartbeat from core client for 30 sec - exiting 13:00:25 (4700): No heartbeat from core client for 30 sec - exiting 13:00:26 (4700): No heartbeat from core client for 30 sec - exiting 13:00:27 (4700): No heartbeat from core client for 30 sec - exiting 13:00:28 (4700): No heartbeat from core client for 30 sec - exiting 13:00:29 (4700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:00:00 (3748): No heartbeat from core client for 30 sec - exiting 14:00:01 (3748): No heartbeat from core client for 30 sec - exiting 14:00:02 (3748): No heartbeat from core client for 30 sec - exiting 14:00:03 (3748): No heartbeat from core client for 30 sec - exiting 14:00:04 (3748): No heartbeat from core client for 30 sec - exiting 14:00:05 (3748): No heartbeat from core client for 30 sec - exiting 14:00:06 (3748): No heartbeat from core client for 30 sec - exiting 14:00:07 (3748): No heartbeat from core client for 30 sec - exiting 14:00:08 (3748): No heartbeat from core client for 30 sec - exiting 14:00:09 (3748): No heartbeat from core client for 30 sec - exiting 14:00:10 (3748): No heartbeat from core client for 30 sec - exiting 14:00:11 (3748): No heartbeat from core client for 30 sec - exiting 14:00:12 (3748): No heartbeat from core client for 30 sec - exiting 14:00:13 (3748): No heartbeat from core client for 30 sec - exiting 14:00:14 (3748): No heartbeat from core client for 30 sec - exiting 14:00:15 (3748): No heartbeat from core client for 30 sec - exiting 14:00:16 (3748): No heartbeat from core client for 30 sec - exiting 14:00:17 (3748): No heartbeat from core client for 30 sec - exiting 14:00:18 (3748): No heartbeat from core client for 30 sec - exiting 14:00:19 (3748): No heartbeat from core client for 30 sec - exiting 14:00:20 (3748): No heartbeat from core client for 30 sec - exiting 14:00:21 (3748): No heartbeat from core client for 30 sec - exiting 14:00:22 (3748): No heartbeat from core client for 30 sec - exiting 14:00:23 (3748): No heartbeat from core client for 30 sec - exiting 14:00:24 (3748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:00:25 (3748): No heartbeat from core client for 30 sec - exiting Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3692, iMonCtr=1 Model crash detected, will try to restart... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3052, selfPID=3052, iMonCtr=1 CPDN Monitor - Quit request from BOINC... 07:35:44 (6876): No heartbeat from core client for 30 sec - exiting 07:35:45 (6876): No heartbeat from core client for 30 sec - exiting 07:35:46 (6876): No heartbeat from core client for 30 sec - exiting 07:35:47 (6876): No heartbeat from core client for 30 sec - exiting 07:35:48 (6876): No heartbeat from core client for 30 sec - exiting 07:35:49 (6876): No heartbeat from core client for 30 sec - exiting 07:35:50 (6876): No heartbeat from core client for 30 sec - exiting 07:35:51 (6876): No heartbeat from core client for 30 sec - exiting 07:35:52 (6876): No heartbeat from core client for 30 sec - exiting 07:35:53 (6876): No heartbeat from core client for 30 sec - exiting 07:35:54 (6876): No heartbeat from core client for 30 sec - exiting 07:35:55 (6876): No heartbeat from core client for 30 sec - exiting 07:35:56 (6876): No heartbeat from core client for 30 sec - exiting 07:35:57 (6876): No heartbeat from core client for 30 sec - exiting 07:35:58 (6876): No heartbeat from core client for 30 sec - exiting 07:35:59 (6876): No heartbeat from core client for 30 sec - exiting 07:36:00 (6876): No heartbeat from core client for 30 sec - exiting 07:36:01 (6876): No heartbeat from core client for 30 sec - exiting 07:36:02 (6876): No heartbeat from core client for 30 sec - exiting 07:36:03 (6876): No heartbeat from core client for 30 sec - exiting 07:36:04 (6876): No heartbeat from core client for 30 sec - exiting 07:36:05 (6876): No heartbeat from core client for 30 sec - exiting 07:36:06 (6876): No heartbeat from core client for 30 sec - exiting 07:36:07 (6876): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:40:16 (6884): No heartbeat from core client for 30 sec - exiting 15:40:17 (6884): No heartbeat from core client for 30 sec - exiting 15:40:18 (6884): No heartbeat from core client for 30 sec - exiting 15:40:19 (6884): No heartbeat from core client for 30 sec - exiting 15:40:20 (6884): No heartbeat from core client for 30 sec - exiting 15:40:21 (6884): No heartbeat from core client for 30 sec - exiting 15:40:22 (6884): No heartbeat from core client for 30 sec - exiting 15:40:23 (6884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=884, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=884, iMonCtr=1 Model crash detected, will try to restart... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6308, selfPID=6308, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:47:55 (4952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x779371F3 read attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Nov 2012 22:49:05	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	259,200	331,575	1.2792
28 Oct 2012 07:03:43	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	233,280	297,992	1.2774
24 Oct 2012 04:20:57	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	207,360	263,986	1.2731
20 Oct 2012 12:21:59	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	181,440	230,406	1.2699
14 Oct 2012 20:12:38	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	155,520	197,476	1.2698
27 Sep 2012 09:09:13	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	129,600	170,663	1.3168
25 Sep 2012 09:53:14	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	103,680	136,578	1.3173
22 Sep 2012 07:27:42	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	77,760	103,027	1.3249
22 Sep 2012 02:06:32	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	51,840	69,274	1.3363
21 Sep 2012 00:15:20	1175227	15280832	hadcm3n_zcfv_1880_40_008198832_2	25,920	34,789	1.3422