Task 12927399

Name	hadcm3n_o7ls_1940_40_007266962_0
Workunit	7465202
Created	3 Jun 2011, 0:39:32 UTC
Sent	3 Jun 2011, 0:39:48 UTC
Report deadline	2 Sep 2011, 8:06:59 UTC
Received	18 Aug 2011, 6:22:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	1131576
Run time	21 days 14 hours 29 min 56 sec
CPU time	20 days 22 hours 50 min 44 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.82 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> - exit code -1073741819 (0xc0000005) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5532, selfPID=5532, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1816, selfPID=1816, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6000, selfPID=6000, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5936, selfPID=5936, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=984, selfPID=984, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4360, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6032, selfPID=6032, iMonCtr=1 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5656, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1856, selfPID=1856, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5788, selfPID=5788, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2224, selfPID=2224, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5856, selfPID=5856, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5168, selfPID=5168, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5780, selfPID=5780, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5720, selfPID=5720, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2576, selfPID=2576, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3028, selfPID=3028, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4536, selfPID=4536, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1640, selfPID=1640, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2764, selfPID=2764, iMonCtr=1 10:27:04 (4676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:27:05 (4676): No heartbeat from core client for 30 sec - exiting 10:27:06 (4676): No heartbeat from core client for 30 sec - exiting 10:27:07 (4676): No heartbeat from core client for 30 sec - exiting 10:27:08 (4676): No heartbeat from core client for 30 sec - exiting 10:27:09 (4676): No heartbeat from core client for 30 sec - exiting 10:27:10 (4676): No heartbeat from core client for 30 sec - exiting 10:27:11 (4676): No heartbeat from core client for 30 sec - exiting 10:27:12 (4676): No heartbeat from core client for 30 sec - exiting 10:27:13 (4676): No heartbeat from core client for 30 sec - exiting 10:27:14 (4676): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2888, selfPID=2888, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4080, selfPID=4080, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4516, selfPID=4516, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5632, selfPID=5632, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=400, selfPID=400, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5240, selfPID=5240, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5164, selfPID=5164, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6892, selfPID=6892, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6260, selfPID=6260, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5588, selfPID=5588, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5048, selfPID=5048, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3380, selfPID=3380, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5776, selfPID=5776, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6456, selfPID=6456, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3100, selfPID=3100, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7100, selfPID=7100, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6168, selfPID=6168, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5264, selfPID=5264, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6668, selfPID=6668, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5836, selfPID=5836, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6604, selfPID=6604, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4980, selfPID=4980, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1980, selfPID=1980, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6956, selfPID=6956, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4648, selfPID=4648, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5908, selfPID=5908, iMonCtr=1 18:10:34 (4884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:10:35 (4884): No heartbeat from core client for 30 sec - exiting 18:10:36 (4884): No heartbeat from core client for 30 sec - exiting 18:10:37 (4884): No heartbeat from core client for 30 sec - exiting 18:10:39 (4884): No heartbeat from core client for 30 sec - exiting 18:10:40 (4884): No heartbeat from core client for 30 sec - exiting 18:10:41 (4884): No heartbeat from core client for 30 sec - exiting 18:10:42 (4884): No heartbeat from core client for 30 sec - exiting 18:10:43 (4884): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4696, selfPID=4696, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3564, selfPID=3564, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=328, selfPID=328, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4772, selfPID=4772, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5548, selfPID=5548, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4408, selfPID=4408, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6664, selfPID=6664, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6484, selfPID=6484, iMonCtr=1 CPDN Monitor - Quit request from BOINC... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2524, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2524, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2524, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5088, selfPID=5088, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4372, selfPID=4372, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6456, selfPID=6456, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6192, selfPID=6192, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6648, selfPID=6648, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5152, selfPID=5152, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2556, selfPID=2556, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5828, selfPID=5828, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5448, selfPID=5448, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3608, selfPID=3608, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1588, selfPID=1588, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6564, selfPID=6564, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3340, selfPID=3340, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4544, selfPID=4544, iMonCtr=1 01:00:52 (4916): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:53 (4916): No heartbeat from core client for 30 sec - exiting 01:00:55 (4916): No heartbeat from core client for 30 sec - exiting 01:00:56 (4916): No heartbeat from core client for 30 sec - exiting 01:00:57 (4916): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2896, selfPID=2896, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5808, selfPID=5808, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6120, selfPID=6120, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2816, selfPID=2816, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1316, selfPID=1316, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5748, selfPID=5748, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6100, selfPID=6100, iMonCtr=1 01:00:35 (3976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:36 (3976): No heartbeat from core client for 30 sec - exiting 01:00:37 (3976): No heartbeat from core client for 30 sec - exiting 01:00:38 (3976): No heartbeat from core client for 30 sec - exiting 01:00:39 (3976): No heartbeat from core client for 30 sec - exiting 01:00:40 (3976): No heartbeat from core client for 30 sec - exiting 01:00:41 (3976): No heartbeat from core client for 30 sec - exiting 01:00:42 (3976): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5464, selfPID=5464, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4688, selfPID=4688, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6520, selfPID=6520, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5068, selfPID=5068, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6764, selfPID=6764, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4408, selfPID=4408, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2524, selfPID=2524, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2664, selfPID=2664, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5296, selfPID=5296, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5736, selfPID=5736, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5372, selfPID=5372, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2860, selfPID=2860, iMonCtr=1 13:11:47 (5040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:11:48 (5040): No heartbeat from core client for 30 sec - exiting 13:11:49 (5040): No heartbeat from core client for 30 sec - exiting 13:11:50 (5040): No heartbeat from core client for 30 sec - exiting 13:11:51 (5040): No heartbeat from core client for 30 sec - exiting 13:11:52 (5040): No heartbeat from core client for 30 sec - exiting 13:11:53 (5040): No heartbeat from core client for 30 sec - exiting 13:11:54 (5040): No heartbeat from core client for 30 sec - exiting 13:11:56 (5040): No heartbeat from core client for 30 sec - exiting 13:11:57 (5040): No heartbeat from core client for 30 sec - exiting 13:11:58 (5040): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6656, selfPID=6656, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4672, selfPID=4672, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=524, selfPID=524, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... 01:00:35 (5552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:36 (5552): No heartbeat from core client for 30 sec - exiting 01:00:37 (5552): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5088, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5088, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5736, selfPID=5736, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5172, selfPID=5172, iMonCtr=1 CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6044, selfPID=6044, iMonCtr=1 CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5468, selfPID=5468, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7745B84B write attempt to address 0xFFFFFFF8 Engaging BOINC Windows Runtime Debugger... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x77433A93 read attempt to address 0x00000000 Engaging BOINC Windows Runtime Debugger... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Aug 2011 09:13:45	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	1,036,800	1,809,567	1.7453
16 Aug 2011 11:02:46	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	1,010,880	1,762,729	1.7438
15 Aug 2011 12:55:44	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	984,960	1,716,059	1.7423
14 Aug 2011 14:35:24	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	959,040	1,669,327	1.7406
14 Aug 2011 01:31:54	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	933,120	1,622,917	1.7392
13 Aug 2011 03:21:22	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	907,200	1,629,229	1.7959
11 Aug 2011 07:17:48	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	881,280	1,581,875	1.7950
10 Aug 2011 08:40:27	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	855,360	1,534,019	1.7934
09 Aug 2011 09:55:23	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	829,440	1,485,312	1.7907
08 Aug 2011 11:13:10	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	803,520	1,436,899	1.7883
05 Aug 2011 03:40:57	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	777,600	1,388,289	1.7854
02 Aug 2011 01:32:04	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	751,680	1,338,772	1.7810
29 Jul 2011 03:48:42	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	725,760	1,304,223	1.7970
26 Jul 2011 02:26:29	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	699,840	1,256,940	1.7960
25 Jul 2011 20:21:23	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	673,920	1,209,166	1.7942
25 Jul 2011 17:49:38	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	648,000	1,162,329	1.7937
25 Jul 2011 14:55:57	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	622,080	1,115,006	1.7924
25 Jul 2011 13:02:51	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	596,160	1,067,340	1.7904
10 Jul 2011 12:53:24	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	570,240	1,082,503	1.8983
09 Jul 2011 19:31:16	1131576	12927399	hadcm3n_o7ls_1940_40_007266962_0	544,320	1,034,059	1.8997