Task 16050652

Name	hadcm3n_zht0_1960_40_008367554_4
Workunit	8518413
Created	30 Sep 2013, 10:47:07 UTC
Sent	30 Sep 2013, 11:12:40 UTC
Report deadline	30 Dec 2013, 18:39:51 UTC
Received	28 Nov 2013, 0:47:33 UTC
Server state	Over
Outcome	Success
Client state	Done
Exit status	0 (0x00000000)
Computer ID	1277518
Run time	42 days 2 hours 42 min 2 sec
CPU time	35 days 11 hours 48 min 31 sec
Validate state	Valid
Credit	12,441.60
Device peak FLOPS	2.10 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:20:17 (5268): No heartbeat from core client for 30 sec - exiting 04:20:19 (5268): No heartbeat from core client for 30 sec - exiting 04:20:20 (5268): No heartbeat from core client for 30 sec - exiting 04:20:21 (5268): No heartbeat from core client for 30 sec - exiting 04:20:22 (5268): No heartbeat from core client for 30 sec - exiting 04:20:23 (5268): No heartbeat from core client for 30 sec - exiting 04:20:24 (5268): No heartbeat from core client for 30 sec - exiting 04:20:25 (5268): No heartbeat from core client for 30 sec - exiting 04:20:26 (5268): No heartbeat from core client for 30 sec - exiting 04:20:27 (5268): No heartbeat from core client for 30 sec - exiting 04:20:28 (5268): No heartbeat from core client for 30 sec - exiting 04:20:29 (5268): No heartbeat from core client for 30 sec - exiting 04:20:30 (5268): No heartbeat from core client for 30 sec - exiting 04:20:31 (5268): No heartbeat from core client for 30 sec - exiting 04:20:32 (5268): No heartbeat from core client for 30 sec - exiting 04:20:33 (5268): No heartbeat from core client for 30 sec - exiting 04:20:34 (5268): No heartbeat from core client for 30 sec - exiting 04:20:36 (5268): No heartbeat from core client for 30 sec - exiting 04:20:37 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:30:16 (1552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:46:03 (1540): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:49:19 (5800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:29 (4020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:25:30 (4020): No heartbeat from core client for 30 sec - exiting 03:25:31 (4020): No heartbeat from core client for 30 sec - exiting 03:25:32 (4020): No heartbeat from core client for 30 sec - exiting 03:25:33 (4020): No heartbeat from core client for 30 sec - exiting 03:25:34 (4020): No heartbeat from core client for 30 sec - exiting 03:25:35 (4020): No heartbeat from core client for 30 sec - exiting 03:25:36 (4020): No heartbeat from core client for 30 sec - exiting 03:25:37 (4020): No heartbeat from core client for 30 sec - exiting 03:25:38 (4020): No heartbeat from core client for 30 sec - exiting 03:25:40 (4020): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4988, iMonCtr=1 Model crash detected, will try to restart... 00:07:58 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:28:30 (1672): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:30 (3708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:58:32 (3708): No heartbeat from core client for 30 sec - exiting 19:58:33 (3708): No heartbeat from core client for 30 sec - exiting 19:58:34 (3708): No heartbeat from core client for 30 sec - exiting 19:58:35 (3708): No heartbeat from core client for 30 sec - exiting 19:58:36 (3708): No heartbeat from core client for 30 sec - exiting 19:58:37 (3708): No heartbeat from core client for 30 sec - exiting 19:58:38 (3708): No heartbeat from core client for 30 sec - exiting 19:58:39 (3708): No heartbeat from core client for 30 sec - exiting 19:58:40 (3708): No heartbeat from core client for 30 sec - exiting 19:58:41 (3708): No heartbeat from core client for 30 sec - exiting 20:15:37 (3360): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:57:09 (4368): No heartbeat from core client for 30 sec - exiting 02:57:10 (4368): No heartbeat from core client for 30 sec - exiting 02:57:11 (4368): No heartbeat from core client for 30 sec - exiting 02:57:12 (4368): No heartbeat from core client for 30 sec - exiting 02:57:13 (4368): No heartbeat from core client for 30 sec - exiting 02:57:14 (4368): No heartbeat from core client for 30 sec - exiting 02:57:15 (4368): No heartbeat from core client for 30 sec - exiting 02:57:16 (4368): No heartbeat from core client for 30 sec - exiting 02:57:17 (4368): No heartbeat from core client for 30 sec - exiting 02:57:18 (4368): No heartbeat from core client for 30 sec - exiting 02:57:19 (4368): No heartbeat from core client for 30 sec - exiting 02:57:20 (4368): No heartbeat from core client for 30 sec - exiting 02:57:21 (4368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:57:22 (4368): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5112, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5112, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5112, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5112, iMonCtr=1 Model crash detected, will try to restart... 02:41:54 (4264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:43:39 (4620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4300, iMonCtr=1 Model crash detected, will try to restart... 12:49:29 (4300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:49:30 (4300): No heartbeat from core client for 30 sec - exiting 12:49:31 (4300): No heartbeat from core client for 30 sec - exiting 12:49:33 (4300): No heartbeat from core client for 30 sec - exiting 12:49:34 (4300): No heartbeat from core client for 30 sec - exiting 12:49:35 (4300): No heartbeat from core client for 30 sec - exiting 14:14:01 (6340): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=8212, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:38:47 (5336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CSuspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4824, iMonCtr=1 Model crash detected, will try to restart... 03:41:41 (5396): No heartbeat from core client for 30 sec - exiting 03:41:43 (5396): No heartbeat from core client for 30 sec - exiting 03:41:44 (5396): No heartbeat from core client for 30 sec - exiting 03:41:45 (5396): No heartbeat from core client for 30 sec - exiting 03:41:46 (5396): No heartbeat from core client for 30 sec - exiting 03:41:47 (5396): No heartbeat from core client for 30 sec - exiting 03:41:48 (5396): No heartbeat from core client for 30 sec - exiting 03:41:49 (5396): No heartbeat from core client for 30 sec - exiting 03:41:50 (5396): No heartbeat from core client for 30 sec - exiting 03:41:51 (5396): No heartbeat from core client for 30 sec - exiting 03:41:52 (5396): No heartbeat from core client for 30 sec - exiting 03:41:53 (5396): No heartbeat from core client for 30 sec - exiting 03:41:54 (5396): No heartbeat from core client for 30 sec - exiting 03:41:55 (5396): No heartbeat from core client for 30 sec - exiting 03:41:56 (5396): No heartbeat from core client for 30 sec - exiting 03:41:57 (5396): No heartbeat from core client for 30 sec - exiting 03:41:58 (5396): No heartbeat from core client for 30 sec - exiting 03:41:59 (5396): No heartbeat from core client for 30 sec - exiting 03:42:00 (5396): No heartbeat from core client for 30 sec - exiting 03:42:01 (5396): No heartbeat from core client for 30 sec - exiting 03:42:03 (5396): No heartbeat from core client for 30 sec - exiting 03:42:04 (5396): No heartbeat from core client for 30 sec - exiting 03:42:05 (5396): No heartbeat from core client for 30 sec - exiting 03:42:06 (5396): No heartbeat from core client for 30 sec - exiting 03:42:07 (5396): No heartbeat from core client for 30 sec - exiting 03:42:08 (5396): No heartbeat from core client for 30 sec - exiting 03:42:09 (5396): No heartbeat from core client for 30 sec - exiting 03:42:10 (5396): No heartbeat from core client for 30 sec - exiting 03:42:11 (5396): No heartbeat from core client for 30 sec - exiting 03:42:12 (5396): No heartbeat from core client for 30 sec - exiting 03:42:13 (5396): No heartbeat from core client for 30 sec - exiting 03:42:14 (5396): No heartbeat from core client for 30 sec - exiting 03:42:15 (5396): No heartbeat from core client for 30 sec - exiting 03:42:16 (5396): No heartbeat from core client for 30 sec - exiting 03:42:17 (5396): No heartbeat from core client for 30 sec - exiting 03:42:18 (5396): No heartbeat from core client for 30 sec - exiting 03:42:19 (5396): No heartbeat from core client for 30 sec - exiting 03:42:20 (5396): No heartbeat from core client for 30 sec - exiting 03:42:21 (5396): No heartbeat from core client for 30 sec - exiting 03:42:22 (5396): No heartbeat from core client for 30 sec - exiting 03:42:23 (5396): No heartbeat from core client for 30 sec - exiting 03:42:25 (5396): No heartbeat from core client for 30 sec - exiting 03:42:26 (5396): No heartbeat from core client for 30 sec - exiting 03:42:27 (5396): No heartbeat from core client for 30 sec - exiting 03:42:28 (5396): No heartbeat from core client for 30 sec - exiting 03:42:29 (5396): No heartbeat from core client for 30 sec - exiting 03:42:30 (5396): No heartbeat from core client for 30 sec - exiting 03:42:31 (5396): No heartbeat from core client for 30 sec - exiting 03:42:32 (5396): No heartbeat from core client for 30 sec - exiting 03:42:33 (5396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:42:35 (5396): No heartbeat from core client for 30 sec - exiting 03:42:36 (5396): No heartbeat from core client for 30 sec - exiting 03:42:37 (5396): No heartbeat from core client for 30 sec - exiting 03:42:38 (5396): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6532, iMonCtr=1 Model crash detected, will try to restart... 17:59:14 (3588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1264, iMonCtr=1 Model crash detected, will try to restart... 18:14:03 (3828): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:12:28 (5424): No heartbeat from core client for 30 sec - exiting 12:12:29 (5424): No heartbeat from core client for 30 sec - exiting 12:12:30 (5424): No heartbeat from core client for 30 sec - exiting 12:12:31 (5424): No heartbeat from core client for 30 sec - exiting 12:12:32 (5424): No heartbeat from core client for 30 sec - exiting 12:12:33 (5424): No heartbeat from core client for 30 sec - exiting 12:12:34 (5424): No heartbeat from core client for 30 sec - exiting 12:12:35 (5424): No heartbeat from core client for 30 sec - exiting 12:12:36 (5424): No heartbeat from core client for 30 sec - exiting 12:12:37 (5424): No heartbeat from core client for 30 sec - exiting 12:12:38 (5424): No heartbeat from core client for 30 sec - exiting 12:12:39 (5424): No heartbeat from core client for 30 sec - exiting 12:12:40 (5424): No heartbeat from core client for 30 sec - exiting 12:12:41 (5424): No heartbeat from core client for 30 sec - exiting 12:12:42 (5424): No heartbeat from core client for 30 sec - exiting 12:12:43 (5424): No heartbeat from core client for 30 sec - exiting 12:12:44 (5424): No heartbeat from core client for 30 sec - exiting 12:12:45 (5424): No heartbeat from core client for 30 sec - exiting 12:12:46 (5424): No heartbeat from core client for 30 sec - exiting 12:12:48 (5424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:27:14 (2392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:03:56 (2908): No heartbeat from core client for 30 sec - exiting 19:03:57 (2908): No heartbeat from core client for 30 sec - exiting 19:03:58 (2908): No heartbeat from core client for 30 sec - exiting 19:03:59 (2908): No heartbeat from core client for 30 sec - exiting 19:04:00 (2908): No heartbeat from core client for 30 sec - exiting 19:04:01 (2908): No heartbeat from core client for 30 sec - exiting 19:04:03 (2908): No heartbeat from core client for 30 sec - exiting 19:04:04 (2908): No heartbeat from core client for 30 sec - exiting 19:04:05 (2908): No heartbeat from core client for 30 sec - exiting 19:04:06 (2908): No heartbeat from core client for 30 sec - exiting 19:04:07 (2908): No heartbeat from core client for 30 sec - exiting 19:04:08 (2908): No heartbeat from core client for 30 sec - exiting 19:04:09 (2908): No heartbeat from core client for 30 sec - exiting 19:04:10 (2908): No heartbeat from core client for 30 sec - exiting 19:04:11 (2908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Nov 2013 14:55:05	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	1,036,800	3,065,363	2.9566
25 Nov 2013 20:52:29	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	1,010,880	2,985,514	2.9534
24 Nov 2013 15:13:08	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	984,960	2,897,993	2.9422
22 Nov 2013 04:29:36	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	959,040	2,819,802	2.9402
19 Nov 2013 07:43:35	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	933,120	2,741,418	2.9379
18 Nov 2013 06:44:16	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	907,200	2,662,542	2.9349
17 Nov 2013 07:48:12	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	881,280	2,583,490	2.9315
16 Nov 2013 09:10:28	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	855,360	2,505,045	2.9286
15 Nov 2013 04:29:58	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	829,440	2,426,770	2.9258
13 Nov 2013 23:31:12	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	803,520	2,347,492	2.9215
12 Nov 2013 14:24:24	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	777,600	2,267,220	2.9157
10 Nov 2013 22:59:16	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	751,680	2,185,974	2.9081
08 Nov 2013 06:11:34	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	725,760	2,109,277	2.9063
06 Nov 2013 15:07:01	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	699,840	2,106,551	3.0100
04 Nov 2013 06:13:28	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	673,920	2,026,442	3.0069
03 Nov 2013 06:52:38	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	648,000	1,946,648	3.0041
02 Nov 2013 07:17:41	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	622,080	1,866,253	3.0000
31 Oct 2013 15:41:23	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	596,160	1,786,711	2.9970
30 Oct 2013 01:39:46	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	570,240	1,708,204	2.9956
28 Oct 2013 12:54:09	1277518	16050652	hadcm3n_zht0_1960_40_008367554_4	544,320	1,629,822	2.9942