Task 14796825

Name	hadcm3n_y8ch_1980_40_007833912_3
Workunit	7989024
Created	18 Jun 2012, 21:36:17 UTC
Sent	18 Jun 2012, 21:36:32 UTC
Report deadline	18 Sep 2012, 5:03:43 UTC
Received	11 Jul 2012, 9:11:49 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	763269
Run time	14 days 19 hours 52 min 27 sec
CPU time	11 days 14 hours 51 min 34 sec
Validate state	Invalid
Credit	6,842.88
Device peak FLOPS	2.76 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> 06:23:31 (3324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:23:33 (3324): No heartbeat from core client for 30 sec - exiting 06:23:34 (3324): No heartbeat from core client for 30 sec - exiting 06:23:35 (3324): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 07:41:41 (7228): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 07:41:42 (7228): No heartbeat from core client for 30 sec - exiting 08:41:49 (17879): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:41:50 (17879): No heartbeat from core client for 30 sec - exiting 08:41:51 (17879): No heartbeat from core client for 30 sec - exiting 08:41:52 (17879): No heartbeat from core client for 30 sec - exiting 08:41:53 (17879): No heartbeat from core client for 30 sec - exiting 08:46:09 (18616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 17:01:26 (6101): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:01:28 (6101): No heartbeat from core client for 30 sec - exiting 17:01:29 (6101): No heartbeat from core client for 30 sec - exiting 17:01:30 (6101): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 23:47:03 (16801): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 23:47:04 (16801): No heartbeat from core client for 30 sec - exiting 23:47:05 (16801): No heartbeat from core client for 30 sec - exiting 23:47:06 (16801): No heartbeat from core client for 30 sec - exiting 23:47:07 (16801): No heartbeat from core client for 30 sec - exiting 23:47:08 (16801): No heartbeat from core client for 30 sec - exiting 03:18:30 (8135): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:01:18 (29345): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:01:20 (29345): No heartbeat from core client for 30 sec - exiting 14:01:21 (29345): No heartbeat from core client for 30 sec - exiting 14:01:22 (29345): No heartbeat from core client for 30 sec - exiting 14:01:23 (29345): No heartbeat from core client for 30 sec - exiting 14:01:24 (29345): No heartbeat from core client for 30 sec - exiting 14:01:25 (29345): No heartbeat from core client for 30 sec - exiting 14:01:26 (29345): No heartbeat from core client for 30 sec - exiting 14:01:27 (29345): No heartbeat from core client for 30 sec - exiting 14:01:28 (29345): No heartbeat from core client for 30 sec - exiting 14:01:29 (29345): No heartbeat from core client for 30 sec - exiting 14:01:30 (29345): No heartbeat from core client for 30 sec - exiting 14:01:31 (29345): No heartbeat from core client for 30 sec - exiting 14:01:32 (29345): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 21:56:11 (478): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 03:43:15 (6852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:16 (6852): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 05:31:18 (31564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:31:38 (31564): No heartbeat from core client for 30 sec - exiting 05:31:39 (31564): No heartbeat from core client for 30 sec - exiting 05:31:40 (31564): No heartbeat from core client for 30 sec - exiting 05:31:41 (31564): No heartbeat from core client for 30 sec - exiting 05:31:43 (31564): No heartbeat from core client for 30 sec - exiting 05:31:44 (31564): No heartbeat from core client for 30 sec - exiting 05:31:45 (31564): No heartbeat from core client for 30 sec - exiting 05:31:46 (31564): No heartbeat from core client for 30 sec - exiting 05:31:47 (31564): No heartbeat from core client for 30 sec - exiting 05:31:48 (31564): No heartbeat from core client for 30 sec - exiting 05:31:49 (31564): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:06:10 (6297): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 23:06:13 (6297): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 18:39:30 (6130): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:53 (14652): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:40:57 (14652): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 04:54:58 (6208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:29:21 (25030): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 08:01:10 (28796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:01:11 (28796): No heartbeat from core client for 30 sec - exiting 08:01:12 (28796): No heartbeat from core client for 30 sec - exiting 08:01:13 (28796): No heartbeat from core client for 30 sec - exiting 08:01:14 (28796): No heartbeat from core client for 30 sec - exiting 08:01:15 (28796): No heartbeat from core client for 30 sec - exiting 08:01:16 (28796): No heartbeat from core client for 30 sec - exiting 08:01:17 (28796): No heartbeat from core client for 30 sec - exiting 08:01:18 (28796): No heartbeat from core client for 30 sec - exiting 08:01:19 (28796): No heartbeat from core client for 30 sec - exiting 08:01:20 (28796): No heartbeat from core client for 30 sec - exiting 08:01:21 (28796): No heartbeat from core client for 30 sec - exiting 08:01:22 (28796): No heartbeat from core client for 30 sec - exiting 08:01:23 (28796): No heartbeat from core client for 30 sec - exiting 08:01:24 (28796): No heartbeat from core client for 30 sec - exiting 08:01:25 (28796): No heartbeat from core client for 30 sec - exiting 08:01:26 (28796): No heartbeat from core client for 30 sec - exiting 08:01:27 (28796): No heartbeat from core client for 30 sec - exiting 08:01:28 (28796): No heartbeat from core client for 30 sec - exiting 08:01:29 (28796): No heartbeat from core client for 30 sec - exiting 08:01:30 (28796): No heartbeat from core client for 30 sec - exiting 08:01:31 (28796): No heartbeat from core client for 30 sec - exiting 08:01:32 (28796): No heartbeat from core client for 30 sec - exiting 08:01:33 (28796): No heartbeat from core client for 30 sec - exiting 08:01:34 (28796): No heartbeat from core client for 30 sec - exiting 08:01:35 (28796): No heartbeat from core client for 30 sec - exiting 08:01:36 (28796): No heartbeat from core client for 30 sec - exiting 08:36:56 (11840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:34:59 (26558): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:24:55 (11337): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:52:59 (31281): No heartbeat from core client for 30 sec - exiting 11:53:00 (31281): No heartbeat from core client for 30 sec - exiting 11:53:01 (31281): No heartbeat from core client for 30 sec - exiting 11:53:02 (31281): No heartbeat from core client for 30 sec - exiting 11:53:03 (31281): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:29 (12660): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:30 (12660): No heartbeat from core client for 30 sec - exiting 12:02:31 (12660): No heartbeat from core client for 30 sec - exiting 12:02:32 (12660): No heartbeat from core client for 30 sec - exiting 12:02:33 (12660): No heartbeat from core client for 30 sec - exiting 12:02:34 (12660): No heartbeat from core client for 30 sec - exiting Atmos Hold Restart file rename failed on atmos_restart.hold 12:16:20 (16749): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:19:30 (23384): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:19:31 (23384): No heartbeat from core client for 30 sec - exiting SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf775e400] [0xf775e430] /lib32/libc.so.6(gsignal+0x51)[0xf75c6391] /lib32/libc.so.6(abort+0x17e)[0xf75c7b6e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75b23c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23941, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf77be400] [0xf77be430] /lib32/libc.so.6(gsignal+0x51)[0xf7626391] /lib32/libc.so.6(abort+0x17e)[0xf7627b6e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf76123c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23941, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7731400] [0xf7731430] /lib32/libc.so.6(gsignal+0x51)[0xf7599391] /lib32/libc.so.6(abort+0x17e)[0xf759ab6e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75853c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23941, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7706400] [0xf7706430] /lib32/libc.so.6(gsignal+0x51)[0xf756e391] /lib32/libc.so.6(abort+0x17e)[0xf756fb6e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf755a3c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23941, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf7725400] [0xf7725430] /lib32/libc.so.6(gsignal+0x51)[0xf758d391] /lib32/libc.so.6(abort+0x17e)[0xf758eb6e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf75793c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23941, iMonCtr=1 Model crash detected, will try to restart... SIGABRT: abort called Stack trace (10 frames): /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x840da8f] [0xf771a400] [0xf771a430] /lib32/libc.so.6(gsignal+0x51)[0xf7582391] /lib32/libc.so.6(abort+0x17e)[0xf7583b6e] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x83400c3] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x838f395] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x839bdf8] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf756e3c6] /var/lib/boinc/projects/climateprediction.net/hadcm3n_um_6.07_i686-pc-linux-gnu[0x804cb11] Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23941, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
11 Jul 2012 03:57:39	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	570,240	1,003,049	1.7590
09 Jul 2012 14:30:55	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	544,320	957,970	1.7599
08 Jul 2012 09:40:08	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	518,400	911,363	1.7580
07 Jul 2012 13:06:02	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	492,480	864,891	1.7562
06 Jul 2012 17:54:35	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	466,560	818,971	1.7553
05 Jul 2012 08:48:28	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	440,640	772,967	1.7542
04 Jul 2012 08:51:51	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	414,720	726,575	1.7520
03 Jul 2012 09:23:09	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	388,800	680,082	1.7492
02 Jul 2012 11:13:05	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	362,880	633,940	1.7470
02 Jul 2012 11:13:05	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	336,960	587,697	1.7441
30 Jun 2012 09:52:48	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	311,040	541,428	1.7407
29 Jun 2012 10:59:55	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	285,120	495,410	1.7375
28 Jun 2012 11:21:25	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	259,200	449,176	1.7329
27 Jun 2012 13:07:07	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	233,280	404,972	1.7360
26 Jun 2012 10:57:53	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	207,360	363,912	1.7550
25 Jun 2012 09:16:00	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	181,440	318,034	1.7528
24 Jun 2012 07:23:15	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	155,520	271,821	1.7478
23 Jun 2012 15:35:58	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	129,600	225,909	1.7431
22 Jun 2012 20:13:14	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	103,680	180,142	1.7375
21 Jun 2012 19:44:09	763269	14796825	hadcm3n_y8ch_1980_40_007833912_3	77,760	134,570	1.7306