Task 19549319

Name	hadcm3n_sdhv_198012_480_375_010415310_2
Workunit	10415310
Created	16 Apr 2016, 5:34:47 UTC
Sent	16 Apr 2016, 5:36:38 UTC
Report deadline	29 Mar 2017, 10:56:38 UTC
Received	27 Sep 2016, 12:11:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1311976
Run time	20 days 11 hours 46 min 51 sec
CPU time	19 days 15 hours 2 min 48 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.68 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Peak working set size	204.36 MB
Peak swap size	1,087.87 MB
Peak disk usage	4.51 MB
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x3800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x1004e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(36991,0xa08351d4) malloc: * error for object 0x1004e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x701f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x703ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x703ac00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x701f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x703ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x703ac00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x5832204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x6825004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x6825000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.10.5 build 14F1808 hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x882ee04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x882ee00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Tue Sep 27 07:09:16 2016 hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x1020604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x1020600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x2021604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53502,0xa04a61d4) malloc: * error for object 0x2021600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9a8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/0/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90090000 - 0x9009efff /usr/lib/libz.1.dylib 0x9009f000 - 0x900a8fff /usr/lib/system/libcopyfile.dylib 0x900a9000 - 0x900b1fff /usr/lib/system/libsystem_pthread.dylib 0x900d6000 - 0x900d8fff /usr/lib/system/libsystem_coreservices.dylib 0x902e4000 - 0x90337fff /usr/lib/libstdc++.6.dylib 0x9068a000 - 0x906bdfff /usr/lib/system/libsystem_m.dylib 0x906c1000 - 0x906c7fff /usr/lib/system/libsystem_trace.dylib 0x9083a000 - 0x908a6fff /usr/lib/system/libcorecrypto.dylib 0x90abb000 - 0x90abdfff /usr/lib/system/libquarantine.dylib 0x91d2b000 - 0x91d42fff /usr/lib/system/libsystem_asl.dylib 0x9237e000 - 0x92384fff /usr/lib/system/libsystem_networkextension.dylib 0x92905000 - 0x92906fff /usr/lib/libSystem.B.dylib 0x92fe8000 - 0x92ff1fff /usr/lib/system/libsystem_notify.dylib 0x93299000 - 0x932a5fff /usr/lib/system/libcommonCrypto.dylib 0x940ec000 - 0x940fdfff /usr/lib/system/libsystem_coretls.dylib 0x946db000 - 0x946e1fff /usr/lib/system/libsystem_platform.dylib 0x948ba000 - 0x948c2fff /usr/lib/system/libsystem_dnssd.dylib 0x949d2000 - 0x949d9fff /usr/lib/system/libunwind.dylib 0x97b9f000 - 0x97bbefff /usr/lib/system/libsystem_kernel.dylib 0x97bbf000 - 0x97bd8fff /usr/lib/system/libsystem_malloc.dylib 0x97ea2000 - 0x97ea5fff /usr/lib/system/libdyld.dylib 0x98361000 - 0x98361fff /usr/lib/system/libkeymgr.dylib 0x983e2000 - 0x983e4fff /usr/lib/system/libsystem_configuration.dylib 0x983e5000 - 0x983e5fff /usr/lib/system/liblaunch.dylib 0x99033000 - 0x990c8fff /usr/lib/system/libsystem_c.dylib 0x990d8000 - 0x99115fff /usr/lib/system/libsystem_network.dylib 0x998a6000 - 0x998a8fff /usr/lib/system/libsystem_secinit.dylib 0x99d4a000 - 0x99d4bfff /usr/lib/system/libsystem_blocks.dylib 0x9a8fd000 - 0x9a902fff /usr/lib/system/libmacho.dylib 0x9ae7e000 - 0x9ae82fff /usr/lib/system/libcache.dylib 0x9b2af000 - 0x9b2d8fff /usr/lib/system/libsystem_info.dylib 0x9b3b2000 - 0x9b3d8fff /usr/lib/libc++abi.dylib 0x9b43a000 - 0x9b460fff /usr/lib/system/libxpc.dylib 0x9ba1b000 - 0x9ba1bfff /usr/lib/system/libunc.dylib 0x9ba1c000 - 0x9ba21fff /usr/lib/system/libcompiler_rt.dylib 0x9ba33000 - 0x9ba35fff /usr/lib/system/libsystem_sandbox.dylib 0x9ba36000 - 0x9ba37fff /usr/lib/system/libremovefile.dylib 0x9bcaa000 - 0x9bcd1fff /usr/lib/system/libdispatch.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Sep 2016 12:11:15	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	1,036,800	1,695,759	1.6356
26 Sep 2016 21:13:42	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	1,010,880	1,653,587	1.6358
26 Sep 2016 06:01:16	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	984,960	1,611,417	1.6360
25 Sep 2016 16:13:29	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	959,040	1,568,998	1.6360
24 Sep 2016 22:14:50	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	933,120	1,526,769	1.6362
24 Sep 2016 09:57:12	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	907,200	1,484,429	1.6363
23 Sep 2016 18:33:49	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	881,280	1,442,151	1.6364
23 Sep 2016 06:11:20	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	855,360	1,399,413	1.6361
22 Sep 2016 17:31:42	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	829,440	1,355,831	1.6346
21 Sep 2016 23:58:20	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	803,520	1,313,230	1.6343
19 May 2016 21:54:31	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	777,600	1,267,261	1.6297
19 May 2016 08:57:22	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	751,680	1,222,598	1.6265
18 May 2016 20:41:24	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	725,760	1,180,268	1.6263
18 May 2016 08:04:56	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	699,840	1,136,747	1.6243
17 May 2016 19:32:30	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	673,920	1,093,424	1.6225
17 May 2016 07:13:53	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	648,000	1,050,938	1.6218
15 May 2016 16:10:39	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	622,080	1,002,216	1.6111
15 May 2016 03:06:51	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	596,160	957,327	1.6058
13 May 2016 20:07:41	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	570,240	915,618	1.6057
13 May 2016 07:13:13	1311976	19549319	hadcm3n_sdhv_198012_480_375_010415310_2	544,320	871,133	1.6004