Task 17803146

Name	hadcm3n_saws_1980_40_009466990_2
Workunit	9548733
Created	16 Jan 2015, 16:04:36 UTC
Sent	16 Jan 2015, 18:42:18 UTC
Report deadline	18 Apr 2015, 2:09:29 UTC
Received	20 Mar 2015, 7:43:28 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	139 (0x0000008B) Unknown error code
Computer ID	1326407
Run time	7 days 11 hours 6 min 8 sec
CPU time	6 days 13 hours 45 min 30 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.16 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process got signal 11 </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:33:00 (180743): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 10:50:17 (6229): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:23:14 (4354): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x089e06e8 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08891218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09064218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09583218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09e49218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09f6f218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08f90218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09a74218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0a0e8218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 13:19:38 (3519): No heartbeat from core client for 30 sec - exiting 13:19:47 (3519): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09ee2218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08e4f218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09220218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09cc0218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08a87218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08ea9218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09d4e218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0a02f218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09f0a218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09173218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09b7d218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0906c218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08d2c218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09a7d218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09d0e218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09045218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08ed2218 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Feb 2015 14:11:23	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	259,200	535,381	2.0655
03 Feb 2015 22:03:18	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	233,280	486,627	2.0860
03 Feb 2015 05:44:29	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	207,360	440,209	2.1229
02 Feb 2015 12:32:39	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	181,440	390,995	2.1550
01 Feb 2015 11:18:34	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	155,520	311,930	2.0057
30 Jan 2015 17:16:43	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	129,600	256,299	1.9776
29 Jan 2015 21:59:51	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	103,680	203,405	1.9619
29 Jan 2015 02:24:36	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	77,760	155,492	1.9996
27 Jan 2015 01:57:37	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	51,840	103,838	2.0030
25 Jan 2015 01:13:59	1326407	17803146	hadcm3n_saws_1980_40_009466990_2	25,920	51,798	1.9984