Task 15823320

Name	hadcm3n_o4f2_1980_40_008387476_0
Workunit	8538335
Created	3 Jun 2013, 10:04:01 UTC
Sent	7 Jun 2013, 13:25:31 UTC
Report deadline	6 Sep 2013, 20:52:42 UTC
Received	31 Aug 2013, 15:30:40 UTC
Server state	Over
Outcome	Computation error
Client state	Aborted by user
Exit status	203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID	1279288
Run time	7 days 2 hours 45 min 18 sec
CPU time	6 days 3 hours 38 min 6 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.39 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:40:24 (3404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:52:24 (3242): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:33:22 (3330): No heartbeat from core client for 30 sec - exiting 21:33:23 (3330): No heartbeat from core client for 30 sec - exiting 21:33:24 (3330): No heartbeat from core client for 30 sec - exiting 21:33:25 (3330): No heartbeat from core client for 30 sec - exiting 21:33:26 (3330): No heartbeat from core client for 30 sec - exiting 21:33:27 (3330): No heartbeat from core client for 30 sec - exiting 21:33:28 (3330): No heartbeat from core client for 30 sec - exiting 21:33:29 (3330): No heartbeat from core client for 30 sec - exiting 21:33:30 (3330): No heartbeat from core client for 30 sec - exiting 21:33:31 (3330): No heartbeat from core client for 30 sec - exiting 21:33:32 (3330): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:58:15 (3292): No heartbeat from core client for 30 sec - exiting 21:58:16 (3292): No heartbeat from core client for 30 sec - exiting 21:58:17 (3292): No heartbeat from core client for 30 sec - exiting 21:58:18 (3292): No heartbeat from core client for 30 sec - exiting 21:58:19 (3292): No heartbeat from core client for 30 sec - exiting 21:58:20 (3292): No heartbeat from core client for 30 sec - exiting 21:58:21 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:58:22 (3292): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:11:03 (3392): No heartbeat from core client for 30 sec - exiting 23:11:04 (3392): No heartbeat from core client for 30 sec - exiting 23:11:05 (3392): No heartbeat from core client for 30 sec - exiting 23:11:06 (3392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 10:02:16 (3433): No heartbeat from core client for 30 sec - exiting 10:02:17 (3433): No heartbeat from core client for 30 sec - exiting 10:02:18 (3433): No heartbeat from core client for 30 sec - exiting 10:02:19 (3433): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 22:56:28 (3417): No heartbeat from core client for 30 sec - exiting 22:56:30 (3417): No heartbeat from core client for 30 sec - exiting 22:56:31 (3417): No heartbeat from core client for 30 sec - exiting 22:56:32 (3417): No heartbeat from core client for 30 sec - exiting 22:56:33 (3417): No heartbeat from core client for 30 sec - exiting 22:56:34 (3417): No heartbeat from core client for 30 sec - exiting 22:56:35 (3417): No heartbeat from core client for 30 sec - exiting 22:56:36 (3417): No heartbeat from core client for 30 sec - exiting 22:56:37 (3417): No heartbeat from core client for 30 sec - exiting 22:56:38 (3417): No heartbeat from core client for 30 sec - exiting 22:56:39 (3417): No heartbeat from core client for 30 sec - exiting 22:56:40 (3417): No heartbeat from core client for 30 sec - exiting 22:56:41 (3417): No heartbeat from core client for 30 sec - exiting 22:56:42 (3417): No heartbeat from core client for 30 sec - exiting 22:56:43 (3417): No heartbeat from core client for 30 sec - exiting 22:56:44 (3417): No heartbeat from core client for 30 sec - exiting 22:56:45 (3417): No heartbeat from core client for 30 sec - exiting 22:56:46 (3417): No heartbeat from core client for 30 sec - exiting 22:56:47 (3417): No heartbeat from core client for 30 sec - exiting 22:56:48 (3417): No heartbeat from core client for 30 sec - exiting 22:56:49 (3417): No heartbeat from core client for 30 sec - exiting 22:56:50 (3417): No heartbeat from core client for 30 sec - exiting 22:56:51 (3417): No heartbeat from core client for 30 sec - exiting 22:56:52 (3417): No heartbeat from core client for 30 sec - exiting 22:56:53 (3417): No heartbeat from core client for 30 sec - exiting 22:56:54 (3417): No heartbeat from core client for 30 sec - exiting 22:56:55 (3417): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:08:54 (3414): No heartbeat from core client for 30 sec - exiting 23:08:55 (3414): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:53:59 (3475): No heartbeat from core client for 30 sec - exiting 19:54:00 (3475): No heartbeat from core client for 30 sec - exiting 19:54:01 (3475): No heartbeat from core client for 30 sec - exiting 19:54:02 (3475): No heartbeat from core client for 30 sec - exiting 19:54:03 (3475): No heartbeat from core client for 30 sec - exiting 19:54:04 (3475): No heartbeat from core client for 30 sec - exiting 19:54:05 (3475): No heartbeat from core client for 30 sec - exiting 19:54:06 (3475): No heartbeat from core client for 30 sec - exiting 19:54:07 (3475): No heartbeat from core client for 30 sec - exiting 19:54:08 (3475): No heartbeat from core client for 30 sec - exiting 19:54:09 (3475): No heartbeat from core client for 30 sec - exiting 19:54:10 (3475): No heartbeat from core client for 30 sec - exiting 19:54:11 (3475): No heartbeat from core client for 30 sec - exiting 19:54:12 (3475): No heartbeat from core client for 30 sec - exiting 19:54:13 (3475): No heartbeat from core client for 30 sec - exiting 19:54:14 (3475): No heartbeat from core client for 30 sec - exiting 19:54:15 (3475): No heartbeat from core client for 30 sec - exiting 19:54:16 (3475): No heartbeat from core client for 30 sec - exiting 19:54:17 (3475): No heartbeat from core client for 30 sec - exiting 19:54:18 (3475): No heartbeat from core client for 30 sec - exiting 19:54:19 (3475): No heartbeat from core client for 30 sec - exiting 19:54:20 (3475): No heartbeat from core client for 30 sec - exiting 19:54:21 (3475): No heartbeat from core client for 30 sec - exiting 19:54:22 (3475): No heartbeat from core client for 30 sec - exiting 19:54:23 (3475): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08db46a8 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0947cf60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09ae1f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08babf60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x083e3f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09ff6f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08366f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 22:25:35 (3352): No heartbeat from core client for 30 sec - exiting 22:25:36 (3352): No heartbeat from core client for 30 sec - exiting 22:25:37 (3352): No heartbeat from core client for 30 sec - exiting 22:25:38 (3352): No heartbeat from core client for 30 sec - exiting 22:25:39 (3352): No heartbeat from core client for 30 sec - exiting 22:25:40 (3352): No heartbeat from core client for 30 sec - exiting 22:25:41 (3352): No heartbeat from core client for 30 sec - exiting 22:25:42 (3352): No heartbeat from core client for 30 sec - exiting 00:52:28 (3352): No heartbeat from core client for 30 sec - exiting 00:52:29 (3352): No heartbeat from core client for 30 sec - exiting 00:52:30 (3352): No heartbeat from core client for 30 sec - exiting 00:52:31 (3352): No heartbeat from core client for 30 sec - exiting 00:52:32 (3352): No heartbeat from core client for 30 sec - exiting 00:52:33 (3352): No heartbeat from core client for 30 sec - exiting 00:52:34 (3352): No heartbeat from core client for 30 sec - exiting 00:52:35 (3352): No heartbeat from core client for 30 sec - exiting 00:52:36 (3352): No heartbeat from core client for 30 sec - exiting 00:52:37 (3352): No heartbeat from core client for 30 sec - exiting 00:52:38 (3352): No heartbeat from core client for 30 sec - exiting 00:53:09 (3352): No heartbeat from core client for 30 sec - exiting 00:53:10 (3352): No heartbeat from core client for 30 sec - exiting 00:53:11 (3352): No heartbeat from core client for 30 sec - exiting 00:53:12 (3352): No heartbeat from core client for 30 sec - exiting 00:53:13 (3352): No heartbeat from core client for 30 sec - exiting 00:53:14 (3352): No heartbeat from core client for 30 sec - exiting 00:53:15 (3352): No heartbeat from core client for 30 sec - exiting 00:53:16 (3352): No heartbeat from core client for 30 sec - exiting 00:53:17 (3352): No heartbeat from core client for 30 sec - exiting 00:53:18 (3352): No heartbeat from core client for 30 sec - exiting 00:53:55 (3352): No heartbeat from core client for 30 sec - exiting 00:53:56 (3352): No heartbeat from core client for 30 sec - exiting 00:53:57 (3352): No heartbeat from core client for 30 sec - exiting 00:53:58 (3352): No heartbeat from core client for 30 sec - exiting 00:53:59 (3352): No heartbeat from core client for 30 sec - exiting 00:54:00 (3352): No heartbeat from core client for 30 sec - exiting 00:54:01 (3352): No heartbeat from core client for 30 sec - exiting 00:54:02 (3352): No heartbeat from core client for 30 sec - exiting 00:54:03 (3352): No heartbeat from core client for 30 sec - exiting 00:54:04 (3352): No heartbeat from core client for 30 sec - exiting 00:54:05 (3352): No heartbeat from core client for 30 sec - exiting 00:54:45 (3352): No heartbeat from core client for 30 sec - exiting 00:54:46 (3352): No heartbeat from core client for 30 sec - exiting 00:54:47 (3352): No heartbeat from core client for 30 sec - exiting 00:54:48 (3352): No heartbeat from core client for 30 sec - exiting 00:54:49 (3352): No heartbeat from core client for 30 sec - exiting 00:54:50 (3352): No heartbeat from core client for 30 sec - exiting 00:54:51 (3352): No heartbeat from core client for 30 sec - exiting 00:54:52 (3352): No heartbeat from core client for 30 sec - exiting 00:54:53 (3352): No heartbeat from core client for 30 sec - exiting 00:54:54 (3352): No heartbeat from core client for 30 sec - exiting 00:54:55 (3352): No heartbeat from core client for 30 sec - exiting 00:54:56 (3352): No heartbeat from core client for 30 sec - exiting 00:54:57 (3352): No heartbeat from core client for 30 sec - exiting 00:54:58 (3352): No heartbeat from core client for 30 sec - exiting 00:54:59 (3352): No heartbeat from core client for 30 sec - exiting 00:55:00 (3352): No heartbeat from core client for 30 sec - exiting 00:55:01 (3352): No heartbeat from core client for 30 sec - exiting 00:55:02 (3352): No heartbeat from core client for 30 sec - exiting 00:55:03 (3352): No heartbeat from core client for 30 sec - exiting 00:55:04 (3352): No heartbeat from core client for 30 sec - exiting 00:55:05 (3352): No heartbeat from core client for 30 sec - exiting 00:55:06 (3352): No heartbeat from core client for 30 sec - exiting 00:55:07 (3352): No heartbeat from core client for 30 sec - exiting 00:55:08 (3352): No heartbeat from core client for 30 sec - exiting 00:55:09 (3352): No heartbeat from core client for 30 sec - exiting 00:55:10 (3352): No heartbeat from core client for 30 sec - exiting 00:55:11 (3352): No heartbeat from core client for 30 sec - exiting 00:55:12 (3352): No heartbeat from core client for 30 sec - exiting 00:55:13 (3352): No heartbeat from core client for 30 sec - exiting 00:55:14 (3352): No heartbeat from core client for 30 sec - exiting 00:55:15 (3352): No heartbeat from core client for 30 sec - exiting 00:55:16 (3352): No heartbeat from core client for 30 sec - exiting 00:55:17 (3352): No heartbeat from core client for 30 sec - exiting 00:55:18 (3352): No heartbeat from core client for 30 sec - exiting 00:55:19 (3352): No heartbeat from core client for 30 sec - exiting 00:55:20 (3352): No heartbeat from core client for 30 sec - exiting 00:55:21 (3352): No heartbeat from core client for 30 sec - exiting 00:55:22 (3352): No heartbeat from core client for 30 sec - exiting 00:55:23 (3352): No heartbeat from core client for 30 sec - exiting 00:55:24 (3352): No heartbeat from core client for 30 sec - exiting 00:55:25 (3352): No heartbeat from core client for 30 sec - exiting 00:55:26 (3352): No heartbeat from core client for 30 sec - exiting 00:55:27 (3352): No heartbeat from core client for 30 sec - exiting 00:55:28 (3352): No heartbeat from core client for 30 sec - exiting 00:55:29 (3352): No heartbeat from core client for 30 sec - exiting 00:55:30 (3352): No heartbeat from core client for 30 sec - exiting 00:55:31 (3352): No heartbeat from core client for 30 sec - exiting 00:55:32 (3352): No heartbeat from core client for 30 sec - exiting 00:55:33 (3352): No heartbeat from core client for 30 sec - exiting 00:55:34 (3352): No heartbeat from core client for 30 sec - exiting 00:55:35 (3352): No heartbeat from core client for 30 sec - exiting 00:56:06 (3352): No heartbeat from core client for 30 sec - exiting 00:56:07 (3352): No heartbeat from core client for 30 sec - exiting 00:56:08 (3352): No heartbeat from core client for 30 sec - exiting 00:56:09 (3352): No heartbeat from core client for 30 sec - exiting 00:56:10 (3352): No heartbeat from core client for 30 sec - exiting 00:56:11 (3352): No heartbeat from core client for 30 sec - exiting 00:56:12 (3352): No heartbeat from core client for 30 sec - exiting 00:56:13 (3352): No heartbeat from core client for 30 sec - exiting 00:56:14 (3352): No heartbeat from core client for 30 sec - exiting 00:56:15 (3352): No heartbeat from core client for 30 sec - exiting 00:56:16 (3352): No heartbeat from core client for 30 sec - exiting 00:56:17 (3352): No heartbeat from core client for 30 sec - exiting 00:56:18 (3352): No heartbeat from core client for 30 sec - exiting 00:56:19 (3352): No heartbeat from core client for 30 sec - exiting 00:56:20 (3352): No heartbeat from core client for 30 sec - exiting 00:56:21 (3352): No heartbeat from core client for 30 sec - exiting 00:56:22 (3352): No heartbeat from core client for 30 sec - exiting 00:56:23 (3352): No heartbeat from core client for 30 sec - exiting 00:56:24 (3352): No heartbeat from core client for 30 sec - exiting 00:56:25 (3352): No heartbeat from core client for 30 sec - exiting 00:56:26 (3352): No heartbeat from core client for 30 sec - exiting 00:56:27 (3352): No heartbeat from core client for 30 sec - exiting 00:56:28 (3352): No heartbeat from core client for 30 sec - exiting 00:56:29 (3352): No heartbeat from core client for 30 sec - exiting 00:56:30 (3352): No heartbeat from core client for 30 sec - exiting 00:56:31 (3352): No heartbeat from core client for 30 sec - exiting 00:56:32 (3352): No heartbeat from core client for 30 sec - exiting 00:56:33 (3352): No heartbeat from core client for 30 sec - exiting 00:56:34 (3352): No heartbeat from core client for 30 sec - exiting 00:56:35 (3352): No heartbeat from core client for 30 sec - exiting 00:56:36 (3352): No heartbeat from core client for 30 sec - exiting 00:56:37 (3352): No heartbeat from core client for 30 sec - exiting 00:56:38 (3352): No heartbeat from core client for 30 sec - exiting 00:56:39 (3352): No heartbeat from core client for 30 sec - exiting 00:56:40 (3352): No heartbeat from core client for 30 sec - exiting 00:56:41 (3352): No heartbeat from core client for 30 sec - exiting 00:56:42 (3352): No heartbeat from core client for 30 sec - exiting 00:56:43 (3352): No heartbeat from core client for 30 sec - exiting 00:56:44 (3352): No heartbeat from core client for 30 sec - exiting 00:56:45 (3352): No heartbeat from core client for 30 sec - exiting 00:56:46 (3352): No heartbeat from core client for 30 sec - exiting 00:56:47 (3352): No heartbeat from core client for 30 sec - exiting 00:56:48 (3352): No heartbeat from core client for 30 sec - exiting 00:56:49 (3352): No heartbeat from core client for 30 sec - exiting 00:56:50 (3352): No heartbeat from core client for 30 sec - exiting 00:56:51 (3352): No heartbeat from core client for 30 sec - exiting 00:56:52 (3352): No heartbeat from core client for 30 sec - exiting 00:56:53 (3352): No heartbeat from core client for 30 sec - exiting 00:56:54 (3352): No heartbeat from core client for 30 sec - exiting 00:56:55 (3352): No heartbeat from core client for 30 sec - exiting 00:57:26 (3352): No heartbeat from core client for 30 sec - exiting 00:57:27 (3352): No heartbeat from core client for 30 sec - exiting 00:57:28 (3352): No heartbeat from core client for 30 sec - exiting 00:57:29 (3352): No heartbeat from core client for 30 sec - exiting 00:57:30 (3352): No heartbeat from core client for 30 sec - exiting 00:57:31 (3352): No heartbeat from core client for 30 sec - exiting 00:57:32 (3352): No heartbeat from core client for 30 sec - exiting 00:57:33 (3352): No heartbeat from core client for 30 sec - exiting 00:57:34 (3352): No heartbeat from core client for 30 sec - exiting 00:57:35 (3352): No heartbeat from core client for 30 sec - exiting 00:57:36 (3352): No heartbeat from core client for 30 sec - exiting 00:57:37 (3352): No heartbeat from core client for 30 sec - exiting 00:57:38 (3352): No heartbeat from core client for 30 sec - exiting 00:57:39 (3352): No heartbeat from core client for 30 sec - exiting 00:57:40 (3352): No heartbeat from core client for 30 sec - exiting 00:57:41 (3352): No heartbeat from core client for 30 sec - exiting 00:57:42 (3352): No heartbeat from core client for 30 sec - exiting 00:57:43 (3352): No heartbeat from core client for 30 sec - exiting 00:57:44 (3352): No heartbeat from core client for 30 sec - exiting 00:57:45 (3352): No heartbeat from core client for 30 sec - exiting 00:57:46 (3352): No heartbeat from core client for 30 sec - exiting 00:57:47 (3352): No heartbeat from core client for 30 sec - exiting 00:57:48 (3352): No heartbeat from core client for 30 sec - exiting 00:57:49 (3352): No heartbeat from core client for 30 sec - exiting 00:57:50 (3352): No heartbeat from core client for 30 sec - exiting 00:57:52 (3352): No heartbeat from core client for 30 sec - exiting 00:57:53 (3352): No heartbeat from core client for 30 sec - exiting 00:57:54 (3352): No heartbeat from core client for 30 sec - exiting 00:57:55 (3352): No heartbeat from core client for 30 sec - exiting 00:57:56 (3352): No heartbeat from core client for 30 sec - exiting 00:57:57 (3352): No heartbeat from core client for 30 sec - exiting 00:57:58 (3352): No heartbeat from core client for 30 sec - exiting 00:57:59 (3352): No heartbeat from core client for 30 sec - exiting 00:58:00 (3352): No heartbeat from core client for 30 sec - exiting 00:58:01 (3352): No heartbeat from core client for 30 sec - exiting 00:58:02 (3352): No heartbeat from core client for 30 sec - exiting 00:58:03 (3352): No heartbeat from core client for 30 sec - exiting 00:58:04 (3352): No heartbeat from core client for 30 sec - exiting 00:58:05 (3352): No heartbeat from core client for 30 sec - exiting 00:58:06 (3352): No heartbeat from core client for 30 sec - exiting 00:58:07 (3352): No heartbeat from core client for 30 sec - exiting 00:58:08 (3352): No heartbeat from core client for 30 sec - exiting 00:58:09 (3352): No heartbeat from core client for 30 sec - exiting 00:58:10 (3352): No heartbeat from core client for 30 sec - exiting 00:58:11 (3352): No heartbeat from core client for 30 sec - exiting 00:58:12 (3352): No heartbeat from core client for 30 sec - exiting 00:58:13 (3352): No heartbeat from core client for 30 sec - exiting 00:58:14 (3352): No heartbeat from core client for 30 sec - exiting 00:58:15 (3352): No heartbeat from core client for 30 sec - exiting 00:58:16 (3352): No heartbeat from core client for 30 sec - exiting 00:58:17 (3352): No heartbeat from core client for 30 sec - exiting 00:58:48 (3352): No heartbeat from core client for 30 sec - exiting 00:58:49 (3352): No heartbeat from core client for 30 sec - exiting 00:58:50 (3352): No heartbeat from core client for 30 sec - exiting 00:58:51 (3352): No heartbeat from core client for 30 sec - exiting 00:58:52 (3352): No heartbeat from core client for 30 sec - exiting 00:58:53 (3352): No heartbeat from core client for 30 sec - exiting 00:58:54 (3352): No heartbeat from core client for 30 sec - exiting 00:58:55 (3352): No heartbeat from core client for 30 sec - exiting 00:58:56 (3352): No heartbeat from core client for 30 sec - exiting 00:58:57 (3352): No heartbeat from core client for 30 sec - exiting 00:58:58 (3352): No heartbeat from core client for 30 sec - exiting 00:58:59 (3352): No heartbeat from core client for 30 sec - exiting 00:59:00 (3352): No heartbeat from core client for 30 sec - exiting 00:59:01 (3352): No heartbeat from core client for 30 sec - exiting 00:59:02 (3352): No heartbeat from core client for 30 sec - exiting 00:59:03 (3352): No heartbeat from core client for 30 sec - exiting 00:59:04 (3352): No heartbeat from core client for 30 sec - exiting 00:59:05 (3352): No heartbeat from core client for 30 sec - exiting 00:59:06 (3352): No heartbeat from core client for 30 sec - exiting 00:59:07 (3352): No heartbeat from core client for 30 sec - exiting 00:59:08 (3352): No heartbeat from core client for 30 sec - exiting 00:59:09 (3352): No heartbeat from core client for 30 sec - exiting 00:59:10 (3352): No heartbeat from core client for 30 sec - exiting 00:59:11 (3352): No heartbeat from core client for 30 sec - exiting 00:59:12 (3352): No heartbeat from core client for 30 sec - exiting 00:59:13 (3352): No heartbeat from core client for 30 sec - exiting 00:59:14 (3352): No heartbeat from core client for 30 sec - exiting 00:59:15 (3352): No heartbeat from core client for 30 sec - exiting 00:59:16 (3352): No heartbeat from core client for 30 sec - exiting 00:59:17 (3352): No heartbeat from core client for 30 sec - exiting 00:59:18 (3352): No heartbeat from core client for 30 sec - exiting 00:59:19 (3352): No heartbeat from core client for 30 sec - exiting 00:59:20 (3352): No heartbeat from core client for 30 sec - exiting 00:59:21 (3352): No heartbeat from core client for 30 sec - exiting 00:59:22 (3352): No heartbeat from core client for 30 sec - exiting 00:59:23 (3352): No heartbeat from core client for 30 sec - exiting 00:59:24 (3352): No heartbeat from core client for 30 sec - exiting 00:59:25 (3352): No heartbeat from core client for 30 sec - exiting 00:59:26 (3352): No heartbeat from core client for 30 sec - exiting 00:59:27 (3352): No heartbeat from core client for 30 sec - exiting 00:59:28 (3352): No heartbeat from core client for 30 sec - exiting 00:59:29 (3352): No heartbeat from core client for 30 sec - exiting 00:59:30 (3352): No heartbeat from core client for 30 sec - exiting 00:59:31 (3352): No heartbeat from core client for 30 sec - exiting 00:59:32 (3352): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x085f4f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0897bf60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0920ff60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08563f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09f7ef60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09cb0f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09021f50 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 14:03:58 (7679): No heartbeat from core client for 30 sec - exiting 14:03:59 (7679): No heartbeat from core client for 30 sec - exiting 14:04:00 (7679): No heartbeat from core client for 30 sec - exiting 14:04:01 (7679): No heartbeat from core client for 30 sec - exiting 14:04:02 (7679): No heartbeat from core client for 30 sec - exiting 14:04:03 (7679): No heartbeat from core client for 30 sec - exiting 14:04:04 (7679): No heartbeat from core client for 30 sec - exiting 14:04:05 (7679): No heartbeat from core client for 30 sec - exiting 14:04:06 (7679): No heartbeat from core client for 30 sec - exiting 14:04:07 (7679): No heartbeat from core client for 30 sec - exiting 14:04:38 (7679): No heartbeat from core client for 30 sec - exiting 14:04:39 (7679): No heartbeat from core client for 30 sec - exiting 14:04:40 (7679): No heartbeat from core client for 30 sec - exiting 14:04:41 (7679): No heartbeat from core client for 30 sec - exiting 14:04:42 (7679): No heartbeat from core client for 30 sec - exiting 14:04:43 (7679): No heartbeat from core client for 30 sec - exiting 14:04:44 (7679): No heartbeat from core client for 30 sec - exiting 14:04:45 (7679): No heartbeat from core client for 30 sec - exiting 14:04:46 (7679): No heartbeat from core client for 30 sec - exiting 14:04:47 (7679): No heartbeat from core client for 30 sec - exiting 14:08:11 (7679): No heartbeat from core client for 30 sec - exiting 14:08:12 (7679): No heartbeat from core client for 30 sec - exiting 14:08:13 (7679): No heartbeat from core client for 30 sec - exiting 14:08:14 (7679): No heartbeat from core client for 30 sec - exiting 14:08:15 (7679): No heartbeat from core client for 30 sec - exiting 14:08:16 (7679): No heartbeat from core client for 30 sec - exiting 14:08:17 (7679): No heartbeat from core client for 30 sec - exiting 14:08:18 (7679): No heartbeat from core client for 30 sec - exiting 14:08:19 (7679): No heartbeat from core client for 30 sec - exiting 14:08:20 (7679): No heartbeat from core client for 30 sec - exiting 14:11:04 (7679): No heartbeat from core client for 30 sec - exiting 14:11:05 (7679): No heartbeat from core client for 30 sec - exiting 14:11:06 (7679): No heartbeat from core client for 30 sec - exiting 14:11:07 (7679): No heartbeat from core client for 30 sec - exiting 14:11:08 (7679): No heartbeat from core client for 30 sec - exiting 14:11:09 (7679): No heartbeat from core client for 30 sec - exiting 14:11:10 (7679): No heartbeat from core client for 30 sec - exiting 14:11:11 (7679): No heartbeat from core client for 30 sec - exiting 14:11:12 (7679): No heartbeat from core client for 30 sec - exiting 14:11:13 (7679): No heartbeat from core client for 30 sec - exiting 19:10:54 (7679): No heartbeat from core client for 30 sec - exiting 19:10:55 (7679): No heartbeat from core client for 30 sec - exiting 19:10:56 (7679): No heartbeat from core client for 30 sec - exiting 19:10:57 (7679): No heartbeat from core client for 30 sec - exiting 19:10:58 (7679): No heartbeat from core client for 30 sec - exiting 19:10:59 (7679): No heartbeat from core client for 30 sec - exiting 19:11:00 (7679): No heartbeat from core client for 30 sec - exiting 19:11:01 (7679): No heartbeat from core client for 30 sec - exiting 19:11:02 (7679): No heartbeat from core client for 30 sec - exiting 19:11:03 (7679): No heartbeat from core client for 30 sec - exiting 19:11:04 (7679): No heartbeat from core client for 30 sec - exiting 19:11:05 (7679): No heartbeat from core client for 30 sec - exiting 19:11:06 (7679): No heartbeat from core client for 30 sec - exiting 19:11:07 (7679): No heartbeat from core client for 30 sec - exiting 19:11:08 (7679): No heartbeat from core client for 30 sec - exiting 19:11:09 (7679): No heartbeat from core client for 30 sec - exiting 19:11:10 (7679): No heartbeat from core client for 30 sec - exiting 19:11:11 (7679): No heartbeat from core client for 30 sec - exiting 19:11:12 (7679): No heartbeat from core client for 30 sec - exiting 19:11:13 (7679): No heartbeat from core client for 30 sec - exiting 19:11:14 (7679): No heartbeat from core client for 30 sec - exiting 19:11:15 (7679): No heartbeat from core client for 30 sec - exiting 19:11:16 (7679): No heartbeat from core client for 30 sec - exiting 19:11:17 (7679): No heartbeat from core client for 30 sec - exiting 19:11:18 (7679): No heartbeat from core client for 30 sec - exiting 19:11:19 (7679): No heartbeat from core client for 30 sec - exiting 19:11:20 (7679): No heartbeat from core client for 30 sec - exiting 19:11:21 (7679): No heartbeat from core client for 30 sec - exiting 19:11:22 (7679): No heartbeat from core client for 30 sec - exiting 19:11:23 (7679): No heartbeat from core client for 30 sec - exiting 19:11:24 (7679): No heartbeat from core client for 30 sec - exiting 19:11:25 (7679): No heartbeat from core client for 30 sec - exiting 19:11:26 (7679): No heartbeat from core client for 30 sec - exiting 19:11:27 (7679): No heartbeat from core client for 30 sec - exiting 19:11:28 (7679): No heartbeat from core client for 30 sec - exiting 19:11:29 (7679): No heartbeat from core client for 30 sec - exiting 19:11:30 (7679): No heartbeat from core client for 30 sec - exiting 19:11:31 (7679): No heartbeat from core client for 30 sec - exiting 19:11:32 (7679): No heartbeat from core client for 30 sec - exiting 19:11:34 (7679): No heartbeat from core client for 30 sec - exiting 19:11:35 (7679): No heartbeat from core client for 30 sec - exiting 19:11:36 (7679): No heartbeat from core client for 30 sec - exiting 19:11:37 (7679): No heartbeat from core client for 30 sec - exiting 19:11:38 (7679): No heartbeat from core client for 30 sec - exiting 19:11:39 (7679): No heartbeat from core client for 30 sec - exiting 19:11:40 (7679): No heartbeat from core client for 30 sec - exiting 19:11:41 (7679): No heartbeat from core client for 30 sec - exiting 19:11:42 (7679): No heartbeat from core client for 30 sec - exiting 19:11:43 (7679): No heartbeat from core client for 30 sec - exiting 19:11:44 (7679): No heartbeat from core client for 30 sec - exiting 19:11:45 (7679): No heartbeat from core client for 30 sec - exiting 19:11:46 (7679): No heartbeat from core client for 30 sec - exiting 19:11:47 (7679): No heartbeat from core client for 30 sec - exiting 19:11:48 (7679): No heartbeat from core client for 30 sec - exiting 19:11:49 (7679): No heartbeat from core client for 30 sec - exiting 19:11:50 (7679): No heartbeat from core client for 30 sec - exiting 19:11:51 (7679): No heartbeat from core client for 30 sec - exiting 19:11:52 (7679): No heartbeat from core client for 30 sec - exiting 19:11:53 (7679): No heartbeat from core client for 30 sec - exiting 19:11:54 (7679): No heartbeat from core client for 30 sec - exiting 19:11:55 (7679): No heartbeat from core client for 30 sec - exiting 19:11:56 (7679): No heartbeat from core client for 30 sec - exiting 19:11:57 (7679): No heartbeat from core client for 30 sec - exiting 19:11:58 (7679): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0875ff60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08be9f60 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 19:55:43 (3279): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 20:29:59	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	259,200	525,877	2.0288
14 Aug 2013 20:29:59	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	233,280	473,549	2.0300
24 Jul 2013 19:00:30	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	207,360	421,764	2.0340
23 Jul 2013 14:24:39	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	181,440	368,741	2.0323
08 Jul 2013 12:33:19	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	155,520	316,441	2.0347
06 Jul 2013 11:11:55	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	129,600	263,647	2.0343
02 Jul 2013 10:01:52	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	103,680	211,300	2.0380
21 Jun 2013 11:54:46	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	77,760	158,474	2.0380
16 Jun 2013 07:50:23	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	51,840	105,553	2.0361
12 Jun 2013 17:08:35	1279288	15823320	hadcm3n_o4f2_1980_40_008387476_0	25,920	52,383	2.0209