Task 15629385

Name	hadcm3n_zlc7_1920_40_008257091_2
Workunit	8412215
Created	23 Feb 2013, 19:14:06 UTC
Sent	23 Feb 2013, 19:14:40 UTC
Report deadline	26 May 2013, 2:41:51 UTC
Received	28 Aug 2013, 0:30:53 UTC
Server state	Over
Outcome	Computation error
Client state	Aborted by user
Exit status	203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID	1304053
Run time	31 days 8 hours 19 min 5 sec
CPU time	22 days 20 hours 38 min 13 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x099e8188 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0943f818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08c65818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08472818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08508818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 15:00:25 (2083): No heartbeat from core client for 30 sec - exiting 15:00:26 (2083): No heartbeat from core client for 30 sec - exiting 15:00:27 (2083): No heartbeat from core client for 30 sec - exiting 15:00:28 (2083): No heartbeat from core client for 30 sec - exiting 15:00:29 (2083): No heartbeat from core client for 30 sec - exiting 15:00:30 (2083): No heartbeat from core client for 30 sec - exiting 15:00:31 (2083): No heartbeat from core client for 30 sec - exiting 15:00:32 (2083): No heartbeat from core client for 30 sec - exiting 15:00:33 (2083): No heartbeat from core client for 30 sec - exiting 15:00:34 (2083): No heartbeat from core client for 30 sec - exiting 15:00:35 (2083): No heartbeat from core client for 30 sec - exiting 15:00:36 (2083): No heartbeat from core client for 30 sec - exiting 15:00:37 (2083): No heartbeat from core client for 30 sec - exiting 15:00:38 (2083): No heartbeat from core client for 30 sec - exiting 15:00:39 (2083): No heartbeat from core client for 30 sec - exiting 15:00:40 (2083): No heartbeat from core client for 30 sec - exiting 15:00:41 (2083): No heartbeat from core client for 30 sec - exiting 15:00:42 (2083): No heartbeat from core client for 30 sec - exiting 15:00:43 (2083): No heartbeat from core client for 30 sec - exiting 15:00:44 (2083): No heartbeat from core client for 30 sec - exiting 15:00:45 (2083): No heartbeat from core client for 30 sec - exiting 15:00:46 (2083): No heartbeat from core client for 30 sec - exiting 15:00:47 (2083): No heartbeat from core client for 30 sec - exiting 15:00:48 (2083): No heartbeat from core client for 30 sec - exiting 15:00:49 (2083): No heartbeat from core client for 30 sec - exiting 15:00:50 (2083): No heartbeat from core client for 30 sec - exiting 15:00:51 (2083): No heartbeat from core client for 30 sec - exiting 15:00:52 (2083): No heartbeat from core client for 30 sec - exiting 15:00:53 (2083): No heartbeat from core client for 30 sec - exiting 15:00:54 (2083): No heartbeat from core client for 30 sec - exiting 15:00:55 (2083): No heartbeat from core client for 30 sec - exiting 15:00:56 (2083): No heartbeat from core client for 30 sec - exiting 15:00:57 (2083): No heartbeat from core client for 30 sec - exiting 15:00:58 (2083): No heartbeat from core client for 30 sec - exiting 15:00:59 (2083): No heartbeat from core client for 30 sec - exiting 15:01:00 (2083): No heartbeat from core client for 30 sec - exiting 15:01:01 (2083): No heartbeat from core client for 30 sec - exiting 15:01:02 (2083): No heartbeat from core client for 30 sec - exiting 15:01:03 (2083): No heartbeat from core client for 30 sec - exiting 15:01:04 (2083): No heartbeat from core client for 30 sec - exiting 15:01:05 (2083): No heartbeat from core client for 30 sec - exiting 15:01:06 (2083): No heartbeat from core client for 30 sec - exiting 15:01:07 (2083): No heartbeat from core client for 30 sec - exiting 15:01:08 (2083): No heartbeat from core client for 30 sec - exiting 15:01:09 (2083): No heartbeat from core client for 30 sec - exiting 15:01:10 (2083): No heartbeat from core client for 30 sec - exiting 15:01:11 (2083): No heartbeat from core client for 30 sec - exiting 15:01:12 (2083): No heartbeat from core client for 30 sec - exiting 15:01:13 (2083): No heartbeat from core client for 30 sec - exiting 15:01:14 (2083): No heartbeat from core client for 30 sec - exiting 15:02:17 (2083): No heartbeat from core client for 30 sec - exiting 15:02:18 (2083): No heartbeat from core client for 30 sec - exiting 15:02:19 (2083): No heartbeat from core client for 30 sec - exiting 15:02:20 (2083): No heartbeat from core client for 30 sec - exiting 15:02:21 (2083): No heartbeat from core client for 30 sec - exiting 15:02:22 (2083): No heartbeat from core client for 30 sec - exiting 15:02:23 (2083): No heartbeat from core client for 30 sec - exiting 15:02:24 (2083): No heartbeat from core client for 30 sec - exiting 15:02:25 (2083): No heartbeat from core client for 30 sec - exiting 15:02:26 (2083): No heartbeat from core client for 30 sec - exiting 15:02:58 (2083): No heartbeat from core client for 30 sec - exiting 15:02:59 (2083): No heartbeat from core client for 30 sec - exiting 15:03:00 (2083): No heartbeat from core client for 30 sec - exiting 15:03:01 (2083): No heartbeat from core client for 30 sec - exiting 15:03:02 (2083): No heartbeat from core client for 30 sec - exiting 15:03:03 (2083): No heartbeat from core client for 30 sec - exiting 15:03:04 (2083): No heartbeat from core client for 30 sec - exiting 15:03:05 (2083): No heartbeat from core client for 30 sec - exiting 15:03:06 (2083): No heartbeat from core client for 30 sec - exiting 15:03:07 (2083): No heartbeat from core client for 30 sec - exiting 15:03:08 (2083): No heartbeat from core client for 30 sec - exiting 15:03:09 (2083): No heartbeat from core client for 30 sec - exiting 15:03:10 (2083): No heartbeat from core client for 30 sec - exiting 15:03:11 (2083): No heartbeat from core client for 30 sec - exiting 15:03:12 (2083): No heartbeat from core client for 30 sec - exiting 15:03:13 (2083): No heartbeat from core client for 30 sec - exiting 15:03:14 (2083): No heartbeat from core client for 30 sec - exiting 15:03:15 (2083): No heartbeat from core client for 30 sec - exiting 15:03:16 (2083): No heartbeat from core client for 30 sec - exiting 15:03:17 (2083): No heartbeat from core client for 30 sec - exiting 15:03:18 (2083): No heartbeat from core client for 30 sec - exiting 15:03:19 (2083): No heartbeat from core client for 30 sec - exiting 15:03:20 (2083): No heartbeat from core client for 30 sec - exiting 15:03:21 (2083): No heartbeat from core client for 30 sec - exiting 15:03:22 (2083): No heartbeat from core client for 30 sec - exiting 15:03:23 (2083): No heartbeat from core client for 30 sec - exiting 15:03:24 (2083): No heartbeat from core client for 30 sec - exiting 15:03:25 (2083): No heartbeat from core client for 30 sec - exiting 15:03:26 (2083): No heartbeat from core client for 30 sec - exiting 15:03:27 (2083): No heartbeat from core client for 30 sec - exiting 15:03:28 (2083): No heartbeat from core client for 30 sec - exiting 15:03:29 (2083): No heartbeat from core client for 30 sec - exiting 15:03:30 (2083): No heartbeat from core client for 30 sec - exiting 15:03:31 (2083): No heartbeat from core client for 30 sec - exiting 15:03:32 (2083): No heartbeat from core client for 30 sec - exiting 15:03:33 (2083): No heartbeat from core client for 30 sec - exiting 15:03:34 (2083): No heartbeat from core client for 30 sec - exiting 15:03:35 (2083): No heartbeat from core client for 30 sec - exiting 15:03:36 (2083): No heartbeat from core client for 30 sec - exiting 15:03:37 (2083): No heartbeat from core client for 30 sec - exiting 15:03:38 (2083): No heartbeat from core client for 30 sec - exiting 15:03:39 (2083): No heartbeat from core client for 30 sec - exiting 15:03:40 (2083): No heartbeat from core client for 30 sec - exiting 15:03:41 (2083): No heartbeat from core client for 30 sec - exiting 15:03:42 (2083): No heartbeat from core client for 30 sec - exiting 15:03:43 (2083): No heartbeat from core client for 30 sec - exiting 15:03:44 (2083): No heartbeat from core client for 30 sec - exiting 15:03:45 (2083): No heartbeat from core client for 30 sec - exiting 15:03:46 (2083): No heartbeat from core client for 30 sec - exiting 15:03:47 (2083): No heartbeat from core client for 30 sec - exiting 15:03:48 (2083): No heartbeat from core client for 30 sec - exiting 15:06:21 (2083): No heartbeat from core client for 30 sec - exiting 15:06:22 (2083): No heartbeat from core client for 30 sec - exiting 15:06:23 (2083): No heartbeat from core client for 30 sec - exiting 15:06:24 (2083): No heartbeat from core client for 30 sec - exiting 15:06:25 (2083): No heartbeat from core client for 30 sec - exiting 15:06:26 (2083): No heartbeat from core client for 30 sec - exiting 15:06:27 (2083): No heartbeat from core client for 30 sec - exiting 15:06:28 (2083): No heartbeat from core client for 30 sec - exiting 15:06:29 (2083): No heartbeat from core client for 30 sec - exiting 15:06:30 (2083): No heartbeat from core client for 30 sec - exiting 15:06:31 (2083): No heartbeat from core client for 30 sec - exiting 15:06:32 (2083): No heartbeat from core client for 30 sec - exiting 15:06:33 (2083): No heartbeat from core client for 30 sec - exiting 15:06:34 (2083): No heartbeat from core client for 30 sec - exiting 15:06:35 (2083): No heartbeat from core client for 30 sec - exiting 15:06:36 (2083): No heartbeat from core client for 30 sec - exiting 15:06:37 (2083): No heartbeat from core client for 30 sec - exiting 15:06:38 (2083): No heartbeat from core client for 30 sec - exiting 15:06:39 (2083): No heartbeat from core client for 30 sec - exiting 15:06:40 (2083): No heartbeat from core client for 30 sec - exiting 15:06:41 (2083): No heartbeat from core client for 30 sec - exiting 15:06:42 (2083): No heartbeat from core client for 30 sec - exiting 15:06:43 (2083): No heartbeat from core client for 30 sec - exiting 15:06:44 (2083): No heartbeat from core client for 30 sec - exiting 15:06:45 (2083): No heartbeat from core client for 30 sec - exiting 15:06:46 (2083): No heartbeat from core client for 30 sec - exiting 15:06:47 (2083): No heartbeat from core client for 30 sec - exiting 15:06:48 (2083): No heartbeat from core client for 30 sec - exiting 15:06:49 (2083): No heartbeat from core client for 30 sec - exiting 15:06:50 (2083): No heartbeat from core client for 30 sec - exiting 15:06:51 (2083): No heartbeat from core client for 30 sec - exiting 15:06:52 (2083): No heartbeat from core client for 30 sec - exiting 15:06:53 (2083): No heartbeat from core client for 30 sec - exiting 15:06:54 (2083): No heartbeat from core client for 30 sec - exiting 15:06:55 (2083): No heartbeat from core client for 30 sec - exiting 15:06:56 (2083): No heartbeat from core client for 30 sec - exiting 15:06:57 (2083): No heartbeat from core client for 30 sec - exiting 15:06:58 (2083): No heartbeat from core client for 30 sec - exiting 15:06:59 (2083): No heartbeat from core client for 30 sec - exiting 15:07:00 (2083): No heartbeat from core client for 30 sec - exiting 15:07:01 (2083): No heartbeat from core client for 30 sec - exiting 15:07:02 (2083): No heartbeat from core client for 30 sec - exiting 15:07:03 (2083): No heartbeat from core client for 30 sec - exiting 15:07:04 (2083): No heartbeat from core client for 30 sec - exiting 15:07:05 (2083): No heartbeat from core client for 30 sec - exiting 15:07:06 (2083): No heartbeat from core client for 30 sec - exiting 15:07:07 (2083): No heartbeat from core client for 30 sec - exiting 15:07:08 (2083): No heartbeat from core client for 30 sec - exiting 15:07:09 (2083): No heartbeat from core client for 30 sec - exiting 15:07:10 (2083): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x092cd818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x089b3818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09dc6818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 05:59:52 (1940): No heartbeat from core client for 30 sec - exiting 05:59:53 (1940): No heartbeat from core client for 30 sec - exiting 05:59:54 (1940): No heartbeat from core client for 30 sec - exiting 05:59:55 (1940): No heartbeat from core client for 30 sec - exiting 05:59:56 (1940): No heartbeat from core client for 30 sec - exiting 05:59:57 (1940): No heartbeat from core client for 30 sec - exiting 05:59:58 (1940): No heartbeat from core client for 30 sec - exiting 05:59:59 (1940): No heartbeat from core client for 30 sec - exiting 06:00:00 (1940): No heartbeat from core client for 30 sec - exiting 06:00:01 (1940): No heartbeat from core client for 30 sec - exiting 06:00:32 (1940): No heartbeat from core client for 30 sec - exiting 06:00:33 (1940): No heartbeat from core client for 30 sec - exiting 06:00:34 (1940): No heartbeat from core client for 30 sec - exiting 06:00:35 (1940): No heartbeat from core client for 30 sec - exiting 06:00:36 (1940): No heartbeat from core client for 30 sec - exiting 06:00:37 (1940): No heartbeat from core client for 30 sec - exiting 06:00:38 (1940): No heartbeat from core client for 30 sec - exiting 06:00:39 (1940): No heartbeat from core client for 30 sec - exiting 06:00:40 (1940): No heartbeat from core client for 30 sec - exiting 06:00:41 (1940): No heartbeat from core client for 30 sec - exiting 06:04:13 (1940): No heartbeat from core client for 30 sec - exiting 06:04:14 (1940): No heartbeat from core client for 30 sec - exiting 06:04:15 (1940): No heartbeat from core client for 30 sec - exiting 06:04:16 (1940): No heartbeat from core client for 30 sec - exiting 06:04:17 (1940): No heartbeat from core client for 30 sec - exiting 06:04:18 (1940): No heartbeat from core client for 30 sec - exiting 06:04:19 (1940): No heartbeat from core client for 30 sec - exiting 06:04:20 (1940): No heartbeat from core client for 30 sec - exiting 06:04:21 (1940): No heartbeat from core client for 30 sec - exiting 06:04:22 (1940): No heartbeat from core client for 30 sec - exiting 06:04:23 (1940): No heartbeat from core client for 30 sec - exiting 06:08:33 (1940): No heartbeat from core client for 30 sec - exiting 06:08:34 (1940): No heartbeat from core client for 30 sec - exiting 06:08:35 (1940): No heartbeat from core client for 30 sec - exiting 06:08:36 (1940): No heartbeat from core client for 30 sec - exiting 06:08:37 (1940): No heartbeat from core client for 30 sec - exiting 06:08:38 (1940): No heartbeat from core client for 30 sec - exiting 06:08:39 (1940): No heartbeat from core client for 30 sec - exiting 06:08:40 (1940): No heartbeat from core client for 30 sec - exiting 06:08:41 (1940): No heartbeat from core client for 30 sec - exiting 06:08:42 (1940): No heartbeat from core client for 30 sec - exiting 06:08:43 (1940): No heartbeat from core client for 30 sec - exiting 06:24:00 (1940): No heartbeat from core client for 30 sec - exiting 06:24:01 (1940): No heartbeat from core client for 30 sec - exiting 06:24:02 (1940): No heartbeat from core client for 30 sec - exiting 06:24:03 (1940): No heartbeat from core client for 30 sec - exiting 06:24:04 (1940): No heartbeat from core client for 30 sec - exiting 06:24:05 (1940): No heartbeat from core client for 30 sec - exiting 06:24:06 (1940): No heartbeat from core client for 30 sec - exiting 06:24:07 (1940): No heartbeat from core client for 30 sec - exiting 06:24:08 (1940): No heartbeat from core client for 30 sec - exiting 06:24:09 (1940): No heartbeat from core client for 30 sec - exiting 06:24:10 (1940): No heartbeat from core client for 30 sec - exiting 06:50:11 (1940): No heartbeat from core client for 30 sec - exiting 06:50:12 (1940): No heartbeat from core client for 30 sec - exiting 06:50:13 (1940): No heartbeat from core client for 30 sec - exiting 06:50:14 (1940): No heartbeat from core client for 30 sec - exiting 06:50:15 (1940): No heartbeat from core client for 30 sec - exiting 06:50:16 (1940): No heartbeat from core client for 30 sec - exiting 06:50:17 (1940): No heartbeat from core client for 30 sec - exiting 06:50:18 (1940): No heartbeat from core client for 30 sec - exiting 06:50:19 (1940): No heartbeat from core client for 30 sec - exiting 06:50:20 (1940): No heartbeat from core client for 30 sec - exiting 07:44:59 (1940): No heartbeat from core client for 30 sec - exiting 07:45:00 (1940): No heartbeat from core client for 30 sec - exiting 07:45:01 (1940): No heartbeat from core client for 30 sec - exiting 07:45:02 (1940): No heartbeat from core client for 30 sec - exiting 07:45:03 (1940): No heartbeat from core client for 30 sec - exiting 07:45:04 (1940): No heartbeat from core client for 30 sec - exiting 07:45:05 (1940): No heartbeat from core client for 30 sec - exiting 07:45:06 (1940): No heartbeat from core client for 30 sec - exiting 07:45:07 (1940): No heartbeat from core client for 30 sec - exiting 07:45:08 (1940): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09c5b818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 21:28:09 (1905): No heartbeat from core client for 30 sec - exiting 21:28:10 (1905): No heartbeat from core client for 30 sec - exiting 21:28:11 (1905): No heartbeat from core client for 30 sec - exiting 21:28:12 (1905): No heartbeat from core client for 30 sec - exiting 21:28:13 (1905): No heartbeat from core client for 30 sec - exiting 21:28:14 (1905): No heartbeat from core client for 30 sec - exiting 21:28:15 (1905): No heartbeat from core client for 30 sec - exiting 21:28:16 (1905): No heartbeat from core client for 30 sec - exiting 21:28:17 (1905): No heartbeat from core client for 30 sec - exiting 21:28:18 (1905): No heartbeat from core client for 30 sec - exiting 21:28:19 (1905): No heartbeat from core client for 30 sec - exiting 21:29:21 (1905): No heartbeat from core client for 30 sec - exiting 21:29:22 (1905): No heartbeat from core client for 30 sec - exiting 21:29:23 (1905): No heartbeat from core client for 30 sec - exiting 21:29:24 (1905): No heartbeat from core client for 30 sec - exiting 21:29:25 (1905): No heartbeat from core client for 30 sec - exiting 21:29:26 (1905): No heartbeat from core client for 30 sec - exiting 21:29:27 (1905): No heartbeat from core client for 30 sec - exiting 21:29:28 (1905): No heartbeat from core client for 30 sec - exiting 21:29:29 (1905): No heartbeat from core client for 30 sec - exiting 21:29:30 (1905): No heartbeat from core client for 30 sec - exiting 21:29:31 (1905): No heartbeat from core client for 30 sec - exiting 07:56:21 (1905): No heartbeat from core client for 30 sec - exiting 07:56:22 (1905): No heartbeat from core client for 30 sec - exiting 07:56:23 (1905): No heartbeat from core client for 30 sec - exiting 07:56:24 (1905): No heartbeat from core client for 30 sec - exiting 07:56:25 (1905): No heartbeat from core client for 30 sec - exiting 07:56:26 (1905): No heartbeat from core client for 30 sec - exiting 07:56:27 (1905): No heartbeat from core client for 30 sec - exiting 07:56:28 (1905): No heartbeat from core client for 30 sec - exiting 07:56:29 (1905): No heartbeat from core client for 30 sec - exiting 07:56:30 (1905): No heartbeat from core client for 30 sec - exiting 07:56:31 (1905): No heartbeat from core client for 30 sec - exiting 10:10:57 (1905): No heartbeat from core client for 30 sec - exiting 10:10:58 (1905): No heartbeat from core client for 30 sec - exiting 10:10:59 (1905): No heartbeat from core client for 30 sec - exiting 10:11:00 (1905): No heartbeat from core client for 30 sec - exiting 10:11:01 (1905): No heartbeat from core client for 30 sec - exiting 10:11:02 (1905): No heartbeat from core client for 30 sec - exiting 10:11:03 (1905): No heartbeat from core client for 30 sec - exiting 10:11:04 (1905): No heartbeat from core client for 30 sec - exiting 10:11:05 (1905): No heartbeat from core client for 30 sec - exiting 10:11:06 (1905): No heartbeat from core client for 30 sec - exiting 10:11:07 (1905): No heartbeat from core client for 30 sec - exiting * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08bf6818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08883818 *** hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Apr 2013 21:44:56	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	777,600	1,920,736	2.4701
17 Apr 2013 04:00:12	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	751,680	1,857,883	2.4716
16 Apr 2013 10:05:06	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	725,760	1,794,904	2.4731
15 Apr 2013 15:45:53	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	699,840	1,731,663	2.4744
14 Apr 2013 22:01:41	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	673,920	1,668,784	2.4762
14 Apr 2013 03:54:56	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	648,000	1,605,740	2.4780
13 Apr 2013 09:03:31	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	622,080	1,542,277	2.4792
12 Apr 2013 12:00:29	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	596,160	1,478,999	2.4809
10 Apr 2013 13:40:04	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	570,240	1,414,375	2.4803
07 Apr 2013 15:45:39	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	544,320	1,349,888	2.4800
06 Apr 2013 18:26:53	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	518,400	1,285,803	2.4803
03 Apr 2013 07:50:06	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	492,480	1,221,441	2.4802
01 Apr 2013 10:04:52	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	466,560	1,157,263	2.4804
29 Mar 2013 15:00:33	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	440,640	1,092,999	2.4805
28 Mar 2013 14:55:42	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	414,720	1,029,245	2.4818
26 Mar 2013 06:00:00	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	388,800	964,145	2.4798
23 Mar 2013 00:54:43	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	362,880	899,387	2.4785
21 Mar 2013 11:09:49	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	336,960	834,316	2.4760
20 Mar 2013 00:09:56	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	311,040	769,559	2.4741
19 Mar 2013 01:02:10	1215258	15629385	hadcm3n_zlc7_1920_40_008257091_2	285,120	704,292	2.4702