Task 19324540

Name	hadcm3n_xdoz_198012_480_351_010333693_1
Workunit	10333693
Created	3 Mar 2016, 15:33:24 UTC
Sent	3 Mar 2016, 15:33:50 UTC
Report deadline	13 Feb 2017, 20:53:50 UTC
Received	18 May 2016, 7:02:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1377934
Run time	14 days 6 hours 57 min 9 sec
CPU time	13 days 18 hours 13 min 51 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	1.92 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.6.31</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:41:50 (4696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0a154a80 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x085fb110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08316110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 18:06:06 (15644): No heartbeat from core client for 30 sec - exiting 18:06:07 (15644): No heartbeat from core client for 30 sec - exiting 18:06:08 (15644): No heartbeat from core client for 30 sec - exiting 18:06:09 (15644): No heartbeat from core client for 30 sec - exiting 18:06:10 (15644): No heartbeat from core client for 30 sec - exiting 18:06:11 (15644): No heartbeat from core client for 30 sec - exiting 18:06:12 (15644): No heartbeat from core client for 30 sec - exiting 18:06:13 (15644): No heartbeat from core client for 30 sec - exiting 18:06:14 (15644): No heartbeat from core client for 30 sec - exiting 18:06:15 (15644): No heartbeat from core client for 30 sec - exiting 18:06:16 (15644): No heartbeat from core client for 30 sec - exiting 18:06:17 (15644): No heartbeat from core client for 30 sec - exiting 18:06:18 (15644): No heartbeat from core client for 30 sec - exiting 18:06:19 (15644): No heartbeat from core client for 30 sec - exiting 18:06:20 (15644): No heartbeat from core client for 30 sec - exiting 18:06:21 (15644): No heartbeat from core client for 30 sec - exiting 18:06:22 (15644): No heartbeat from core client for 30 sec - exiting 18:06:23 (15644): No heartbeat from core client for 30 sec - exiting 18:06:24 (15644): No heartbeat from core client for 30 sec - exiting 18:06:25 (15644): No heartbeat from core client for 30 sec - exiting 18:06:26 (15644): No heartbeat from core client for 30 sec - exiting 18:06:27 (15644): No heartbeat from core client for 30 sec - exiting 18:06:28 (15644): No heartbeat from core client for 30 sec - exiting 18:06:29 (15644): No heartbeat from core client for 30 sec - exiting 18:06:30 (15644): No heartbeat from core client for 30 sec - exiting 18:06:31 (15644): No heartbeat from core client for 30 sec - exiting 18:06:32 (15644): No heartbeat from core client for 30 sec - exiting 18:06:33 (15644): No heartbeat from core client for 30 sec - exiting 18:06:34 (15644): No heartbeat from core client for 30 sec - exiting 18:06:35 (15644): No heartbeat from core client for 30 sec - exiting 18:06:36 (15644): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09885110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09e85110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08a46110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09d6a110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0919d110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x091cb110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0848e110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09fb6110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09bb0110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09042110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08729110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09749110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x082d1110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09aeb110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 16:13:01 (13387): No heartbeat from core client for 30 sec - exiting 16:13:02 (13387): No heartbeat from core client for 30 sec - exiting 16:13:03 (13387): No heartbeat from core client for 30 sec - exiting 16:13:04 (13387): No heartbeat from core client for 30 sec - exiting 16:13:05 (13387): No heartbeat from core client for 30 sec - exiting 16:13:06 (13387): No heartbeat from core client for 30 sec - exiting 16:13:07 (13387): No heartbeat from core client for 30 sec - exiting 16:13:08 (13387): No heartbeat from core client for 30 sec - exiting 16:13:09 (13387): No heartbeat from core client for 30 sec - exiting 16:13:10 (13387): No heartbeat from core client for 30 sec - exiting 16:13:11 (13387): No heartbeat from core client for 30 sec - exiting 16:13:12 (13387): No heartbeat from core client for 30 sec - exiting 16:13:13 (13387): No heartbeat from core client for 30 sec - exiting 16:13:14 (13387): No heartbeat from core client for 30 sec - exiting 16:13:15 (13387): No heartbeat from core client for 30 sec - exiting 16:13:16 (13387): No heartbeat from core client for 30 sec - exiting 16:13:17 (13387): No heartbeat from core client for 30 sec - exiting 16:13:18 (13387): No heartbeat from core client for 30 sec - exiting 16:13:19 (13387): No heartbeat from core client for 30 sec - exiting 16:13:20 (13387): No heartbeat from core client for 30 sec - exiting 16:13:21 (13387): No heartbeat from core client for 30 sec - exiting 16:13:22 (13387): No heartbeat from core client for 30 sec - exiting 16:13:23 (13387): No heartbeat from core client for 30 sec - exiting 16:13:24 (13387): No heartbeat from core client for 30 sec - exiting 16:13:25 (13387): No heartbeat from core client for 30 sec - exiting 16:13:26 (13387): No heartbeat from core client for 30 sec - exiting 16:13:27 (13387): No heartbeat from core client for 30 sec - exiting 16:13:28 (13387): No heartbeat from core client for 30 sec - exiting 16:13:29 (13387): No heartbeat from core client for 30 sec - exiting 16:13:30 (13387): No heartbeat from core client for 30 sec - exiting 16:13:31 (13387): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... Called boinc_finish * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09639110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08a7a110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0851e110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x090ee110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 18:33:00 (7697): No heartbeat from core client for 30 sec - exiting 18:33:01 (7697): No heartbeat from core client for 30 sec - exiting 18:33:02 (7697): No heartbeat from core client for 30 sec - exiting 18:33:03 (7697): No heartbeat from core client for 30 sec - exiting 18:33:04 (7697): No heartbeat from core client for 30 sec - exiting 18:33:05 (7697): No heartbeat from core client for 30 sec - exiting 18:33:06 (7697): No heartbeat from core client for 30 sec - exiting 18:33:07 (7697): No heartbeat from core client for 30 sec - exiting 18:33:08 (7697): No heartbeat from core client for 30 sec - exiting 18:33:09 (7697): No heartbeat from core client for 30 sec - exiting 18:33:10 (7697): No heartbeat from core client for 30 sec - exiting 18:33:11 (7697): No heartbeat from core client for 30 sec - exiting 18:33:12 (7697): No heartbeat from core client for 30 sec - exiting 18:33:13 (7697): No heartbeat from core client for 30 sec - exiting 18:33:14 (7697): No heartbeat from core client for 30 sec - exiting 18:33:15 (7697): No heartbeat from core client for 30 sec - exiting 18:33:16 (7697): No heartbeat from core client for 30 sec - exiting 18:33:17 (7697): No heartbeat from core client for 30 sec - exiting 18:33:18 (7697): No heartbeat from core client for 30 sec - exiting 18:33:19 (7697): No heartbeat from core client for 30 sec - exiting 18:33:20 (7697): No heartbeat from core client for 30 sec - exiting 18:33:21 (7697): No heartbeat from core client for 30 sec - exiting 18:33:22 (7697): No heartbeat from core client for 30 sec - exiting 18:33:23 (7697): No heartbeat from core client for 30 sec - exiting 18:33:24 (7697): No heartbeat from core client for 30 sec - exiting 18:33:25 (7697): No heartbeat from core client for 30 sec - exiting 18:33:26 (7697): No heartbeat from core client for 30 sec - exiting 18:33:27 (7697): No heartbeat from core client for 30 sec - exiting 18:33:28 (7697): No heartbeat from core client for 30 sec - exiting 18:33:29 (7697): No heartbeat from core client for 30 sec - exiting 18:33:30 (7697): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... Called boinc_finish * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09b3a110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09a59110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 17:57:35 (18780): No heartbeat from core client for 30 sec - exiting 17:57:36 (18780): No heartbeat from core client for 30 sec - exiting 17:57:37 (18780): No heartbeat from core client for 30 sec - exiting 17:57:38 (18780): No heartbeat from core client for 30 sec - exiting 17:57:39 (18780): No heartbeat from core client for 30 sec - exiting 17:57:40 (18780): No heartbeat from core client for 30 sec - exiting 17:57:41 (18780): No heartbeat from core client for 30 sec - exiting 17:57:42 (18780): No heartbeat from core client for 30 sec - exiting 17:57:43 (18780): No heartbeat from core client for 30 sec - exiting 17:57:44 (18780): No heartbeat from core client for 30 sec - exiting 17:57:45 (18780): No heartbeat from core client for 30 sec - exiting 17:57:46 (18780): No heartbeat from core client for 30 sec - exiting 17:57:47 (18780): No heartbeat from core client for 30 sec - exiting 17:57:48 (18780): No heartbeat from core client for 30 sec - exiting 17:57:49 (18780): No heartbeat from core client for 30 sec - exiting 17:57:50 (18780): No heartbeat from core client for 30 sec - exiting 17:57:51 (18780): No heartbeat from core client for 30 sec - exiting 17:57:52 (18780): No heartbeat from core client for 30 sec - exiting 17:57:53 (18780): No heartbeat from core client for 30 sec - exiting 17:57:54 (18780): No heartbeat from core client for 30 sec - exiting 17:57:55 (18780): No heartbeat from core client for 30 sec - exiting 17:57:56 (18780): No heartbeat from core client for 30 sec - exiting 17:57:57 (18780): No heartbeat from core client for 30 sec - exiting 17:57:58 (18780): No heartbeat from core client for 30 sec - exiting 17:57:59 (18780): No heartbeat from core client for 30 sec - exiting 17:58:00 (18780): No heartbeat from core client for 30 sec - exiting 17:58:01 (18780): No heartbeat from core client for 30 sec - exiting 17:58:02 (18780): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... Called boinc_finish 17:58:03 (18780): No heartbeat from core client for 30 sec - exiting 17:58:04 (18780): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x090e6110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 18:41:23 (4612): No heartbeat from core client for 30 sec - exiting 18:41:25 (4612): No heartbeat from core client for 30 sec - exiting 18:41:26 (4612): No heartbeat from core client for 30 sec - exiting 18:41:27 (4612): No heartbeat from core client for 30 sec - exiting 18:41:28 (4612): No heartbeat from core client for 30 sec - exiting 18:41:29 (4612): No heartbeat from core client for 30 sec - exiting 18:41:30 (4612): No heartbeat from core client for 30 sec - exiting 18:41:31 (4612): No heartbeat from core client for 30 sec - exiting 18:41:32 (4612): No heartbeat from core client for 30 sec - exiting 18:41:33 (4612): No heartbeat from core client for 30 sec - exiting 18:41:34 (4612): No heartbeat from core client for 30 sec - exiting 18:41:35 (4612): No heartbeat from core client for 30 sec - exiting 18:41:36 (4612): No heartbeat from core client for 30 sec - exiting 18:41:37 (4612): No heartbeat from core client for 30 sec - exiting 18:41:38 (4612): No heartbeat from core client for 30 sec - exiting 18:41:39 (4612): No heartbeat from core client for 30 sec - exiting 18:41:40 (4612): No heartbeat from core client for 30 sec - exiting 18:41:41 (4612): No heartbeat from core client for 30 sec - exiting 18:41:42 (4612): No heartbeat from core client for 30 sec - exiting 18:41:43 (4612): No heartbeat from core client for 30 sec - exiting 18:41:44 (4612): No heartbeat from core client for 30 sec - exiting 18:41:45 (4612): No heartbeat from core client for 30 sec - exiting 18:41:46 (4612): No heartbeat from core client for 30 sec - exiting 18:41:47 (4612): No heartbeat from core client for 30 sec - exiting 18:41:48 (4612): No heartbeat from core client for 30 sec - exiting 18:41:49 (4612): No heartbeat from core client for 30 sec - exiting 18:41:50 (4612): No heartbeat from core client for 30 sec - exiting 18:41:51 (4612): No heartbeat from core client for 30 sec - exiting 18:41:52 (4612): No heartbeat from core client for 30 sec - exiting 18:41:53 (4612): No heartbeat from core client for 30 sec - exiting 18:41:54 (4612): No heartbeat from core client for 30 sec - exiting Signal 15 received, exiting... Called boinc_finish 18:41:55 (4612): No heartbeat from core client for 30 sec - exiting 18:41:56 (4612): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08d8a110 * SIGABRT: abort called hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2372: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 18:42:13 (4772): No heartbeat from core client for 30 sec - exiting 18:42:14 (4772): No heartbeat from core client for 30 sec - exiting 18:42:15 (4772): No heartbeat from core client for 30 sec - exiting 18:42:16 (4772): No heartbeat from core client for 30 sec - exiting 18:42:17 (4772): No heartbeat from core client for 30 sec - exiting 18:42:18 (4772): No heartbeat from core client for 30 sec - exiting 18:42:19 (4772): No heartbeat from core client for 30 sec - exiting 18:42:20 (4772): No heartbeat from core client for 30 sec - exiting 18:42:21 (4772): No heartbeat from core client for 30 sec - exiting 18:42:22 (4772): No heartbeat from core client for 30 sec - exiting 18:42:23 (4772): No heartbeat from core client for 30 sec - exiting 18:42:24 (4772): No heartbeat from core client for 30 sec - exiting 18:42:25 (4772): No heartbeat from core client for 30 sec - exiting 18:42:26 (4772): No heartbeat from core client for 30 sec - exiting 18:42:27 (4772): No heartbeat from core client for 30 sec - exiting 18:42:28 (4772): No heartbeat from core client for 30 sec - exiting 18:42:29 (4772): No heartbeat from core client for 30 sec - exiting 18:42:30 (4772): No heartbeat from core client for 30 sec - exiting 18:42:31 (4772): No heartbeat from core client for 30 sec - exiting 18:42:32 (4772): No heartbeat from core client for 30 sec - exiting 18:42:33 (4772): No heartbeat from core client for 30 sec - exiting 18:42:34 (4772): No heartbeat from core client for 30 sec - exiting 18:42:35 (4772): No heartbeat from core client for 30 sec - exiting 18:42:36 (4772): No heartbeat from core client for 30 sec - exiting Signal 3 received, exiting... Called boinc_finish SIGSEGV: segmentation violation Stack trace (7 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] linux-gate.so.1(__kernel_sigreturn+0x0)[0xf77b0d40] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804ceb0] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050a03] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/i686/cmov/libc.so.6(__libc_start_main+0xf3)[0xf746da63] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Apr 2016 11:48:54	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	518,400	1,186,443	2.2887
25 Apr 2016 08:49:58	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	492,480	1,133,116	2.3008
21 Apr 2016 10:58:41	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	466,560	1,076,386	2.3071
20 Apr 2016 03:50:21	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	440,640	1,019,583	2.3139
19 Apr 2016 10:35:34	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	414,720	961,344	2.3181
15 Apr 2016 12:17:21	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	388,800	903,973	2.3250
11 Apr 2016 12:49:27	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	362,880	844,771	2.3280
06 Apr 2016 12:26:20	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	336,960	783,861	2.3263
24 Mar 2016 03:57:26	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	311,040	724,581	2.3295
22 Mar 2016 10:23:55	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	285,120	662,839	2.3248
17 Mar 2016 09:42:04	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	259,200	604,522	2.3323
14 Mar 2016 15:59:58	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	233,280	545,744	2.3394
14 Mar 2016 15:59:59	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	207,360	487,403	2.3505
14 Mar 2016 15:59:59	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	181,440	424,945	2.3421
14 Mar 2016 15:59:59	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	155,520	364,880	2.3462
14 Mar 2016 16:00:00	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	129,600	304,378	2.3486
10 Mar 2016 07:10:05	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	103,680	242,962	2.3434
09 Mar 2016 05:53:38	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	77,760	181,825	2.3383
06 Mar 2016 22:38:59	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	51,840	122,712	2.3671
05 Mar 2016 20:33:48	1377934	19324540	hadcm3n_xdoz_198012_480_351_010333693_1	25,920	62,762	2.4214