climateprediction.net home page
Task 15881840

Task 15881840

Name hadcm3n_o3fd_1980_40_008384178_3
Workunit 8535037
Created 5 Jul 2013, 10:21:02 UTC
Sent 5 Jul 2013, 10:58:14 UTC
Report deadline 4 Oct 2013, 18:25:25 UTC
Received 28 Dec 2013, 21:56:01 UTC
Server state Over
Outcome Computation error
Client state Aborted by user
Exit status 203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID 1369231
Run time 21 days 10 hours 8 min 48 sec
CPU time 11 days 5 hours 9 min 54 sec
Validate state Invalid
Credit 6,220.80
Device peak FLOPS 2.62 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>7.2.33</core_client_version>
<![CDATA[
<message>
aborted by user
</message>
<stderr_txt>
nt for 30 sec - exiting
21:17:05 (6006): No heartbeat from core client for 30 sec - exiting
21:17:06 (6006): No heartbeat from core client for 30 sec - exiting
21:17:07 (6006): No heartbeat from core client for 30 sec - exiting
21:17:08 (6006): No heartbeat from core client for 30 sec - exiting
21:17:09 (6006): No heartbeat from core client for 30 sec - exiting
21:17:10 (6006): No heartbeat from core client for 30 sec - exiting
21:17:11 (6006): No heartbeat from core client for 30 sec - exiting
21:17:12 (6006): No heartbeat from core client for 30 sec - exiting
21:17:13 (6006): No heartbeat from core client for 30 sec - exiting
21:17:14 (6006): No heartbeat from core client for 30 sec - exiting
21:17:15 (6006): No heartbeat from core client for 30 sec - exiting
02:21:08 (6006): No heartbeat from core client for 30 sec - exiting
02:21:09 (6006): No heartbeat from core client for 30 sec - exiting
02:21:10 (6006): No heartbeat from core client for 30 sec - exiting
02:21:11 (6006): No heartbeat from core client for 30 sec - exiting
02:21:12 (6006): No heartbeat from core client for 30 sec - exiting
02:21:13 (6006): No heartbeat from core client for 30 sec - exiting
02:21:14 (6006): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0895b4e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
09:42:08 (3377): No heartbeat from core client for 30 sec - exiting
09:42:09 (3377): No heartbeat from core client for 30 sec - exiting
09:42:10 (3377): No heartbeat from core client for 30 sec - exiting
09:42:11 (3377): No heartbeat from core client for 30 sec - exiting
21:33:58 (3377): No heartbeat from core client for 30 sec - exiting
21:33:59 (3377): No heartbeat from core client for 30 sec - exiting
21:34:00 (3377): No heartbeat from core client for 30 sec - exiting
21:34:01 (3377): No heartbeat from core client for 30 sec - exiting
21:34:02 (3377): No heartbeat from core client for 30 sec - exiting
21:34:03 (3377): No heartbeat from core client for 30 sec - exiting
02:29:28 (3377): No heartbeat from core client for 30 sec - exiting
02:29:29 (3377): No heartbeat from core client for 30 sec - exiting
02:29:30 (3377): No heartbeat from core client for 30 sec - exiting
02:29:31 (3377): No heartbeat from core client for 30 sec - exiting
02:29:32 (3377): No heartbeat from core client for 30 sec - exiting
02:29:33 (3377): No heartbeat from core client for 30 sec - exiting
02:29:34 (3377): No heartbeat from core client for 30 sec - exiting
02:29:35 (3377): No heartbeat from core client for 30 sec - exiting
20:53:13 (3377): No heartbeat from core client for 30 sec - exiting
20:53:14 (3377): No heartbeat from core client for 30 sec - exiting
20:53:15 (3377): No heartbeat from core client for 30 sec - exiting
20:53:16 (3377): No heartbeat from core client for 30 sec - exiting
20:53:17 (3377): No heartbeat from core client for 30 sec - exiting
20:53:18 (3377): No heartbeat from core client for 30 sec - exiting
20:53:19 (3377): No heartbeat from core client for 30 sec - exiting
20:53:20 (3377): No heartbeat from core client for 30 sec - exiting
19:37:13 (3377): No heartbeat from core client for 30 sec - exiting
19:37:14 (3377): No heartbeat from core client for 30 sec - exiting
19:37:15 (3377): No heartbeat from core client for 30 sec - exiting
19:37:16 (3377): No heartbeat from core client for 30 sec - exiting
19:37:17 (3377): No heartbeat from core client for 30 sec - exiting
19:37:18 (3377): No heartbeat from core client for 30 sec - exiting
19:37:19 (3377): No heartbeat from core client for 30 sec - exiting
19:37:20 (3377): No heartbeat from core client for 30 sec - exiting
19:37:21 (3377): No heartbeat from core client for 30 sec - exiting
19:37:22 (3377): No heartbeat from core client for 30 sec - exiting
19:37:23 (3377): No heartbeat from core client for 30 sec - exiting
19:37:24 (3377): No heartbeat from core client for 30 sec - exiting
19:37:25 (3377): No heartbeat from core client for 30 sec - exiting
19:37:26 (3377): No heartbeat from core client for 30 sec - exiting
19:37:27 (3377): No heartbeat from core client for 30 sec - exiting
19:37:28 (3377): No heartbeat from core client for 30 sec - exiting
19:37:29 (3377): No heartbeat from core client for 30 sec - exiting
19:37:30 (3377): No heartbeat from core client for 30 sec - exiting
19:37:31 (3377): No heartbeat from core client for 30 sec - exiting
19:37:32 (3377): No heartbeat from core client for 30 sec - exiting
19:37:33 (3377): No heartbeat from core client for 30 sec - exiting
19:37:34 (3377): No heartbeat from core client for 30 sec - exiting
19:37:35 (3377): No heartbeat from core client for 30 sec - exiting
23:56:48 (3377): No heartbeat from core client for 30 sec - exiting
23:56:49 (3377): No heartbeat from core client for 30 sec - exiting
23:56:50 (3377): No heartbeat from core client for 30 sec - exiting
23:56:51 (3377): No heartbeat from core client for 30 sec - exiting
23:56:52 (3377): No heartbeat from core client for 30 sec - exiting
23:56:53 (3377): No heartbeat from core client for 30 sec - exiting
23:56:54 (3377): No heartbeat from core client for 30 sec - exiting
23:56:55 (3377): No heartbeat from core client for 30 sec - exiting
23:56:56 (3377): No heartbeat from core client for 30 sec - exiting
21:18:26 (3377): No heartbeat from core client for 30 sec - exiting
21:18:27 (3377): No heartbeat from core client for 30 sec - exiting
21:18:28 (3377): No heartbeat from core client for 30 sec - exiting
21:18:29 (3377): No heartbeat from core client for 30 sec - exiting
21:18:30 (3377): No heartbeat from core client for 30 sec - exiting
21:18:31 (3377): No heartbeat from core client for 30 sec - exiting
21:18:32 (3377): No heartbeat from core client for 30 sec - exiting
16:56:35 (3377): No heartbeat from core client for 30 sec - exiting
16:56:36 (3377): No heartbeat from core client for 30 sec - exiting
16:56:37 (3377): No heartbeat from core client for 30 sec - exiting
16:56:38 (3377): No heartbeat from core client for 30 sec - exiting
16:56:39 (3377): No heartbeat from core client for 30 sec - exiting
16:56:40 (3377): No heartbeat from core client for 30 sec - exiting
16:56:41 (3377): No heartbeat from core client for 30 sec - exiting
16:56:42 (3377): No heartbeat from core client for 30 sec - exiting
05:19:35 (3377): No heartbeat from core client for 30 sec - exiting
05:19:36 (3377): No heartbeat from core client for 30 sec - exiting
05:19:37 (3377): No heartbeat from core client for 30 sec - exiting
05:19:38 (3377): No heartbeat from core client for 30 sec - exiting
05:19:39 (3377): No heartbeat from core client for 30 sec - exiting
05:19:40 (3377): No heartbeat from core client for 30 sec - exiting
05:19:41 (3377): No heartbeat from core client for 30 sec - exiting
02:21:41 (3377): No heartbeat from core client for 30 sec - exiting
02:21:43 (3377): No heartbeat from core client for 30 sec - exiting
02:21:44 (3377): No heartbeat from core client for 30 sec - exiting
02:21:45 (3377): No heartbeat from core client for 30 sec - exiting
02:21:46 (3377): No heartbeat from core client for 30 sec - exiting
02:21:47 (3377): No heartbeat from core client for 30 sec - exiting
02:21:48 (3377): No heartbeat from core client for 30 sec - exiting
06:13:29 (3377): No heartbeat from core client for 30 sec - exiting
06:13:31 (3377): No heartbeat from core client for 30 sec - exiting
06:13:32 (3377): No heartbeat from core client for 30 sec - exiting
06:13:33 (3377): No heartbeat from core client for 30 sec - exiting
06:13:34 (3377): No heartbeat from core client for 30 sec - exiting
06:13:35 (3377): No heartbeat from core client for 30 sec - exiting
06:13:36 (3377): No heartbeat from core client for 30 sec - exiting
02:49:02 (3377): No heartbeat from core client for 30 sec - exiting
02:49:03 (3377): No heartbeat from core client for 30 sec - exiting
02:49:04 (3377): No heartbeat from core client for 30 sec - exiting
02:49:05 (3377): No heartbeat from core client for 30 sec - exiting
02:49:06 (3377): No heartbeat from core client for 30 sec - exiting
02:49:07 (3377): No heartbeat from core client for 30 sec - exiting
02:49:08 (3377): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x092054e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
15:50:34 (3683): No heartbeat from core client for 30 sec - exiting
15:50:35 (3683): No heartbeat from core client for 30 sec - exiting
15:50:36 (3683): No heartbeat from core client for 30 sec - exiting
15:50:37 (3683): No heartbeat from core client for 30 sec - exiting
15:50:38 (3683): No heartbeat from core client for 30 sec - exiting
15:50:39 (3683): No heartbeat from core client for 30 sec - exiting
15:50:40 (3683): No heartbeat from core client for 30 sec - exiting
15:50:41 (3683): No heartbeat from core client for 30 sec - exiting
15:50:42 (3683): No heartbeat from core client for 30 sec - exiting
15:50:43 (3683): No heartbeat from core client for 30 sec - exiting
15:50:44 (3683): No heartbeat from core client for 30 sec - exiting
15:50:45 (3683): No heartbeat from core client for 30 sec - exiting
17:16:51 (3683): No heartbeat from core client for 30 sec - exiting
17:16:52 (3683): No heartbeat from core client for 30 sec - exiting
17:16:53 (3683): No heartbeat from core client for 30 sec - exiting
17:16:54 (3683): No heartbeat from core client for 30 sec - exiting
17:16:55 (3683): No heartbeat from core client for 30 sec - exiting
17:16:56 (3683): No heartbeat from core client for 30 sec - exiting
17:16:57 (3683): No heartbeat from core client for 30 sec - exiting
17:16:58 (3683): No heartbeat from core client for 30 sec - exiting
17:16:59 (3683): No heartbeat from core client for 30 sec - exiting
17:17:00 (3683): No heartbeat from core client for 30 sec - exiting
17:17:01 (3683): No heartbeat from core client for 30 sec - exiting
17:17:02 (3683): No heartbeat from core client for 30 sec - exiting
17:17:03 (3683): No heartbeat from core client for 30 sec - exiting
17:17:04 (3683): No heartbeat from core client for 30 sec - exiting
17:17:05 (3683): No heartbeat from core client for 30 sec - exiting
17:17:06 (3683): No heartbeat from core client for 30 sec - exiting
17:17:07 (3683): No heartbeat from core client for 30 sec - exiting
17:17:08 (3683): No heartbeat from core client for 30 sec - exiting
17:17:09 (3683): No heartbeat from core client for 30 sec - exiting
17:17:10 (3683): No heartbeat from core client for 30 sec - exiting
18:11:55 (3683): No heartbeat from core client for 30 sec - exiting
18:11:56 (3683): No heartbeat from core client for 30 sec - exiting
18:11:57 (3683): No heartbeat from core client for 30 sec - exiting
18:11:58 (3683): No heartbeat from core client for 30 sec - exiting
18:11:59 (3683): No heartbeat from core client for 30 sec - exiting
18:12:00 (3683): No heartbeat from core client for 30 sec - exiting
14:18:05 (3683): No heartbeat from core client for 30 sec - exiting
14:18:06 (3683): No heartbeat from core client for 30 sec - exiting
14:18:07 (3683): No heartbeat from core client for 30 sec - exiting
14:18:08 (3683): No heartbeat from core client for 30 sec - exiting
14:18:09 (3683): No heartbeat from core client for 30 sec - exiting
14:18:10 (3683): No heartbeat from core client for 30 sec - exiting
14:18:11 (3683): No heartbeat from core client for 30 sec - exiting
14:18:12 (3683): No heartbeat from core client for 30 sec - exiting
14:18:13 (3683): No heartbeat from core client for 30 sec - exiting
14:18:14 (3683): No heartbeat from core client for 30 sec - exiting
14:18:15 (3683): No heartbeat from core client for 30 sec - exiting
14:18:16 (3683): No heartbeat from core client for 30 sec - exiting
14:18:17 (3683): No heartbeat from core client for 30 sec - exiting
14:18:18 (3683): No heartbeat from core client for 30 sec - exiting
20:18:12 (3683): No heartbeat from core client for 30 sec - exiting
20:18:14 (3683): No heartbeat from core client for 30 sec - exiting
20:18:15 (3683): No heartbeat from core client for 30 sec - exiting
20:18:16 (3683): No heartbeat from core client for 30 sec - exiting
20:18:17 (3683): No heartbeat from core client for 30 sec - exiting
20:18:18 (3683): No heartbeat from core client for 30 sec - exiting
20:18:19 (3683): No heartbeat from core client for 30 sec - exiting
15:12:49 (3683): No heartbeat from core client for 30 sec - exiting
15:12:50 (3683): No heartbeat from core client for 30 sec - exiting
15:12:51 (3683): No heartbeat from core client for 30 sec - exiting
15:12:52 (3683): No heartbeat from core client for 30 sec - exiting
15:12:53 (3683): No heartbeat from core client for 30 sec - exiting
15:12:54 (3683): No heartbeat from core client for 30 sec - exiting
21:06:38 (3683): No heartbeat from core client for 30 sec - exiting
21:06:39 (3683): No heartbeat from core client for 30 sec - exiting
21:06:40 (3683): No heartbeat from core client for 30 sec - exiting
21:06:41 (3683): No heartbeat from core client for 30 sec - exiting
21:06:42 (3683): No heartbeat from core client for 30 sec - exiting
21:06:43 (3683): No heartbeat from core client for 30 sec - exiting
21:06:44 (3683): No heartbeat from core client for 30 sec - exiting
21:06:45 (3683): No heartbeat from core client for 30 sec - exiting
03:20:09 (3683): No heartbeat from core client for 30 sec - exiting
03:20:10 (3683): No heartbeat from core client for 30 sec - exiting
03:20:11 (3683): No heartbeat from core client for 30 sec - exiting
03:20:12 (3683): No heartbeat from core client for 30 sec - exiting
03:20:13 (3683): No heartbeat from core client for 30 sec - exiting
03:20:14 (3683): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x086614e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
23:09:43 (4053): No heartbeat from core client for 30 sec - exiting
23:09:44 (4053): No heartbeat from core client for 30 sec - exiting
23:09:45 (4053): No heartbeat from core client for 30 sec - exiting
23:09:46 (4053): No heartbeat from core client for 30 sec - exiting
23:09:47 (4053): No heartbeat from core client for 30 sec - exiting
23:09:48 (4053): No heartbeat from core client for 30 sec - exiting
23:09:49 (4053): No heartbeat from core client for 30 sec - exiting
23:09:50 (4053): No heartbeat from core client for 30 sec - exiting
05:24:19 (4053): No heartbeat from core client for 30 sec - exiting
05:24:20 (4053): No heartbeat from core client for 30 sec - exiting
05:24:21 (4053): No heartbeat from core client for 30 sec - exiting
05:24:22 (4053): No heartbeat from core client for 30 sec - exiting
05:24:23 (4053): No heartbeat from core client for 30 sec - exiting
05:24:24 (4053): No heartbeat from core client for 30 sec - exiting
00:22:43 (4053): No heartbeat from core client for 30 sec - exiting
00:22:44 (4053): No heartbeat from core client for 30 sec - exiting
00:22:45 (4053): No heartbeat from core client for 30 sec - exiting
00:22:46 (4053): No heartbeat from core client for 30 sec - exiting
00:22:47 (4053): No heartbeat from core client for 30 sec - exiting
00:22:48 (4053): No heartbeat from core client for 30 sec - exiting
00:22:49 (4053): No heartbeat from core client for 30 sec - exiting
00:22:50 (4053): No heartbeat from core client for 30 sec - exiting
21:39:40 (4053): No heartbeat from core client for 30 sec - exiting
21:39:41 (4053): No heartbeat from core client for 30 sec - exiting
21:39:42 (4053): No heartbeat from core client for 30 sec - exiting
21:39:43 (4053): No heartbeat from core client for 30 sec - exiting
21:39:44 (4053): No heartbeat from core client for 30 sec - exiting
21:39:45 (4053): No heartbeat from core client for 30 sec - exiting
02:08:22 (4053): No heartbeat from core client for 30 sec - exiting
02:08:23 (4053): No heartbeat from core client for 30 sec - exiting
02:08:24 (4053): No heartbeat from core client for 30 sec - exiting
02:08:25 (4053): No heartbeat from core client for 30 sec - exiting
02:08:26 (4053): No heartbeat from core client for 30 sec - exiting
02:08:27 (4053): No heartbeat from core client for 30 sec - exiting
02:08:28 (4053): No heartbeat from core client for 30 sec - exiting
02:08:29 (4053): No heartbeat from core client for 30 sec - exiting
11:02:19 (4053): No heartbeat from core client for 30 sec - exiting
11:02:20 (4053): No heartbeat from core client for 30 sec - exiting
11:02:21 (4053): No heartbeat from core client for 30 sec - exiting
11:02:22 (4053): No heartbeat from core client for 30 sec - exiting
11:02:23 (4053): No heartbeat from core client for 30 sec - exiting
11:02:24 (4053): No heartbeat from core client for 30 sec - exiting
11:02:25 (4053): No heartbeat from core client for 30 sec - exiting
01:10:00 (4053): No heartbeat from core client for 30 sec - exiting
01:10:02 (4053): No heartbeat from core client for 30 sec - exiting
01:10:03 (4053): No heartbeat from core client for 30 sec - exiting
01:10:04 (4053): No heartbeat from core client for 30 sec - exiting
01:10:05 (4053): No heartbeat from core client for 30 sec - exiting
01:10:06 (4053): No heartbeat from core client for 30 sec - exiting
01:10:07 (4053): No heartbeat from core client for 30 sec - exiting
01:10:08 (4053): No heartbeat from core client for 30 sec - exiting
20:25:57 (4053): No heartbeat from core client for 30 sec - exiting
20:25:58 (4053): No heartbeat from core client for 30 sec - exiting
20:25:59 (4053): No heartbeat from core client for 30 sec - exiting
20:26:00 (4053): No heartbeat from core client for 30 sec - exiting
20:26:01 (4053): No heartbeat from core client for 30 sec - exiting
20:26:02 (4053): No heartbeat from core client for 30 sec - exiting
20:26:03 (4053): No heartbeat from core client for 30 sec - exiting
20:26:04 (4053): No heartbeat from core client for 30 sec - exiting
20:26:05 (4053): No heartbeat from core client for 30 sec - exiting
20:26:06 (4053): No heartbeat from core client for 30 sec - exiting
20:26:07 (4053): No heartbeat from core client for 30 sec - exiting
20:26:08 (4053): No heartbeat from core client for 30 sec - exiting
20:26:09 (4053): No heartbeat from core client for 30 sec - exiting
20:26:10 (4053): No heartbeat from core client for 30 sec - exiting
20:26:11 (4053): No heartbeat from core client for 30 sec - exiting
20:26:12 (4053): No heartbeat from core client for 30 sec - exiting
20:26:13 (4053): No heartbeat from core client for 30 sec - exiting
06:09:13 (4053): No heartbeat from core client for 30 sec - exiting
06:09:14 (4053): No heartbeat from core client for 30 sec - exiting
06:09:15 (4053): No heartbeat from core client for 30 sec - exiting
06:09:16 (4053): No heartbeat from core client for 30 sec - exiting
06:09:17 (4053): No heartbeat from core client for 30 sec - exiting
06:09:18 (4053): No heartbeat from core client for 30 sec - exiting
06:09:19 (4053): No heartbeat from core client for 30 sec - exiting
20:29:28 (4053): No heartbeat from core client for 30 sec - exiting
20:29:30 (4053): No heartbeat from core client for 30 sec - exiting
20:29:31 (4053): No heartbeat from core client for 30 sec - exiting
20:29:32 (4053): No heartbeat from core client for 30 sec - exiting
20:29:33 (4053): No heartbeat from core client for 30 sec - exiting
20:29:34 (4053): No heartbeat from core client for 30 sec - exiting
20:29:35 (4053): No heartbeat from core client for 30 sec - exiting
20:29:36 (4053): No heartbeat from core client for 30 sec - exiting
20:29:37 (4053): No heartbeat from core client for 30 sec - exiting
20:29:38 (4053): No heartbeat from core client for 30 sec - exiting
20:29:39 (4053): No heartbeat from core client for 30 sec - exiting
20:29:40 (4053): No heartbeat from core client for 30 sec - exiting
20:29:41 (4053): No heartbeat from core client for 30 sec - exiting
20:29:42 (4053): No heartbeat from core client for 30 sec - exiting
20:29:43 (4053): No heartbeat from core client for 30 sec - exiting
20:29:44 (4053): No heartbeat from core client for 30 sec - exiting
20:29:45 (4053): No heartbeat from core client for 30 sec - exiting
20:29:46 (4053): No heartbeat from core client for 30 sec - exiting
20:29:47 (4053): No heartbeat from core client for 30 sec - exiting
20:29:48 (4053): No heartbeat from core client for 30 sec - exiting
20:29:49 (4053): No heartbeat from core client for 30 sec - exiting
20:29:50 (4053): No heartbeat from core client for 30 sec - exiting
20:29:51 (4053): No heartbeat from core client for 30 sec - exiting
20:29:52 (4053): No heartbeat from core client for 30 sec - exiting
20:29:53 (4053): No heartbeat from core client for 30 sec - exiting
20:29:54 (4053): No heartbeat from core client for 30 sec - exiting
20:29:55 (4053): No heartbeat from core client for 30 sec - exiting
20:29:56 (4053): No heartbeat from core client for 30 sec - exiting
20:29:57 (4053): No heartbeat from core client for 30 sec - exiting
20:29:58 (4053): No heartbeat from core client for 30 sec - exiting
20:29:59 (4053): No heartbeat from core client for 30 sec - exiting
20:30:00 (4053): No heartbeat from core client for 30 sec - exiting
20:30:01 (4053): No heartbeat from core client for 30 sec - exiting
20:30:02 (4053): No heartbeat from core client for 30 sec - exiting
20:30:03 (4053): No heartbeat from core client for 30 sec - exiting
20:30:05 (4053): No heartbeat from core client for 30 sec - exiting
20:30:06 (4053): No heartbeat from core client for 30 sec - exiting
20:30:07 (4053): No heartbeat from core client for 30 sec - exiting
20:30:09 (4053): No heartbeat from core client for 30 sec - exiting
20:30:10 (4053): No heartbeat from core client for 30 sec - exiting
20:30:11 (4053): No heartbeat from core client for 30 sec - exiting
20:30:12 (4053): No heartbeat from core client for 30 sec - exiting
20:30:13 (4053): No heartbeat from core client for 30 sec - exiting
20:30:14 (4053): No heartbeat from core client for 30 sec - exiting
20:30:15 (4053): No heartbeat from core client for 30 sec - exiting
20:30:16 (4053): No heartbeat from core client for 30 sec - exiting
20:30:17 (4053): No heartbeat from core client for 30 sec - exiting
20:30:18 (4053): No heartbeat from core client for 30 sec - exiting
20:30:19 (4053): No heartbeat from core client for 30 sec - exiting
20:30:20 (4053): No heartbeat from core client for 30 sec - exiting
20:30:21 (4053): No heartbeat from core client for 30 sec - exiting
20:30:22 (4053): No heartbeat from core client for 30 sec - exiting
20:30:24 (4053): No heartbeat from core client for 30 sec - exiting
20:30:25 (4053): No heartbeat from core client for 30 sec - exiting
20:30:26 (4053): No heartbeat from core client for 30 sec - exiting
20:30:27 (4053): No heartbeat from core client for 30 sec - exiting
20:30:28 (4053): No heartbeat from core client for 30 sec - exiting
20:30:29 (4053): No heartbeat from core client for 30 sec - exiting
20:30:30 (4053): No heartbeat from core client for 30 sec - exiting
20:30:31 (4053): No heartbeat from core client for 30 sec - exiting
20:30:32 (4053): No heartbeat from core client for 30 sec - exiting
20:30:33 (4053): No heartbeat from core client for 30 sec - exiting
20:30:35 (4053): No heartbeat from core client for 30 sec - exiting
20:30:36 (4053): No heartbeat from core client for 30 sec - exiting
20:30:37 (4053): No heartbeat from core client for 30 sec - exiting
20:30:38 (4053): No heartbeat from core client for 30 sec - exiting
20:30:41 (4053): No heartbeat from core client for 30 sec - exiting
20:30:42 (4053): No heartbeat from core client for 30 sec - exiting
20:30:43 (4053): No heartbeat from core client for 30 sec - exiting
20:30:44 (4053): No heartbeat from core client for 30 sec - exiting
20:30:45 (4053): No heartbeat from core client for 30 sec - exiting
20:30:46 (4053): No heartbeat from core client for 30 sec - exiting
20:30:47 (4053): No heartbeat from core client for 30 sec - exiting
20:30:48 (4053): No heartbeat from core client for 30 sec - exiting
20:30:49 (4053): No heartbeat from core client for 30 sec - exiting
20:30:50 (4053): No heartbeat from core client for 30 sec - exiting
20:30:51 (4053): No heartbeat from core client for 30 sec - exiting
20:30:52 (4053): No heartbeat from core client for 30 sec - exiting
20:30:53 (4053): No heartbeat from core client for 30 sec - exiting
20:30:54 (4053): No heartbeat from core client for 30 sec - exiting
20:30:55 (4053): No heartbeat from core client for 30 sec - exiting
20:30:56 (4053): No heartbeat from core client for 30 sec - exiting
20:30:57 (4053): No heartbeat from core client for 30 sec - exiting
20:30:58 (4053): No heartbeat from core client for 30 sec - exiting
20:30:59 (4053): No heartbeat from core client for 30 sec - exiting
20:31:00 (4053): No heartbeat from core client for 30 sec - exiting
20:31:01 (4053): No heartbeat from core client for 30 sec - exiting
20:31:02 (4053): No heartbeat from core client for 30 sec - exiting
20:31:03 (4053): No heartbeat from core client for 30 sec - exiting
20:31:04 (4053): No heartbeat from core client for 30 sec - exiting
20:31:05 (4053): No heartbeat from core client for 30 sec - exiting
21:35:39 (4053): No heartbeat from core client for 30 sec - exiting
21:35:41 (4053): No heartbeat from core client for 30 sec - exiting
21:35:42 (4053): No heartbeat from core client for 30 sec - exiting
21:35:43 (4053): No heartbeat from core client for 30 sec - exiting
21:35:44 (4053): No heartbeat from core client for 30 sec - exiting
21:35:57 (4053): No heartbeat from core client for 30 sec - exiting
21:35:58 (4053): No heartbeat from core client for 30 sec - exiting
21:35:59 (4053): No heartbeat from core client for 30 sec - exiting
21:36:00 (4053): No heartbeat from core client for 30 sec - exiting
21:36:02 (4053): No heartbeat from core client for 30 sec - exiting
21:36:03 (4053): No heartbeat from core client for 30 sec - exiting
21:36:04 (4053): No heartbeat from core client for 30 sec - exiting
21:36:05 (4053): No heartbeat from core client for 30 sec - exiting
21:36:06 (4053): No heartbeat from core client for 30 sec - exiting
21:36:07 (4053): No heartbeat from core client for 30 sec - exiting
21:36:08 (4053): No heartbeat from core client for 30 sec - exiting
21:36:09 (4053): No heartbeat from core client for 30 sec - exiting
21:36:10 (4053): No heartbeat from core client for 30 sec - exiting
21:36:11 (4053): No heartbeat from core client for 30 sec - exiting
21:36:12 (4053): No heartbeat from core client for 30 sec - exiting
21:36:13 (4053): No heartbeat from core client for 30 sec - exiting
21:36:14 (4053): No heartbeat from core client for 30 sec - exiting
21:36:15 (4053): No heartbeat from core client for 30 sec - exiting
21:36:16 (4053): No heartbeat from core client for 30 sec - exiting
21:36:17 (4053): No heartbeat from core client for 30 sec - exiting
21:36:18 (4053): No heartbeat from core client for 30 sec - exiting
21:36:19 (4053): No heartbeat from core client for 30 sec - exiting
21:36:20 (4053): No heartbeat from core client for 30 sec - exiting
21:36:21 (4053): No heartbeat from core client for 30 sec - exiting
21:36:22 (4053): No heartbeat from core client for 30 sec - exiting
21:36:23 (4053): No heartbeat from core client for 30 sec - exiting
21:36:24 (4053): No heartbeat from core client for 30 sec - exiting
21:36:25 (4053): No heartbeat from core client for 30 sec - exiting
21:36:26 (4053): No heartbeat from core client for 30 sec - exiting
21:36:27 (4053): No heartbeat from core client for 30 sec - exiting
21:36:28 (4053): No heartbeat from core client for 30 sec - exiting
21:36:29 (4053): No heartbeat from core client for 30 sec - exiting
21:36:30 (4053): No heartbeat from core client for 30 sec - exiting
21:36:31 (4053): No heartbeat from core client for 30 sec - exiting
21:36:32 (4053): No heartbeat from core client for 30 sec - exiting
21:36:33 (4053): No heartbeat from core client for 30 sec - exiting
21:36:34 (4053): No heartbeat from core client for 30 sec - exiting
21:36:35 (4053): No heartbeat from core client for 30 sec - exiting
21:36:36 (4053): No heartbeat from core client for 30 sec - exiting
21:36:37 (4053): No heartbeat from core client for 30 sec - exiting
21:36:38 (4053): No heartbeat from core client for 30 sec - exiting
21:36:39 (4053): No heartbeat from core client for 30 sec - exiting
21:36:40 (4053): No heartbeat from core client for 30 sec - exiting
21:36:42 (4053): No heartbeat from core client for 30 sec - exiting
21:36:43 (4053): No heartbeat from core client for 30 sec - exiting
21:36:44 (4053): No heartbeat from core client for 30 sec - exiting
21:36:45 (4053): No heartbeat from core client for 30 sec - exiting
21:36:46 (4053): No heartbeat from core client for 30 sec - exiting
21:36:47 (4053): No heartbeat from core client for 30 sec - exiting
21:36:48 (4053): No heartbeat from core client for 30 sec - exiting
21:36:49 (4053): No heartbeat from core client for 30 sec - exiting
21:36:50 (4053): No heartbeat from core client for 30 sec - exiting
21:36:51 (4053): No heartbeat from core client for 30 sec - exiting
21:36:52 (4053): No heartbeat from core client for 30 sec - exiting
21:36:53 (4053): No heartbeat from core client for 30 sec - exiting
21:36:54 (4053): No heartbeat from core client for 30 sec - exiting
21:36:55 (4053): No heartbeat from core client for 30 sec - exiting
21:36:56 (4053): No heartbeat from core client for 30 sec - exiting
21:36:57 (4053): No heartbeat from core client for 30 sec - exiting
21:36:58 (4053): No heartbeat from core client for 30 sec - exiting
21:36:59 (4053): No heartbeat from core client for 30 sec - exiting
21:37:00 (4053): No heartbeat from core client for 30 sec - exiting
21:37:01 (4053): No heartbeat from core client for 30 sec - exiting
21:37:02 (4053): No heartbeat from core client for 30 sec - exiting
21:37:03 (4053): No heartbeat from core client for 30 sec - exiting
21:37:04 (4053): No heartbeat from core client for 30 sec - exiting
21:37:05 (4053): No heartbeat from core client for 30 sec - exiting
21:37:06 (4053): No heartbeat from core client for 30 sec - exiting
21:37:08 (4053): No heartbeat from core client for 30 sec - exiting
21:37:09 (4053): No heartbeat from core client for 30 sec - exiting
21:37:10 (4053): No heartbeat from core client for 30 sec - exiting
21:37:11 (4053): No heartbeat from core client for 30 sec - exiting
21:37:13 (4053): No heartbeat from core client for 30 sec - exiting
21:37:14 (4053): No heartbeat from core client for 30 sec - exiting
21:37:15 (4053): No heartbeat from core client for 30 sec - exiting
21:37:16 (4053): No heartbeat from core client for 30 sec - exiting
21:37:17 (4053): No heartbeat from core client for 30 sec - exiting
21:37:18 (4053): No heartbeat from core client for 30 sec - exiting
21:37:19 (4053): No heartbeat from core client for 30 sec - exiting
21:37:20 (4053): No heartbeat from core client for 30 sec - exiting
21:37:21 (4053): No heartbeat from core client for 30 sec - exiting
21:37:23 (4053): No heartbeat from core client for 30 sec - exiting
21:37:24 (4053): No heartbeat from core client for 30 sec - exiting
21:37:25 (4053): No heartbeat from core client for 30 sec - exiting
21:37:26 (4053): No heartbeat from core client for 30 sec - exiting
21:37:27 (4053): No heartbeat from core client for 30 sec - exiting
21:37:28 (4053): No heartbeat from core client for 30 sec - exiting
21:37:29 (4053): No heartbeat from core client for 30 sec - exiting
21:37:30 (4053): No heartbeat from core client for 30 sec - exiting
21:37:31 (4053): No heartbeat from core client for 30 sec - exiting
21:37:32 (4053): No heartbeat from core client for 30 sec - exiting
21:37:33 (4053): No heartbeat from core client for 30 sec - exiting
21:37:34 (4053): No heartbeat from core client for 30 sec - exiting
21:37:35 (4053): No heartbeat from core client for 30 sec - exiting
21:37:36 (4053): No heartbeat from core client for 30 sec - exiting
21:37:38 (4053): No heartbeat from core client for 30 sec - exiting
21:37:39 (4053): No heartbeat from core client for 30 sec - exiting
21:37:40 (4053): No heartbeat from core client for 30 sec - exiting
21:37:41 (4053): No heartbeat from core client for 30 sec - exiting
21:37:42 (4053): No heartbeat from core client for 30 sec - exiting
21:37:43 (4053): No heartbeat from core client for 30 sec - exiting
21:37:44 (4053): No heartbeat from core client for 30 sec - exiting
21:37:45 (4053): No heartbeat from core client for 30 sec - exiting
21:37:46 (4053): No heartbeat from core client for 30 sec - exiting
21:37:47 (4053): No heartbeat from core client for 30 sec - exiting
21:37:48 (4053): No heartbeat from core client for 30 sec - exiting
21:37:49 (4053): No heartbeat from core client for 30 sec - exiting
21:37:50 (4053): No heartbeat from core client for 30 sec - exiting
21:37:51 (4053): No heartbeat from core client for 30 sec - exiting
21:37:52 (4053): No heartbeat from core client for 30 sec - exiting
21:37:53 (4053): No heartbeat from core client for 30 sec - exiting
21:37:54 (4053): No heartbeat from core client for 30 sec - exiting
21:37:55 (4053): No heartbeat from core client for 30 sec - exiting
21:37:56 (4053): No heartbeat from core client for 30 sec - exiting
21:37:57 (4053): No heartbeat from core client for 30 sec - exiting
21:37:58 (4053): No heartbeat from core client for 30 sec - exiting
21:37:59 (4053): No heartbeat from core client for 30 sec - exiting
21:38:00 (4053): No heartbeat from core client for 30 sec - exiting
21:38:01 (4053): No heartbeat from core client for 30 sec - exiting
21:38:02 (4053): No heartbeat from core client for 30 sec - exiting
21:38:03 (4053): No heartbeat from core client for 30 sec - exiting
21:38:04 (4053): No heartbeat from core client for 30 sec - exiting
21:38:05 (4053): No heartbeat from core client for 30 sec - exiting
21:38:07 (4053): No heartbeat from core client for 30 sec - exiting
21:38:09 (4053): No heartbeat from core client for 30 sec - exiting
21:38:10 (4053): No heartbeat from core client for 30 sec - exiting
21:38:11 (4053): No heartbeat from core client for 30 sec - exiting
21:38:12 (4053): No heartbeat from core client for 30 sec - exiting
21:38:13 (4053): No heartbeat from core client for 30 sec - exiting
21:38:14 (4053): No heartbeat from core client for 30 sec - exiting
21:38:15 (4053): No heartbeat from core client for 30 sec - exiting
21:38:16 (4053): No heartbeat from core client for 30 sec - exiting
21:38:17 (4053): No heartbeat from core client for 30 sec - exiting
21:38:19 (4053): No heartbeat from core client for 30 sec - exiting
21:38:20 (4053): No heartbeat from core client for 30 sec - exiting
21:38:21 (4053): No heartbeat from core client for 30 sec - exiting
21:38:22 (4053): No heartbeat from core client for 30 sec - exiting
21:38:24 (4053): No heartbeat from core client for 30 sec - exiting
21:38:25 (4053): No heartbeat from core client for 30 sec - exiting
21:38:26 (4053): No heartbeat from core client for 30 sec - exiting
21:38:27 (4053): No heartbeat from core client for 30 sec - exiting
21:38:28 (4053): No heartbeat from core client for 30 sec - exiting
21:38:29 (4053): No heartbeat from core client for 30 sec - exiting
21:38:30 (4053): No heartbeat from core client for 30 sec - exiting
21:38:31 (4053): No heartbeat from core client for 30 sec - exiting
21:38:32 (4053): No heartbeat from core client for 30 sec - exiting
21:38:33 (4053): No heartbeat from core client for 30 sec - exiting
21:38:34 (4053): No heartbeat from core client for 30 sec - exiting
21:38:35 (4053): No heartbeat from core client for 30 sec - exiting
21:38:36 (4053): No heartbeat from core client for 30 sec - exiting
21:38:37 (4053): No heartbeat from core client for 30 sec - exiting
21:38:38 (4053): No heartbeat from core client for 30 sec - exiting
21:38:39 (4053): No heartbeat from core client for 30 sec - exiting
21:38:42 (4053): No heartbeat from core client for 30 sec - exiting
21:38:43 (4053): No heartbeat from core client for 30 sec - exiting
21:38:44 (4053): No heartbeat from core client for 30 sec - exiting
21:38:45 (4053): No heartbeat from core client for 30 sec - exiting
21:38:46 (4053): No heartbeat from core client for 30 sec - exiting
21:38:47 (4053): No heartbeat from core client for 30 sec - exiting
21:38:48 (4053): No heartbeat from core client for 30 sec - exiting
21:38:49 (4053): No heartbeat from core client for 30 sec - exiting
21:38:50 (4053): No heartbeat from core client for 30 sec - exiting
21:38:51 (4053): No heartbeat from core client for 30 sec - exiting
21:38:52 (4053): No heartbeat from core client for 30 sec - exiting
21:38:53 (4053): No heartbeat from core client for 30 sec - exiting
21:38:55 (4053): No heartbeat from core client for 30 sec - exiting
21:38:56 (4053): No heartbeat from core client for 30 sec - exiting
21:38:57 (4053): No heartbeat from core client for 30 sec - exiting
21:38:58 (4053): No heartbeat from core client for 30 sec - exiting
21:38:59 (4053): No heartbeat from core client for 30 sec - exiting
21:39:01 (4053): No heartbeat from core client for 30 sec - exiting
21:39:02 (4053): No heartbeat from core client for 30 sec - exiting
21:39:03 (4053): No heartbeat from core client for 30 sec - exiting
21:39:05 (4053): No heartbeat from core client for 30 sec - exiting
21:39:06 (4053): No heartbeat from core client for 30 sec - exiting
21:39:07 (4053): No heartbeat from core client for 30 sec - exiting
21:39:08 (4053): No heartbeat from core client for 30 sec - exiting
21:39:09 (4053): No heartbeat from core client for 30 sec - exiting
21:39:10 (4053): No heartbeat from core client for 30 sec - exiting
21:39:11 (4053): No heartbeat from core client for 30 sec - exiting
21:39:12 (4053): No heartbeat from core client for 30 sec - exiting
21:39:13 (4053): No heartbeat from core client for 30 sec - exiting
21:39:14 (4053): No heartbeat from core client for 30 sec - exiting
21:39:17 (4053): No heartbeat from core client for 30 sec - exiting
21:39:18 (4053): No heartbeat from core client for 30 sec - exiting
21:39:19 (4053): No heartbeat from core client for 30 sec - exiting
21:39:22 (4053): No heartbeat from core client for 30 sec - exiting
21:39:23 (4053): No heartbeat from core client for 30 sec - exiting
21:39:24 (4053): No heartbeat from core client for 30 sec - exiting
21:39:25 (4053): No heartbeat from core client for 30 sec - exiting
21:39:26 (4053): No heartbeat from core client for 30 sec - exiting
21:39:27 (4053): No heartbeat from core client for 30 sec - exiting
21:39:28 (4053): No heartbeat from core client for 30 sec - exiting
21:39:29 (4053): No heartbeat from core client for 30 sec - exiting
21:39:30 (4053): No heartbeat from core client for 30 sec - exiting
21:39:31 (4053): No heartbeat from core client for 30 sec - exiting
21:39:32 (4053): No heartbeat from core client for 30 sec - exiting
21:39:33 (4053): No heartbeat from core client for 30 sec - exiting
21:39:34 (4053): No heartbeat from core client for 30 sec - exiting
06:58:30 (4053): No heartbeat from core client for 30 sec - exiting
06:58:31 (4053): No heartbeat from core client for 30 sec - exiting
06:58:32 (4053): No heartbeat from core client for 30 sec - exiting
06:58:33 (4053): No heartbeat from core client for 30 sec - exiting
06:58:34 (4053): No heartbeat from core client for 30 sec - exiting
06:58:35 (4053): No heartbeat from core client for 30 sec - exiting
06:58:36 (4053): No heartbeat from core client for 30 sec - exiting
06:58:37 (4053): No heartbeat from core client for 30 sec - exiting
06:58:38 (4053): No heartbeat from core client for 30 sec - exiting
06:58:39 (4053): No heartbeat from core client for 30 sec - exiting
01:18:57 (4053): No heartbeat from core client for 30 sec - exiting
01:18:58 (4053): No heartbeat from core client for 30 sec - exiting
01:18:59 (4053): No heartbeat from core client for 30 sec - exiting
01:19:00 (4053): No heartbeat from core client for 30 sec - exiting
01:19:01 (4053): No heartbeat from core client for 30 sec - exiting
01:19:02 (4053): No heartbeat from core client for 30 sec - exiting
01:19:03 (4053): No heartbeat from core client for 30 sec - exiting
01:19:04 (4053): No heartbeat from core client for 30 sec - exiting
10:53:27 (4053): No heartbeat from core client for 30 sec - exiting
10:53:28 (4053): No heartbeat from core client for 30 sec - exiting
10:53:29 (4053): No heartbeat from core client for 30 sec - exiting
10:53:30 (4053): No heartbeat from core client for 30 sec - exiting
10:53:31 (4053): No heartbeat from core client for 30 sec - exiting
10:53:32 (4053): No heartbeat from core client for 30 sec - exiting
17:43:31 (4053): No heartbeat from core client for 30 sec - exiting
17:43:32 (4053): No heartbeat from core client for 30 sec - exiting
17:43:33 (4053): No heartbeat from core client for 30 sec - exiting
17:43:34 (4053): No heartbeat from core client for 30 sec - exiting
17:43:35 (4053): No heartbeat from core client for 30 sec - exiting
17:43:36 (4053): No heartbeat from core client for 30 sec - exiting
17:43:37 (4053): No heartbeat from core client for 30 sec - exiting
17:43:38 (4053): No heartbeat from core client for 30 sec - exiting
11:11:41 (4053): No heartbeat from core client for 30 sec - exiting
11:11:42 (4053): No heartbeat from core client for 30 sec - exiting
11:11:43 (4053): No heartbeat from core client for 30 sec - exiting
11:11:44 (4053): No heartbeat from core client for 30 sec - exiting
11:11:45 (4053): No heartbeat from core client for 30 sec - exiting
11:11:46 (4053): No heartbeat from core client for 30 sec - exiting
01:02:24 (4053): No heartbeat from core client for 30 sec - exiting
01:02:25 (4053): No heartbeat from core client for 30 sec - exiting
01:02:26 (4053): No heartbeat from core client for 30 sec - exiting
01:02:27 (4053): No heartbeat from core client for 30 sec - exiting
01:02:28 (4053): No heartbeat from core client for 30 sec - exiting
01:02:29 (4053): No heartbeat from core client for 30 sec - exiting
01:02:30 (4053): No heartbeat from core client for 30 sec - exiting
01:02:31 (4053): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08c0c4e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
18:00:02 (8984): No heartbeat from core client for 30 sec - exiting
18:00:03 (8984): No heartbeat from core client for 30 sec - exiting
18:00:04 (8984): No heartbeat from core client for 30 sec - exiting
18:00:05 (8984): No heartbeat from core client for 30 sec - exiting
18:00:06 (8984): No heartbeat from core client for 30 sec - exiting
18:00:07 (8984): No heartbeat from core client for 30 sec - exiting
22:34:03 (8984): No heartbeat from core client for 30 sec - exiting
22:34:04 (8984): No heartbeat from core client for 30 sec - exiting
22:34:05 (8984): No heartbeat from core client for 30 sec - exiting
22:34:06 (8984): No heartbeat from core client for 30 sec - exiting
22:34:07 (8984): No heartbeat from core client for 30 sec - exiting
22:34:08 (8984): No heartbeat from core client for 30 sec - exiting
22:34:09 (8984): No heartbeat from core client for 30 sec - exiting
22:34:10 (8984): No heartbeat from core client for 30 sec - exiting
14:48:35 (8984): No heartbeat from core client for 30 sec - exiting
14:48:36 (8984): No heartbeat from core client for 30 sec - exiting
14:48:37 (8984): No heartbeat from core client for 30 sec - exiting
14:48:38 (8984): No heartbeat from core client for 30 sec - exiting
14:48:39 (8984): No heartbeat from core client for 30 sec - exiting
14:48:40 (8984): No heartbeat from core client for 30 sec - exiting
14:48:41 (8984): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0970d4e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
00:59:57 (5575): No heartbeat from core client for 30 sec - exiting
00:59:58 (5575): No heartbeat from core client for 30 sec - exiting
00:59:59 (5575): No heartbeat from core client for 30 sec - exiting
01:00:00 (5575): No heartbeat from core client for 30 sec - exiting
01:00:01 (5575): No heartbeat from core client for 30 sec - exiting
01:00:02 (5575): No heartbeat from core client for 30 sec - exiting
01:00:03 (5575): No heartbeat from core client for 30 sec - exiting
01:00:04 (5575): No heartbeat from core client for 30 sec - exiting
19:39:32 (5575): No heartbeat from core client for 30 sec - exiting
19:39:33 (5575): No heartbeat from core client for 30 sec - exiting
19:39:34 (5575): No heartbeat from core client for 30 sec - exiting
19:39:35 (5575): No heartbeat from core client for 30 sec - exiting
19:39:36 (5575): No heartbeat from core client for 30 sec - exiting
19:39:37 (5575): No heartbeat from core client for 30 sec - exiting
00:35:23 (5575): No heartbeat from core client for 30 sec - exiting
00:35:24 (5575): No heartbeat from core client for 30 sec - exiting
00:35:25 (5575): No heartbeat from core client for 30 sec - exiting
00:35:26 (5575): No heartbeat from core client for 30 sec - exiting
00:35:27 (5575): No heartbeat from core client for 30 sec - exiting
00:35:28 (5575): No heartbeat from core client for 30 sec - exiting
00:35:29 (5575): No heartbeat from core client for 30 sec - exiting
00:35:30 (5575): No heartbeat from core client for 30 sec - exiting
09:44:55 (5575): No heartbeat from core client for 30 sec - exiting
09:44:56 (5575): No heartbeat from core client for 30 sec - exiting
09:44:57 (5575): No heartbeat from core client for 30 sec - exiting
09:44:58 (5575): No heartbeat from core client for 30 sec - exiting
09:44:59 (5575): No heartbeat from core client for 30 sec - exiting
09:45:00 (5575): No heartbeat from core client for 30 sec - exiting
09:45:01 (5575): No heartbeat from core client for 30 sec - exiting
09:45:02 (5575): No heartbeat from core client for 30 sec - exiting
01:15:24 (5575): No heartbeat from core client for 30 sec - exiting
01:15:25 (5575): No heartbeat from core client for 30 sec - exiting
01:15:26 (5575): No heartbeat from core client for 30 sec - exiting
01:15:27 (5575): No heartbeat from core client for 30 sec - exiting
01:15:28 (5575): No heartbeat from core client for 30 sec - exiting
01:15:29 (5575): No heartbeat from core client for 30 sec - exiting
01:15:30 (5575): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x094c04e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
22:46:09 (3505): No heartbeat from core client for 30 sec - exiting
22:46:10 (3505): No heartbeat from core client for 30 sec - exiting
22:46:11 (3505): No heartbeat from core client for 30 sec - exiting
22:46:12 (3505): No heartbeat from core client for 30 sec - exiting
22:46:13 (3505): No heartbeat from core client for 30 sec - exiting
22:46:14 (3505): No heartbeat from core client for 30 sec - exiting
22:46:15 (3505): No heartbeat from core client for 30 sec - exiting
22:46:16 (3505): No heartbeat from core client for 30 sec - exiting
22:46:17 (3505): No heartbeat from core client for 30 sec - exiting
22:46:18 (3505): No heartbeat from core client for 30 sec - exiting
22:46:19 (3505): No heartbeat from core client for 30 sec - exiting
22:46:20 (3505): No heartbeat from core client for 30 sec - exiting
22:46:21 (3505): No heartbeat from core client for 30 sec - exiting
22:46:22 (3505): No heartbeat from core client for 30 sec - exiting
22:46:23 (3505): No heartbeat from core client for 30 sec - exiting
22:46:24 (3505): No heartbeat from core client for 30 sec - exiting
22:46:25 (3505): No heartbeat from core client for 30 sec - exiting
22:46:26 (3505): No heartbeat from core client for 30 sec - exiting
22:46:27 (3505): No heartbeat from core client for 30 sec - exiting
22:46:28 (3505): No heartbeat from core client for 30 sec - exiting
22:46:29 (3505): No heartbeat from core client for 30 sec - exiting
22:46:30 (3505): No heartbeat from core client for 30 sec - exiting
22:46:31 (3505): No heartbeat from core client for 30 sec - exiting
23:03:44 (3505): No heartbeat from core client for 30 sec - exiting
23:03:45 (3505): No heartbeat from core client for 30 sec - exiting
23:03:46 (3505): No heartbeat from core client for 30 sec - exiting
23:03:47 (3505): No heartbeat from core client for 30 sec - exiting
23:03:48 (3505): No heartbeat from core client for 30 sec - exiting
23:03:49 (3505): No heartbeat from core client for 30 sec - exiting
23:03:50 (3505): No heartbeat from core client for 30 sec - exiting
23:03:51 (3505): No heartbeat from core client for 30 sec - exiting
10:22:33 (3505): No heartbeat from core client for 30 sec - exiting
03:30:18 (3505): No heartbeat from core client for 30 sec - exiting
03:30:19 (3505): No heartbeat from core client for 30 sec - exiting
03:30:20 (3505): No heartbeat from core client for 30 sec - exiting
19:34:29 (3505): No heartbeat from core client for 30 sec - exiting
19:34:30 (3505): No heartbeat from core client for 30 sec - exiting
19:34:31 (3505): No heartbeat from core client for 30 sec - exiting
19:34:32 (3505): No heartbeat from core client for 30 sec - exiting
19:34:33 (3505): No heartbeat from core client for 30 sec - exiting
19:34:34 (3505): No heartbeat from core client for 30 sec - exiting
19:34:35 (3505): No heartbeat from core client for 30 sec - exiting
23:30:56 (3505): No heartbeat from core client for 30 sec - exiting
23:30:57 (3505): No heartbeat from core client for 30 sec - exiting
23:30:58 (3505): No heartbeat from core client for 30 sec - exiting
23:30:59 (3505): No heartbeat from core client for 30 sec - exiting
23:31:00 (3505): No heartbeat from core client for 30 sec - exiting
23:31:01 (3505): No heartbeat from core client for 30 sec - exiting
23:31:02 (3505): No heartbeat from core client for 30 sec - exiting
23:31:03 (3505): No heartbeat from core client for 30 sec - exiting
23:31:04 (3505): No heartbeat from core client for 30 sec - exiting
23:31:05 (3505): No heartbeat from core client for 30 sec - exiting
23:31:06 (3505): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x089224e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
03:47:30 (13505): No heartbeat from core client for 30 sec - exiting
03:47:31 (13505): No heartbeat from core client for 30 sec - exiting
03:47:32 (13505): No heartbeat from core client for 30 sec - exiting
03:47:33 (13505): No heartbeat from core client for 30 sec - exiting
03:47:34 (13505): No heartbeat from core client for 30 sec - exiting
03:47:35 (13505): No heartbeat from core client for 30 sec - exiting
03:47:36 (13505): No heartbeat from core client for 30 sec - exiting
09:53:04 (13505): No heartbeat from core client for 30 sec - exiting
09:53:05 (13505): No heartbeat from core client for 30 sec - exiting
09:53:06 (13505): No heartbeat from core client for 30 sec - exiting
09:53:07 (13505): No heartbeat from core client for 30 sec - exiting
09:53:08 (13505): No heartbeat from core client for 30 sec - exiting
09:53:09 (13505): No heartbeat from core client for 30 sec - exiting
09:53:10 (13505): No heartbeat from core client for 30 sec - exiting
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x094a84e8 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
23:54:08 (11134): No heartbeat from core client for 30 sec - exiting
23:54:09 (11134): No heartbeat from core client for 30 sec - exiting
23:54:10 (11134): No heartbeat from core client for 30 sec - exiting
23:54:11 (11134): No heartbeat from core client for 30 sec - exiting
23:54:12 (11134): No heartbeat from core client for 30 sec - exiting
23:54:13 (11134): No heartbeat from core client for 30 sec - exiting
23:54:14 (11134): No heartbeat from core client for 30 sec - exiting
18:58:15 (11134): No heartbeat from core client for 30 sec - exiting
18:58:16 (11134): No heartbeat from core client for 30 sec - exiting
18:58:17 (11134): No heartbeat from core client for 30 sec - exiting
18:58:18 (11134): No heartbeat from core client for 30 sec - exiting
18:58:19 (11134): No heartbeat from core client for 30 sec - exiting
00:10:50 (11134): No heartbeat from core client for 30 sec - exiting
00:10:51 (11134): No heartbeat from core client for 30 sec - exiting
00:10:52 (11134): No heartbeat from core client for 30 sec - exiting
00:10:53 (11134): No heartbeat from core client for 30 sec - exiting
00:10:54 (11134): No heartbeat from core client for 30 sec - exiting
00:10:55 (11134): No heartbeat from core client for 30 sec - exiting
00:59:20 (11134): No heartbeat from core client for 30 sec - exiting
00:59:21 (11134): No heartbeat from core client for 30 sec - exiting
00:59:22 (11134): No heartbeat from core client for 30 sec - exiting
00:59:23 (11134): No heartbeat from core client for 30 sec - exiting
00:59:24 (11134): No heartbeat from core client for 30 sec - exiting
00:59:25 (11134): No heartbeat from core client for 30 sec - exiting
00:59:26 (11134): No heartbeat from core client for 30 sec - exiting
00:59:27 (11134): No heartbeat from core client for 30 sec - exiting
23:58:31 (11134): No heartbeat from core client for 30 sec - exiting
23:58:33 (11134): No heartbeat from core client for 30 sec - exiting
23:58:34 (11134): No heartbeat from core client for 30 sec - exiting
23:58:35 (11134): No heartbeat from core client for 30 sec - exiting
23:58:36 (11134): No heartbeat from core client for 30 sec - exiting
23:58:37 (11134): No heartbeat from core client for 30 sec - exiting
23:58:38 (11134): No heartbeat from core client for 30 sec - exiting
23:58:39 (11134): No heartbeat from core client for 30 sec - exiting
01:01:17 (11134): No heartbeat from core client for 30 sec - exiting
01:01:18 (11134): No heartbeat from core client for 30 sec - exiting
01:01:19 (11134): No heartbeat from core client for 30 sec - exiting
01:01:20 (11134): No heartbeat from core client for 30 sec - exiting
01:01:21 (11134): No heartbeat from core client for 30 sec - exiting
01:01:22 (11134): No heartbeat from core client for 30 sec - exiting
01:01:23 (11134): No heartbeat from core client for 30 sec - exiting
01:01:24 (11134): No heartbeat from core client for 30 sec - exiting
19:52:51 (11134): No heartbeat from core client for 30 sec - exiting
19:52:52 (11134): No heartbeat from core client for 30 sec - exiting
19:52:53 (11134): No heartbeat from core client for 30 sec - exiting
19:52:54 (11134): No heartbeat from core client for 30 sec - exiting
19:52:55 (11134): No heartbeat from core client for 30 sec - exiting
19:52:56 (11134): No heartbeat from core client for 30 sec - exiting
19:52:57 (11134): No heartbeat from core client for 30 sec - exiting
19:52:59 (11134): No heartbeat from core client for 30 sec - exiting
Signal 15 received, exiting...
Called boinc_finish
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x091de4d0 ***
hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed.
SIGABRT: abort called
20:37:48 (3474): No heartbeat from core client for 30 sec - exiting
20:37:49 (3474): No heartbeat from core client for 30 sec - exiting
20:37:50 (3474): No heartbeat from core client for 30 sec - exiting
20:37:51 (3474): No heartbeat from core client for 30 sec - exiting
20:37:52 (3474): No heartbeat from core client for 30 sec - exiting
20:37:53 (3474): No heartbeat from core client for 30 sec - exiting
19:55:28 (3474): No heartbeat from core client for 30 sec - exiting
19:55:29 (3474): No heartbeat from core client for 30 sec - exiting
19:55:30 (3474): No heartbeat from core client for 30 sec - exiting
19:55:31 (3474): No heartbeat from core client for 30 sec - exiting
19:55:32 (3474): No heartbeat from core client for 30 sec - exiting
19:55:33 (3474): No heartbeat from core client for 30 sec - exiting
19:55:34 (3474): No heartbeat from core client for 30 sec - exiting
19:55:35 (3474): No heartbeat from core client for 30 sec - exiting
00:38:53 (3474): No heartbeat from core client for 30 sec - exiting
00:38:54 (3474): No heartbeat from core client for 30 sec - exiting
00:38:55 (3474): No heartbeat from core client for 30 sec - exiting
00:38:56 (3474): No heartbeat from core client for 30 sec - exiting
00:38:57 (3474): No heartbeat from core client for 30 sec - exiting
00:38:58 (3474): No heartbeat from core client for 30 sec - exiting
00:38:59 (3474): No heartbeat from core client for 30 sec - exiting
19:14:11 (3474): No heartbeat from core client for 30 sec - exiting
19:14:12 (3474): No heartbeat from core client for 30 sec - exiting
19:14:13 (3474): No heartbeat from core client for 30 sec - exiting
19:14:14 (3474): No heartbeat from core client for 30 sec - exiting
19:14:15 (3474): No heartbeat from core client for 30 sec - exiting
19:14:16 (3474): No heartbeat from core client for 30 sec - exiting
19:14:17 (3474): No heartbeat from core client for 30 sec - exiting
19:14:18 (3474): No heartbeat from core client for 30 sec - exiting
19:14:19 (3474): No heartbeat from core client for 30 sec - exiting
09:30:46 (3474): No heartbeat from core client for 30 sec - exiting
09:30:47 (3474): No heartbeat from core client for 30 sec - exiting
09:30:48 (3474): No heartbeat from core client for 30 sec - exiting
09:30:49 (3474): No heartbeat from core client for 30 sec - exiting
09:30:50 (3474): No heartbeat from core client for 30 sec - exiting
09:30:51 (3474): No heartbeat from core client for 30 sec - exiting
09:30:52 (3474): No heartbeat from core client for 30 sec - exiting
09:30:53 (3474): No heartbeat from core client for 30 sec - exiting
09:30:54 (3474): No heartbeat from core client for 30 sec - exiting
09:30:55 (3474): No heartbeat from core client for 30 sec - exiting
09:30:56 (3474): No heartbeat from core client for 30 sec - exiting
09:30:57 (3474): No heartbeat from core client for 30 sec - exiting
09:30:58 (3474): No heartbeat from core client for 30 sec - exiting
09:30:59 (3474): No heartbeat from core client for 30 sec - exiting
09:31:00 (3474): No heartbeat from core client for 30 sec - exiting
00:09:03 (3474): No heartbeat from core client for 30 sec - exiting
00:09:04 (3474): No heartbeat from core client for 30 sec - exiting
00:09:05 (3474): No heartbeat from core client for 30 sec - exiting
00:09:06 (3474): No heartbeat from core client for 30 sec - exiting
00:09:07 (3474): No heartbeat from core client for 30 sec - exiting
00:09:08 (3474): No heartbeat from core client for 30 sec - exiting
00:09:09 (3474): No heartbeat from core client for 30 sec - exiting
00:09:10 (3474): No heartbeat from core client for 30 sec - exiting
01:56:34 (3474): No heartbeat from core client for 30 sec - exiting
01:56:35 (3474): No heartbeat from core client for 30 sec - exiting
01:56:36 (3474): No heartbeat from core client for 30 sec - exiting
01:56:37 (3474): No heartbeat from core client for 30 sec - exiting
01:56:38 (3474): No heartbeat from core client for 30 sec - exiting
01:56:39 (3474): No heartbeat from core client for 30 sec - exiting
01:56:40 (3474): No heartbeat from core client for 30 sec - exiting
21:44:20 (3474): No heartbeat from core client for 30 sec - exiting
21:44:21 (3474): No heartbeat from core client for 30 sec - exiting
21:44:22 (3474): No heartbeat from core client for 30 sec - exiting
21:44:23 (3474): No heartbeat from core client for 30 sec - exiting
21:44:24 (3474): No heartbeat from core client for 30 sec - exiting
21:44:25 (3474): No heartbeat from core client for 30 sec - exiting
21:44:26 (3474): No heartbeat from core client for 30 sec - exiting
21:44:27 (3474): No heartbeat from core client for 30 sec - exiting
20:38:38 (3474): No heartbeat from core client for 30 sec - exiting
20:38:39 (3474): No heartbeat from core client for 30 sec - exiting
20:38:40 (3474): No heartbeat from core client for 30 sec - exiting
20:38:41 (3474): No heartbeat from core client for 30 sec - exiting
20:38:42 (3474): No heartbeat from core client for 30 sec - exiting
20:38:43 (3474): No heartbeat from core client for 30 sec - exiting
20:38:44 (3474): No heartbeat from core client for 30 sec - exiting
20:38:45 (3474): No heartbeat from core client for 30 sec - exiting

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 Jul 2013 22:03:32 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 518,400 968,644 1.8685
23 Jul 2013 21:29:51 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 492,480 922,868 1.8739
23 Jul 2013 20:47:42 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 466,560 876,675 1.8790
23 Jul 2013 20:10:02 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 440,640 830,089 1.8838
23 Jul 2013 18:55:55 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 414,720 783,094 1.8882
23 Jul 2013 15:36:38 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 388,800 736,129 1.8933
23 Jul 2013 15:36:38 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 362,880 688,846 1.8983
23 Jul 2013 15:36:38 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 336,960 641,987 1.9052
23 Jul 2013 15:36:38 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 311,040 595,186 1.9135
23 Jul 2013 15:36:39 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 285,120 548,199 1.9227
23 Jul 2013 15:36:39 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 259,200 501,222 1.9337
23 Jul 2013 15:36:38 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 233,280 453,647 1.9446
23 Jul 2013 15:36:38 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 207,360 405,675 1.9564
23 Jul 2013 15:36:37 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 181,440 357,718 1.9715
23 Jul 2013 15:36:39 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 155,520 309,522 1.9902
11 Jul 2013 20:29:41 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 129,600 261,426 2.0172
08 Jul 2013 21:45:47 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 103,680 211,634 2.0412
08 Jul 2013 05:46:36 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 77,760 158,994 2.0447
07 Jul 2013 13:41:57 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 51,840 106,120 2.0471
06 Jul 2013 20:24:32 1284277 15881840 hadcm3n_o3fd_1980_40_008384178_3 25,920 53,200 2.0525


©2024 climateprediction.net