Task 15899390

Name	hadcm3n_o1ya_1980_40_008387471_4
Workunit	8538330
Created	22 Jul 2013, 0:33:38 UTC
Sent	22 Jul 2013, 0:34:03 UTC
Report deadline	21 Oct 2013, 8:01:14 UTC
Received	21 Aug 2013, 9:00:03 UTC
Server state	Over
Outcome	Computation error
Client state	Aborted by user
Exit status	203 (0x000000CB) EXIT_ABORTED_VIA_GUI
Computer ID	1266285
Run time	11 days 15 hours 40 min 46 sec
CPU time	11 days 8 hours 7 min 3 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.98 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> aborted by user </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0983b948 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08bfff98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 05:13:19 (12375): No heartbeat from core client for 30 sec - exiting 05:13:20 (12375): No heartbeat from core client for 30 sec - exiting 05:15:02 (12375): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09236f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08527f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09424f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09e05f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called 03:33:41 (1639): No heartbeat from core client for 30 sec - exiting 03:33:42 (1639): No heartbeat from core client for 30 sec - exiting 03:33:43 (1639): No heartbeat from core client for 30 sec - exiting 03:33:44 (1639): No heartbeat from core client for 30 sec - exiting 03:33:45 (1639): No heartbeat from core client for 30 sec - exiting 03:33:46 (1639): No heartbeat from core client for 30 sec - exiting 03:33:47 (1639): No heartbeat from core client for 30 sec - exiting 03:33:48 (1639): No heartbeat from core client for 30 sec - exiting 03:33:49 (1639): No heartbeat from core client for 30 sec - exiting 03:33:50 (1639): No heartbeat from core client for 30 sec - exiting 03:33:51 (1639): No heartbeat from core client for 30 sec - exiting 03:34:22 (1639): No heartbeat from core client for 30 sec - exiting 03:34:23 (1639): No heartbeat from core client for 30 sec - exiting 03:34:24 (1639): No heartbeat from core client for 30 sec - exiting 03:34:25 (1639): No heartbeat from core client for 30 sec - exiting 03:34:26 (1639): No heartbeat from core client for 30 sec - exiting 03:34:28 (1639): No heartbeat from core client for 30 sec - exiting 03:34:29 (1639): No heartbeat from core client for 30 sec - exiting 03:34:30 (1639): No heartbeat from core client for 30 sec - exiting 03:34:31 (1639): No heartbeat from core client for 30 sec - exiting 03:34:32 (1639): No heartbeat from core client for 30 sec - exiting 03:34:33 (1639): No heartbeat from core client for 30 sec - exiting 03:34:34 (1639): No heartbeat from core client for 30 sec - exiting 03:34:35 (1639): No heartbeat from core client for 30 sec - exiting 03:34:36 (1639): No heartbeat from core client for 30 sec - exiting 03:34:37 (1639): No heartbeat from core client for 30 sec - exiting 03:34:38 (1639): No heartbeat from core client for 30 sec - exiting 03:34:39 (1639): No heartbeat from core client for 30 sec - exiting 03:34:40 (1639): No heartbeat from core client for 30 sec - exiting 03:34:41 (1639): No heartbeat from core client for 30 sec - exiting 03:34:42 (1639): No heartbeat from core client for 30 sec - exiting 03:34:43 (1639): No heartbeat from core client for 30 sec - exiting 03:34:44 (1639): No heartbeat from core client for 30 sec - exiting 03:34:45 (1639): No heartbeat from core client for 30 sec - exiting 03:34:46 (1639): No heartbeat from core client for 30 sec - exiting 03:34:47 (1639): No heartbeat from core client for 30 sec - exiting 03:34:48 (1639): No heartbeat from core client for 30 sec - exiting 03:34:49 (1639): No heartbeat from core client for 30 sec - exiting 03:34:50 (1639): No heartbeat from core client for 30 sec - exiting 03:34:51 (1639): No heartbeat from core client for 30 sec - exiting 03:34:52 (1639): No heartbeat from core client for 30 sec - exiting 03:34:53 (1639): No heartbeat from core client for 30 sec - exiting 03:58:13 (1639): No heartbeat from core client for 30 sec - exiting 03:58:14 (1639): No heartbeat from core client for 30 sec - exiting 03:58:15 (1639): No heartbeat from core client for 30 sec - exiting 03:58:16 (1639): No heartbeat from core client for 30 sec - exiting 03:58:17 (1639): No heartbeat from core client for 30 sec - exiting 03:58:18 (1639): No heartbeat from core client for 30 sec - exiting 03:58:19 (1639): No heartbeat from core client for 30 sec - exiting 03:58:20 (1639): No heartbeat from core client for 30 sec - exiting 03:58:21 (1639): No heartbeat from core client for 30 sec - exiting 03:58:22 (1639): No heartbeat from core client for 30 sec - exiting 03:58:23 (1639): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x094a5f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x099d5f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x0970ef98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09c81f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x082daf98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08ae1f98 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2369: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. SIGABRT: abort called </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	1,036,800	979,228	0.9445
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	1,010,880	954,516	0.9442
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	984,960	929,790	0.9440
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	959,040	905,027	0.9437
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	933,120	880,350	0.9434
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	907,200	855,752	0.9433
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	881,280	831,302	0.9433
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	855,360	806,898	0.9433
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	829,440	782,476	0.9434
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	803,520	757,999	0.9433
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	777,600	733,355	0.9431
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	751,680	708,535	0.9426
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	725,760	683,413	0.9417
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	699,840	658,298	0.9406
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	673,920	633,273	0.9397
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	648,000	608,092	0.9384
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	622,080	583,281	0.9376
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	596,160	558,695	0.9372
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	570,240	533,815	0.9361
14 Aug 2013 23:22:05	1266285	15899390	hadcm3n_o1ya_1980_40_008387471_4	544,320	508,936	0.9350