Task 17353394

Name	hadcm3n_x1hm_1940_40_009149197_0
Workunit	9279533
Created	6 Nov 2014, 12:46:59 UTC
Sent	6 Nov 2014, 12:58:18 UTC
Report deadline	5 Feb 2015, 20:25:29 UTC
Received	29 Apr 2016, 23:04:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	197 (0x000000C5) EXIT_TIME_LIMIT_EXCEEDED
Computer ID	1329810
Run time	223 days 5 hours 36 min 12 sec
CPU time	27 days 12 hours 36 min 54 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	2.19 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.4.42</core_client_version> <![CDATA[ <message> exceeded elapsed time limit 19287355.77 (42289620.00G/2.20G) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09dce278 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09d0aaa8 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x091c0ac0 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x087feac0 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09076ac0 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09126ac8 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09b35ac8 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09eb8ac8 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2368: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 21:34:29 (2370): No heartbeat from core client for 30 sec - exiting 21:34:31 (2370): No heartbeat from core client for 30 sec - exiting 21:34:32 (2370): No heartbeat from core client for 30 sec - exiting 21:34:33 (2370): No heartbeat from core client for 30 sec - exiting 03:03:58 (2370): No heartbeat from core client for 30 sec - exiting 03:03:59 (2370): No heartbeat from core client for 30 sec - exiting 03:04:00 (2370): No heartbeat from core client for 30 sec - exiting 21:42:21 (2370): No heartbeat from core client for 30 sec - exiting 21:42:26 (2370): No heartbeat from core client for 30 sec - exiting 21:42:28 (2370): No heartbeat from core client for 30 sec - exiting * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x09bbaac0 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called * Error in `../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu': double free or corruption (out): 0x08b06ac0 * hadcm3n_6.07_i686-pc-linux-gnu: malloc.c:2370: sysmalloc: Assertion `(old_top == (((mbinptr) (((char ) &((av)->bins[((1) - 1) 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) \|\| ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 (sizeof(size_t))) - 1)) & ~((2 (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long) old_end & pagemask) == 0)' failed. SIGABRT: abort called 10:21:15 (2327): No heartbeat from core client for 30 sec - exiting 10:21:16 (2327): No heartbeat from core client for 30 sec - exiting 10:21:17 (2327): No heartbeat from core client for 30 sec - exiting 10:21:18 (2327): No heartbeat from core client for 30 sec - exiting 10:21:19 (2327): No heartbeat from core client for 30 sec - exiting 10:21:20 (2327): No heartbeat from core client for 30 sec - exiting 10:21:21 (2327): No heartbeat from core client for 30 sec - exiting 10:21:22 (2327): No heartbeat from core client for 30 sec - exiting 10:21:23 (2327): No heartbeat from core client for 30 sec - exiting 10:21:24 (2327): No heartbeat from core client for 30 sec - exiting 10:21:25 (2327): No heartbeat from core client for 30 sec - exiting 15:13:10 (2327): No heartbeat from core client for 30 sec - exiting 04:58:37 (2327): No heartbeat from core client for 30 sec - exiting 04:58:42 (2327): No heartbeat from core client for 30 sec - exiting 04:58:43 (2327): No heartbeat from core client for 30 sec - exiting 04:58:44 (2327): No heartbeat from core client for 30 sec - exiting 04:58:45 (2327): No heartbeat from core client for 30 sec - exiting 04:58:46 (2327): No heartbeat from core client for 30 sec - exiting 04:58:47 (2327): No heartbeat from core client for 30 sec - exiting 04:58:48 (2327): No heartbeat from core client for 30 sec - exiting 04:58:49 (2327): No heartbeat from core client for 30 sec - exiting </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Dec 2014 14:36:08	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	1,036,800	2,377,270	2.2929
09 Dec 2014 18:45:31	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	1,010,880	2,320,213	2.2952
08 Dec 2014 13:29:18	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	984,960	2,258,320	2.2928
07 Dec 2014 17:51:37	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	959,040	2,198,414	2.2923
06 Dec 2014 21:19:31	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	933,120	2,142,670	2.2962
06 Dec 2014 01:33:34	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	907,200	2,086,008	2.2994
05 Dec 2014 06:50:02	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	881,280	2,027,784	2.3010
04 Dec 2014 02:17:47	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	855,360	1,970,065	2.3032
02 Dec 2014 12:44:55	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	829,440	1,908,175	2.3006
01 Dec 2014 17:52:41	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	803,520	1,846,192	2.2976
30 Nov 2014 22:04:20	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	777,600	1,784,792	2.2953
30 Nov 2014 01:15:50	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	751,680	1,723,092	2.2923
29 Nov 2014 05:39:20	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	725,760	1,661,346	2.2891
28 Nov 2014 08:20:09	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	699,840	1,600,061	2.2863
27 Nov 2014 13:39:04	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	673,920	1,538,753	2.2833
26 Nov 2014 18:47:50	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	648,000	1,476,830	2.2791
25 Nov 2014 23:44:58	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	622,080	1,415,058	2.2747
25 Nov 2014 02:05:56	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	596,160	1,353,327	2.2701
24 Nov 2014 07:21:09	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	570,240	1,291,045	2.2640
23 Nov 2014 12:31:11	1329810	17353394	hadcm3n_x1hm_1940_40_009149197_0	544,320	1,228,616	2.2572