Task 13624599

Name	hadcm3n_o6ij_1940_40_007538935_4
Workunit	7736167
Created	9 Nov 2011, 21:11:58 UTC
Sent	9 Nov 2011, 21:28:43 UTC
Report deadline	9 Feb 2012, 4:55:54 UTC
Received	12 Dec 2011, 2:12:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1179305
Run time	8 days 5 hours 41 min 46 sec
CPU time	7 days 4 hours 54 min 49 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.73 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:25:16 (3085): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 12:39:27 (1423): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:40:26 (3074): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:42:18 (3083): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:43:41 (3093): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:46:02 (3104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:48:24 (3114): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:50:48 (3124): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:53:09 (3135): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:55:32 (3145): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:57:54 (3156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:00:17 (3167): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:05:03 (3177): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:06:46 (3190): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:08:09 (3200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:10:04 (3209): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:17:50 (1395): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:32:11 (1371): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:34:04 (3073): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:35:27 (3084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:37:50 (3094): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:40:12 (3133): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0a144828 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0xd73591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0xd74de8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0xd77ecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x781741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x78179d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xd1ebd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 002d6000-002eb000 r-xp 00000000 08:01 3801245 /lib/tls/i686/cmov/libpthread-2.11.1.so 002eb000-002ec000 r--p 00014000 08:01 3801245 /lib/tls/i686/cmov/libpthread-2.11.1.so 002ec000-002ed000 rw-p 00015000 08:01 3801245 /lib/tls/i686/cmov/libpthread-2.11.1.so 002ed000-002ef000 rw-p 00000000 00:00 0 00402000-00404000 r-xp 00000000 08:01 3801151 /lib/tls/i686/cmov/libdl-2.11.1.so 00404000-00405000 r--p 00001000 08:01 3801151 /lib/tls/i686/cmov/libdl-2.11.1.so 00405000-00406000 rw-p 00002000 08:01 3801151 /lib/tls/i686/cmov/libdl-2.11.1.so 004e1000-004fc000 r-xp 00000000 08:01 3805114 /lib/ld-2.11.1.so 004fc000-004fd000 r--p 0001a000 08:01 3805114 /lib/ld-2.11.1.so 004fd000-004fe000 rw-p 0001b000 08:01 3805114 /lib/ld-2.11.1.so 005fe000-005ff000 r-xp 00000000 00:00 0 [vdso] 006c6000-007af000 r-xp 00000000 08:01 17829555 /usr/lib/libstdc++.so.6.0.13 007af000-007b0000 ---p 000e9000 08:01 17829555 /usr/lib/libstdc++.so.6.0.13 007b0000-007b4000 r--p 000e9000 08:01 17829555 /usr/lib/libstdc++.so.6.0.13 007b4000-007b5000 rw-p 000ed000 08:01 17829555 /usr/lib/libstdc++.so.6.0.13 007b5000-007bc000 rw-p 00000000 00:00 0 00823000-00840000 r-xp 00000000 08:01 3801171 /lib/libgcc_s.so.1 00840000-00841000 r--p 0001c000 08:01 3801171 /lib/libgcc_s.so.1 00841000-00842000 rw-p 0001d000 08:01 3801171 /lib/libgcc_s.so.1 00d08000-00e5b000 r-xp 00000000 08:01 3801137 /lib/tls/i686/cmov/libc-2.11.1.so 00e5b000-00e5c000 ---p 00153000 08:01 3801137 /lib/tls/i686/cmov/libc-2.11.1.so 00e5c000-00e5e000 r--p 00153000 08:01 3801137 /lib/tls/i686/cmov/libc-2.11.1.so 00e5e000-00e5f000 rw-p 00155000 08:01 3801137 /lib/tls/i686/cmov/libc-2.11.1.so 00e5f000-00e62000 rw-p 00000000 00:00 0 00ef5000-00f19000 r-xp 00000000 08:01 3801186 /lib/tls/i686/cmov/libm-2.11.1.so 00f19000-00f1a000 r--p 00023000 08:01 3801186 /lib/tls/i686/cmov/libm-2.11.1.so 00f1a000-00f1b000 rw-p 00024000 08:01 3801186 /lib/tls/i686/cmov/libm-2.11.1.so 08048000-080e3000 r-xp 00000000 08:01 26870615 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 26870615 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0a0ef000-0a155000 rw-p 00000000 00:00 0 [heap] b7100000-b7121000 rw-p 00000000 00:00 0 b7121000-b7200000 ---p 00000000 00:00 0 b72ff000-b777a000 rw-s 00000000 08:01 26870790 /var/lib/boinc-client/slots/0/137610 b777a000-b777d000 rw-p 00000000 00:00 0 b7793000-b7794000 rw-p 00000000 00:00 0 b7794000-b7795000 ---p 00000000 00:00 0 b7795000-b7798000 rw-p 00000000 00:00 0 b7798000-b779a000 rw-s 00000000 08:01 26870662 /var/lib/boinc-client/slots/0/boinc_mmap_file b779a000-b779c000 rw-p 00000000 00:00 0 bf96a000-bf9da000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x5fe400] [0x5fe422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0xd32651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0xd35a82] /lib/tls/i686/cmov/libc.so.6(+0x6149d)[0xd6949d] /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0xd73591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0xd74de8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0xd77ecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x781741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x78179d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0xd1ebd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Dec 2011 02:16:43	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	259,200	622,482	2.4016
05 Dec 2011 22:46:23	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	233,280	560,045	2.4007
04 Dec 2011 06:34:26	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	207,360	497,504	2.3992
30 Nov 2011 21:21:26	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	181,440	435,392	2.3996
26 Nov 2011 19:20:48	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	155,520	373,096	2.3990
24 Nov 2011 23:43:27	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	129,600	310,627	2.3968
20 Nov 2011 22:55:59	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	103,680	248,465	2.3965
18 Nov 2011 16:36:16	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	77,760	186,111	2.3934
17 Nov 2011 02:21:43	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	51,840	123,880	2.3897
17 Nov 2011 02:21:43	1179305	13624599	hadcm3n_o6ij_1940_40_007538935_4	25,920	61,776	2.3833