Task 15914980

Name	hadcm3n_n1sx_1960_40_008363614_1
Workunit	8514473
Created	14 Aug 2013, 11:39:26 UTC
Sent	14 Aug 2013, 15:25:36 UTC
Report deadline	13 Nov 2013, 22:52:47 UTC
Received	2 Sep 2013, 9:46:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1285553
Run time	5 days 11 hours 35 min 58 sec
CPU time	5 days 6 hours 13 min 7 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.52 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.45</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:40:06 (27290): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:43:38 (28432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:52:44 (28910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:55:17 (28928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:00:11 (29051): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:07:36 (29204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:57:14 (29439): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09183b20 *** ======= Backtrace: ========= /lib/libc.so.6[0xb73e31] /lib/libc.so.6[0xb76571] /usr/lib/libstdc++.so.6(_ZdlPv+0x22)[0x228df2] /usr/lib/libstdc++.so.6(_ZdaPv+0x1e)[0x228e4e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe6)[0xb19ce6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 0017c000-0025b000 r-xp 00000000 103:01 421245 /usr/lib/libstdc++.so.6.0.13 0025b000-0025f000 r--p 000de000 103:01 421245 /usr/lib/libstdc++.so.6.0.13 0025f000-00260000 rw-p 000e2000 103:01 421245 /usr/lib/libstdc++.so.6.0.13 00260000-00267000 rw-p 00000000 00:00 0 00267000-00284000 r-xp 00000000 103:01 134417 /lib/libgcc_s-4.4.7-20120601.so.1 00284000-00285000 rw-p 0001d000 103:01 134417 /lib/libgcc_s-4.4.7-20120601.so.1 0028d000-0028e000 r-xp 00000000 00:00 0 [vdso] 00ae1000-00aff000 r-xp 00000000 103:01 134433 /lib/ld-2.12.so 00aff000-00b00000 r--p 0001d000 103:01 134433 /lib/ld-2.12.so 00b00000-00b01000 rw-p 0001e000 103:01 134433 /lib/ld-2.12.so 00b03000-00c93000 r-xp 00000000 103:01 134440 /lib/libc-2.12.so 00c93000-00c94000 ---p 00190000 103:01 134440 /lib/libc-2.12.so 00c94000-00c96000 r--p 00190000 103:01 134440 /lib/libc-2.12.so 00c96000-00c97000 rw-p 00192000 103:01 134440 /lib/libc-2.12.so 00c97000-00c9a000 rw-p 00000000 00:00 0 00c9c000-00cc4000 r-xp 00000000 103:01 134491 /lib/libm-2.12.so 00cc4000-00cc5000 r--p 00027000 103:01 134491 /lib/libm-2.12.so 00cc5000-00cc6000 rw-p 00028000 103:01 134491 /lib/libm-2.12.so 00cc8000-00ccb000 r-xp 00000000 103:01 134487 /lib/libdl-2.12.so 00ccb000-00ccc000 r--p 00002000 103:01 134487 /lib/libdl-2.12.so 00ccc000-00ccd000 rw-p 00003000 103:01 134487 /lib/libdl-2.12.so 00ccf000-00ce6000 r-xp 00000000 103:01 134240 /lib/libpthread-2.12.so 00ce6000-00ce7000 r--p 00016000 103:01 134240 /lib/libpthread-2.12.so 00ce7000-00ce8000 rw-p 00017000 103:01 134240 /lib/libpthread-2.12.so 00ce8000-00cea000 rw-p 00000000 00:00 0 08048000-080e3000 r-xp 00000000 103:04 10354961 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 103:04 10354961 /var/lib/boinc/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0912e000-091b5000 rw-p 00000000 00:00 0 [heap] f7100000-f7121000 rw-p 00000000 00:00 0 f7121000-f7200000 ---p 00000000 00:00 0 f72c3000-f773e000 rw-s 00000000 103:04 10356105 /var/lib/boinc/slots/0/137535 f773e000-f7741000 rw-p 00000000 00:00 0 f7750000-f7751000 rw-p 00000000 00:00 0 f7751000-f7752000 ---p 00000000 00:00 0 f7752000-f7755000 rw-p 00000000 00:00 0 f7755000-f7757000 rw-s 00000000 103:04 10356044 /var/lib/boinc/slots/0/boinc_mmap_file f7757000-f7759000 rw-p 00000000 00:00 0 ff7a3000-ff814000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (18 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x28d400] [0x28d430] /lib/libc.so.6(gsignal+0x51)[0xb2db11] /lib/libc.so.6(abort+0x17a)[0xb2f3ea] /lib/libc.so.6[0xb6d9d5] /lib/libc.so.6[0xb73e31] /lib/libc.so.6[0xb76571] /usr/lib/libstdc++.so.6(_ZdlPv+0x22)[0x228df2] /usr/lib/libstdc++.so.6(_ZdaPv+0x1e)[0x228e4e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe6)[0xb19ce6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Sep 2013 09:49:32	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	259,200	454,383	1.7530
31 Aug 2013 18:14:18	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	233,280	408,207	1.7499
29 Aug 2013 22:51:54	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	207,360	362,779	1.7495
27 Aug 2013 22:50:01	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	181,440	317,928	1.7522
26 Aug 2013 00:36:52	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	155,520	271,559	1.7461
23 Aug 2013 23:15:24	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	129,600	225,679	1.7414
21 Aug 2013 08:56:47	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	103,680	176,940	1.7066
19 Aug 2013 20:32:16	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	77,760	133,440	1.7160
18 Aug 2013 03:28:30	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	51,840	89,747	1.7312
16 Aug 2013 11:45:15	1285553	15914980	hadcm3n_n1sx_1960_40_008363614_1	25,920	44,432	1.7142