Task 13013108

Name	hadcm3n_t1g2_1940_40_007311207_1
Workunit	7508637
Created	27 Jun 2011, 22:13:35 UTC
Sent	27 Jun 2011, 22:13:48 UTC
Report deadline	27 Sep 2011, 5:40:59 UTC
Received	15 Jul 2011, 17:37:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1045545
Run time	14 days 17 hours 56 min 53 sec
CPU time	13 days 18 hours 27 min 16 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27402, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0841ffb0 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6b961)[0x270961] /lib/i386-linux-gnu/libc.so.6(+0x6d28b)[0x27228b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x27541d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0x9bd4d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0x9bd52d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x21be37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 001e7000-00203000 r-xp 00000000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 00203000-00204000 r--p 0001b000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 00204000-00205000 rw-p 0001c000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 00205000-0035f000 r-xp 00000000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 0035f000-00360000 ---p 0015a000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 00360000-00362000 r--p 0015a000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 00362000-00363000 rw-p 0015c000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 00363000-00366000 rw-p 00000000 00:00 0 003a3000-003bd000 r-xp 00000000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 003bd000-003be000 r--p 00019000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 003be000-003bf000 rw-p 0001a000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 003de000-003df000 r-xp 00000000 00:00 0 [vdso] 00913000-009f2000 r-xp 00000000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 009f2000-009f6000 r--p 000de000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 009f6000-009f7000 rw-p 000e2000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 009f7000-009fe000 rw-p 00000000 00:00 0 00c73000-00c88000 r-xp 00000000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00c88000-00c89000 r--p 00015000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00c89000-00c8a000 rw-p 00016000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00c8a000-00c8c000 rw-p 00000000 00:00 0 00c9e000-00cc2000 r-xp 00000000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 00cc2000-00cc3000 r--p 00023000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 00cc3000-00cc4000 rw-p 00024000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 00ddd000-00ddf000 r-xp 00000000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 00ddf000-00de0000 r--p 00001000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 00de0000-00de1000 rw-p 00002000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 08048000-080e3000 r-xp 00000000 08:01 262884 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 262884 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 083ca000-0842f000 rw-p 00000000 00:00 0 [heap] b7300000-b7321000 rw-p 00000000 00:00 0 b7321000-b7400000 ---p 00000000 00:00 0 b7437000-b78b2000 rw-s 00000000 08:01 527793 /var/lib/boinc-client/slots/2/131930 b78b2000-b78b6000 rw-p 00000000 00:00 0 b78c8000-b78c9000 rw-p 00000000 00:00 0 b78c9000-b78ca000 ---p 00000000 00:00 0 b78ca000-b78cd000 rw-p 00000000 00:00 0 b78cd000-b78cf000 rw-s 00000000 08:01 527619 /var/lib/boinc-client/slots/2/boinc_mmap_file b78cf000-b78d1000 rw-p 00000000 00:00 0 bfb6a000-bfbda000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x3de400] [0x3de416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x51)[0x22fe71] /lib/i386-linux-gnu/libc.so.6(abort+0x17e)[0x23334e] /lib/i386-linux-gnu/libc.so.6(+0x61577)[0x266577] /lib/i386-linux-gnu/libc.so.6(+0x6b961)[0x270961] /lib/i386-linux-gnu/libc.so.6(+0x6d28b)[0x27228b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x27541d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0x9bd4d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0x9bd52d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x21be37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 14:30:38	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	259,200	1,189,623	4.5896
25 Jul 2011 14:30:38	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	233,280	1,070,775	4.5901
09 Jul 2011 22:17:35	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	207,360	951,159	4.5870
08 Jul 2011 14:23:56	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	181,440	832,980	4.5909
07 Jul 2011 15:39:43	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	155,520	714,443	4.5939
05 Jul 2011 12:50:35	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	129,600	595,925	4.5982
03 Jul 2011 21:51:02	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	103,680	476,854	4.5993
02 Jul 2011 10:11:52	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	77,760	357,760	4.6008
30 Jun 2011 22:29:03	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	51,840	238,651	4.6036
29 Jun 2011 10:29:39	1045545	13013108	hadcm3n_t1g2_1940_40_007311207_1	25,920	119,528	4.6114