Task 13263082

Name	hadcm3n_o4rg_1980_40_007411347_1
Workunit	7608977
Created	16 Aug 2011, 18:24:33 UTC
Sent	16 Aug 2011, 18:25:04 UTC
Report deadline	16 Nov 2011, 1:52:15 UTC
Received	3 Sep 2011, 15:46:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1045545
Run time	15 days 1 hours 11 min 30 sec
CPU time	13 days 14 hours 30 min 47 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.13 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=30813, selfPID=30813, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:51:55 (16454): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x08248c58 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6b961)[0xc44961] /lib/i386-linux-gnu/libc.so.6(+0x6d28b)[0xc4628b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xc4941d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0xa214d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0xa2152d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0xbefe37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00124000-00148000 r-xp 00000000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 00148000-00149000 r--p 00023000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 00149000-0014a000 rw-p 00024000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 00333000-00348000 r-xp 00000000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00348000-00349000 r--p 00015000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00349000-0034a000 rw-p 00016000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 0034a000-0034c000 rw-p 00000000 00:00 0 0049f000-004a1000 r-xp 00000000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 004a1000-004a2000 r--p 00001000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 004a2000-004a3000 rw-p 00002000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 0062d000-0062e000 r-xp 00000000 00:00 0 [vdso] 006e1000-006fb000 r-xp 00000000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 006fb000-006fc000 r--p 00019000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 006fc000-006fd000 rw-p 0001a000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 00977000-00a56000 r-xp 00000000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00a56000-00a5a000 r--p 000de000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00a5a000-00a5b000 rw-p 000e2000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00a5b000-00a62000 rw-p 00000000 00:00 0 00b85000-00ba1000 r-xp 00000000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 00ba1000-00ba2000 r--p 0001b000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 00ba2000-00ba3000 rw-p 0001c000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 00bd9000-00d33000 r-xp 00000000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 00d33000-00d34000 ---p 0015a000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 00d34000-00d36000 r--p 0015a000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 00d36000-00d37000 rw-p 0015c000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 00d37000-00d3a000 rw-p 00000000 00:00 0 08048000-080e3000 r-xp 00000000 08:01 262884 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 262884 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 081f3000-08258000 rw-p 00000000 00:00 0 [heap] b7100000-b7121000 rw-p 00000000 00:00 0 b7121000-b7200000 ---p 00000000 00:00 0 b7269000-b76e4000 rw-s 00000000 08:01 539414 /var/lib/boinc-client/slots/2/136445 b76e4000-b76e8000 rw-p 00000000 00:00 0 b76fa000-b76fb000 rw-p 00000000 00:00 0 b76fb000-b76fc000 ---p 00000000 00:00 0 b76fc000-b76ff000 rw-p 00000000 00:00 0 b76ff000-b7701000 rw-s 00000000 08:01 527522 /var/lib/boinc-client/slots/2/boinc_mmap_file b7701000-b7703000 rw-p 00000000 00:00 0 bff00000-bff70000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x62d400] [0x62d416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x51)[0xc03e71] /lib/i386-linux-gnu/libc.so.6(abort+0x17e)[0xc0734e] /lib/i386-linux-gnu/libc.so.6(+0x61577)[0xc3a577] /lib/i386-linux-gnu/libc.so.6(+0x6b961)[0xc44961] /lib/i386-linux-gnu/libc.so.6(+0x6d28b)[0xc4628b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xc4941d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0xa214d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0xa2152d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0xbefe37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Sep 2011 14:44:45	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	259,200	1,175,440	4.5349
01 Sep 2011 09:51:59	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	233,280	1,056,446	4.5287
30 Aug 2011 15:21:39	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	207,360	939,822	4.5323
29 Aug 2011 00:54:58	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	181,440	821,918	4.5300
27 Aug 2011 12:27:26	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	155,520	704,029	4.5269
25 Aug 2011 22:28:30	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	129,600	586,110	4.5225
24 Aug 2011 08:49:12	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	103,680	467,883	4.5128
22 Aug 2011 20:17:18	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	77,760	350,194	4.5035
21 Aug 2011 07:15:42	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	51,840	232,242	4.4800
18 Aug 2011 09:23:14	1045545	13263082	hadcm3n_o4rg_1980_40_007411347_1	25,920	115,764	4.4662