Task 13407845

Name	hadcm3n_u5dh_1980_40_007458799_1
Workunit	7656302
Created	22 Sep 2011, 16:28:46 UTC
Sent	25 Sep 2011, 19:09:37 UTC
Report deadline	26 Dec 2011, 2:36:48 UTC
Received	13 Oct 2011, 4:35:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1132881
Run time	6 days 15 hours 13 min 37 sec
CPU time	6 days 8 hours 34 min 34 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:51:52 (4258): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09d60f50 *** ======= Backtrace: ========= /lib32/libc.so.6(+0x6c231)[0xf7512231] /lib32/libc.so.6(+0x6dab8)[0xf7513ab8] /lib32/libc.so.6(cfree+0x6d)[0xf7516b9d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7700a91] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf7700aed] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf74bcbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 08:01 34735704 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 34735704 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 09d0b000-09d71000 rw-p 00000000 00:00 0 [heap] f6f00000-f6f21000 rw-p 00000000 00:00 0 f6f21000-f7000000 ---p 00000000 00:00 0 f7029000-f74a4000 rw-s 00000000 08:01 34744406 /var/lib/boinc-client/slots/0/139025 f74a4000-f74a6000 rw-p 00000000 00:00 0 f74a6000-f75f9000 r-xp 00000000 08:01 17090749 /lib32/libc-2.11.1.so f75f9000-f75fa000 ---p 00153000 08:01 17090749 /lib32/libc-2.11.1.so f75fa000-f75fc000 r--p 00153000 08:01 17090749 /lib32/libc-2.11.1.so f75fc000-f75fd000 rw-p 00155000 08:01 17090749 /lib32/libc-2.11.1.so f75fd000-f7600000 rw-p 00000000 00:00 0 f7600000-f761d000 r-xp 00000000 08:01 35651812 /usr/lib32/libgcc_s.so.1 f761d000-f761e000 r--p 0001c000 08:01 35651812 /usr/lib32/libgcc_s.so.1 f761e000-f761f000 rw-p 0001d000 08:01 35651812 /usr/lib32/libgcc_s.so.1 f761f000-f7643000 r-xp 00000000 08:01 17090753 /lib32/libm-2.11.1.so f7643000-f7644000 r--p 00023000 08:01 17090753 /lib32/libm-2.11.1.so f7644000-f7645000 rw-p 00024000 08:01 17090753 /lib32/libm-2.11.1.so f7645000-f772e000 r-xp 00000000 08:01 35651903 /usr/lib32/libstdc++.so.6.0.13 f772e000-f772f000 ---p 000e9000 08:01 35651903 /usr/lib32/libstdc++.so.6.0.13 f772f000-f7733000 r--p 000e9000 08:01 35651903 /usr/lib32/libstdc++.so.6.0.13 f7733000-f7734000 rw-p 000ed000 08:01 35651903 /usr/lib32/libstdc++.so.6.0.13 f7734000-f773b000 rw-p 00000000 00:00 0 f773b000-f773d000 r-xp 00000000 08:01 17090752 /lib32/libdl-2.11.1.so f773d000-f773e000 r--p 00001000 08:01 17090752 /lib32/libdl-2.11.1.so f773e000-f773f000 rw-p 00002000 08:01 17090752 /lib32/libdl-2.11.1.so f773f000-f7740000 rw-p 00000000 00:00 0 f7740000-f7755000 r-xp 00000000 08:01 17090763 /lib32/libpthread-2.11.1.so f7755000-f7756000 r--p 00014000 08:01 17090763 /lib32/libpthread-2.11.1.so f7756000-f7757000 rw-p 00015000 08:01 17090763 /lib32/libpthread-2.11.1.so f7757000-f7759000 rw-p 00000000 00:00 0 f776f000-f7770000 rw-p 00000000 00:00 0 f7770000-f7771000 ---p 00000000 00:00 0 f7771000-f7774000 rw-p 00000000 00:00 0 f7774000-f7776000 rw-s 00000000 08:01 34743857 /var/lib/boinc-client/slots/0/boinc_mmap_file f7776000-f7778000 rw-p 00000000 00:00 0 f7778000-f7779000 r-xp 00000000 00:00 0 [vdso] f7779000-f7795000 r-xp 00000000 08:01 17049571 /lib32/ld-2.11.1.so f7795000-f7796000 r--p 0001b000 08:01 17049571 /lib32/ld-2.11.1.so f7796000-f7797000 rw-p 0001c000 08:01 17049571 /lib32/ld-2.11.1.so ff841000-ff8b1000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xf7778400] [0xf7778430] /lib32/libc.so.6(gsignal+0x51)[0xf74d0921] /lib32/libc.so.6(abort+0x182)[0xf74d3d52] /lib32/libc.so.6(+0x6213d)[0xf750813d] /lib32/libc.so.6(+0x6c231)[0xf7512231] /lib32/libc.so.6(+0x6dab8)[0xf7513ab8] /lib32/libc.so.6(cfree+0x6d)[0xf7516b9d] /usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf7700a91] /usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf7700aed] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib32/libc.so.6(__libc_start_main+0xe6)[0xf74bcbd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Oct 2011 03:39:02	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	259,200	549,269	2.1191
10 Oct 2011 19:53:45	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	233,280	494,279	2.1188
09 Oct 2011 18:59:57	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	207,360	439,513	2.1196
08 Oct 2011 08:33:19	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	181,440	384,611	2.1198
07 Oct 2011 07:03:56	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	155,520	329,061	2.1159
05 Oct 2011 12:33:54	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	129,600	274,275	2.1163
04 Oct 2011 02:00:14	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	103,680	219,406	2.1162
30 Sep 2011 18:32:16	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	77,760	164,193	2.1115
30 Sep 2011 01:08:41	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	51,840	109,262	2.1077
27 Sep 2011 03:54:56	1132881	13407845	hadcm3n_u5dh_1980_40_007458799_1	25,920	54,534	2.1039