Task 16915429

Name	hadcm3n_8d2w_1980_40_008726371_3
Workunit	8872349
Created	22 Aug 2014, 7:26:25 UTC
Sent	22 Aug 2014, 7:28:19 UTC
Report deadline	21 Nov 2014, 14:55:30 UTC
Received	3 Sep 2014, 12:09:32 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1217152
Run time	6 days 16 hours 36 min 40 sec
CPU time	4 days 12 hours 17 min 56 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.90 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... 12:30:21 (59310): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:31:11 (66699): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:04:58 (37959): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 14:36:10 (62569): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:36:12 (62569): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Signal 15 received, exiting... Called boinc_finish Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:51:08 (80129): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... 14:06:27 (45851): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish Suspended CPDN Monitor - Suspend request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x0818abe0 *** ======= Backtrace: ========= /lib/libc.so.6[0x282c00f5] /lib/libc.so.6(cfree+0x96)[0x282c2146] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x281dff71] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x281dffcd] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe5)[0x282666e5] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 001c4000 00:00 235 080e3000-0815c000 rw-p 00079000 00:00 0 0815c000-081a0000 rw-p 00044000 00:00 0 280e3000-28103000 r-xp 00026000 00:00 78093 /compat/linux/lib/ld-2.9.so 28103000-28104000 r-xp 00026000 00:00 78093 /compat/linux/lib/ld-2.9.so 28104000-28105000 rw-p 00001000 00:00 0 28105000-28107000 rwxp 00002000 00:00 12533 /var/db/boinc/slots/2/boinc_mmap_file 28107000-28108000 rwxp 00001000 00:00 0 28109000-2810a000 rwxp 00001000 00:00 0 2810a000-28120000 r-xp 00021000 00:00 78067 /compat/linux/lib/libpthread-2.9.so 28120000-28121000 r-xp 00021000 00:00 78067 /compat/linux/lib/libpthread-2.9.so 28121000-28122000 rwxp 00021000 00:00 78067 /compat/linux/lib/libpthread-2.9.so 28122000-28124000 rwxp 00002000 00:00 0 28124000-28127000 r-xp 00005000 00:00 78064 /compat/linux/lib/libdl-2.9.so 28127000-28128000 r-xp 00005000 00:00 78064 /compat/linux/lib/libdl-2.9.so 28128000-28129000 rwxp 00005000 00:00 78064 /compat/linux/lib/libdl-2.9.so 28129000-2820e000 r-xp 000ea000 00:00 79826 /compat/linux/usr/lib/libstdc++.so.6.0.10 2820e000-28212000 r-xp 000ea000 00:00 79826 /compat/linux/usr/lib/libstdc++.so.6.0.10 28212000-28213000 rwxp 000ea000 00:00 79826 /compat/linux/usr/lib/libstdc++.so.6.0.10 28213000-28219000 rwxp 00006000 00:00 0 28219000-28240000 r-xp 00033000 00:00 78005 /compat/linux/lib/libm-2.9.so 28240000-28241000 r-xp 00033000 00:00 78005 /compat/linux/lib/libm-2.9.so 28241000-28242000 rwxp 00033000 00:00 78005 /compat/linux/lib/libm-2.9.so 28242000-2824f000 r-xp 0000e000 00:00 78001 /compat/linux/lib/libgcc_s-4.3.2-20081105.so.1 2824f000-28250000 rwxp 0000e000 00:00 78001 /compat/linux/lib/libgcc_s-4.3.2-20081105.so.1 28250000-283c2000 r-xp 001be000 00:00 78081 /compat/linux/lib/libc-2.9.so 283c2000-283c4000 r-xp 001be000 00:00 78081 /compat/linux/lib/libc-2.9.so 283c4000-283c5000 rwxp 001be000 00:00 78081 /compat/linux/lib/libc-2.9.so 283c5000-283ca000 rwxp 00005000 00:00 0 283ca000-283cb000 ---p 00001000 00:00 0 283cb000-283ce000 rwxp 00009000 00:00 0 283ce000-28849000 rwxp 0047b000 00:00 12641 /var/db/boinc/slots/2/134885 28900000-28921000 rwxp 001b7000 00:00 0 28921000-28a00000 ---p 001b7000 00:00 0 fff7e000-fffde000 rw-p 00060000 00:00 0 fffde000-ffffe000 rw-p 00020000 00:00 0 ffffe000-fffff000 r-xp 00001000 00:00 0 SIGABRT: abort called Stack trace (16 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xffffe1b7] /lib/libc.so.6(abort+0x188)[0x2827d158] /lib/libc.so.6[0x282b9c9d] /lib/libc.so.6[0x282c00f5] /lib/libc.so.6(cfree+0x96)[0x282c2146] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x281dff71] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x281dffcd] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/libc.so.6(__libc_start_main+0xe5)[0x282666e5] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Sep 2014 11:35:56	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	259,200	389,867	1.5041
02 Sep 2014 20:06:29	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	233,280	346,583	1.4857
01 Sep 2014 21:55:51	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	207,360	303,251	1.4624
01 Sep 2014 02:09:37	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	181,440	259,600	1.4308
30 Aug 2014 10:19:39	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	155,520	242,376	1.5585
29 Aug 2014 10:07:08	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	129,600	192,794	1.4876
28 Aug 2014 15:01:17	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	103,680	170,281	1.6424
25 Aug 2014 17:11:28	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	77,760	118,646	1.5258
24 Aug 2014 08:56:48	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	51,840	100,370	1.9361
23 Aug 2014 07:12:21	1217152	16915429	hadcm3n_8d2w_1980_40_008726371_3	25,920	49,384	1.9052