Task 13600590

Name	hadcm3n_o2jr_1940_40_007536091_0
Workunit	7733323
Created	5 Nov 2011, 11:28:41 UTC
Sent	17 Nov 2011, 12:10:30 UTC
Report deadline	16 Feb 2012, 19:37:41 UTC
Received	30 Nov 2011, 6:24:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1167372
Run time	6 days 18 hours 17 min 25 sec
CPU time	6 days 15 hours 7 min 13 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.99 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 18:33:49 (24666): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:46:44 (29878): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:19:29 (30777): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:48:01 (2993): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:57:55 (3017): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:08:48 (3055): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:46 (3102): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:29:23 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:05 (4221): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:37:42 (4249): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:53:32 (4259): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:28:56 (6469): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 01:37:37 (8393): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 03:59:58 (9518): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:02:21 (9535): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:03:57 (9549): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:04:43 (9562): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:39:35 (9580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 06:40:57 (10042): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 06:41:43 (10056): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Invalid argument BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 07:03:17 (10067): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:03:59 (10115): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:05:25 (10125): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:08:13 (10137): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:24:38 (10604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:06 (10662): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:44:59 (10730): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:20:02 (11178): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:20:57 (11208): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:22:28 (11224): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:23:45 (11237): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:26:38 (11253): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:01:15 (11276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:07 (11377): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:07:05 (11390): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:14:45 (11415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:24:00 (13637): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x094d04a8 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb74aabc2] /lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb74ab862] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb74ae94d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xb76ad80f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xb76ad86b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb7455113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 08048000-080e3000 r-xp 00000000 fc:03 795628 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 fc:03 795628 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0947b000-094e0000 rw-p 00000000 00:00 0 [heap] b6e00000-b6e21000 rw-p 00000000 00:00 0 b6e21000-b6f00000 ---p 00000000 00:00 0 b6fbe000-b7439000 rw-s 00000000 fc:03 1705963 /var/lib/boinc-client/slots/5/136705 b7439000-b743c000 rw-p 00000000 00:00 0 b743c000-b75b2000 r-xp 00000000 fc:03 9965631 /lib/i386-linux-gnu/libc-2.13.so b75b2000-b75b4000 r--p 00176000 fc:03 9965631 /lib/i386-linux-gnu/libc-2.13.so b75b4000-b75b5000 rw-p 00178000 fc:03 9965631 /lib/i386-linux-gnu/libc-2.13.so b75b5000-b75b8000 rw-p 00000000 00:00 0 b75b8000-b75d4000 r-xp 00000000 fc:03 9965627 /lib/i386-linux-gnu/libgcc_s.so.1 b75d4000-b75d5000 r--p 0001b000 fc:03 9965627 /lib/i386-linux-gnu/libgcc_s.so.1 b75d5000-b75d6000 rw-p 0001c000 fc:03 9965627 /lib/i386-linux-gnu/libgcc_s.so.1 b75d6000-b75fe000 r-xp 00000000 fc:03 9965635 /lib/i386-linux-gnu/libm-2.13.so b75fe000-b75ff000 r--p 00028000 fc:03 9965635 /lib/i386-linux-gnu/libm-2.13.so b75ff000-b7600000 rw-p 00029000 fc:03 9965635 /lib/i386-linux-gnu/libm-2.13.so b7600000-b76de000 r-xp 00000000 fc:03 15729125 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76de000-b76df000 ---p 000de000 fc:03 15729125 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76df000-b76e3000 r--p 000de000 fc:03 15729125 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76e3000-b76e4000 rw-p 000e2000 fc:03 15729125 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.16 b76e4000-b76ec000 rw-p 00000000 00:00 0 b76ec000-b76ef000 r-xp 00000000 fc:03 9965634 /lib/i386-linux-gnu/libdl-2.13.so b76ef000-b76f0000 r--p 00002000 fc:03 9965634 /lib/i386-linux-gnu/libdl-2.13.so b76f0000-b76f1000 rw-p 00003000 fc:03 9965634 /lib/i386-linux-gnu/libdl-2.13.so b76f1000-b7708000 r-xp 00000000 fc:03 9965645 /lib/i386-linux-gnu/libpthread-2.13.so b7708000-b7709000 r--p 00016000 fc:03 9965645 /lib/i386-linux-gnu/libpthread-2.13.so b7709000-b770a000 rw-p 00017000 fc:03 9965645 /lib/i386-linux-gnu/libpthread-2.13.so b770a000-b770c000 rw-p 00000000 00:00 0 b771b000-b771c000 rw-p 00000000 00:00 0 b771c000-b771d000 ---p 00000000 00:00 0 b771d000-b7720000 rw-p 00000000 00:00 0 b7720000-b7722000 rw-s 00000000 fc:03 1705960 /var/lib/boinc-client/slots/5/boinc_mmap_file b7722000-b7724000 rw-p 00000000 00:00 0 b7724000-b7725000 r-xp 00000000 00:00 0 [vdso] b7725000-b7743000 r-xp 00000000 fc:03 9965628 /lib/i386-linux-gnu/ld-2.13.so b7743000-b7744000 r--p 0001d000 fc:03 9965628 /lib/i386-linux-gnu/ld-2.13.so b7744000-b7745000 rw-p 0001e000 fc:03 9965628 /lib/i386-linux-gnu/ld-2.13.so bfd82000-bfdf1000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb7724400] [0xb7724424] /lib/i386-linux-gnu/libc.so.6(gsignal+0x4f)[0xb7469c8f] /lib/i386-linux-gnu/libc.so.6(abort+0x175)[0xb746d2b5] /lib/i386-linux-gnu/libc.so.6(+0x63dfc)[0xb749fdfc] /lib/i386-linux-gnu/libc.so.6(+0x6ebc2)[0xb74aabc2] /lib/i386-linux-gnu/libc.so.6(+0x6f862)[0xb74ab862] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0xb74ae94d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x1f)[0xb76ad80f] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1b)[0xb76ad86b] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xf3)[0xb7455113] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Nov 2011 23:48:46	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	259,200	572,823	2.2100
28 Nov 2011 17:22:55	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	233,280	516,554	2.2143
27 Nov 2011 00:33:13	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	207,360	460,138	2.2190
22 Nov 2011 09:54:04	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	181,440	403,192	2.2222
21 Nov 2011 17:05:43	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	155,520	344,643	2.2161
20 Nov 2011 22:45:57	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	129,600	287,342	2.2171
20 Nov 2011 06:12:17	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	103,680	230,045	2.2188
19 Nov 2011 13:35:05	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	77,760	171,522	2.2058
18 Nov 2011 20:59:56	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	51,840	113,894	2.1970
18 Nov 2011 04:33:05	1167372	13600590	hadcm3n_o2jr_1940_40_007536091_0	25,920	56,843	2.1930