Task 12733373

Name	hadcm3n_o0lb_1900_40_007196098_1
Workunit	7394378
Created	28 Mar 2011, 13:57:25 UTC
Sent	2 Apr 2011, 23:36:25 UTC
Report deadline	3 Jul 2011, 7:03:36 UTC
Received	31 May 2011, 9:30:08 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	978632
Run time	55 days 21 hours 35 min 23 sec
CPU time	23 days 20 hours 10 min 45 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	0.76 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:07:06 (6563): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:07:47 (10430): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o0lbko.pja6c10 is not a valid UM file. Error converting file to netcdf: dataout/o0lbko.pja6c10 Error: Input file: dataout/o0lbko.pia6c10 is not a valid UM file. Error converting file to netcdf: dataout/o0lbko.pia6c10 Error: Input file: dataout/o0lbko.pfa6c10 is not a valid UM file. Error converting file to netcdf: dataout/o0lbko.pfa6c10 Error: Input file: dataout/o0lbka.pha6c10 is not a valid UM file. Error converting file to netcdf: dataout/o0lbka.pha6c10 Error: Input file: dataout/o0lbka.pga6c10 is not a valid UM file. Error converting file to netcdf: dataout/o0lbka.pga6c10 Error: Input file: dataout/o0lbka.pea6c10 is not a valid UM file. Error converting file to netcdf: dataout/o0lbka.pea6c10 Error: Input file: dataout/o0lbka.pda6c10 is not a valid UM file. Error converting file to netcdf: dataout/o0lbka.pda6c10 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:56:25 (1463): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x089d7620 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x17b591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x17cde8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x17fecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x53b741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x53b79d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x126bd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00110000-00263000 r-xp 00000000 08:01 1583930 /lib/tls/i686/cmov/libc-2.11.1.so 00263000-00264000 ---p 00153000 08:01 1583930 /lib/tls/i686/cmov/libc-2.11.1.so 00264000-00266000 r--p 00153000 08:01 1583930 /lib/tls/i686/cmov/libc-2.11.1.so 00266000-00267000 rw-p 00155000 08:01 1583930 /lib/tls/i686/cmov/libc-2.11.1.so 00267000-0026a000 rw-p 00000000 00:00 0 00450000-0046d000 r-xp 00000000 08:01 1564813 /lib/libgcc_s.so.1 0046d000-0046e000 r--p 0001c000 08:01 1564813 /lib/libgcc_s.so.1 0046e000-0046f000 rw-p 0001d000 08:01 1564813 /lib/libgcc_s.so.1 00480000-00569000 r-xp 00000000 08:01 509998 /usr/lib/libstdc++.so.6.0.13 00569000-0056a000 ---p 000e9000 08:01 509998 /usr/lib/libstdc++.so.6.0.13 0056a000-0056e000 r--p 000e9000 08:01 509998 /usr/lib/libstdc++.so.6.0.13 0056e000-0056f000 rw-p 000ed000 08:01 509998 /usr/lib/libstdc++.so.6.0.13 0056f000-00576000 rw-p 00000000 00:00 0 005ae000-005c3000 r-xp 00000000 08:01 1583973 /lib/tls/i686/cmov/libpthread-2.11.1.so 005c3000-005c4000 r--p 00014000 08:01 1583973 /lib/tls/i686/cmov/libpthread-2.11.1.so 005c4000-005c5000 rw-p 00015000 08:01 1583973 /lib/tls/i686/cmov/libpthread-2.11.1.so 005c5000-005c7000 rw-p 00000000 00:00 0 006c8000-006e3000 r-xp 00000000 08:01 1564677 /lib/ld-2.11.1.so 006e3000-006e4000 r--p 0001a000 08:01 1564677 /lib/ld-2.11.1.so 006e4000-006e5000 rw-p 0001b000 08:01 1564677 /lib/ld-2.11.1.so 00997000-009bb000 r-xp 00000000 08:01 1583934 /lib/tls/i686/cmov/libm-2.11.1.so 009bb000-009bc000 r--p 00023000 08:01 1583934 /lib/tls/i686/cmov/libm-2.11.1.so 009bc000-009bd000 rw-p 00024000 08:01 1583934 /lib/tls/i686/cmov/libm-2.11.1.so 00bd1000-00bd2000 r-xp 00000000 00:00 0 [vdso] 00cee000-00cf0000 r-xp 00000000 08:01 1583933 /lib/tls/i686/cmov/libdl-2.11.1.so 00cf0000-00cf1000 r--p 00001000 08:01 1583933 /lib/tls/i686/cmov/libdl-2.11.1.so 00cf1000-00cf2000 rw-p 00002000 08:01 1583933 /lib/tls/i686/cmov/libdl-2.11.1.so 08048000-080e3000 r-xp 00000000 08:01 966841 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 966841 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 08982000-089e8000 rw-p 00000000 00:00 0 [heap] b7200000-b7221000 rw-p 00000000 00:00 0 b7221000-b7300000 ---p 00000000 00:00 0 b739b000-b7816000 rw-s 00000000 08:01 999803 /var/lib/boinc-client/slots/0/136820 b7816000-b7819000 rw-p 00000000 00:00 0 b7829000-b782a000 rw-p 00000000 00:00 0 b782a000-b782b000 ---p 00000000 00:00 0 b782b000-b782e000 rw-p 00000000 00:00 0 b782e000-b7830000 rw-s 00000000 08:01 999797 /var/lib/boinc-client/slots/0/boinc_mmap_file b7830000-b7832000 rw-p 00000000 00:00 0 bfa26000-bfa95000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xbd1400] [0xbd1422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0x13a651] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0x13da82] /lib/tls/i686/cmov/libc.so.6(+0x6149d)[0x17149d] /lib/tls/i686/cmov/libc.so.6(+0x6b591)[0x17b591] /lib/tls/i686/cmov/libc.so.6(+0x6cde8)[0x17cde8] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x17fecd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x53b741] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x53b79d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x126bd6] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 May 2011 09:34:14	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	259,200	2,059,831	7.9469
25 May 2011 06:37:57	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	233,280	1,853,358	7.9448
19 May 2011 07:32:50	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	207,360	1,645,983	7.9378
13 May 2011 11:34:07	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	181,440	1,441,603	7.9453
07 May 2011 21:35:47	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	155,520	1,236,867	7.9531
02 May 2011 00:55:46	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	129,600	1,032,840	7.9694
26 Apr 2011 08:33:41	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	103,680	828,020	7.9863
20 Apr 2011 17:46:34	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	77,760	623,119	8.0134
20 Apr 2011 17:47:01	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	51,840	418,829	8.0793
08 Apr 2011 18:19:04	978632	12733373	hadcm3n_o0lb_1900_40_007196098_1	25,920	208,986	8.0627