Task 13013706

Name	hadcm3n_t1xc_1940_40_007311449_1
Workunit	7508879
Created	27 Jun 2011, 23:14:22 UTC
Sent	27 Jun 2011, 23:14:30 UTC
Report deadline	27 Sep 2011, 6:41:41 UTC
Received	15 Jul 2011, 17:37:00 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1045545
Run time	14 days 14 hours 47 min 52 sec
CPU time	13 days 14 hours 30 min 34 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	1.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.59</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=27403, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t1xcko.pjf0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcko.pjf0c10 Error: Input file: dataout/t1xcko.pif0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcko.pif0c10 Error: Input file: dataout/t1xcko.pff0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcko.pff0c10 Error: Input file: dataout/t1xcko.pcf0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcko.pcf0c10 Error: Input file: dataout/t1xcko.pbf0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcko.pbf0c10 Error: Input file: dataout/t1xcko.paf0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcko.paf0c10 Error: Input file: dataout/t1xcka.phf0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcka.phf0c10 Error: Input file: dataout/t1xcka.pgf0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcka.pgf0c10 Error: Input file: dataout/t1xcka.pef0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcka.pef0c10 Error: Input file: dataout/t1xcka.pdf0c10 is not a valid UM file. Error converting file to netcdf: dataout/t1xcka.pdf0c10 * glibc detected * ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x097d3f98 *** ======= Backtrace: ========= /lib/i386-linux-gnu/libc.so.6(+0x6b961)[0x17b961] /lib/i386-linux-gnu/libc.so.6(+0x6d28b)[0x17d28b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x18041d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0x8ec4d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0x8ec52d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x126e37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00110000-0026a000 r-xp 00000000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 0026a000-0026b000 ---p 0015a000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 0026b000-0026d000 r--p 0015a000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 0026d000-0026e000 rw-p 0015c000 08:01 1006 /lib/i386-linux-gnu/libc-2.13.so 0026e000-00271000 rw-p 00000000 00:00 0 00292000-002ae000 r-xp 00000000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 002ae000-002af000 r--p 0001b000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 002af000-002b0000 rw-p 0001c000 08:01 5596 /lib/i386-linux-gnu/ld-2.13.so 004c2000-004c4000 r-xp 00000000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 004c4000-004c5000 r--p 00001000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 004c5000-004c6000 rw-p 00002000 08:01 5593 /lib/i386-linux-gnu/libdl-2.13.so 00842000-00921000 r-xp 00000000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00921000-00925000 r--p 000de000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00925000-00926000 rw-p 000e2000 08:01 140335 /usr/lib/i386-linux-gnu/libstdc++.so.6.0.14 00926000-0092d000 rw-p 00000000 00:00 0 0098d000-0098e000 r-xp 00000000 00:00 0 [vdso] 009b6000-009da000 r-xp 00000000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 009da000-009db000 r--p 00023000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 009db000-009dc000 rw-p 00024000 08:01 5595 /lib/i386-linux-gnu/libm-2.13.so 00a92000-00aac000 r-xp 00000000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 00aac000-00aad000 r--p 00019000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 00aad000-00aae000 rw-p 0001a000 08:01 900 /lib/i386-linux-gnu/libgcc_s.so.1 00b99000-00bae000 r-xp 00000000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00bae000-00baf000 r--p 00015000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00baf000-00bb0000 rw-p 00016000 08:01 5591 /lib/i386-linux-gnu/libpthread-2.13.so 00bb0000-00bb2000 rw-p 00000000 00:00 0 08048000-080e3000 r-xp 00000000 08:01 262884 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 262884 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 0977e000-097e4000 rw-p 00000000 00:00 0 [heap] b7200000-b7221000 rw-p 00000000 00:00 0 b7221000-b7300000 ---p 00000000 00:00 0 b73bd000-b7838000 rw-s 00000000 08:01 528221 /var/lib/boinc-client/slots/3/136595 b7838000-b783c000 rw-p 00000000 00:00 0 b784e000-b784f000 rw-p 00000000 00:00 0 b784f000-b7850000 ---p 00000000 00:00 0 b7850000-b7853000 rw-p 00000000 00:00 0 b7853000-b7855000 rw-s 00000000 08:01 527864 /var/lib/boinc-client/slots/3/boinc_mmap_file b7855000-b7857000 rw-p 00000000 00:00 0 bfddb000-bfe4a000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0x98d400] [0x98d416] /lib/i386-linux-gnu/libc.so.6(gsignal+0x51)[0x13ae71] /lib/i386-linux-gnu/libc.so.6(abort+0x17e)[0x13e34e] /lib/i386-linux-gnu/libc.so.6(+0x61577)[0x171577] /lib/i386-linux-gnu/libc.so.6(+0x6b961)[0x17b961] /lib/i386-linux-gnu/libc.so.6(+0x6d28b)[0x17d28b] /lib/i386-linux-gnu/libc.so.6(cfree+0x6d)[0x18041d] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdlPv+0x21)[0x8ec4d1] /usr/lib/i386-linux-gnu/libstdc++.so.6(_ZdaPv+0x1d)[0x8ec52d] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/i386-linux-gnu/libc.so.6(__libc_start_main+0xe7)[0x126e37] ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 14:30:39	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	259,200	1,175,424	4.5348
25 Jul 2011 14:30:38	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	233,280	1,058,339	4.5368
09 Jul 2011 21:02:06	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	207,360	941,043	4.5382
08 Jul 2011 14:29:25	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	181,440	824,126	4.5421
07 Jul 2011 15:39:43	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	155,520	706,824	4.5449
05 Jul 2011 11:49:55	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	129,600	589,749	4.5505
03 Jul 2011 22:51:37	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	103,680	472,103	4.5535
02 Jul 2011 11:12:26	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	77,760	353,766	4.5495
30 Jun 2011 23:29:25	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	51,840	235,495	4.5427
29 Jun 2011 11:30:04	1045545	13013706	hadcm3n_t1xc_1940_40_007311449_1	25,920	117,666	4.5396