Task 13558192

Name	hadcm3n_yj25_1900_40_007526217_0
Workunit	7723692
Created	28 Oct 2011, 13:44:50 UTC
Sent	29 Oct 2011, 17:39:05 UTC
Report deadline	29 Jan 2012, 1:06:16 UTC
Received	7 Nov 2011, 10:10:40 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1177195
Run time	6 days 5 hours 34 min 47 sec
CPU time	5 days 8 hours 42 min 49 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	3.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Atmos Hold Restart file rename failed on atmos_restart.hold CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/yj25ko.pjb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pjb0c10 Error: Input file: dataout/yj25ko.pib0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pib0c10 Error: Input file: dataout/yj25ko.pfb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pfb0c10 Error: Input file: dataout/yj25ko.pcb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pcb0c10 Error: Input file: dataout/yj25ko.pbb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pbb0c10 Error: Input file: dataout/yj25ko.pab0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pab0c10 Error: Input file: dataout/yj25ka.phb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.phb0c10 Error: Input file: dataout/yj25ka.pgb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.pgb0c10 Error: Input file: dataout/yj25ka.peb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.peb0c10 Error: Input file: dataout/yj25ka.pdb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.pdb0c10 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/yj25ko.pjb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pjb0c10 Error: Input file: dataout/yj25ko.pib0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pib0c10 Error: Input file: dataout/yj25ko.pfb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pfb0c10 Error: Input file: dataout/yj25ko.pcb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pcb0c10 Error: Input file: dataout/yj25ko.pbb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pbb0c10 Error: Input file: dataout/yj25ko.pab0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ko.pab0c10 Error: Input file: dataout/yj25ka.phb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.phb0c10 Error: Input file: dataout/yj25ka.pgb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.pgb0c10 Error: Input file: dataout/yj25ka.peb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.peb0c10 Error: Input file: dataout/yj25ka.pdb0c10 is not a valid UM file. Error converting file to netcdf: dataout/yj25ka.pdb0c10 hadcm3n_6.07_i686-apple-darwin(26451,0xa053d540) malloc: * error for object 0x5006604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26451,0xa053d540) malloc: * error for object 0x5006600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26451,0xa053d540) malloc: * error for object 0x5006604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26451,0xa053d540) malloc: * error for object 0x5006600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(26451,0xa053d540) malloc: * error for object 0x5006604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Nov 6 15:04:06 2011 Thread 0 Crashed: Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=26459, selfPID=26459, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(27092,0xa053d540) malloc: * error for object 0x804e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27092,0xa053d540) malloc: * error for object 0x804e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27092,0xa053d540) malloc: * error for object 0x280e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27092,0xa053d540) malloc: * error for object 0x280e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27092,0xa053d540) malloc: * error for object 0x280e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27092,0xa053d540) malloc: * error for object 0x280e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(27092,0xa053d540) malloc: * error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(29103,0xa053d540) malloc: * error for object 0x4009e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31802,0xa053d540) malloc: * error for object 0x3029c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(32559,0xa053d540) malloc: * error for object 0x382de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon Nov 7 04:01:34 2011 Thread 0 Crashed: 0 libSystem.B.dylib 0x94131b03 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x9412e5cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x9412d5e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a __Z6workerv + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x94126c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x941ae429 _nanosleep + 345 2 libSystem.B.dylib 0x941ae2ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x94154259 __pthread_start + 345 6 libSystem.B.dylib 0x941540de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9f8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/2/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x91a34000 - 0x91a42fff /usr/lib/libz.1.dylib 0x94126000 - 0x942cdfff /usr/lib/libSystem.B.dylib 0x95a6e000 - 0x95a71fff /usr/lib/system/libmathCommon.A.dylib 0x9708c000 - 0x970f6fff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Nov 2011 21:05:22	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	259,200	463,268	1.7873
05 Nov 2011 16:32:08	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	233,280	417,232	1.7885
04 Nov 2011 20:58:58	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	207,360	371,720	1.7926
03 Nov 2011 17:06:54	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	181,440	323,854	1.7849
02 Nov 2011 21:07:59	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	155,520	279,670	1.7983
02 Nov 2011 05:17:25	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	129,600	233,358	1.8006
01 Nov 2011 14:18:00	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	103,680	185,727	1.7913
31 Oct 2011 21:06:36	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	77,760	138,720	1.7840
31 Oct 2011 19:36:11	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	51,840	89,933	1.7348
31 Oct 2011 19:06:54	1177195	13558192	hadcm3n_yj25_1900_40_007526217_0	25,920	43,290	1.6701