Task 13028819

Name	hadcm3n_t34m_1940_40_007317403_0
Workunit	7514833
Created	29 Jun 2011, 7:50:41 UTC
Sent	29 Jun 2011, 9:28:07 UTC
Report deadline	28 Sep 2011, 16:55:18 UTC
Received	15 Jul 2011, 14:52:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	810656
Run time	4 days 18 hours 54 min 23 sec
CPU time	4 days 18 hours 54 min 23 sec
Validate state	Invalid
Credit	3,110.40
Device peak FLOPS	2.86 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.2.15</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... OPEN: File Creation Failed: No space left on device OPEN: Unable to Open File dataout/t34mko.dae49e0 for Read/Write Model crashed: DUMPCTL : Fail to open output dump - may already exist tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t34mko.pje4c10 is not a valid UM file. Error converting file to netcdf: dataout/t34mko.pje4c10 Error: Input file: dataout/t34mko.pie4c10 is not a valid UM file. Error converting file to netcdf: dataout/t34mko.pie4c10 Error: Input file: dataout/t34mko.pfe4c10 is not a valid UM file. Error converting file to netcdf: dataout/t34mko.pfe4c10 Error: Input file: dataout/t34mka.phe4c10 is not a valid UM file. Error converting file to netcdf: dataout/t34mka.phe4c10 Error: Input file: dataout/t34mka.pge4c10 is not a valid UM file. Error converting file to netcdf: dataout/t34mka.pge4c10 Error: Input file: dataout/t34mka.pee4c10 is not a valid UM file. Error converting file to netcdf: dataout/t34mka.pee4c10 Error: Input file: dataout/t34mka.pde4c10 is not a valid UM file. Error converting file to netcdf: dataout/t34mka.pde4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 17:19:07 (26589): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(49662,0xa0808720) malloc: * error for object 0x805e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) hadcm3n_6.07_i686-apple-darwin(49662,0xa0808720) malloc: * error for object 0x805e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(49662,0xa0808720) malloc: * error for object 0x805e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=49667, selfPID=49667, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(69959,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(70512,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(70902,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(72055,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(72550,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(74625,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(75898,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(81435,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a hadcm3n_6.07_i686-apple-darwin(83244,0xa0808720) malloc: * error for object 0x803000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a 10:05:51 (337): No heartbeat from core client for 30 sec - exiting 10:05:52 (337): No heartbeat from core client for 30 sec - exiting 10:05:53 (337): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(337,0xa0808720) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Fri Jul 15 10:06:21 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin. 0 hadcm3n_6.07_i686-apple-darwin 0x00074953 1 hadcm3n_6.07_i686-apple-darwin 0x0006a936 2 libSystem.B.dylib 0x9602c1cb 3 ??? 0xffffffff 4 libSystem.B.dylib 0x95fc6e26 5 libSystem.B.dylib 0x95fc626d 6 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 7 hadcm3n_6.07_i686-apple-darwin 0x0000d36b 8 hadcm3n_6.07_i686-apple-darwin 0x000067ff 9 hadcm3n_6.07_i686-apple-darwin 0x0000876a 10 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 11 hadcm3n_6.07_i686-apple-darwin 0x00002676 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x95ff47c2 ecx: 0xbff905ac edx: 0x95fc0166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff905e8 esp: 0xbff905ac ss: 0x0000001f efl: 0x00000206 eip: 0x95fc0166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/1/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90a37000 - 0x90a94fff /usr/lib/libstdc++.6.dylib 0x93fc0000 - 0x93fc7fff /usr/lib/libgcc_s.1.dylib 0x95b2f000 - 0x95b33fff /usr/lib/system/libmathCommon.A.dylib 0x95fbf000 - 0x96126fff /usr/lib/libSystem.B.dylib 0x97083000 - 0x97091fff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Jul 2011 13:09:05	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	259,200	400,427	1.5449
25 Jul 2011 13:09:05	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	233,280	355,734	1.5249
25 Jul 2011 13:09:05	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	207,360	311,575	1.5026
25 Jul 2011 13:09:05	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	181,440	267,774	1.4758
07 Jul 2011 15:43:27	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	155,520	223,853	1.4394
07 Jul 2011 15:43:27	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	129,600	180,154	1.3901
05 Jul 2011 14:11:27	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	103,680	136,276	1.3144
04 Jul 2011 19:56:49	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	77,760	132,971	1.7100
30 Jun 2011 22:44:09	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	51,840	88,494	1.7071
30 Jun 2011 03:07:30	810656	13028819	hadcm3n_t34m_1940_40_007317403_0	25,920	44,009	1.6979