Task 15841362

Name	hadcm3n_o38f_1980_40_008387468_2
Workunit	8538327
Created	13 Jun 2013, 10:36:40 UTC
Sent	13 Jun 2013, 10:59:54 UTC
Report deadline	12 Sep 2013, 18:27:05 UTC
Received	6 Jul 2013, 14:51:05 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1169903
Run time	18 days 23 hours 53 min 59 sec
CPU time	15 days 17 hours 41 min 22 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.15 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 23:07:40 (80148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:00:50 (99545): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:23:23 (2112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 04:01:44 (32621): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:07:02 (15495): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:57:09 (702): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(28797,0xa0829540) malloc: * error for object 0x1001804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28797,0xa0829540) malloc: * error for object 0x1002604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28797,0xa0829540) malloc: * error for object 0x1002600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28797,0xa0829540) malloc: * error for object 0x1002604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28797,0xa0829540) malloc: * error for object 0x1002600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28797,0xa0829540) malloc: * error for object 0x1002600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Jun 30 01:01:15 2013 hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x7836600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x801c404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x801d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(53938,0xa0829540) malloc: * error for object 0x801d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... 23:13:26 (72436): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:01:29 (29684): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:48:53 (83611): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 06:41:56 (5092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x383e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x383e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x383e604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x383e600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x681f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x6020604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x6020600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x4032604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x1006404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x1006400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(54378,0xa0829540) malloc: * error for object 0x801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sat Jul 6 09:41:28 2013 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. 0 libSystem.B.dylib 0x95bd5b0f small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x95bd25cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x95bd15e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c mainLoop() + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 worker() + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x95bcac0e mach_wait_until + 10 1 libSystem.B.dylib 0x95c52429 nanosleep + 345 2 libSystem.B.dylib 0x95c522ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x95bf8259 _pthread_start + 345 6 libSystem.B.dylib 0x95bf80de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f6a8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/1/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x916d8000 - 0x91742fff /usr/lib/libstdc++.6.dylib 0x95bca000 - 0x95d71fff /usr/lib/libSystem.B.dylib 0x9aa74000 - 0x9aa77fff /usr/lib/system/libmathCommon.A.dylib 0x9aadc000 - 0x9aaeafff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Jul 2013 14:54:10	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	777,600	1,359,676	1.7486
06 Jul 2013 05:38:42	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	751,680	1,317,177	1.7523
06 Jul 2013 04:57:07	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	725,760	1,274,844	1.7566
04 Jul 2013 17:48:05	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	699,840	1,230,123	1.7577
04 Jul 2013 14:20:55	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	673,920	1,187,353	1.7619
03 Jul 2013 15:23:44	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	648,000	1,145,318	1.7675
02 Jul 2013 22:38:03	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	622,080	1,101,978	1.7714
02 Jul 2013 12:05:51	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	596,160	1,058,340	1.7753
02 Jul 2013 11:48:28	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	570,240	1,013,775	1.7778
02 Jul 2013 11:03:57	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	544,320	970,171	1.7824
02 Jul 2013 10:31:52	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	518,400	925,367	1.7850
02 Jul 2013 10:12:39	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	492,480	880,975	1.7889
02 Jul 2013 09:50:24	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	466,560	835,760	1.7913
28 Jun 2013 04:36:44	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	440,640	790,419	1.7938
27 Jun 2013 12:05:26	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	414,720	746,515	1.8000
26 Jun 2013 15:02:35	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	388,800	699,442	1.7990
25 Jun 2013 22:27:22	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	362,880	652,948	1.7993
25 Jun 2013 03:43:55	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	336,960	608,282	1.8052
24 Jun 2013 12:36:42	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	311,040	563,275	1.8109
23 Jun 2013 11:17:47	1169903	15841362	hadcm3n_o38f_1980_40_008387468_2	285,120	518,405	1.8182