Task 15693131

Name	hadcm3n_4doa_1980_40_008341081_1
Workunit	8491942
Created	29 Mar 2013, 10:33:43 UTC
Sent	29 Mar 2013, 10:36:13 UTC
Report deadline	28 Jun 2013, 18:03:24 UTC
Received	8 Jun 2013, 22:30:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1168218
Run time	13 days 2 hours 16 min 56 sec
CPU time	10 days 21 hours 4 min 58 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.65</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(76638,0xa0608540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(76638,0xa0608540) malloc: * error for object 0x1810e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(76638,0xa0608540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(76638,0xa0608540) malloc: * error for object 0x1810e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(76638,0xa0608540) malloc: * error for object 0x1810e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(24705,0xa0608540) malloc: * error for object 0x3039200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Thu Apr 18 02:24:29 2013 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. 16:32:04 (24705): No heartbeat from core client for 30 sec - exiting 16:32:05 (24705): No heartbeat from core client for 30 sec - exiting 16:32:06 (24705): No heartbeat from core client for 30 sec - exiting 16:32:07 (24705): No heartbeat from core client for 30 sec - exiting 16:34:59 (24705): No heartbeat from core client for 30 sec - exiting 16:35:00 (24705): No heartbeat from core client for 30 sec - exiting 16:35:01 (24705): No heartbeat from core client for 30 sec - exiting 16:35:02 (24705): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(693,0xa04a9540) malloc: * error for object 0x2800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(693,0xa04a9540) malloc: * error for object 0x2800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(693,0xa04a9540) malloc: * error for object 0x481d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Mon May 13 13:49:20 2013 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. 0 libSystem.B.dylib 0x947ffb03 small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x947fc5cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x947fb5e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a worker() + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: hadcm3n_6.07_i686-apple-darwin(73265,0xa04a9540) malloc: error for object 0x4829c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sat Jun 8 10:01:02 2013 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. 0 libSystem.B.dylib 0x947ffb03 small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x947fc5cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x947fb5e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a worker() + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x947f4c0e mach_wait_until + 10 1 libSystem.B.dylib 0x9487c429 nanosleep + 345 2 libSystem.B.dylib 0x9487c2ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x94822259 _pthread_start + 345 6 libSystem.B.dylib 0x948220de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f978 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/4/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x947f4000 - 0x9499bfff /usr/lib/libSystem.B.dylib 0x949e1000 - 0x949effff /usr/lib/libz.1.dylib 0x95371000 - 0x95374fff /usr/lib/system/libmathCommon.A.dylib 0x96cdd000 - 0x96d47fff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Apr 2013 18:55:09	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	518,400	899,359	1.7349
16 Apr 2013 12:41:00	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	492,480	855,507	1.7371
14 Apr 2013 21:21:21	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	466,560	810,951	1.7381
13 Apr 2013 20:06:07	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	440,640	765,498	1.7372
13 Apr 2013 00:35:46	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	414,720	720,434	1.7372
12 Apr 2013 07:54:01	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	388,800	675,402	1.7371
11 Apr 2013 15:15:30	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	362,880	630,153	1.7365
10 Apr 2013 20:34:04	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	336,960	586,088	1.7393
10 Apr 2013 04:46:45	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	311,040	540,413	1.7374
09 Apr 2013 11:10:53	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	285,120	493,172	1.7297
08 Apr 2013 18:46:56	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	259,200	446,332	1.7220
08 Apr 2013 02:39:48	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	233,280	399,765	1.7137
07 Apr 2013 11:48:53	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	207,360	354,290	1.7086
06 Apr 2013 18:40:26	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	181,440	309,430	1.7054
06 Apr 2013 02:51:16	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	155,520	264,054	1.6979
03 Apr 2013 18:52:07	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	129,600	220,163	1.6988
02 Apr 2013 10:39:20	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	103,680	175,775	1.6954
31 Mar 2013 06:08:28	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	77,760	131,776	1.6947
30 Mar 2013 15:50:41	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	51,840	87,839	1.6944
30 Mar 2013 02:02:12	1168218	15693131	hadcm3n_4doa_1980_40_008341081_1	25,920	43,832	1.6910