Task 12916568

Name	hadcm3n_p53u_1900_40_007224202_2
Workunit	7422442
Created	1 Jun 2011, 2:11:06 UTC
Sent	1 Jun 2011, 2:11:09 UTC
Report deadline	31 Aug 2011, 9:38:20 UTC
Received	17 Jun 2011, 8:37:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1077752
Run time	12 days 16 hours 49 min 29 sec
CPU time	9 days 3 hours 39 min 45 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.21 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(99967,0xa01ed540) malloc: * error for object 0x1012204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99967,0xa01ed540) malloc: * error for object 0x1012200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(99967,0xa01ed540) malloc: * error for object 0x1012200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=99972, selfPID=99972, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(5093,0xa01ed540) malloc: * error for object 0x103de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5093,0xa01ed540) malloc: * error for object 0x103de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(5093,0xa01ed540) malloc: * error for object 0x103de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(9117,0xa01ed540) malloc: * error for object 0x201dc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9117,0xa01ed540) malloc: * error for object 0x201dc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(9117,0xa01ed540) malloc: * error for object 0x801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(18854,0xa01ed540) malloc: * error for object 0x183e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(18854,0xa01ed540) malloc: * error for object 0x183e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(18854,0xa01ed540) malloc: * error for object 0x802004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(18854,0xa01ed540) malloc: * error for object 0x802000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(18854,0xa01ed540) malloc: * error for object 0x802004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x826204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x826200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x1015e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x1015e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x826204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x826200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x826204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x826200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(28826,0xa01ed540) malloc: * error for object 0x826200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x2001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x2001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(29728,0xa01ed540) malloc: * error for object 0x281de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(95714,0xa01ed540) malloc: * error for object 0x81fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95714,0xa01ed540) malloc: * error for object 0x83b004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95714,0xa01ed540) malloc: * error for object 0x83b000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95714,0xa01ed540) malloc: * error for object 0x282b004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95714,0xa01ed540) malloc: * error for object 0x282b000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95714,0xa01ed540) malloc: * error for object 0x1038a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(95714,0xa01ed540) malloc: * error for object 0x2039e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Thu Jun 16 18:16:43 2011 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin. Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=95717, selfPID=95717, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x1054604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x1054604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x82ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x281f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x82ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x2028604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x1054604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1118,0xa01ed540) malloc: * error for object 0x281de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Fri Jun 17 04:36:49 2011 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin. 0 libSystem.B.dylib 0x922d80a3 small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x922d4b6c szone_free_definite_size + 3457 2 libSystem.B.dylib 0x922d3b88 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a worker() + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x922cd1ae mach_wait_until + 10 1 libSystem.B.dylib 0x92354809 nanosleep + 345 2 libSystem.B.dylib 0x923546aa usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x922fa7fd _pthread_start + 345 6 libSystem.B.dylib 0x922fa682 thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9f8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/2/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x91ca3000 - 0x91cb1fff /usr/lib/libz.1.dylib 0x922cc000 - 0x92473fff /usr/lib/libSystem.B.dylib 0x942cb000 - 0x94335fff /usr/lib/libstdc++.6.dylib 0x959a5000 - 0x959a8fff /usr/lib/system/libmathCommon.A.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Jun 2011 22:19:44	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	518,400	761,605	1.4691
16 Jun 2011 09:35:57	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	492,480	723,681	1.4695
15 Jun 2011 20:25:22	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	466,560	685,366	1.4690
15 Jun 2011 02:52:17	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	440,640	646,464	1.4671
14 Jun 2011 11:27:23	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	414,720	608,173	1.4665
13 Jun 2011 17:57:59	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	388,800	568,035	1.4610
13 Jun 2011 02:01:56	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	362,880	528,862	1.4574
12 Jun 2011 06:37:25	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	336,960	487,958	1.4481
11 Jun 2011 17:35:28	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	311,040	449,139	1.4440
11 Jun 2011 04:18:16	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	285,120	410,420	1.4395
07 Jun 2011 23:58:22	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	259,200	371,272	1.4324
07 Jun 2011 04:31:16	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	233,280	332,287	1.4244
06 Jun 2011 14:44:46	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	207,360	293,759	1.4167
06 Jun 2011 04:48:20	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	181,440	260,650	1.4366
05 Jun 2011 18:22:58	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	155,520	226,626	1.4572
05 Jun 2011 06:34:56	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	129,600	189,151	1.4595
04 Jun 2011 19:08:18	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	103,680	151,539	1.4616
04 Jun 2011 06:32:06	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	77,760	113,273	1.4567
03 Jun 2011 18:36:53	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	51,840	75,134	1.4493
03 Jun 2011 02:56:41	1077752	12916568	hadcm3n_p53u_1900_40_007224202_2	25,920	37,375	1.4419