Task 12759238

Name	hadcm3n_o305_1900_40_007199224_2
Workunit	7397504
Created	31 Mar 2011, 16:29:36 UTC
Sent	31 Mar 2011, 16:35:01 UTC
Report deadline	1 Jul 2011, 0:02:12 UTC
Received	15 Jul 2011, 14:16:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1122605
Run time	17 days 8 hours 32 min 30 sec
CPU time	16 days 1 hours 0 min 19 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x601f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x603ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x603ac00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x501f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x501f600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x603ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x603ac00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x5837a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x8831e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x802e404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x802e400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x7800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x7800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(37590,0xa0a99540) malloc: * error for object 0x7800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=37611, selfPID=37611, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(62073,0xa0a99540) malloc: * error for object 0x701b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62073,0xa0a99540) malloc: * error for object 0x8021200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(8027,0xa0a99540) malloc: * error for object 0x501b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8027,0xa0a99540) malloc: * error for object 0x501b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8027,0xa0a99540) malloc: * error for object 0x4027604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8027,0xa0a99540) malloc: * error for object 0x4027600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8027,0xa0a99540) malloc: * error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.7 build 10J869 Tue Jul 12 00:01:41 2011 hadcm3n_6.07_i686-apple-darwin(8027,0xa0a99540) malloc: * error for object 0x8020404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Thread 0 Crashed: 0 libSystem.B.dylib 0x986a10a3 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x9869db6c _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x9869cb88 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a __Z6workerv + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x986961ae _mach_wait_until + 10 1 libSystem.B.dylib 0x9871d809 _nanosleep + 345 2 libSystem.B.dylib 0x9871d6aa _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x986c37fd __pthread_start + 345 6 libSystem.B.dylib 0x986c3682 _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fc48 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/14/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x932a4000 - 0x932b2fff /usr/lib/libz.1.dylib 0x98695000 - 0x9883cfff /usr/lib/libSystem.B.dylib 0x98aa4000 - 0x98b0efff /usr/lib/libstdc++.6.dylib 0x98b21000 - 0x98b24fff /usr/lib/system/libmathCommon.A.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Jul 2011 15:40:25	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	777,600	1,385,790	1.7821
30 Jun 2011 21:13:28	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	751,680	1,338,697	1.7809
28 Jun 2011 00:18:05	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	725,760	1,291,219	1.7791
22 Jun 2011 18:56:28	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	699,840	1,243,908	1.7774
20 Jun 2011 17:46:06	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	673,920	1,196,265	1.7751
15 Jun 2011 17:35:32	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	648,000	1,149,055	1.7732
12 Jun 2011 16:05:43	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	622,080	1,102,047	1.7716
10 Jun 2011 16:02:37	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	596,160	1,058,908	1.7762
07 Jun 2011 14:57:59	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	570,240	1,011,798	1.7743
01 Jun 2011 21:22:00	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	544,320	964,657	1.7722
29 May 2011 20:59:59	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	518,400	917,426	1.7697
27 May 2011 20:20:11	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	492,480	867,969	1.7624
25 May 2011 21:02:42	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	466,560	826,648	1.7718
23 May 2011 20:33:35	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	440,640	776,586	1.7624
21 May 2011 14:52:12	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	414,720	735,263	1.7729
17 May 2011 21:16:20	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	388,800	685,020	1.7619
12 May 2011 21:14:01	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	362,880	638,081	1.7584
10 May 2011 20:10:17	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	336,960	589,949	1.7508
08 May 2011 20:01:11	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	311,040	543,443	1.7472
06 May 2011 20:43:04	1122605	12759238	hadcm3n_o305_1900_40_007199224_2	285,120	500,625	1.7558