Task 16008324

Name	hadcm3n_8202_1980_40_008460533_1
Workunit	8611389
Created	8 Sep 2013, 15:23:47 UTC
Sent	8 Sep 2013, 16:05:42 UTC
Report deadline	8 Dec 2013, 23:32:53 UTC
Received	2 Nov 2013, 11:01:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1001397
Run time	16 days 23 hours 30 min 21 sec
CPU time	15 days 16 hours 25 min 11 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(61178,0xa0c87540) malloc: * error for object 0x3027804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(61178,0xa0c87540) malloc: * error for object 0x3042e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(61178,0xa0c87540) malloc: * error for object 0x3042e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(61178,0xa0c87540) malloc: * error for object 0x839804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(61178,0xa0c87540) malloc: * error for object 0x3820a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(61178,0xa0c87540) malloc: * error for object 0x3820a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=61185, selfPID=61185, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x301f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x103ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x4800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x4800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x3001c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(211,0xa0c87540) malloc: * error for object 0x3001c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(38022,0xa0c87540) malloc: * error for object 0x381f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(38022,0xa0c87540) malloc: * error for object 0x4029c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(38022,0xa0c87540) malloc: * error for object 0x4029c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(38022,0xa0c87540) malloc: * error for object 0x4029c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sat Nov 2 22:00:04 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x9a4c4b03 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x9a4c15cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x9a4c05e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000876a __Z6workerv + 2896 7 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 8 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x9a4b9c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x9a541429 _nanosleep + 345 2 libSystem.B.dylib 0x9a5412ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9a4e7259 __pthread_start + 345 6 libSystem.B.dylib 0x9a4e70de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fb58 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/2/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9173a000 - 0x917a4fff /usr/lib/libstdc++.6.dylib 0x940e8000 - 0x940f6fff /usr/lib/libz.1.dylib 0x9a4b9000 - 0x9a660fff /usr/lib/libSystem.B.dylib 0x9abda000 - 0x9abddfff /usr/lib/system/libmathCommon.A.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Oct 2013 13:15:23	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	777,600	1,332,391	1.7135
29 Oct 2013 18:06:43	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	751,680	1,288,093	1.7136
28 Oct 2013 14:05:03	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	725,760	1,243,652	1.7136
26 Oct 2013 18:07:10	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	699,840	1,199,266	1.7136
25 Oct 2013 14:05:29	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	673,920	1,154,888	1.7137
23 Oct 2013 18:55:31	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	648,000	1,110,481	1.7137
22 Oct 2013 13:40:42	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	622,080	1,065,984	1.7136
20 Oct 2013 18:18:02	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	596,160	1,021,654	1.7137
19 Oct 2013 13:52:58	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	570,240	977,267	1.7138
17 Oct 2013 18:29:07	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	544,320	932,867	1.7138
16 Oct 2013 14:02:42	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	518,400	888,397	1.7137
14 Oct 2013 18:27:22	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	492,480	843,839	1.7134
13 Oct 2013 14:09:09	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	466,560	799,315	1.7132
11 Oct 2013 18:47:58	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	440,640	754,791	1.7129
10 Oct 2013 14:39:58	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	414,720	710,310	1.7127
08 Oct 2013 19:25:47	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	388,800	665,910	1.7127
07 Oct 2013 15:26:24	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	362,880	621,485	1.7126
05 Oct 2013 20:18:14	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	336,960	577,056	1.7125
04 Oct 2013 15:12:18	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	311,040	532,812	1.7130
02 Oct 2013 19:54:49	1001397	16008324	hadcm3n_8202_1980_40_008460533_1	285,120	488,596	1.7137