Task 13384086

Name	hadcm3n_ydro_1940_40_007450089_4
Workunit	7647592
Created	13 Sep 2011, 19:48:23 UTC
Sent	13 Sep 2011, 20:03:36 UTC
Report deadline	14 Dec 2011, 3:30:47 UTC
Received	5 Nov 2011, 4:46:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1115260
Run time	11 days 6 hours 5 min 37 sec
CPU time	10 days 5 hours 4 min 49 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.88 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:27:16 (31807): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3056804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x1030604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x1030600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3056804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3056800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3820604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3821604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3821600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3821604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3821600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3821604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3821600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(57113,0xa0066540) malloc: * error for object 0x3821600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sat Nov 5 03:45:48 2011 Thread 0 Crashed: atos cannot load symbols for the file ../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin. 0 libSystem.B.dylib 0x90bd4b0f small_free_list_remove_ptr + 246 1 libSystem.B.dylib 0x90bd15cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x90bd05e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c mainLoop() + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 worker() + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x90bc9c0e mach_wait_until + 10 1 libSystem.B.dylib 0x90c51429 nanosleep + 345 2 libSystem.B.dylib 0x90c512ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x90bf7259 _pthread_start + 345 6 libSystem.B.dylib 0x90bf70de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f5d8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/5/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90bc9000 - 0x90d70fff /usr/lib/libSystem.B.dylib 0x95adc000 - 0x95b46fff /usr/lib/libstdc++.6.dylib 0x9647f000 - 0x96482fff /usr/lib/system/libmathCommon.A.dylib 0x9783e000 - 0x9784cfff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
05 Nov 2011 03:49:12	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	518,400	882,280	1.7019
03 Nov 2011 03:03:54	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	492,480	837,576	1.7007
02 Nov 2011 00:57:47	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	466,560	792,875	1.6994
31 Oct 2011 23:26:05	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	440,640	748,374	1.6984
31 Oct 2011 17:18:03	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	414,720	703,557	1.6965
31 Oct 2011 16:41:33	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	388,800	658,718	1.6942
31 Oct 2011 15:16:49	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	362,880	614,501	1.6934
31 Oct 2011 15:16:49	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	336,960	570,679	1.6936
07 Oct 2011 04:51:10	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	311,040	527,299	1.6953
06 Oct 2011 03:07:31	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	285,120	482,130	1.6910
05 Oct 2011 01:12:32	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	259,200	437,307	1.6871
03 Oct 2011 23:02:36	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	233,280	393,709	1.6877
02 Oct 2011 20:41:01	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	207,360	349,690	1.6864
01 Oct 2011 20:40:17	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	181,440	307,870	1.6968
30 Sep 2011 07:04:24	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	155,520	264,495	1.7007
29 Sep 2011 04:34:29	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	129,600	220,576	1.7020
28 Sep 2011 02:24:29	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	103,680	175,604	1.6937
27 Sep 2011 00:07:20	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	77,760	129,907	1.6706
19 Sep 2011 22:05:06	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	51,840	85,008	1.6398
15 Sep 2011 20:45:39	1115260	13384086	hadcm3n_ydro_1940_40_007450089_4	25,920	41,488	1.6006