Task 16153091

Name	hadcm3n_zh4q_1960_40_008402414_4
Workunit	8553270
Created	21 Dec 2013, 22:27:35 UTC
Sent	21 Dec 2013, 22:27:41 UTC
Report deadline	23 Mar 2014, 5:54:52 UTC
Received	8 Mar 2014, 0:02:33 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1232508
Run time	14 days 20 hours 6 min 54 sec
CPU time	14 days 11 hours 58 min 14 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.2.39</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(1394,0xa05d0540) malloc: * error for object 0x4059004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1394,0xa05d0540) malloc: * error for object 0x4059000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1394,0xa05d0540) malloc: * error for object 0x303ce04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1394,0xa05d0540) malloc: * error for object 0x4824604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1394,0xa05d0540) malloc: * error for object 0x483fc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1394,0xa05d0540) malloc: * error for object 0x483fc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1394,0xa05d0540) malloc: * error for object 0x303ce04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Fri Mar 7 23:59:21 2014 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin. 0 libSystem.B.dylib 0x95de9b03 small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x95de65cc szone_free_definite_size + 3457 2 libSystem.B.dylib 0x95de55e8 free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x00022341 ncio_px_free + 97 4 hadcm3n_6.07_i686-apple-darwin 0x0002290c ncio_free + 28 5 hadcm3n_6.07_i686-apple-darwin 0x00022f09 ncio_close + 57 6 hadcm3n_6.07_i686-apple-darwin 0x0001cdd7 nc_close + 103 7 hadcm3n_6.07_i686-apple-darwin 0x0000bab9 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const, char const, int, int) + 3579 8 hadcm3n_6.07_i686-apple-darwin 0x0000d36b decadalMeans(int, char const) + 957 9 hadcm3n_6.07_i686-apple-darwin 0x000067ff doCM3Proc() + 185 10 hadcm3n_6.07_i686-apple-darwin 0x0000791c mainLoop() + 410 11 hadcm3n_6.07_i686-apple-darwin 0x000087c7 worker() + 2989 12 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 13 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x95ddec0e mach_wait_until + 10 1 libSystem.B.dylib 0x95e66429 nanosleep + 345 2 libSystem.B.dylib 0x95e662ca usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c boinc_sleep(double) + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 timer_thread(void) + 78 5 libSystem.B.dylib 0x95e0c259 _pthread_start + 345 6 libSystem.B.dylib 0x95e0c0de thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f5d8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/1/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x94c3e000 - 0x94c41fff /usr/lib/system/libmathCommon.A.dylib 0x95dde000 - 0x95f85fff /usr/lib/libSystem.B.dylib 0x96167000 - 0x96175fff /usr/lib/libz.1.dylib 0x99624000 - 0x9968efff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
07 Mar 2014 23:01:28	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	1,036,800	1,252,692	1.2082
06 Mar 2014 20:29:18	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	1,010,880	1,217,544	1.2044
01 Mar 2014 19:11:02	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	984,960	1,184,763	1.2029
01 Mar 2014 09:54:25	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	959,040	1,152,350	1.2016
01 Mar 2014 01:23:15	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	933,120	1,120,147	1.2004
25 Feb 2014 17:56:41	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	907,200	1,088,510	1.1999
23 Feb 2014 17:00:53	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	881,280	1,056,176	1.1985
18 Feb 2014 17:29:39	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	855,360	1,023,804	1.1969
15 Feb 2014 00:10:29	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	829,440	993,421	1.1977
13 Feb 2014 21:13:52	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	803,520	963,143	1.1987
11 Feb 2014 19:29:07	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	777,600	931,226	1.1976
10 Feb 2014 17:57:43	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	751,680	904,299	1.2030
08 Feb 2014 16:20:03	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	725,760	877,980	1.2097
07 Feb 2014 16:31:11	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	699,840	846,214	1.2092
02 Feb 2014 17:52:50	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	673,920	819,060	1.2154
31 Jan 2014 16:52:50	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	648,000	789,048	1.2177
26 Jan 2014 14:28:09	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	622,080	756,408	1.2159
26 Jan 2014 06:41:42	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	596,160	727,502	1.2203
25 Jan 2014 00:00:33	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	570,240	699,870	1.2273
21 Jan 2014 19:55:42	1232508	16153091	hadcm3n_zh4q_1960_40_008402414_4	544,320	667,593	1.2265