Task 15465436

Name	hadcm3n_o3ua_1980_40_008254596_1
Workunit	8409720
Created	28 Nov 2012, 13:54:06 UTC
Sent	28 Nov 2012, 13:59:51 UTC
Report deadline	27 Feb 2013, 21:27:02 UTC
Received	6 Jan 2013, 5:25:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1197259
Run time	35 days 12 hours 19 min 38 sec
CPU time	20 days 21 hours 7 min 57 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	3.19 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:52:57 (21135): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:15:53 (2085): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x49bee04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x49bee00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x59b9804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x71a5404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x71a5400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x59b9804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x51a3c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x51a3c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(48013,0xacee62c0) malloc: * error for object 0x81b3c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Sat Jan 5 22:43:41 2013 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9e8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/10/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90931000 - 0x90960fff /usr/lib/system/libsystem_info.dylib 0x90a0b000 - 0x90a21fff /usr/lib/system/libxpc.dylib 0x90d46000 - 0x90d74fff /usr/lib/libSystem.B.dylib 0x9143e000 - 0x9144cfff /usr/lib/libz.1.dylib 0x925c4000 - 0x925c5fff /usr/lib/system/libsystem_blocks.dylib 0x926dd000 - 0x926ddfff /usr/lib/system/libdnsinfo.dylib 0x92a51000 - 0x92a55fff /usr/lib/system/libsystem_network.dylib 0x93681000 - 0x93686fff /usr/lib/system/libmacho.dylib 0x93754000 - 0x93755fff /usr/lib/system/libunc.dylib 0x93a2f000 - 0x93a38fff /usr/lib/libc++abi.dylib 0x93a39000 - 0x93a47fff /usr/lib/system/libdispatch.dylib 0x93e81000 - 0x93e82fff /usr/lib/system/libquarantine.dylib 0x9556c000 - 0x95574fff /usr/lib/system/libcopyfile.dylib 0x95a0f000 - 0x95a16fff /usr/lib/system/libsystem_dnssd.dylib 0x95a6e000 - 0x95a71fff /usr/lib/system/libcompiler_rt.dylib 0x95ca0000 - 0x95ca1fff /usr/lib/system/libremovefile.dylib 0x95cf5000 - 0x95cf6fff /usr/lib/system/libsystem_sandbox.dylib 0x95cf7000 - 0x95cfffff /usr/lib/system/libunwind.dylib 0x964fb000 - 0x96502fff /usr/lib/system/libsystem_notify.dylib 0x96abd000 - 0x96ac5fff /usr/lib/system/liblaunch.dylib 0x97422000 - 0x97426fff /usr/lib/system/libcache.dylib 0x98431000 - 0x98433fff /usr/lib/system/libdyld.dylib 0x98aa5000 - 0x98aa8fff /usr/lib/system/libmathCommon.A.dylib 0x9a0d3000 - 0x9a135fff /usr/lib/libstdc++.6.dylib 0x9a789000 - 0x9a789fff /usr/lib/system/libkeymgr.dylib 0x9a78f000 - 0x9a85afff /usr/lib/system/libsystem_c.dylib 0x9c266000 - 0x9c284fff /usr/lib/system/libsystem_kernel.dylib 0x9c2a6000 - 0x9c2e9fff /usr/lib/system/libcommonCrypto.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Jan 2013 05:30:21	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	1,036,800	1,804,073	1.7400
04 Jan 2013 15:40:56	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	1,010,880	1,764,387	1.7454
03 Jan 2013 20:05:21	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	984,960	1,723,351	1.7497
03 Jan 2013 01:39:21	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	959,040	1,682,416	1.7543
02 Jan 2013 01:43:35	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	933,120	1,635,654	1.7529
01 Jan 2013 04:50:33	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	907,200	1,591,864	1.7547
31 Dec 2012 07:58:17	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	881,280	1,544,398	1.7524
30 Dec 2012 11:30:08	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	855,360	1,497,137	1.7503
29 Dec 2012 16:37:32	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	829,440	1,450,014	1.7482
28 Dec 2012 21:11:51	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	803,520	1,402,917	1.7460
28 Dec 2012 03:18:17	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	777,600	1,355,839	1.7436
27 Dec 2012 08:29:20	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	751,680	1,308,462	1.7407
26 Dec 2012 12:29:44	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	725,760	1,261,792	1.7386
25 Dec 2012 14:50:49	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	699,840	1,215,017	1.7361
24 Dec 2012 16:01:07	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	673,920	1,168,252	1.7335
23 Dec 2012 16:34:49	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	648,000	1,123,118	1.7332
22 Dec 2012 16:15:54	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	622,080	1,077,187	1.7316
21 Dec 2012 16:47:31	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	596,160	1,029,850	1.7275
20 Dec 2012 16:52:45	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	570,240	982,750	1.7234
19 Dec 2012 16:53:21	1197259	15465436	hadcm3n_o3ua_1980_40_008254596_1	544,320	936,206	1.7200