Task 15433034

Name	hadcm3n_o5fa_2020_40_008203292_3
Workunit	8358416
Created	11 Nov 2012, 20:57:05 UTC
Sent	11 Nov 2012, 20:57:11 UTC
Report deadline	11 Feb 2013, 4:24:22 UTC
Received	1 Dec 2012, 19:46:03 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1228742
Run time	9 days 14 hours 49 min 35 sec
CPU time	7 days 12 hours 9 min 2 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.31 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:13:07 (3499): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:01:48 (3428): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:29:45 (29481): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:56 (29690): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x22bbe04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x22bbe00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0xa9fa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0xabb004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0xabb000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x2aac404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x2ac7a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x2ac7a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0xabb004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0xabb000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x2ac7a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x2ac7a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x22a0804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x22bbe04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(10971,0xac3ea2c0) malloc: * error for object 0x22bbe00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63b Sat Dec 1 13:45:23 2012 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fa38 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/1/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9007e000 - 0x9008cfff /usr/lib/system/libdispatch.dylib 0x908a3000 - 0x908e6fff /usr/lib/system/libcommonCrypto.dylib 0x909df000 - 0x909fdfff /usr/lib/system/libsystem_kernel.dylib 0x9138e000 - 0x9138ffff /usr/lib/system/libremovefile.dylib 0x92c4b000 - 0x92c4efff /usr/lib/system/libcompiler_rt.dylib 0x93712000 - 0x93715fff /usr/lib/system/libmathCommon.A.dylib 0x93a39000 - 0x93b04fff /usr/lib/system/libsystem_c.dylib 0x94a41000 - 0x94a49fff /usr/lib/system/liblaunch.dylib 0x94b91000 - 0x94b92fff /usr/lib/system/libquarantine.dylib 0x94b93000 - 0x94b94fff /usr/lib/system/libunc.dylib 0x950cb000 - 0x950d4fff /usr/lib/libc++abi.dylib 0x95626000 - 0x9562afff /usr/lib/system/libsystem_network.dylib 0x95656000 - 0x9565dfff /usr/lib/system/libsystem_notify.dylib 0x963d7000 - 0x96406fff /usr/lib/system/libsystem_info.dylib 0x96b84000 - 0x96b86fff /usr/lib/system/libdyld.dylib 0x96db3000 - 0x96db4fff /usr/lib/system/libsystem_blocks.dylib 0x96db5000 - 0x96db5fff /usr/lib/system/libkeymgr.dylib 0x97c3e000 - 0x97c45fff /usr/lib/system/libsystem_dnssd.dylib 0x990f5000 - 0x990f5fff /usr/lib/system/libdnsinfo.dylib 0x990f9000 - 0x99101fff /usr/lib/system/libcopyfile.dylib 0x99102000 - 0x99106fff /usr/lib/system/libcache.dylib 0x99c38000 - 0x99c3dfff /usr/lib/system/libmacho.dylib 0x9ad51000 - 0x9ad5ffff /usr/lib/libz.1.dylib 0x9af79000 - 0x9af81fff /usr/lib/system/libunwind.dylib 0x9c12e000 - 0x9c190fff /usr/lib/libstdc++.6.dylib 0x9c201000 - 0x9c202fff /usr/lib/system/libsystem_sandbox.dylib 0x9c203000 - 0x9c219fff /usr/lib/system/libxpc.dylib 0x9c34a000 - 0x9c378fff /usr/lib/libSystem.B.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Dec 2012 18:47:55	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	777,600	648,538	0.8340
01 Dec 2012 09:21:23	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	751,680	624,959	0.8314
01 Dec 2012 00:59:30	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	725,760	601,430	0.8287
26 Nov 2012 09:45:22	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	699,840	577,842	0.8257
25 Nov 2012 18:38:19	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	673,920	554,234	0.8224
25 Nov 2012 10:15:41	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	648,000	530,823	0.8192
25 Nov 2012 01:38:45	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	622,080	507,268	0.8154
24 Nov 2012 17:15:24	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	596,160	483,738	0.8114
24 Nov 2012 08:33:23	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	570,240	460,203	0.8070
23 Nov 2012 18:20:27	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	544,320	436,686	0.8023
23 Nov 2012 09:55:47	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	518,400	413,089	0.7969
23 Nov 2012 01:03:50	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	492,480	389,462	0.7908
22 Nov 2012 16:25:44	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	466,560	365,827	0.7841
22 Nov 2012 07:42:17	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	440,640	342,240	0.7767
21 Nov 2012 22:55:42	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	414,720	318,580	0.7682
21 Nov 2012 13:59:14	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	388,800	294,868	0.7584
21 Nov 2012 05:08:07	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	362,880	271,182	0.7473
15 Nov 2012 16:45:31	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	336,960	248,967	0.7389
15 Nov 2012 11:44:42	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	311,040	230,573	0.7413
15 Nov 2012 04:48:53	1228742	15433034	hadcm3n_o5fa_2020_40_008203292_3	285,120	212,204	0.7443