Task 19039144

Name	hadcm3n_sdne_1940_40_010020615_3
Workunit	10018686
Created	26 Oct 2015, 17:33:24 UTC
Sent	26 Oct 2015, 17:33:41 UTC
Report deadline	26 Jan 2016, 1:00:52 UTC
Received	15 Dec 2015, 12:05:50 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1408816
Run time	22 days 20 hours 7 min 54 sec
CPU time	17 days 5 hours 40 min 36 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.06 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.12</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:30:37 (2085): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 10:30:13 (74003): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 18:05:44 (746): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:37:21 (712): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:11:48 (77066): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:59:55 (935): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:01:18 (6261): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(693,0xa3562000) malloc: * error for object 0x200a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(693,0xa3562000) malloc: * error for object 0x200a600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(693,0xa3562000) malloc: * error for object 0x200a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(693,0xa3562000) malloc: * error for object 0x200a600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(693,0xa3562000) malloc: * error for object 0x200a600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(697,0xa3562000) malloc: * error for object 0x100a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(697,0xa3562000) malloc: * error for object 0x100a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(697,0xa3562000) malloc: * error for object 0x280e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(697,0xa3562000) malloc: * error for object 0x280e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(697,0xa3562000) malloc: * error for object 0x280e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(697,0xa3562000) malloc: * error for object 0x280e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(697,0xa3562000) malloc: * error for object 0x280e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) 09:14:23 (697): No heartbeat from core client for 30 sec - exiting hadcm3n_6.07_i686-apple-darwin(717,0xa3562000) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(717,0xa3562000) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(717,0xa3562000) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(717,0xa3562000) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(717,0xa3562000) malloc: * error for object 0x81e804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(717,0xa3562000) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(717,0xa3562000) malloc: * error for object 0x81e800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(1054,0xa3562000) malloc: * error for object 0x1002804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1054,0xa3562000) malloc: * error for object 0x1002800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1054,0xa3562000) malloc: * error for object 0x1002804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1054,0xa3562000) malloc: * error for object 0x1002800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1054,0xa3562000) malloc: * error for object 0x1002800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.1 build 15B42 hadcm3n_6.07_i686-apple-darwin(1054,0xa3562000) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1054,0xa3562000) malloc: * error for object 0x800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Tue Dec 8 05:37:55 2015 hadcm3n_6.07_i686-apple-darwin(859,0xa3562000) malloc: * error for object 0x833204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(859,0xa3562000) malloc: * error for object 0x833200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(859,0xa3562000) malloc: * error for object 0x833204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(859,0xa3562000) malloc: * error for object 0x833200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(859,0xa3562000) malloc: * error for object 0x833204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(859,0xa3562000) malloc: * error for object 0x833200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(859,0xa3562000) malloc: * error for object 0x833200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.1 build 15B42 Mon Dec 14 19:27:59 2015 hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x1015c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x1015c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x2800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4675,0xa3562000) malloc: * error for object 0x2800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.1 build 15B42 Tue Dec 15 04:04:14 2015 Thread 0 Crashed: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 1: Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fd48 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/12/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90010000 - 0x90038fff /usr/lib/system/libxpc.dylib 0x9016d000 - 0x90202fff /usr/lib/system/libsystem_c.dylib 0x90203000 - 0x90206fff /usr/lib/system/libdyld.dylib 0x9091e000 - 0x90937fff /usr/lib/system/libsystem_asl.dylib 0x90d5f000 - 0x90d60fff /usr/lib/system/libremovefile.dylib 0x90d61000 - 0x90d66fff /usr/lib/system/libmacho.dylib 0x90d67000 - 0x90d67fff /usr/lib/system/libunc.dylib 0x90dfc000 - 0x90e11fff /usr/lib/system/libsystem_coretls.dylib 0x91721000 - 0x9172ffff /usr/lib/libz.1.dylib 0x945be000 - 0x945c1fff /usr/lib/system/libsystem_sandbox.dylib 0x946f1000 - 0x94714fff /usr/lib/libc++abi.dylib 0x94767000 - 0x94767fff /usr/lib/system/liblaunch.dylib 0x95810000 - 0x95829fff /usr/lib/system/libsystem_malloc.dylib 0x95d24000 - 0x95d2dfff /usr/lib/system/libsystem_networkextension.dylib 0x962a4000 - 0x962acfff /usr/lib/system/libsystem_pthread.dylib 0x9722e000 - 0x97281fff /usr/lib/libstdc++.6.dylib 0x97901000 - 0x97909fff /usr/lib/system/libsystem_dnssd.dylib 0x97a10000 - 0x97a12fff /usr/lib/system/libquarantine.dylib 0x981d3000 - 0x981d3fff /usr/lib/system/libsystem_blocks.dylib 0x981d4000 - 0x981ddfff /usr/lib/system/libcopyfile.dylib 0x981de000 - 0x98208fff /usr/lib/system/libdispatch.dylib 0x98209000 - 0x9826ffff /usr/lib/system/libsystem_network.dylib 0x983d7000 - 0x983d9fff /usr/lib/system/libsystem_secinit.dylib 0x989a0000 - 0x989a0fff /usr/lib/system/libkeymgr.dylib 0x99019000 - 0x9901bfff /usr/lib/system/libsystem_configuration.dylib 0x994e8000 - 0x99508fff /usr/lib/system/libsystem_kernel.dylib 0x9a05e000 - 0x9a063fff /usr/lib/system/libcompiler_rt.dylib 0x9bb33000 - 0x9bb39fff /usr/lib/system/libunwind.dylib 0x9bf28000 - 0x9bf32fff /usr/lib/system/libsystem_notify.dylib 0x9c1b8000 - 0x9c1bcfff /usr/lib/system/libcache.dylib 0x9c1c8000 - 0x9c22ffff /usr/lib/system/libcorecrypto.dylib 0x9c230000 - 0x9c263fff /usr/lib/system/libsystem_m.dylib 0x9cb7e000 - 0x9cb89fff /usr/lib/system/libcommonCrypto.dylib 0x9d035000 - 0x9d03bfff /usr/lib/system/libsystem_platform.dylib 0x9d05f000 - 0x9d088fff /usr/lib/system/libsystem_info.dylib 0x9e516000 - 0x9e517fff /usr/lib/libSystem.B.dylib 0x9eb2f000 - 0x9eb40fff /usr/lib/system/libsystem_trace.dylib 0x9ed37000 - 0x9ed39fff /usr/lib/system/libsystem_coreservices.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Nov 2015 08:13:12	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	777,600	1,466,104	1.8854
28 Nov 2015 03:32:30	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	751,680	1,416,491	1.8844
27 Nov 2015 05:25:14	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	725,760	1,367,917	1.8848
26 Nov 2015 09:42:13	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	699,840	1,316,029	1.8805
25 Nov 2015 14:12:10	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	673,920	1,264,009	1.8756
24 Nov 2015 18:44:08	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	648,000	1,212,279	1.8708
23 Nov 2015 22:58:14	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	622,080	1,160,426	1.8654
23 Nov 2015 02:32:13	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	596,160	1,107,089	1.8570
22 Nov 2015 06:10:06	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	570,240	1,054,558	1.8493
21 Nov 2015 11:25:31	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	544,320	1,005,849	1.8479
20 Nov 2015 17:16:31	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	518,400	955,876	1.8439
17 Nov 2015 17:18:17	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	492,480	905,745	1.8392
16 Nov 2015 21:55:41	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	466,560	855,291	1.8332
16 Nov 2015 04:36:07	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	440,640	803,927	1.8245
15 Nov 2015 12:15:15	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	414,720	754,656	1.8197
14 Nov 2015 17:26:26	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	388,800	705,714	1.8151
13 Nov 2015 22:45:49	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	362,880	658,175	1.8138
13 Nov 2015 14:19:18	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	336,960	610,205	1.8109
12 Nov 2015 10:46:09	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	311,040	562,659	1.8090
11 Nov 2015 17:06:04	1287225	19039144	hadcm3n_sdne_1940_40_010020615_3	285,120	515,934	1.8095