Task 16294982

Name	hadcm3n_86ck_1980_40_008514731_0
Workunit	8662243
Created	26 Feb 2014, 16:04:32 UTC
Sent	26 Feb 2014, 17:14:15 UTC
Report deadline	29 May 2014, 0:41:26 UTC
Received	21 Mar 2014, 20:18:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1197259
Run time	21 days 12 hours 56 min 41 sec
CPU time	15 days 17 hours 46 min 11 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.20 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> 01:30:10 (35924): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:50:31 (55329): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:36:33 (14505): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:59:05 (72116): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x697ba04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x697ba00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(96086,0xacc932c0) malloc: * error for object 0x6180c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:40:58 (2778): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:59:33 (9795): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(56805,0xacc932c0) malloc: * error for object 0x2090a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(56805,0xacc932c0) malloc: * error for object 0x2090a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(56805,0xacc932c0) malloc: * error for object 0x887b000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(56805,0xacc932c0) malloc: * error for object 0xa873a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Fri Mar 21 05:59:28 2014 hadcm3n_6.07_i686-apple-darwin(98320,0xacc932c0) malloc: * error for object 0x4b4de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98320,0xacc932c0) malloc: * error for object 0x4b4de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98320,0xacc932c0) malloc: * error for object 0x3b4de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98320,0xacc932c0) malloc: * error for object 0x3b4de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98320,0xacc932c0) malloc: * error for object 0x3b4de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98320,0xacc932c0) malloc: * error for object 0x3b4de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(98320,0xacc932c0) malloc: * error for object 0x3b4de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Fri Mar 21 15:10:54 2014 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9f8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/5/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x901d6000 - 0x901d7fff /usr/lib/system/libquarantine.dylib 0x903eb000 - 0x903edfff /usr/lib/system/libdyld.dylib 0x9045b000 - 0x9045ffff /usr/lib/system/libcache.dylib 0x9169f000 - 0x916a7fff /usr/lib/system/libcopyfile.dylib 0x91ab9000 - 0x91ae7fff /usr/lib/libSystem.B.dylib 0x91b76000 - 0x91b77fff /usr/lib/system/libsystem_blocks.dylib 0x9225a000 - 0x92262fff /usr/lib/system/libunwind.dylib 0x922cf000 - 0x922d8fff /usr/lib/libc++abi.dylib 0x922d9000 - 0x922e7fff /usr/lib/system/libdispatch.dylib 0x92995000 - 0x92996fff /usr/lib/system/libunc.dylib 0x92b5b000 - 0x92b71fff /usr/lib/system/libxpc.dylib 0x931b5000 - 0x931f8fff /usr/lib/system/libcommonCrypto.dylib 0x95672000 - 0x95680fff /usr/lib/libz.1.dylib 0x95bb0000 - 0x95bb3fff /usr/lib/system/libmathCommon.A.dylib 0x95bb4000 - 0x95bb5fff /usr/lib/system/libsystem_sandbox.dylib 0x95c42000 - 0x95c46fff /usr/lib/system/libsystem_network.dylib 0x96626000 - 0x96688fff /usr/lib/libstdc++.6.dylib 0x96691000 - 0x96694fff /usr/lib/system/libcompiler_rt.dylib 0x974b6000 - 0x974d4fff /usr/lib/system/libsystem_kernel.dylib 0x974e5000 - 0x975b0fff /usr/lib/system/libsystem_c.dylib 0x978fb000 - 0x97902fff /usr/lib/system/libsystem_notify.dylib 0x98875000 - 0x9887cfff /usr/lib/system/libsystem_dnssd.dylib 0x9887d000 - 0x9887efff /usr/lib/system/libremovefile.dylib 0x98d93000 - 0x98dc2fff /usr/lib/system/libsystem_info.dylib 0x99113000 - 0x9911bfff /usr/lib/system/liblaunch.dylib 0x995e3000 - 0x995e3fff /usr/lib/system/libdnsinfo.dylib 0x995e4000 - 0x995e4fff /usr/lib/system/libkeymgr.dylib 0x9c49d000 - 0x9c4a2fff /usr/lib/system/libmacho.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Mar 2014 11:46:03	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	777,600	1,341,883	1.7257
20 Mar 2014 18:17:41	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	751,680	1,293,815	1.7212
20 Mar 2014 01:39:28	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	725,760	1,245,340	1.7159
19 Mar 2014 05:01:09	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	699,840	1,197,087	1.7105
18 Mar 2014 11:38:38	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	673,920	1,148,567	1.7043
17 Mar 2014 18:11:51	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	648,000	1,099,995	1.6975
17 Mar 2014 02:14:28	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	622,080	1,051,469	1.6902
16 Mar 2014 07:01:47	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	596,160	1,002,878	1.6822
15 Mar 2014 13:21:32	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	570,240	954,350	1.6736
14 Mar 2014 19:32:49	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	544,320	905,963	1.6644
14 Mar 2014 02:22:51	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	518,400	857,231	1.6536
13 Mar 2014 09:43:14	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	492,480	808,865	1.6424
12 Mar 2014 20:20:03	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	466,560	774,985	1.6611
12 Mar 2014 04:01:44	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	440,640	731,348	1.6597
10 Mar 2014 19:40:33	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	414,720	684,604	1.6508
10 Mar 2014 01:12:58	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	388,800	637,455	1.6395
09 Mar 2014 03:24:12	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	362,880	590,369	1.6269
08 Mar 2014 09:35:06	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	336,960	543,548	1.6131
07 Mar 2014 14:39:41	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	311,040	496,584	1.5965
06 Mar 2014 19:44:01	1197259	16294982	hadcm3n_86ck_1980_40_008514731_0	285,120	449,191	1.5754