Task 19477611

Name	hadcm3n_sdn3_198012_480_375_010415396_1
Workunit	10415396
Created	4 Apr 2016, 12:41:06 UTC
Sent	4 Apr 2016, 12:41:18 UTC
Report deadline	17 Mar 2017, 18:01:18 UTC
Received	24 Apr 2016, 12:57:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1223244
Run time	9 days 12 hours 56 min 36 sec
CPU time	8 days 9 hours 46 min 23 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.83 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:45:38 (750): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:46:18 (853): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(1836,0xa353d000) malloc: * error for object 0x802204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1836,0xa353d000) malloc: * error for object 0x802200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1836,0xa353d000) malloc: * error for object 0x802204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(1836,0xa353d000) malloc: * error for object 0x181a604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) hadcm3n_6.07_i686-apple-darwin(11167,0xa353d000) malloc: * error for object 0x100e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(11167,0xa353d000) malloc: * error for object 0x100e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(11167,0xa353d000) malloc: * error for object 0x100e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(11167,0xa353d000) malloc: * error for object 0x100e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(11167,0xa353d000) malloc: * error for object 0x100e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(601,0xa4435000) malloc: * error for object 0x100b204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(601,0xa4435000) malloc: * error for object 0x100b200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(601,0xa4435000) malloc: * error for object 0x807004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(601,0xa4435000) malloc: * error for object 0x807000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(601,0xa4435000) malloc: * error for object 0x800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.4 build 15E65 Fri Apr 22 07:25:56 2016 hadcm3n_6.07_i686-apple-darwin(712,0xa4435000) malloc: * error for object 0x380dc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(712,0xa4435000) malloc: * error for object 0x380dc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(712,0xa4435000) malloc: * error for object 0x380dc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(712,0xa4435000) malloc: * error for object 0x380dc00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(712,0xa4435000) malloc: * error for object 0x380dc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(1162,0xa4435000) malloc: * error for object 0x480ec04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1162,0xa4435000) malloc: * error for object 0x480ec00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1162,0xa4435000) malloc: * error for object 0x4800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1162,0xa4435000) malloc: * error for object 0x4800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1162,0xa4435000) malloc: * error for object 0x4800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(663,0xa4435000) malloc: * error for object 0x4801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(663,0xa4435000) malloc: * error for object 0x4801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(663,0xa4435000) malloc: * error for object 0x801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(663,0xa4435000) malloc: * error for object 0x3005e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(663,0xa4435000) malloc: * error for object 0x3007004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1442,0xa4435000) malloc: * error for object 0x380e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1442,0xa4435000) malloc: * error for object 0x380e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1442,0xa4435000) malloc: * error for object 0x380e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1442,0xa4435000) malloc: * error for object 0x380e200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1442,0xa4435000) malloc: * error for object 0x380e204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.4 build 15E65 Sat Apr 23 07:55:59 2016 hadcm3n_6.07_i686-apple-darwin(12740,0xa4435000) malloc: * error for object 0x816404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12740,0xa4435000) malloc: * error for object 0x816400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12740,0xa4435000) malloc: * error for object 0x816404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12740,0xa4435000) malloc: * error for object 0x816400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(12740,0xa4435000) malloc: * error for object 0x816404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(666,0xa4435000) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(666,0xa4435000) malloc: * error for object 0x2801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(666,0xa4435000) malloc: * error for object 0x2801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) hadcm3n_6.07_i686-apple-darwin(666,0xa4435000) malloc: * error for object 0x4821404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug System version: Macintosh OS 10.11.4 build 15E65 Sun Apr 24 08:21:34 2016 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fd98 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/0/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90651000 - 0x9065afff /usr/lib/system/libsystem_networkextension.dylib 0x9065b000 - 0x9068efff /usr/lib/system/libsystem_m.dylib 0x906a0000 - 0x906a0fff /usr/lib/system/libkeymgr.dylib 0x90cf2000 - 0x90d00fff /usr/lib/libz.1.dylib 0x9101a000 - 0x9101afff /usr/lib/system/libsystem_blocks.dylib 0x9101b000 - 0x91020fff /usr/lib/system/libmacho.dylib 0x91021000 - 0x91074fff /usr/lib/libstdc++.6.dylib 0x91c11000 - 0x91c7ffff /usr/lib/system/libsystem_network.dylib 0x920a2000 - 0x920a8fff /usr/lib/system/libunwind.dylib 0x94345000 - 0x94350fff /usr/lib/system/libcommonCrypto.dylib 0x943a2000 - 0x943cafff /usr/lib/system/libxpc.dylib 0x94418000 - 0x94418fff /usr/lib/system/liblaunch.dylib 0x94484000 - 0x94486fff /usr/lib/system/libsystem_coreservices.dylib 0x95636000 - 0x95660fff /usr/lib/system/libdispatch.dylib 0x95ad4000 - 0x95ad6fff /usr/lib/system/libsystem_secinit.dylib 0x95b0b000 - 0x95b13fff /usr/lib/system/libsystem_dnssd.dylib 0x95b16000 - 0x95b20fff /usr/lib/system/libsystem_notify.dylib 0x966d9000 - 0x966dafff /usr/lib/system/libremovefile.dylib 0x972ec000 - 0x97381fff /usr/lib/system/libsystem_c.dylib 0x97b1a000 - 0x97b1dfff /usr/lib/system/libdyld.dylib 0x982f7000 - 0x98310fff /usr/lib/system/libsystem_malloc.dylib 0x98341000 - 0x98341fff /usr/lib/system/libunc.dylib 0x98dab000 - 0x98dadfff /usr/lib/system/libquarantine.dylib 0x98db2000 - 0x98dc9fff /usr/lib/system/libsystem_coretls.dylib 0x98dd3000 - 0x98de4fff /usr/lib/system/libsystem_trace.dylib 0x98f46000 - 0x98f47fff /usr/lib/libSystem.B.dylib 0x99c42000 - 0x99c47fff /usr/lib/system/libcompiler_rt.dylib 0x9a561000 - 0x9a564fff /usr/lib/system/libsystem_sandbox.dylib 0x9bf6e000 - 0x9bf77fff /usr/lib/system/libcopyfile.dylib 0x9c0e8000 - 0x9c14ffff /usr/lib/system/libcorecrypto.dylib 0x9cc35000 - 0x9cc5ffff /usr/lib/system/libsystem_info.dylib 0x9cc60000 - 0x9cc83fff /usr/lib/libc++abi.dylib 0x9d877000 - 0x9d898fff /usr/lib/system/libsystem_kernel.dylib 0x9d985000 - 0x9d98bfff /usr/lib/system/libsystem_platform.dylib 0x9da44000 - 0x9da46fff /usr/lib/system/libsystem_configuration.dylib 0x9dc57000 - 0x9dc5ffff /usr/lib/system/libsystem_pthread.dylib 0x9ea30000 - 0x9ea34fff /usr/lib/system/libcache.dylib 0x9ed69000 - 0x9ed82fff /usr/lib/system/libsystem_asl.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Apr 2016 19:31:45	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	518,400	707,582	1.3649
20 Apr 2016 18:35:09	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	492,480	671,283	1.3631
19 Apr 2016 20:05:27	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	466,560	634,276	1.3595
18 Apr 2016 21:31:44	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	440,640	598,142	1.3574
18 Apr 2016 09:59:26	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	414,720	562,410	1.3561
17 Apr 2016 14:27:35	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	388,800	526,112	1.3532
16 Apr 2016 14:31:07	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	362,880	491,341	1.3540
15 Apr 2016 15:35:11	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	336,960	456,704	1.3554
15 Apr 2016 12:16:03	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	311,040	421,942	1.3566
15 Apr 2016 12:14:16	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	285,120	386,752	1.3565
15 Apr 2016 12:12:48	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	259,200	352,476	1.3599
15 Apr 2016 12:10:06	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	233,280	317,512	1.3611
11 Apr 2016 12:27:29	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	207,360	279,710	1.3489
10 Apr 2016 16:22:09	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	181,440	244,643	1.3483
09 Apr 2016 18:47:09	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	155,520	210,188	1.3515
08 Apr 2016 19:30:24	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	129,600	172,193	1.3286
07 Apr 2016 22:28:56	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	103,680	136,087	1.3126
07 Apr 2016 13:49:19	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	77,760	108,515	1.3955
06 Apr 2016 14:07:19	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	51,840	72,850	1.4053
05 Apr 2016 17:27:28	1223244	19477611	hadcm3n_sdn3_198012_480_375_010415396_1	25,920	35,471	1.3685