Task 15456956

Name	hadcm3n_zkgy_1880_40_008252371_0
Workunit	8407495
Created	23 Nov 2012, 13:11:24 UTC
Sent	23 Nov 2012, 13:12:02 UTC
Report deadline	22 Feb 2013, 20:39:13 UTC
Received	23 Dec 2012, 8:05:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1110299
Run time	21 days 6 hours 13 min 16 sec
CPU time	17 days 13 hours 1 min 32 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.93 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:18:55 (6401): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:18:57 (6401): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:06:27 (8901): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:02:37 (16631): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:04:57 (25084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:28:31 (27092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:16:31 (31424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:42:07 (36132): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:36:17 (45697): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 19:03:14 (284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:03:57 (12211): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:25:35 (14964): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:51 (18190): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:20:52 (18190): No heartbeat from core client for 30 sec - exiting 05:20:53 (18190): No heartbeat from core client for 30 sec - exiting 14:55:56 (26119): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:47:45 (67659): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:36:28 (11142): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:26:56 (21506): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 06:11:30 (42857): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:37:43 (59898): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:37:44 (59898): No heartbeat from core client for 30 sec - exiting 12:37:45 (59898): No heartbeat from core client for 30 sec - exiting 21:44:38 (91154): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:47:57 (1332): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:43:57 (15347): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:02:35 (20751): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/zkgyko.pj97c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyko.pj97c10 Error: Input file: dataout/zkgyko.pi97c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyko.pi97c10 Error: Input file: dataout/zkgyko.pf97c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyko.pf97c10 Error: Input file: dataout/zkgyka.ph97c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.ph97c10 Error: Input file: dataout/zkgyka.pg97c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.pg97c10 Error: Input file: dataout/zkgyka.pe97c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.pe97c10 Error: Input file: dataout/zkgyka.pd97c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.pd97c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/zkgyko.pja3c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyko.pja3c10 Error: Input file: dataout/zkgyko.pia3c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyko.pia3c10 Error: Input file: dataout/zkgyko.pfa3c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyko.pfa3c10 Error: Input file: dataout/zkgyka.pha3c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.pha3c10 Error: Input file: dataout/zkgyka.pga3c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.pga3c10 Error: Input file: dataout/zkgyka.pea3c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.pea3c10 Error: Input file: dataout/zkgyka.pda3c10 is not a valid UM file. Error converting file to netcdf: dataout/zkgyka.pda3c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(6056,0xac4502c0) malloc: * error for object 0x6b75804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6056,0xac4502c0) malloc: * error for object 0x6b75800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6056,0xac4502c0) malloc: * error for object 0x2371804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6056,0xac4502c0) malloc: * error for object 0x7b8d804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6056,0xac4502c0) malloc: * error for object 0x7b8d800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 Sun Dec 23 07:34:03 2012 hadcm3n_6.07_i686-apple-darwin(6056,0xac4502c0) malloc: * error for object 0x7b72204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6056,0xac4502c0) malloc: * error for object 0x7b72200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8fa08 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/0/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90da2000 - 0x90db0fff /usr/lib/libz.1.dylib 0x91f4c000 - 0x91f50fff /usr/lib/system/libsystem_network.dylib 0x923b1000 - 0x923b2fff /usr/lib/system/libremovefile.dylib 0x92880000 - 0x92881fff /usr/lib/system/libunc.dylib 0x93188000 - 0x93253fff /usr/lib/system/libsystem_c.dylib 0x93917000 - 0x9391afff /usr/lib/system/libcompiler_rt.dylib 0x94391000 - 0x94391fff /usr/lib/system/libdnsinfo.dylib 0x94c98000 - 0x94cc7fff /usr/lib/system/libsystem_info.dylib 0x94cc9000 - 0x94d0cfff /usr/lib/system/libcommonCrypto.dylib 0x96597000 - 0x9659ffff /usr/lib/system/liblaunch.dylib 0x96650000 - 0x96666fff /usr/lib/system/libxpc.dylib 0x9740b000 - 0x97412fff /usr/lib/system/libsystem_dnssd.dylib 0x9744f000 - 0x97451fff /usr/lib/system/libdyld.dylib 0x984cb000 - 0x984ccfff /usr/lib/system/libquarantine.dylib 0x98e73000 - 0x98e91fff /usr/lib/system/libsystem_kernel.dylib 0x9a7b0000 - 0x9a7b4fff /usr/lib/system/libcache.dylib 0x9a908000 - 0x9a909fff /usr/lib/system/libsystem_sandbox.dylib 0x9a912000 - 0x9a915fff /usr/lib/system/libmathCommon.A.dylib 0x9ac38000 - 0x9ac40fff /usr/lib/system/libunwind.dylib 0x9ade4000 - 0x9ae12fff /usr/lib/libSystem.B.dylib 0x9ae1b000 - 0x9ae7dfff /usr/lib/libstdc++.6.dylib 0x9b10b000 - 0x9b10cfff /usr/lib/system/libsystem_blocks.dylib 0x9c113000 - 0x9c11cfff /usr/lib/libc++abi.dylib 0x9c2f9000 - 0x9c301fff /usr/lib/system/libcopyfile.dylib 0x9c33e000 - 0x9c345fff /usr/lib/system/libsystem_notify.dylib 0x9c796000 - 0x9c796fff /usr/lib/system/libkeymgr.dylib 0x9ceff000 - 0x9cf0dfff /usr/lib/system/libdispatch.dylib 0x9cf2c000 - 0x9cf31fff /usr/lib/system/libmacho.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Dec 2012 08:07:52	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	777,600	1,515,688	1.9492
22 Dec 2012 17:06:08	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	751,680	1,467,105	1.9518
21 Dec 2012 18:15:13	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	725,760	1,416,818	1.9522
20 Dec 2012 15:47:32	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	699,840	1,366,695	1.9529
19 Dec 2012 13:52:52	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	673,920	1,316,470	1.9535
18 Dec 2012 13:20:19	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	648,000	1,266,963	1.9552
17 Dec 2012 06:29:14	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	622,080	1,217,258	1.9568
16 Dec 2012 11:30:04	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	596,160	1,166,446	1.9566
15 Dec 2012 06:08:03	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	570,240	1,116,940	1.9587
14 Dec 2012 07:38:50	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	544,320	1,066,586	1.9595
14 Dec 2012 07:38:50	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	518,400	1,017,082	1.9620
14 Dec 2012 07:38:50	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	492,480	966,609	1.9627
14 Dec 2012 07:38:50	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	466,560	916,874	1.9652
14 Dec 2012 07:38:50	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	440,640	866,490	1.9664
14 Dec 2012 07:38:50	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	414,720	816,072	1.9678
05 Dec 2012 22:21:13	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	388,800	766,097	1.9704
05 Dec 2012 06:49:03	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	362,880	719,696	1.9833
04 Dec 2012 14:48:34	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	336,960	674,355	2.0013
03 Dec 2012 19:47:17	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	311,040	628,001	2.0190
03 Dec 2012 00:57:51	1110299	15456956	hadcm3n_zkgy_1880_40_008252371_0	285,120	576,707	2.0227