Task 14647613

Name	hadcm3n_206q_1980_40_007956980_1
Workunit	8112092
Created	9 May 2012, 11:04:28 UTC
Sent	9 May 2012, 11:29:35 UTC
Report deadline	8 Aug 2012, 18:56:46 UTC
Received	23 May 2012, 11:26:54 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	976458
Run time	11 days 6 hours 9 min 59 sec
CPU time	10 days 16 hours 41 min 42 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:26:52 (319): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/206qko.pjk0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qko.pjk0c10 Error: Input file: dataout/206qko.pik0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qko.pik0c10 Error: Input file: dataout/206qko.pfk0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qko.pfk0c10 Error: Input file: dataout/206qko.pck0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qko.pck0c10 Error: Input file: dataout/206qko.pbk0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qko.pbk0c10 Error: Input file: dataout/206qko.pak0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qko.pak0c10 Error: Input file: dataout/206qka.phk0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qka.phk0c10 Error: Input file: dataout/206qka.pgk0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qka.pgk0c10 Error: Input file: dataout/206qka.pek0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qka.pek0c10 Error: Input file: dataout/206qka.pdk0c10 is not a valid UM file. Error converting file to netcdf: dataout/206qka.pdk0c10 hadcm3n_6.07_i686-apple-darwin(14624,0xacce82c0) malloc: * error for object 0x8bcaa04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14624,0xacce82c0) malloc: * error for object 0x8bcaa00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14624,0xacce82c0) malloc: * error for object 0x7bcdc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(14624,0xacce82c0) malloc: * error for object 0x3bcdc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14624,0xacce82c0) malloc: * error for object 0x3bce604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(14624,0xacce82c0) malloc: * error for object 0x3bce600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.4 build 11E53 Wed May 23 06:27:32 2012 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8faa8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/10/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9004e000 - 0x9006cfff /usr/lib/system/libsystem_kernel.dylib 0x91128000 - 0x91128fff /usr/lib/system/libdnsinfo.dylib 0x91129000 - 0x91137fff /usr/lib/libz.1.dylib 0x914e0000 - 0x914e4fff /usr/lib/system/libsystem_network.dylib 0x923d5000 - 0x923d6fff /usr/lib/system/libremovefile.dylib 0x93886000 - 0x9388efff /usr/lib/system/liblaunch.dylib 0x94291000 - 0x94299fff /usr/lib/system/libunwind.dylib 0x94cdf000 - 0x94ce2fff /usr/lib/system/libcompiler_rt.dylib 0x96093000 - 0x960c2fff /usr/lib/system/libsystem_info.dylib 0x96b5b000 - 0x96b62fff /usr/lib/system/libsystem_notify.dylib 0x97131000 - 0x97138fff /usr/lib/system/libsystem_dnssd.dylib 0x97359000 - 0x9735afff /usr/lib/system/libquarantine.dylib 0x97b21000 - 0x97b64fff /usr/lib/system/libcommonCrypto.dylib 0x9937c000 - 0x9938afff /usr/lib/system/libdispatch.dylib 0x993a4000 - 0x99406fff /usr/lib/libstdc++.6.dylib 0x996bb000 - 0x996befff /usr/lib/system/libmathCommon.A.dylib 0x996bf000 - 0x996c4fff /usr/lib/system/libmacho.dylib 0x9a41e000 - 0x9a422fff /usr/lib/system/libcache.dylib 0x9a4fe000 - 0x9a507fff /usr/lib/libc++abi.dylib 0x9a910000 - 0x9a911fff /usr/lib/system/libsystem_sandbox.dylib 0x9a965000 - 0x9a993fff /usr/lib/libSystem.B.dylib 0x9ab01000 - 0x9ab01fff /usr/lib/system/libkeymgr.dylib 0x9aec2000 - 0x9af8dfff /usr/lib/system/libsystem_c.dylib 0x9b006000 - 0x9b008fff /usr/lib/system/libdyld.dylib 0x9ca40000 - 0x9ca48fff /usr/lib/system/libcopyfile.dylib 0x9cae6000 - 0x9cae7fff /usr/lib/system/libsystem_blocks.dylib 0x9cae8000 - 0x9cae9fff /usr/lib/system/libunc.dylib 0x9ced6000 - 0x9ceecfff /usr/lib/system/libxpc.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 May 2012 11:30:19	976458	14647613	hadcm3n_206q_1980_40_007956980_1	518,400	924,095	1.7826
22 May 2012 15:38:31	976458	14647613	hadcm3n_206q_1980_40_007956980_1	492,480	877,907	1.7826
21 May 2012 19:24:00	976458	14647613	hadcm3n_206q_1980_40_007956980_1	466,560	831,269	1.7817
21 May 2012 01:08:27	976458	14647613	hadcm3n_206q_1980_40_007956980_1	440,640	784,580	1.7805
20 May 2012 10:28:38	976458	14647613	hadcm3n_206q_1980_40_007956980_1	414,720	738,079	1.7797
19 May 2012 18:01:53	976458	14647613	hadcm3n_206q_1980_40_007956980_1	388,800	691,525	1.7786
19 May 2012 00:19:27	976458	14647613	hadcm3n_206q_1980_40_007956980_1	362,880	644,979	1.7774
18 May 2012 09:27:26	976458	14647613	hadcm3n_206q_1980_40_007956980_1	336,960	598,274	1.7755
17 May 2012 19:07:30	976458	14647613	hadcm3n_206q_1980_40_007956980_1	311,040	551,672	1.7736
16 May 2012 20:06:35	976458	14647613	hadcm3n_206q_1980_40_007956980_1	285,120	504,509	1.7695
16 May 2012 05:53:58	976458	14647613	hadcm3n_206q_1980_40_007956980_1	259,200	458,332	1.7683
15 May 2012 15:31:03	976458	14647613	hadcm3n_206q_1980_40_007956980_1	233,280	412,890	1.7699
14 May 2012 21:29:10	976458	14647613	hadcm3n_206q_1980_40_007956980_1	207,360	367,226	1.7710
14 May 2012 03:25:57	976458	14647613	hadcm3n_206q_1980_40_007956980_1	181,440	321,219	1.7704
13 May 2012 14:29:01	976458	14647613	hadcm3n_206q_1980_40_007956980_1	155,520	274,952	1.7680
13 May 2012 01:08:15	976458	14647613	hadcm3n_206q_1980_40_007956980_1	129,600	228,822	1.7656
12 May 2012 10:48:39	976458	14647613	hadcm3n_206q_1980_40_007956980_1	103,680	182,908	1.7642
11 May 2012 21:57:11	976458	14647613	hadcm3n_206q_1980_40_007956980_1	77,760	137,012	1.7620
11 May 2012 03:19:46	976458	14647613	hadcm3n_206q_1980_40_007956980_1	51,840	90,784	1.7512
10 May 2012 14:21:13	976458	14647613	hadcm3n_206q_1980_40_007956980_1	25,920	45,311	1.7481