Task 15312370

Name	hadcm3n_o3gw_2060_40_008203709_1
Workunit	8358833
Created	27 Sep 2012, 1:05:20 UTC
Sent	27 Sep 2012, 1:06:07 UTC
Report deadline	27 Dec 2012, 8:33:18 UTC
Received	28 Oct 2012, 12:17:24 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1172598
Run time	18 days 6 hours 23 min 44 sec
CPU time	16 days 23 hours 51 min 57 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:45:29 (50630): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 1 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:47:53 (77655): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/o3gwko.pjr8c10 is not a valid UM file. Error converting file to netcdf: dataout/o3gwko.pjr8c10 Error: Input file: dataout/o3gwko.pir8c10 is not a valid UM file. Error converting file to netcdf: dataout/o3gwko.pir8c10 Error: Input file: dataout/o3gwko.pfr8c10 is not a valid UM file. Error converting file to netcdf: dataout/o3gwko.pfr8c10 Error: Input file: dataout/o3gwka.phr8c10 is not a valid UM file. Error converting file to netcdf: dataout/o3gwka.phr8c10 Error: Input file: dataout/o3gwka.pgr8c10 is not a valid UM file. Error converting file to netcdf: dataout/o3gwka.pgr8c10 Error: Input file: dataout/o3gwka.per8c10 is not a valid UM file. Error converting file to netcdf: dataout/o3gwka.per8c10 Error: Input file: dataout/o3gwka.pdr8c10 is not a valid UM file. Error converting file to netcdf: dataout/o3gwka.pdr8c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(21135,0xa07ec540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Oct 21 08:33:15 2012 Thread 0 Crashed: CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x501f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x832604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x1030604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6115,0xa07ec540) malloc: * error for object 0x3801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Sun Oct 28 07:22:50 2012 Thread 0 Crashed: 0 libSystem.B.dylib 0x959ecb03 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x959e95cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x959e85e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c __Z8mainLoopv + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 __Z6workerv + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x959e1c0e _mach_wait_until + 10 1 libSystem.B.dylib 0x95a69429 _nanosleep + 345 2 libSystem.B.dylib 0x95a692ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x95a0f259 __pthread_start + 345 6 libSystem.B.dylib 0x95a0f0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f688 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/14/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x9224f000 - 0x92252fff /usr/lib/system/libmathCommon.A.dylib 0x959e1000 - 0x95b88fff /usr/lib/libSystem.B.dylib 0x985d2000 - 0x9863cfff /usr/lib/libstdc++.6.dylib 0x9956b000 - 0x99579fff /usr/lib/libz.1.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
28 Oct 2012 12:19:18	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	777,600	1,468,311	1.8883
27 Oct 2012 19:32:31	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	751,680	1,422,260	1.8921
27 Oct 2012 03:33:18	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	725,760	1,372,019	1.8905
26 Oct 2012 10:58:19	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	699,840	1,320,181	1.8864
25 Oct 2012 12:05:22	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	673,920	1,273,038	1.8890
24 Oct 2012 19:28:10	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	648,000	1,228,526	1.8959
24 Oct 2012 03:25:46	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	622,080	1,183,454	1.9024
23 Oct 2012 13:13:46	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	596,160	1,139,199	1.9109
22 Oct 2012 21:35:22	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	570,240	1,094,591	1.9195
22 Oct 2012 04:52:12	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	544,320	1,049,480	1.9281
21 Oct 2012 12:55:25	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	518,400	1,002,647	1.9341
20 Oct 2012 23:13:33	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	492,480	956,569	1.9424
18 Oct 2012 13:37:46	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	466,560	910,885	1.9523
16 Oct 2012 18:35:09	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	440,640	872,846	1.9809
12 Oct 2012 22:15:36	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	414,720	823,179	1.9849
11 Oct 2012 19:26:12	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	388,800	775,818	1.9954
10 Oct 2012 10:36:37	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	362,880	724,975	1.9978
09 Oct 2012 10:05:47	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	336,960	670,997	1.9913
08 Oct 2012 09:36:12	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	311,040	617,068	1.9839
07 Oct 2012 12:10:42	1172598	15312370	hadcm3n_o3gw_2060_40_008203709_1	285,120	563,240	1.9754