Task 15716784

Name	hadcm3n_4gf0_1980_40_008341254_4
Workunit	8492115
Created	8 Apr 2013, 8:38:25 UTC
Sent	8 Apr 2013, 8:38:43 UTC
Report deadline	8 Jul 2013, 16:05:54 UTC
Received	2 May 2013, 17:31:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1172598
Run time	13 days 10 hours 34 min 43 sec
CPU time	11 days 10 hours 13 min 40 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:47:15 (49513): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 62 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/4gf0ko.pjk0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ko.pjk0c10 Error: Input file: dataout/4gf0ko.pik0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ko.pik0c10 Error: Input file: dataout/4gf0ko.pfk0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ko.pfk0c10 Error: Input file: dataout/4gf0ko.pck0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ko.pck0c10 Error: Input file: dataout/4gf0ko.pbk0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ko.pbk0c10 Error: Input file: dataout/4gf0ko.pak0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ko.pak0c10 Error: Input file: dataout/4gf0ka.phk0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ka.phk0c10 Error: Input file: dataout/4gf0ka.pgk0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ka.pgk0c10 Error: Input file: dataout/4gf0ka.pek0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ka.pek0c10 Error: Input file: dataout/4gf0ka.pdk0c10 is not a valid UM file. Error converting file to netcdf: dataout/4gf0ka.pdk0c10 hadcm3n_6.07_i686-apple-darwin(55873,0xa04a5540) malloc: * error for object 0x2832804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55873,0xa04a5540) malloc: * error for object 0x4827604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55873,0xa04a5540) malloc: * error for object 0x4827600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55873,0xa04a5540) malloc: * error for object 0x601b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55873,0xa04a5540) malloc: * error for object 0x601b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(55873,0xa04a5540) malloc: * error for object 0x6801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Thu May 2 13:06:54 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x93157b03 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x931545cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x931535e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x0000ba58 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3482 4 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 5 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 6 hadcm3n_6.07_i686-apple-darwin 0x0000791c __Z8mainLoopv + 410 7 hadcm3n_6.07_i686-apple-darwin 0x000087c7 __Z6workerv + 2989 8 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 9 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: 0 libSystem.B.dylib 0x9314cc0e _mach_wait_until + 10 1 libSystem.B.dylib 0x931d4429 _nanosleep + 345 2 libSystem.B.dylib 0x931d42ca _usleep + 61 3 hadcm3n_6.07_i686-apple-darwin 0x00071a7c __Z11boinc_sleepd + 188 4 hadcm3n_6.07_i686-apple-darwin 0x00067282 __Z12timer_threadPv + 78 5 libSystem.B.dylib 0x9317a259 __pthread_start + 345 6 libSystem.B.dylib 0x9317a0de _thread_start + 34 Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f678 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/1/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90003000 - 0x90006fff /usr/lib/system/libmathCommon.A.dylib 0x9314c000 - 0x932f3fff /usr/lib/libSystem.B.dylib 0x946eb000 - 0x946f9fff /usr/lib/libz.1.dylib 0x99176000 - 0x991e0fff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 May 2013 17:34:26	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	518,400	987,215	1.9043
01 May 2013 10:39:24	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	492,480	941,777	1.9123
30 Apr 2013 20:54:40	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	466,560	904,930	1.9396
29 Apr 2013 23:02:32	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	440,640	859,683	1.9510
28 Apr 2013 10:11:36	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	414,720	810,778	1.9550
27 Apr 2013 17:17:32	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	388,800	760,463	1.9559
26 Apr 2013 22:34:34	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	362,880	711,753	1.9614
25 Apr 2013 09:42:19	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	336,960	659,400	1.9569
24 Apr 2013 12:21:16	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	311,040	607,888	1.9544
23 Apr 2013 18:01:53	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	285,120	556,066	1.9503
22 Apr 2013 22:40:35	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	259,200	503,227	1.9415
22 Apr 2013 03:11:52	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	233,280	450,854	1.9327
21 Apr 2013 06:41:30	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	207,360	398,739	1.9229
20 Apr 2013 00:26:01	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	181,440	347,272	1.9140
18 Apr 2013 23:55:36	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	155,520	296,140	1.9042
16 Apr 2013 06:54:07	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	129,600	247,579	1.9103
15 Apr 2013 14:01:47	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	103,680	198,203	1.9117
14 Apr 2013 04:30:08	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	77,760	148,580	1.9108
13 Apr 2013 01:46:09	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	51,840	99,141	1.9124
10 Apr 2013 01:55:38	1172598	15716784	hadcm3n_4gf0_1980_40_008341254_4	25,920	49,575	1.9126