Task 15599476

Name	hadcm3n_4hgn_1940_40_008311782_0
Workunit	8462917
Created	8 Feb 2013, 5:45:00 UTC
Sent	8 Feb 2013, 5:46:35 UTC
Report deadline	10 May 2013, 13:13:46 UTC
Received	19 Mar 2013, 21:07:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1256345
Run time	24 days 22 hours 24 min
CPU time	20 days 0 hours 55 min 5 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x184b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x184b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x103c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x383c004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x383c000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x103c604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x1830004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x4829004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x4829000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x281b604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58433,0xa050f540) malloc: * error for object 0x281b600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Fri Mar 8 04:37:36 2013 Thread 0 Crashed: 0 libSystem.B.dylib 0x945dab03 _small_free_list_remove_ptr + 234 1 libSystem.B.dylib 0x945d75cc _szone_free_definite_size + 3457 2 libSystem.B.dylib 0x945d65e8 _free + 244 3 hadcm3n_6.07_i686-apple-darwin 0x000158d6 __ZNSt6vectorIN62_GLOBAL__N_.._hadcm3_NetCdfAddAttributes.cpp_00000000_ABF61B0D9attributeESaIS1_EE13_M_insert_auxEN9__gnu_cxx17__normal_iteratorIPS1_S3_EERKS1_ + 608 4 hadcm3n_6.07_i686-apple-darwin 0x0001593a __ZNSt6vectorIN62_GLOBAL__N_.._hadcm3_NetCdfAddAttributes.cpp_00000000_ABF61B0D9attributeESaIS1_EE9push_backERKS1_ + 62 5 hadcm3n_6.07_i686-apple-darwin 0x00013b8f __Z14add_attributesPKcS0_ + 639 6 hadcm3n_6.07_i686-apple-darwin 0x0000bb26 __Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3688 7 hadcm3n_6.07_i686-apple-darwin 0x0000d36b __Z12decadalMeansiPKc + 957 8 hadcm3n_6.07_i686-apple-darwin 0x000067ff __Z9doCM3Procv + 185 9 hadcm3n_6.07_i686-apple-darwin 0x0000791c __Z8mainLoopv + 410 10 hadcm3n_6.07_i686-apple-darwin 0x000087c7 __Z6workerv + 2989 11 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 _main + 491 12 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 1: hadcm3n_6.07_i686-apple-darwin(76824,0xa050f540) malloc: * error for object 0x482c404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.8 build 10K549 Fri Mar 8 17:05:04 2013 Thread 0 Crashed: SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x202ac04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3034604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3004e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7090,0xa050f540) malloc: * error for object 0x3005c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Mar 2013 05:38:18	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	984,960	1,723,811	1.7501
18 Mar 2013 18:12:19	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	959,040	1,686,419	1.7584
17 Mar 2013 22:55:49	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	933,120	1,653,315	1.7718
17 Mar 2013 11:44:39	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	907,200	1,618,977	1.7846
16 Mar 2013 22:36:26	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	881,280	1,579,461	1.7922
16 Mar 2013 06:47:33	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	855,360	1,536,298	1.7961
14 Mar 2013 07:19:25	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	829,440	1,492,614	1.7995
13 Mar 2013 06:09:16	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	803,520	1,445,741	1.7993
08 Mar 2013 04:40:08	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	777,600	1,394,306	1.7931
07 Mar 2013 03:38:45	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	751,680	1,348,771	1.7943
06 Mar 2013 02:19:20	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	725,760	1,301,634	1.7935
04 Mar 2013 22:27:39	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	699,840	1,244,617	1.7784
03 Mar 2013 20:27:21	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	673,920	1,194,170	1.7720
03 Mar 2013 04:49:59	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	648,000	1,149,055	1.7732
02 Mar 2013 13:30:15	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	622,080	1,103,960	1.7746
01 Mar 2013 21:37:59	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	596,160	1,058,329	1.7752
28 Feb 2013 20:53:29	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	570,240	1,012,922	1.7763
27 Feb 2013 20:07:27	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	544,320	967,313	1.7771
26 Feb 2013 19:12:06	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	518,400	921,158	1.7769
25 Feb 2013 18:24:39	1256345	15599476	hadcm3n_4hgn_1940_40_008311782_0	492,480	875,886	1.7785