Task 15638662

Name	hadcm3n_4ig1_1980_40_008322881_0
Workunit	8474016
Created	25 Feb 2013, 2:15:19 UTC
Sent	25 Feb 2013, 2:24:25 UTC
Report deadline	27 May 2013, 9:51:36 UTC
Received	26 Mar 2013, 18:07:25 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1190785
Run time	11 days 0 hours 36 min 46 sec
CPU time	9 days 1 hours 31 min 50 sec
Validate state	Invalid
Credit	10,886.40
Device peak FLOPS	3.33 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10ad604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10ad600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x209e004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10ad604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10c8c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x209e004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x20b9604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x20b9600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x10ad604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1663,0xac1bd2c0) malloc: * error for object 0x28a1000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(4306,0xac1bd2c0) malloc: * error for object 0x2a3dc04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4306,0xac1bd2c0) malloc: * error for object 0xa3ec04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4306,0xac1bd2c0) malloc: * error for object 0x2a1fe04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4306,0xac1bd2c0) malloc: * error for object 0x2a20c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(4815,0xac1bd2c0) malloc: * error for object 0x1198204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4815,0xac1bd2c0) malloc: * error for object 0x1198200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4815,0xac1bd2c0) malloc: * error for object 0x9a1a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4815,0xac1bd2c0) malloc: * error for object 0x9a1a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(4815,0xac1bd2c0) malloc: * error for object 0x984004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.5 build 11G63 hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x118f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x2983004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x115b800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x1170804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(8939,0xac1bd2c0) malloc: * error for object 0x117ee04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Mar 2013 13:22:30	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	907,200	799,763	0.8816
24 Mar 2013 10:37:38	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	881,280	776,302	0.8809
24 Mar 2013 02:24:58	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	855,360	752,917	0.8802
23 Mar 2013 19:01:12	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	829,440	729,633	0.8797
23 Mar 2013 10:49:04	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	803,520	706,313	0.8790
23 Mar 2013 01:35:06	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	777,600	682,791	0.8781
09 Mar 2013 18:52:36	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	751,680	659,464	0.8773
09 Mar 2013 12:03:02	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	725,760	639,169	0.8807
09 Mar 2013 05:31:30	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	699,840	618,905	0.8844
08 Mar 2013 22:59:45	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	673,920	598,615	0.8883
08 Mar 2013 16:50:58	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	648,000	578,322	0.8925
08 Mar 2013 09:21:38	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	622,080	558,055	0.8971
08 Mar 2013 01:44:16	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	596,160	535,509	0.8983
06 Mar 2013 05:55:09	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	570,240	512,235	0.8983
05 Mar 2013 23:08:28	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	544,320	488,971	0.8983
03 Mar 2013 15:20:34	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	518,400	465,590	0.8981
03 Mar 2013 07:05:29	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	492,480	442,359	0.8982
02 Mar 2013 23:43:38	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	466,560	419,113	0.8983
02 Mar 2013 15:40:45	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	440,640	395,801	0.8982
02 Mar 2013 05:41:54	1190785	15638662	hadcm3n_4ig1_1980_40_008322881_0	414,720	372,293	0.8977