Task 15646390

Name	hadcm3n_4830_1980_40_008324394_0
Workunit	8475529
Created	2 Mar 2013, 6:36:50 UTC
Sent	2 Mar 2013, 6:39:35 UTC
Report deadline	1 Jun 2013, 14:06:46 UTC
Received	31 Mar 2013, 22:21:11 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	843946
Run time	10 days 14 hours 26 min 25 sec
CPU time	10 days 0 hours 11 min 57 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.86 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2ada004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2ada000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0xad8604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0xad8600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0xad8604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0xad8600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x12f5c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x12f5c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x12f5c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x12f5c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x12f5c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x12f5c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2abea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2ada004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2ada000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0xabd004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x12da604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0xabd004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2a90204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2a90200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2a90204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2a90200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(3190,0xac0572c0) malloc: * error for object 0x2a90204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0xa2d804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x19f4204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x2a00204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x2a00200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9e7400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9d5a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(21926,0xac0572c0) malloc: * error for object 0x9d5a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x10e3204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x10e3200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x8e9404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18e1804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18fce04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18fce00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x28d2e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x28ee404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x28ee400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18e1804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18fce04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18fce00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18fce04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x18fce00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x8e9404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x8b1000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation hadcm3n_6.07_i686-apple-darwin(237,0xac0572c0) malloc: * error for object 0x10bf600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 hadcm3n_6.07_i686-apple-darwin(6447,0xac0572c0) malloc: * error for object 0x140a804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(6447,0xac0572c0) malloc: * error for object 0x140a800: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
31 Mar 2013 19:43:01	843946	15646390	hadcm3n_4830_1980_40_008324394_0	518,400	857,125	1.6534
31 Mar 2013 05:48:23	843946	15646390	hadcm3n_4830_1980_40_008324394_0	492,480	813,994	1.6528
30 Mar 2013 18:06:28	843946	15646390	hadcm3n_4830_1980_40_008324394_0	466,560	773,485	1.6578
30 Mar 2013 06:38:45	843946	15646390	hadcm3n_4830_1980_40_008324394_0	440,640	733,880	1.6655
29 Mar 2013 19:04:01	843946	15646390	hadcm3n_4830_1980_40_008324394_0	414,720	693,863	1.6731
29 Mar 2013 06:55:03	843946	15646390	hadcm3n_4830_1980_40_008324394_0	388,800	653,290	1.6803
28 Mar 2013 18:25:49	843946	15646390	hadcm3n_4830_1980_40_008324394_0	362,880	613,645	1.6910
28 Mar 2013 05:37:34	843946	15646390	hadcm3n_4830_1980_40_008324394_0	336,960	573,331	1.7015
27 Mar 2013 16:20:46	843946	15646390	hadcm3n_4830_1980_40_008324394_0	311,040	530,853	1.7067
27 Mar 2013 03:40:30	843946	15646390	hadcm3n_4830_1980_40_008324394_0	285,120	491,163	1.7227
21 Mar 2013 12:25:08	843946	15646390	hadcm3n_4830_1980_40_008324394_0	259,200	451,604	1.7423
15 Mar 2013 11:09:08	843946	15646390	hadcm3n_4830_1980_40_008324394_0	233,280	411,893	1.7657
07 Mar 2013 15:20:32	843946	15646390	hadcm3n_4830_1980_40_008324394_0	207,360	371,431	1.7912
06 Mar 2013 09:56:03	843946	15646390	hadcm3n_4830_1980_40_008324394_0	181,440	327,659	1.8059
05 Mar 2013 19:40:07	843946	15646390	hadcm3n_4830_1980_40_008324394_0	155,520	280,390	1.8029
05 Mar 2013 05:55:04	843946	15646390	hadcm3n_4830_1980_40_008324394_0	129,600	233,766	1.8038
04 Mar 2013 16:00:45	843946	15646390	hadcm3n_4830_1980_40_008324394_0	103,680	187,048	1.8041
04 Mar 2013 02:11:38	843946	15646390	hadcm3n_4830_1980_40_008324394_0	77,760	140,276	1.8040
03 Mar 2013 11:56:34	843946	15646390	hadcm3n_4830_1980_40_008324394_0	51,840	92,964	1.7933
02 Mar 2013 21:11:23	843946	15646390	hadcm3n_4830_1980_40_008324394_0	25,920	45,548	1.7573