Task 13535535

Name	hadcm3n_yj8x_1900_40_007514934_1
Workunit	7712409
Created	28 Oct 2011, 12:39:33 UTC
Sent	28 Oct 2011, 12:45:03 UTC
Report deadline	27 Jan 2012, 20:12:14 UTC
Received	9 Dec 2011, 12:42:15 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1076515
Run time	12 days 2 hours 11 min 19 sec
CPU time	11 days 0 hours 43 min 53 sec
Validate state	Invalid
Credit	5,909.76
Device peak FLOPS	3.14 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2028a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2028a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x823a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x281f604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x281f600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2028a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x823a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x83f000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2028a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x823a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x815204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x816004: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x816000: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2002200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7321,0xa0c7f540) malloc: * error for object 0x2000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Dec 2011 07:57:26	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	492,480	956,825	1.9429
04 Dec 2011 14:16:19	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	466,560	902,167	1.9337
03 Dec 2011 22:07:04	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	440,640	851,906	1.9333
03 Dec 2011 20:51:36	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	414,720	796,556	1.9207
03 Dec 2011 20:51:36	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	388,800	740,834	1.9054
30 Nov 2011 19:05:04	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	362,880	689,874	1.9011
30 Nov 2011 00:03:53	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	336,960	641,235	1.9030
29 Nov 2011 06:05:19	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	311,040	587,233	1.8880
28 Nov 2011 15:12:20	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	285,120	541,952	1.9008
27 Nov 2011 15:17:26	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	259,200	491,940	1.8979
24 Nov 2011 11:37:24	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	233,280	441,305	1.8917
23 Nov 2011 14:43:44	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	207,360	394,756	1.9037
22 Nov 2011 21:19:21	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	181,440	342,081	1.8854
22 Nov 2011 00:41:40	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	155,520	295,331	1.8990
21 Nov 2011 06:57:59	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	129,600	248,949	1.9209
20 Nov 2011 03:11:37	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	103,680	197,874	1.9085
19 Nov 2011 02:27:04	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	77,760	153,212	1.9703
16 Nov 2011 01:16:34	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	51,840	105,767	2.0403
09 Nov 2011 12:22:18	1076515	13535535	hadcm3n_yj8x_1900_40_007514934_1	25,920	54,214	2.0916