Task 13735307

Name	hadcm3n_u2ia_1980_40_007608108_2
Workunit	7786238
Created	6 Dec 2011, 5:32:40 UTC
Sent	6 Dec 2011, 7:14:20 UTC
Report deadline	6 Mar 2012, 14:41:31 UTC
Received	20 Dec 2011, 2:20:34 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1169510
Run time	13 days 8 hours 51 min 15 sec
CPU time	7 days 11 hours 41 min 11 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	3.00 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x3000e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x3000e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x823404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x823400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62723,0xa0774540) malloc: * error for object 0x1801c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
19 Dec 2011 21:14:43	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	648,000	1,048,948	1.6187
19 Dec 2011 06:37:08	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	622,080	1,000,778	1.6088
18 Dec 2011 16:59:13	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	596,160	955,743	1.6032
18 Dec 2011 05:11:53	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	570,240	914,653	1.6040
17 Dec 2011 15:50:28	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	544,320	873,205	1.6042
17 Dec 2011 03:27:35	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	518,400	831,702	1.6044
16 Dec 2011 15:08:29	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	492,480	790,304	1.6047
16 Dec 2011 02:50:00	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	466,560	748,894	1.6051
15 Dec 2011 14:32:06	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	440,640	707,455	1.6055
15 Dec 2011 02:12:36	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	414,720	666,054	1.6060
14 Dec 2011 13:04:03	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	388,800	621,578	1.5987
13 Dec 2011 22:55:05	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	362,880	575,735	1.5866
13 Dec 2011 08:45:49	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	336,960	529,439	1.5712
12 Dec 2011 18:37:55	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	311,040	484,465	1.5576
12 Dec 2011 07:28:03	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	285,120	445,612	1.5629
11 Dec 2011 19:03:13	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	259,200	406,640	1.5688
11 Dec 2011 08:16:40	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	233,280	369,779	1.5851
10 Dec 2011 21:08:31	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	207,360	332,991	1.6059
10 Dec 2011 09:01:07	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	181,440	294,335	1.6222
09 Dec 2011 20:42:50	1169510	13735307	hadcm3n_u2ia_1980_40_007608108_2	155,520	253,447	1.6297