Task 15701608

Name	hadcm3n_4f55_1980_40_008322587_1
Workunit	8473722
Created	4 Apr 2013, 6:38:47 UTC
Sent	4 Apr 2013, 6:39:22 UTC
Report deadline	4 Jul 2013, 14:06:33 UTC
Received	26 Apr 2013, 6:25:44 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	864106
Run time	17 days 2 hours 31 min 22 sec
CPU time	11 days 18 hours 20 min 39 sec
Validate state	Invalid
Credit	8,709.12
Device peak FLOPS	3.21 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0x3b68e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0xb68e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0xb77404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(15832,0xac2482c0) malloc: * error for object 0xb77400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
25 Apr 2013 21:23:43	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	725,760	1,015,309	1.3990
24 Apr 2013 21:42:57	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	699,840	980,210	1.4006
24 Apr 2013 04:33:45	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	673,920	943,717	1.4003
23 Apr 2013 10:44:41	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	648,000	907,301	1.4002
22 Apr 2013 20:33:16	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	622,080	870,724	1.3997
22 Apr 2013 06:12:55	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	596,160	834,107	1.3991
21 Apr 2013 16:36:02	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	570,240	797,481	1.3985
21 Apr 2013 01:55:03	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	544,320	760,886	1.3979
20 Apr 2013 11:34:48	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	518,400	724,400	1.3974
19 Apr 2013 19:22:19	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	492,480	687,858	1.3967
18 Apr 2013 22:09:58	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	466,560	651,014	1.3953
18 Apr 2013 06:35:25	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	440,640	614,401	1.3943
17 Apr 2013 12:33:14	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	414,720	577,933	1.3935
16 Apr 2013 17:36:37	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	388,800	540,544	1.3903
16 Apr 2013 00:26:00	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	362,880	502,541	1.3849
15 Apr 2013 08:25:09	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	336,960	466,091	1.3832
14 Apr 2013 13:40:07	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	311,040	430,042	1.3826
13 Apr 2013 20:01:02	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	285,120	393,995	1.3819
12 Apr 2013 13:50:15	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	259,200	356,509	1.3754
11 Apr 2013 03:32:25	864106	15701608	hadcm3n_4f55_1980_40_008322587_1	233,280	320,475	1.3738