Task 18876089

Name	hadcm3n_xdk8_1940_40_010021359_3
Workunit	10019430
Created	6 Sep 2015, 12:59:52 UTC
Sent	6 Sep 2015, 13:00:01 UTC
Report deadline	6 Dec 2015, 20:27:12 UTC
Received	22 Sep 2015, 14:52:19 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1015995
Run time	11 days 21 hours 37 min 59 sec
CPU time	11 days 8 hours 38 min 3 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.35 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.4.42</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1043404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x105ea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x105ea00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1043404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x105ea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x105ea00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x105ea04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x105ea00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x834e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x850400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x834e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(58732,0xa0d7a1d4) malloc: * error for object 0x1040c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/xdk8ko.pjg9c10 is not a valid UM file. Error converting file to netcdf: dataout/xdk8ko.pjg9c10 Error: Input file: dataout/xdk8ko.pig9c10 is not a valid UM file. Error converting file to netcdf: dataout/xdk8ko.pig9c10 Error: Input file: dataout/xdk8ko.pfg9c10 is not a valid UM file. Error converting file to netcdf: dataout/xdk8ko.pfg9c10 Error: Input file: dataout/xdk8ka.phg9c10 is not a valid UM file. Error converting file to netcdf: dataout/xdk8ka.phg9c10 Error: Input file: dataout/xdk8ka.pgg9c10 is not a valid UM file. Error converting file to netcdf: dataout/xdk8ka.pgg9c10 Error: Input file: dataout/xdk8ka.peg9c10 is not a valid UM file. Error converting file to netcdf: dataout/xdk8ka.peg9c10 Error: Input file: dataout/xdk8ka.pdg9c10 is not a valid UM file. Error converting file to netcdf: dataout/xdk8ka.pdg9c10 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Sep 2015 11:48:47	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	777,600	976,602	1.2559
21 Sep 2015 19:58:24	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	751,680	943,428	1.2551
20 Sep 2015 23:29:24	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	725,760	911,120	1.2554
20 Sep 2015 04:38:20	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	699,840	878,209	1.2549
19 Sep 2015 19:09:25	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	673,920	845,412	1.2545
19 Sep 2015 09:41:25	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	648,000	812,886	1.2545
19 Sep 2015 00:13:48	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	622,080	780,385	1.2545
18 Sep 2015 11:10:33	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	596,160	747,774	1.2543
17 Sep 2015 22:03:51	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	570,240	715,583	1.2549
17 Sep 2015 09:00:45	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	544,320	683,769	1.2562
16 Sep 2015 17:12:28	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	518,400	650,656	1.2551
16 Sep 2015 02:15:59	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	492,480	617,992	1.2549
15 Sep 2015 12:23:25	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	466,560	585,550	1.2550
15 Sep 2015 01:19:17	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	440,640	553,927	1.2571
14 Sep 2015 10:33:56	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	414,720	521,318	1.2570
13 Sep 2015 19:17:34	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	388,800	488,316	1.2560
13 Sep 2015 04:37:27	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	362,880	455,403	1.2550
12 Sep 2015 16:11:36	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	336,960	423,066	1.2555
12 Sep 2015 02:11:03	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	311,040	390,685	1.2561
11 Sep 2015 13:01:47	1015995	18876089	hadcm3n_xdk8_1940_40_010021359_3	285,120	359,065	1.2593