Task 15590876

Name	hadcm3n_4n59_1940_40_008304902_0
Workunit	8456037
Created	7 Feb 2013, 4:14:50 UTC
Sent	7 Feb 2013, 4:16:47 UTC
Report deadline	9 May 2013, 11:43:58 UTC
Received	17 Feb 2013, 5:10:18 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1244925
Run time	8 days 20 hours 34 min 55 sec
CPU time	7 days 9 hours 42 min 12 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	3.55 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.0.31</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/4n59ko.pje6c10 is not a valid UM file. Error converting file to netcdf: dataout/4n59ko.pje6c10 Error: Input file: dataout/4n59ko.pie6c10 is not a valid UM file. Error converting file to netcdf: dataout/4n59ko.pie6c10 Error: Input file: dataout/4n59ko.pfe6c10 is not a valid UM file. Error converting file to netcdf: dataout/4n59ko.pfe6c10 Error: Input file: dataout/4n59ka.phe6c10 is not a valid UM file. Error converting file to netcdf: dataout/4n59ka.phe6c10 Error: Input file: dataout/4n59ka.pge6c10 is not a valid UM file. Error converting file to netcdf: dataout/4n59ka.pge6c10 Error: Input file: dataout/4n59ka.pee6c10 is not a valid UM file. Error converting file to netcdf: dataout/4n59ka.pee6c10 Error: Input file: dataout/4n59ka.pde6c10 is not a valid UM file. Error converting file to netcdf: dataout/4n59ka.pde6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30fe404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x2925404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x2925400: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x40fe604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x8fae04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x8fae00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30fe404: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x30e4a00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x40dae04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(86991,0xac0962c0) malloc: * error for object 0x40dae00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Feb 2013 18:42:11	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	518,400	612,045	1.1806
16 Feb 2013 07:03:36	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	492,480	580,248	1.1782
15 Feb 2013 19:59:09	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	466,560	548,840	1.1764
15 Feb 2013 07:21:53	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	440,640	517,186	1.1737
14 Feb 2013 20:58:40	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	414,720	485,560	1.1708
14 Feb 2013 10:36:05	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	388,800	454,143	1.1681
13 Feb 2013 14:50:31	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	362,880	422,577	1.1645
13 Feb 2013 03:25:14	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	336,960	391,859	1.1629
12 Feb 2013 16:21:32	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	311,040	360,756	1.1598
12 Feb 2013 05:18:52	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	285,120	329,927	1.1572
11 Feb 2013 17:36:26	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	259,200	299,531	1.1556
11 Feb 2013 08:16:05	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	233,280	268,971	1.1530
10 Feb 2013 20:05:47	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	207,360	237,929	1.1474
10 Feb 2013 10:37:35	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	181,440	206,474	1.1380
09 Feb 2013 21:53:54	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	155,520	175,406	1.1279
09 Feb 2013 07:11:26	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	129,600	145,057	1.1193
08 Feb 2013 21:38:20	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	103,680	116,849	1.1270
08 Feb 2013 10:12:34	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	77,760	86,793	1.1162
07 Feb 2013 23:53:33	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	51,840	56,348	1.0870
07 Feb 2013 13:15:35	1244925	15590876	hadcm3n_4n59_1940_40_008304902_0	25,920	27,304	1.0534