Task 19387097

Name	hadcm3n_xd8u_198012_480_010220905_2
Workunit	10220905
Created	18 Mar 2016, 1:11:38 UTC
Sent	18 Mar 2016, 1:12:01 UTC
Report deadline	28 Feb 2017, 6:32:01 UTC
Received	12 Apr 2016, 20:59:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	961934
Run time	14 days 13 hours 18 min 5 sec
CPU time	13 days 17 hours 9 min 37 sec
Validate state	Invalid
Credit	9,020.16
Device peak FLOPS	3.49 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>7.6.22</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/xd8uko.pji1c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uko.pji1c10 Error: Input file: dataout/xd8uko.pii1c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uko.pii1c10 Error: Input file: dataout/xd8uko.pfi1c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uko.pfi1c10 Error: Input file: dataout/xd8uka.phi1c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.phi1c10 Error: Input file: dataout/xd8uka.pgi1c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.pgi1c10 Error: Input file: dataout/xd8uka.pei1c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.pei1c10 Error: Input file: dataout/xd8uka.pdi1c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.pdi1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/xd8uko.pji6c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uko.pji6c10 Error: Input file: dataout/xd8uko.pii6c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uko.pii6c10 Error: Input file: dataout/xd8uko.pfi6c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uko.pfi6c10 Error: Input file: dataout/xd8uka.phi6c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.phi6c10 Error: Input file: dataout/xd8uka.pgi6c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.pgi6c10 Error: Input file: dataout/xd8uka.pei6c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.pei6c10 Error: Input file: dataout/xd8uka.pdi6c10 is not a valid UM file. Error converting file to netcdf: dataout/xd8uka.pdi6c10 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x101a204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x101a200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x2816600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x1809204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x1817804: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x1826204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x1834c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(1132,0xa3331000) malloc: * error for object 0x1834c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x180d204: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x180d200: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x5002604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x5002600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x5002604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x5002600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x5002604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x5002600: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x5002604: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.11.4 build 15E65 hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x1800e04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(77825,0xa3331000) malloc: * error for object 0x1800e00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug Mon Apr 4 18:51:06 2016 hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x282de04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x282de00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x1011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x1011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x1011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x1011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x1011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x1011c00: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(78743,0xa3331000) malloc: * error for object 0x1011c04: incorrect checksum for freed object - object was probably modified after being freed. * set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Apr 2016 12:10:58	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	751,680	1,184,962	1.5764
11 Apr 2016 07:32:30	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	725,760	1,142,353	1.5740
10 Apr 2016 19:24:55	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	699,840	1,101,846	1.5744
09 Apr 2016 01:54:50	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	673,920	1,061,769	1.5755
08 Apr 2016 19:03:10	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	648,000	1,021,735	1.5768
07 Apr 2016 18:13:56	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	622,080	981,662	1.5780
07 Apr 2016 02:01:31	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	596,160	942,538	1.5810
06 Apr 2016 05:41:01	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	570,240	902,206	1.5822
05 Apr 2016 10:46:44	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	544,320	858,509	1.5772
05 Apr 2016 00:18:00	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	518,400	818,145	1.5782
03 Apr 2016 23:33:00	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	492,480	777,809	1.5794
03 Apr 2016 07:21:24	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	466,560	737,140	1.5799
02 Apr 2016 19:19:36	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	440,640	696,467	1.5806
02 Apr 2016 02:09:13	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	414,720	656,404	1.5828
01 Apr 2016 01:22:33	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	388,800	616,402	1.5854
31 Mar 2016 05:54:05	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	362,880	576,880	1.5897
30 Mar 2016 14:12:29	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	336,960	535,843	1.5902
29 Mar 2016 22:07:03	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	311,040	495,591	1.5933
29 Mar 2016 03:00:26	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	285,120	455,383	1.5972
28 Mar 2016 13:25:52	961934	19387097	hadcm3n_xd8u_198012_480_010220905_2	259,200	412,590	1.5918