Task 13627294

Name	hadcm3n_t6rh_1940_40_007538942_3
Workunit	7736174
Created	10 Nov 2011, 2:14:24 UTC
Sent	16 Nov 2011, 6:39:46 UTC
Report deadline	15 Feb 2012, 14:06:57 UTC
Received	20 Dec 2011, 20:08:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	962203
Run time	20 days 9 hours 52 min 44 sec
CPU time	20 days 9 hours 52 min 44 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	1.97 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin
Stderr	<core_client_version>6.2.18</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=43216, selfPID=43216, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t6rhko.pje5c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pje5c10 Error: Input file: dataout/t6rhko.pie5c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pie5c10 Error: Input file: dataout/t6rhko.pfe5c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pfe5c10 Error: Input file: dataout/t6rhka.phe5c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.phe5c10 Error: Input file: dataout/t6rhka.pge5c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.pge5c10 Error: Input file: dataout/t6rhka.pee5c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.pee5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t6rhko.pjf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pjf6c10 Error: Input file: dataout/t6rhko.pif6c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pif6c10 Error: Input file: dataout/t6rhko.pff6c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pff6c10 Error: Input file: dataout/t6rhka.phf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.phf6c10 Error: Input file: dataout/t6rhka.pgf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.pgf6c10 Error: Input file: dataout/t6rhka.pef6c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.pef6c10 Error: Input file: dataout/t6rhka.pdf6c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.pdf6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 63 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 64 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 65 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 66 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 67 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/t6rhko.pjg4c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pjg4c10 Error: Input file: dataout/t6rhko.pig4c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pig4c10 Error: Input file: dataout/t6rhko.pfg4c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhko.pfg4c10 Error: Input file: dataout/t6rhka.phg4c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.phg4c10 Error: Input file: dataout/t6rhka.pgg4c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.pgg4c10 Error: Input file: dataout/t6rhka.peg4c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.peg4c10 Error: Input file: dataout/t6rhka.pdg4c10 is not a valid UM file. Error converting file to netcdf: dataout/t6rhka.pdg4c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.5.8 build 9L31a Tue Dec 20 14:36:33 2011 BOINC backtrace under OS 10.5.x only shows exported (global) symbols and may not show the final location which caused a crash. For a better backtrace, either run under OS 10.4.x or run under OS 10.6.x or later. 0 hadcm3n_6.07_i686-apple-darwin 0x00074953 version + 61229 1 hadcm3n_6.07_i686-apple-darwin 0x0006a936 version + 20240 2 libSystem.B.dylib 0x949081cb _sigtramp + 43 3 ??? 0xffffffff 0x0 + 4294967295 4 hadcm3n_6.07_i686-apple-darwin 0x00020602 ncx_getn_float_float + 34 5 hadcm3n_6.07_i686-apple-darwin 0x00027264 fill_NC_var + 17204 6 hadcm3n_6.07_i686-apple-darwin 0x000294e3 fill_NC_var + 26035 7 hadcm3n_6.07_i686-apple-darwin 0x0002f733 nc_get_var_float + 355 8 hadcm3n_6.07_i686-apple-darwin 0x0000b927 _Z12annual_cyclePKSt6vectorISsSaISsEEPKcii + 3177 9 hadcm3n_6.07_i686-apple-darwin 0x0000d36b _Z12decadalMeansiPKc + 957 10 hadcm3n_6.07_i686-apple-darwin 0x000067ff _Z9doCM3Procv + 185 11 hadcm3n_6.07_i686-apple-darwin 0x0000791c _Z8mainLoopv + 410 12 hadcm3n_6.07_i686-apple-darwin 0x000087c7 _Z6workerv + 2989 13 hadcm3n_6.07_i686-apple-darwin 0x00008aa9 main + 491 14 hadcm3n_6.07_i686-apple-darwin 0x00002676 start + 54 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x948d07c2 ecx: 0xbff901dc edx: 0x9489c166 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff90218 esp: 0xbff901dc ss: 0x0000001f efl: 0x00000206 eip: 0x9489c166 cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/2/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x902ca000 - 0x902cefff /usr/lib/system/libmathCommon.A.dylib 0x90dee000 - 0x90df5fff /usr/lib/libgcc_s.1.dylib 0x93800000 - 0x9380efff /usr/lib/libz.1.dylib 0x9489b000 - 0x94a02fff /usr/lib/libSystem.B.dylib 0x97396000 - 0x973f3fff /usr/lib/libstdc++.6.dylib Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Dec 2011 13:40:30	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	777,600	1,763,560	2.2680
19 Dec 2011 10:07:58	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	751,680	1,705,560	2.2690
17 Dec 2011 23:00:30	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	725,760	1,646,500	2.2687
16 Dec 2011 06:15:52	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	699,840	1,586,066	2.2663
14 Dec 2011 18:06:08	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	673,920	1,525,799	2.2641
13 Dec 2011 13:32:46	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	648,000	1,467,082	2.2640
12 Dec 2011 05:47:36	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	622,080	1,407,375	2.2624
11 Dec 2011 02:55:24	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	596,160	1,346,637	2.2589
10 Dec 2011 08:31:01	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	570,240	1,288,832	2.2602
09 Dec 2011 03:51:29	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	544,320	1,230,454	2.2605
07 Dec 2011 16:39:35	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	518,400	1,171,463	2.2598
06 Dec 2011 05:33:51	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	492,480	1,112,065	2.2581
04 Dec 2011 05:39:10	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	466,560	1,049,763	2.2500
02 Dec 2011 22:43:35	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	440,640	991,205	2.2495
01 Dec 2011 16:18:21	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	414,720	932,873	2.2494
30 Nov 2011 13:22:55	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	388,800	874,466	2.2491
29 Nov 2011 16:48:20	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	362,880	812,787	2.2398
28 Nov 2011 20:16:16	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	336,960	753,069	2.2349
28 Nov 2011 02:58:51	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	311,040	694,536	2.2329
27 Nov 2011 08:35:40	962203	13627294	hadcm3n_t6rh_1940_40_007538942_3	285,120	634,113	2.2240