Task 13382785

Name	hadcm3n_yem0_1940_40_007423836_2
Workunit	7621471
Created	13 Sep 2011, 3:29:12 UTC
Sent	13 Sep 2011, 3:40:32 UTC
Report deadline	13 Dec 2011, 11:07:43 UTC
Received	15 Nov 2011, 16:40:12 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	193 (0x000000C1) EXIT_SIGNAL
Computer ID	1029082
Run time	45 days 1 hours 33 min 5 sec
CPU time	45 days 1 hours 33 min 5 sec
Validate state	Invalid
Credit	12,441.60
Device peak FLOPS	1.40 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.4.5</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1857, selfPID=1857, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/yem0ko.pje1c10 is not a valid UM file. Error converting file to netcdf: dataout/yem0ko.pje1c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9517, selfPID=9517, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/yem0ko.pjh5c10 is not a valid UM file. Error converting file to netcdf: dataout/yem0ko.pjh5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=27182, selfPID=27182, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Error: Input file: dataout/yem0ko.pjh6c10 is not a valid UM file. Error converting file to netcdf: dataout/yem0ko.pjh6c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... * glibc detected * hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09d7d9f0 *** ======= Backtrace: ========= /lib/tls/i686/cmov/libc.so.6[0x17aff1] /lib/tls/i686/cmov/libc.so.6[0x17c6f2] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x17f7cd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x4876f1] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x48774d] hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x126b56] hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] ======= Memory map: ======== 00110000-0024e000 r-xp 00000000 08:01 8831885 /lib/tls/i686/cmov/libc-2.10.1.so 0024e000-0024f000 ---p 0013e000 08:01 8831885 /lib/tls/i686/cmov/libc-2.10.1.so 0024f000-00251000 r--p 0013e000 08:01 8831885 /lib/tls/i686/cmov/libc-2.10.1.so 00251000-00252000 rw-p 00140000 08:01 8831885 /lib/tls/i686/cmov/libc-2.10.1.so 00252000-00255000 rw-p 00000000 00:00 0 00314000-0032f000 r-xp 00000000 08:01 8814654 /lib/ld-2.10.1.so 0032f000-00330000 r--p 0001a000 08:01 8814654 /lib/ld-2.10.1.so 00330000-00331000 rw-p 0001b000 08:01 8814654 /lib/ld-2.10.1.so 003cf000-004b5000 r-xp 00000000 08:01 3260968 /usr/lib/libstdc++.so.6.0.13 004b5000-004b9000 r--p 000e6000 08:01 3260968 /usr/lib/libstdc++.so.6.0.13 004b9000-004ba000 rw-p 000ea000 08:01 3260968 /usr/lib/libstdc++.so.6.0.13 004ba000-004c1000 rw-p 00000000 00:00 0 00684000-00699000 r-xp 00000000 08:01 8832188 /lib/tls/i686/cmov/libpthread-2.10.1.so 00699000-0069a000 r--p 00014000 08:01 8832188 /lib/tls/i686/cmov/libpthread-2.10.1.so 0069a000-0069b000 rw-p 00015000 08:01 8832188 /lib/tls/i686/cmov/libpthread-2.10.1.so 0069b000-0069d000 rw-p 00000000 00:00 0 00b4d000-00b4e000 r-xp 00000000 00:00 0 [vdso] 00b9f000-00ba1000 r-xp 00000000 08:01 8831892 /lib/tls/i686/cmov/libdl-2.10.1.so 00ba1000-00ba2000 r--p 00001000 08:01 8831892 /lib/tls/i686/cmov/libdl-2.10.1.so 00ba2000-00ba3000 rw-p 00002000 08:01 8831892 /lib/tls/i686/cmov/libdl-2.10.1.so 00efc000-00f18000 r-xp 00000000 08:01 8814606 /lib/libgcc_s.so.1 00f18000-00f19000 r--p 0001b000 08:01 8814606 /lib/libgcc_s.so.1 00f19000-00f1a000 rw-p 0001c000 08:01 8814606 /lib/libgcc_s.so.1 00f3e000-00f62000 r-xp 00000000 08:01 8831893 /lib/tls/i686/cmov/libm-2.10.1.so 00f62000-00f63000 r--p 00023000 08:01 8831893 /lib/tls/i686/cmov/libm-2.10.1.so 00f63000-00f64000 rw-p 00024000 08:01 8831893 /lib/tls/i686/cmov/libm-2.10.1.so 08048000-080e3000 r-xp 00000000 08:01 2711609 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e3000-080e4000 rw-p 0009b000 08:01 2711609 /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu 080e4000-0813b000 rw-p 00000000 00:00 0 09d28000-09d8e000 rw-p 00000000 00:00 0 [heap] b7200000-b7221000 rw-p 00000000 00:00 0 b7221000-b7300000 ---p 00000000 00:00 0 b73ac000-b7827000 rw-s 00000000 08:01 2826370 /var/lib/boinc-client/slots/4/136250 b7827000-b782a000 rw-p 00000000 00:00 0 b7835000-b7836000 rw-p 00000000 00:00 0 b7836000-b7837000 ---p 00000000 00:00 0 b7837000-b783a000 rw-p 00000000 00:00 0 b783a000-b783c000 rw-s 00000000 08:01 2826367 /var/lib/boinc-client/slots/4/boinc_mmap_file b783c000-b783e000 rw-p 00000000 00:00 0 bfaf0000-bfb61000 rw-p 00000000 00:00 0 [stack] SIGABRT: abort called Stack trace (19 frames): hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df] [0xb4d400] [0xb4d422] /lib/tls/i686/cmov/libc.so.6(gsignal+0x51)[0x13a4d1] /lib/tls/i686/cmov/libc.so.6(abort+0x182)[0x13d932] /lib/tls/i686/cmov/libc.so.6[0x170ee5] /lib/tls/i686/cmov/libc.so.6[0x17aff1] /lib/tls/i686/cmov/libc.so.6[0x17c6f2] /lib/tls/i686/cmov/libc.so.6(cfree+0x6d)[0x17f7cd] /usr/lib/libstdc++.so.6(_ZdlPv+0x21)[0x4876f1] /usr/lib/libstdc++.so.6(_ZdaPv+0x1d)[0x48774d] hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e] hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4] hadcm3n_6.07_i686-pc-linux-gnu[0x804f232] hadcm3n_6.07_i686-pc-linux-gnu[0x8050491] hadcm3n_6.07_i686-pc-linux-gnu[0x805112c] hadcm3n_6.07_i686-pc-linux-gnu[0x805137a] /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6)[0x126b56] hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51] Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Nov 2011 16:50:25	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	1,036,800	3,893,569	3.7554
09 Nov 2011 00:03:20	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	1,010,880	3,798,882	3.7580
07 Nov 2011 17:09:17	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	984,960	3,704,459	3.7610
06 Nov 2011 13:33:01	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	959,040	3,608,861	3.7630
05 Nov 2011 04:15:52	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	933,120	3,509,149	3.7607
03 Nov 2011 00:25:07	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	907,200	3,410,088	3.7589
01 Nov 2011 07:30:44	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	881,280	3,307,187	3.7527
31 Oct 2011 19:16:33	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	855,360	3,205,396	3.7474
31 Oct 2011 18:35:03	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	829,440	3,102,681	3.7407
31 Oct 2011 17:33:16	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	803,520	3,003,816	3.7383
31 Oct 2011 16:51:30	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	777,600	2,900,334	3.7299
31 Oct 2011 14:43:39	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	751,680	2,805,485	3.7323
31 Oct 2011 14:43:39	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	725,760	2,711,165	3.7356
31 Oct 2011 14:43:39	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	699,840	2,617,081	3.7395
31 Oct 2011 14:43:39	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	673,920	2,522,472	3.7430
31 Oct 2011 14:43:39	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	648,000	2,427,540	3.7462
18 Oct 2011 15:43:12	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	622,080	2,332,940	3.7502
17 Oct 2011 12:25:00	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	596,160	2,238,706	3.7552
15 Oct 2011 23:08:24	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	570,240	2,143,631	3.7592
14 Oct 2011 18:52:42	1029082	13382785	hadcm3n_yem0_1940_40_007423836_2	544,320	2,048,068	3.7626