climateprediction.net home page
Task 15277287

Task 15277287

Name hadcm3n_zmmj_1880_40_008199666_1
Workunit 8354790
Created 13 Sep 2012, 2:12:56 UTC
Sent 13 Sep 2012, 2:20:39 UTC
Report deadline 13 Dec 2012, 9:47:50 UTC
Received 28 Oct 2012, 0:15:53 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1196273
Run time 13 days 6 hours 50 min 57 sec
CPU time 13 days 5 hours 24 min 22 sec
Validate state Invalid
Credit 6,220.80
Device peak FLOPS 2.43 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-pc-linux-gnu
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
*** glibc detected *** ../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu: double free or corruption (out): 0x09662c28 ***
======= Backtrace: =========
/lib32/libc.so.6(+0x6c421)[0xf74c2421]
/lib32/libc.so.6(+0x6dcb0)[0xf74c3cb0]
/lib32/libc.so.6(cfree+0x6d)[0xf74c6d5d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf769d7b1]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf769d80d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe7)[0xf746cce7]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]
======= Memory map: ========
08048000-080e3000 r-xp 00000000 08:01 2889478                            /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e3000-080e4000 rw-p 0009b000 08:01 2889478                            /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu
080e4000-0813b000 rw-p 00000000 00:00 0 
0960d000-09672000 rw-p 00000000 00:00 0                                  [heap]
f6e00000-f6e21000 rw-p 00000000 00:00 0 
f6e21000-f6f00000 ---p 00000000 00:00 0 
f6fd8000-f7453000 rw-s 00000000 08:01 3162567                            /var/lib/boinc-client/slots/1/141630
f7453000-f7456000 rw-p 00000000 00:00 0 
f7456000-f75aa000 r-xp 00000000 08:01 2093207                            /lib32/libc-2.12.1.so
f75aa000-f75ab000 ---p 00154000 08:01 2093207                            /lib32/libc-2.12.1.so
f75ab000-f75ad000 r--p 00154000 08:01 2093207                            /lib32/libc-2.12.1.so
f75ad000-f75ae000 rw-p 00156000 08:01 2093207                            /lib32/libc-2.12.1.so
f75ae000-f75b1000 rw-p 00000000 00:00 0 
f75b1000-f75cb000 r-xp 00000000 08:01 3014967                            /usr/lib32/libgcc_s.so.1
f75cb000-f75cc000 r--p 00019000 08:01 3014967                            /usr/lib32/libgcc_s.so.1
f75cc000-f75cd000 rw-p 0001a000 08:01 3014967                            /usr/lib32/libgcc_s.so.1
f75cd000-f75f1000 r-xp 00000000 08:01 2093214                            /lib32/libm-2.12.1.so
f75f1000-f75f2000 r--p 00023000 08:01 2093214                            /lib32/libm-2.12.1.so
f75f2000-f75f3000 rw-p 00024000 08:01 2093214                            /lib32/libm-2.12.1.so
f75f3000-f76d2000 r-xp 00000000 08:01 3014970                            /usr/lib32/libstdc++.so.6.0.14
f76d2000-f76d6000 r--p 000de000 08:01 3014970                            /usr/lib32/libstdc++.so.6.0.14
f76d6000-f76d7000 rw-p 000e2000 08:01 3014970                            /usr/lib32/libstdc++.so.6.0.14
f76d7000-f76de000 rw-p 00000000 00:00 0 
f76de000-f76e0000 r-xp 00000000 08:01 2093223                            /lib32/libdl-2.12.1.so
f76e0000-f76e1000 r--p 00001000 08:01 2093223                            /lib32/libdl-2.12.1.so
f76e1000-f76e2000 rw-p 00002000 08:01 2093223                            /lib32/libdl-2.12.1.so
f76e2000-f76e3000 rw-p 00000000 00:00 0 
f76e3000-f76f8000 r-xp 00000000 08:01 2093209                            /lib32/libpthread-2.12.1.so
f76f8000-f76f9000 r--p 00014000 08:01 2093209                            /lib32/libpthread-2.12.1.so
f76f9000-f76fa000 rw-p 00015000 08:01 2093209                            /lib32/libpthread-2.12.1.so
f76fa000-f76fc000 rw-p 00000000 00:00 0 
f7718000-f7719000 rw-p 00000000 00:00 0 
f7719000-f771a000 ---p 00000000 00:00 0 
f771a000-f771d000 rw-p 00000000 00:00 0 
f771d000-f771f000 rw-s 00000000 08:01 3162564                            /var/lib/boinc-client/slots/1/boinc_mmap_file
f771f000-f7721000 rw-p 00000000 00:00 0 
f7721000-f7722000 r-xp 00000000 00:00 0                                  [vdso]
f7722000-f773e000 r-xp 00000000 08:01 2093215                            /lib32/ld-2.12.1.so
f773e000-f773f000 r--p 0001b000 08:01 2093215                            /lib32/ld-2.12.1.so
f773f000-f7740000 rw-p 0001c000 08:01 2093215                            /lib32/ld-2.12.1.so
ffd7d000-ffded000 rw-p 00000000 00:00 0                                  [stack]
SIGABRT: abort called
Stack trace (19 frames):
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(boinc_catch_signal+0x6f)[0x80b80df]
[0xf7721400]
[0xf7721430]
/lib32/libc.so.6(gsignal+0x51)[0xf7480a01]
/lib32/libc.so.6(abort+0x182)[0xf7483e42]
/lib32/libc.so.6(+0x62345)[0xf74b8345]
/lib32/libc.so.6(+0x6c421)[0xf74c2421]
/lib32/libc.so.6(+0x6dcb0)[0xf74c3cb0]
/lib32/libc.so.6(cfree+0x6d)[0xf74c6d5d]
/usr/lib32/libstdc++.so.6(_ZdlPv+0x21)[0xf769d7b1]
/usr/lib32/libstdc++.so.6(_ZdaPv+0x1d)[0xf769d80d]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8053e8e]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8057bc4]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x804f232]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x8050491]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805112c]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu[0x805137a]
/lib32/libc.so.6(__libc_start_main+0xe7)[0xf746cce7]
../../projects/climateprediction.net/hadcm3n_6.07_i686-pc-linux-gnu(__gxx_personality_v0+0x169)[0x804cb51]

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
28 Oct 2012 00:18:04 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 518,400 1,142,657 2.2042
24 Oct 2012 22:13:36 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 492,480 1,085,404 2.2040
22 Oct 2012 13:48:22 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 466,560 1,029,225 2.2060
20 Oct 2012 02:20:39 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 440,640 972,768 2.2076
17 Oct 2012 20:04:00 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 414,720 915,690 2.2080
15 Oct 2012 09:40:49 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 388,800 858,445 2.2079
13 Oct 2012 02:16:25 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 362,880 801,268 2.2081
11 Oct 2012 08:30:38 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 336,960 744,922 2.2107
08 Oct 2012 07:25:42 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 311,040 687,963 2.2118
05 Oct 2012 20:43:53 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 285,120 631,262 2.2140
03 Oct 2012 12:07:08 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 259,200 573,712 2.2134
01 Oct 2012 03:39:22 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 233,280 516,559 2.2143
28 Sep 2012 15:42:44 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 207,360 459,711 2.2170
26 Sep 2012 07:37:23 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 181,440 401,582 2.2133
23 Sep 2012 23:52:54 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 155,520 344,124 2.2127
21 Sep 2012 14:29:08 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 129,600 286,724 2.2124
19 Sep 2012 05:17:17 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 103,680 228,034 2.1994
18 Sep 2012 07:20:10 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 77,760 170,712 2.1954
14 Sep 2012 13:53:41 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 51,840 113,175 2.1832
13 Sep 2012 18:34:50 1196273 15277287 hadcm3n_zmmj_1880_40_008199666_1 25,920 56,681 2.1868


©2024 cpdn.org