climateprediction.net home page
Task 13470124

Task 13470124

Name hadcm3n_o5tb_1980_40_007451538_2
Workunit 7649041
Created 10 Oct 2011, 20:59:11 UTC
Sent 10 Oct 2011, 20:59:33 UTC
Report deadline 10 Jan 2012, 4:26:44 UTC
Received 1 Nov 2011, 20:48:57 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 193 (0x000000C1) EXIT_SIGNAL
Computer ID 1168218
Run time 12 days 9 hours 52 min 55 sec
CPU time 10 days 13 hours 26 min 48 sec
Validate state Invalid
Credit 6,220.80
Device peak FLOPS 3.27 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
i686-apple-darwin
Stderr
<core_client_version>6.12.35</core_client_version>
<![CDATA[
<message>
process exited with code 193 (0xc1, -63)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
06:33:13 (4099): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2829204: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2829200: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e04: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(29799,0xa049c540) malloc: *** error for object 0x2804e00: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
22:59:04 (5286): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation
hadcm3n_6.07_i686-apple-darwin(55307,0xa049c540) malloc: *** error for object 0x5027604: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(55307,0xa049c540) malloc: *** error for object 0x5028004: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
hadcm3n_6.07_i686-apple-darwin(55307,0xa049c540) malloc: *** error for object 0x5028000: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug

Crashed executable name: hadcm3n_6.07_i686-apple-darwin
built using BOINC library version 6.13.0
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.8 build 10K549
Tue Nov  1 15:39:48 2011

Thread 0 Crashed:
dyld: Library not loaded: /System/Library/PrivateFrameworks/CoreSymbolication.framework/Versions/A/CoreSymbolication
  Referenced from: /usr/bin/atos
  Reason: image not found
0   libSystem.B.dylib                   0x94f6cac5 small_malloc_from_free_list + 133
1   libSystem.B.dylib                   0x94f694d9 szone_malloc_should_clear + 2271
2   libSystem.B.dylib                   0x94f68ba8 malloc_zone_malloc + 81
3   libSystem.B.dylib                   0x94f66c78 malloc + 50
4   libstdc++.6.dylib                   0x92e06617 operator new(unsigned long) + 36
5   libstdc++.6.dylib                   0x92e06703 operator new[](unsigned long) + 17
6   hadcm3n_6.07_i686-apple-darwin      0x0000b8e7 annual_cycle(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const*, char const*, int, int) + 3113
7   hadcm3n_6.07_i686-apple-darwin      0x0000d36b decadalMeans(int, char const*) + 957
8   hadcm3n_6.07_i686-apple-darwin      0x000067ff doCM3Proc() + 185
9   hadcm3n_6.07_i686-apple-darwin      0x0000791c mainLoop() + 410
10  hadcm3n_6.07_i686-apple-darwin      0x000087c7 worker() + 2989
11  hadcm3n_6.07_i686-apple-darwin      0x00008aa9 main + 491
12  hadcm3n_6.07_i686-apple-darwin      0x00002676 start + 54

Thread 1:
0   libSystem.B.dylib                   0x94f64c0e mach_wait_until + 10
1   libSystem.B.dylib                   0x94fec429 nanosleep + 345
2   libSystem.B.dylib                   0x94fec2ca usleep + 61
3   hadcm3n_6.07_i686-apple-darwin      0x00071a7c boinc_sleep(double) + 188
4   hadcm3n_6.07_i686-apple-darwin      0x00067282 timer_thread(void*) + 78
5   libSystem.B.dylib                   0x94f92259 _pthread_start + 345
6   libSystem.B.dylib                   0x94f920de thread_start + 34

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f6b8 esp: 0x00000000
   ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e  cs: 0x00000000
   ds: 0x00000000  es: 0x00000000  fs: 0x00000000  gs: 0x00000000

Binary Images Description:
    0x1000 -    0x93fff /Library/Application Support/BOINC Data/slots/11/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin
0x92a7c000 - 0x92a8afff /usr/lib/libz.1.dylib
0x92dbc000 - 0x92e26fff /usr/lib/libstdc++.6.dylib
0x94f64000 - 0x9510bfff /usr/lib/libSystem.B.dylib
0x98a75000 - 0x98a78fff /usr/lib/system/libmathCommon.A.dylib


Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
01 Nov 2011 20:51:28 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 518,400 912,398 1.7600
01 Nov 2011 02:32:13 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 492,480 865,433 1.7573
31 Oct 2011 19:43:10 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 466,560 817,340 1.7518
31 Oct 2011 19:19:55 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 440,640 770,426 1.7484
31 Oct 2011 18:52:41 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 414,720 724,534 1.7470
31 Oct 2011 18:28:21 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 388,800 679,190 1.7469
31 Oct 2011 17:25:07 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 362,880 632,569 1.7432
31 Oct 2011 17:01:58 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 336,960 586,086 1.7393
31 Oct 2011 16:32:17 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 311,040 538,884 1.7325
31 Oct 2011 15:23:26 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 285,120 492,789 1.7284
31 Oct 2011 13:29:50 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 259,200 449,988 1.7361
31 Oct 2011 13:29:50 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 233,280 406,709 1.7434
31 Oct 2011 13:29:50 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 207,360 363,981 1.7553
31 Oct 2011 13:29:50 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 181,440 321,312 1.7709
19 Oct 2011 06:33:55 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 155,520 275,622 1.7723
18 Oct 2011 09:58:30 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 129,600 231,124 1.7834
17 Oct 2011 16:40:36 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 103,680 187,207 1.8056
15 Oct 2011 06:35:18 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 77,760 141,764 1.8231
12 Oct 2011 06:48:29 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 51,840 97,033 1.8718
11 Oct 2011 12:54:24 1168218 13470124 hadcm3n_o5tb_1980_40_007451538_2 25,920 48,173 1.8585


©2024 cpdn.org