Name | hadcm3n_yf4w_1900_40_007526285_1 |
Workunit | 7723760 |
Created | 28 Oct 2011, 13:45:10 UTC |
Sent | 29 Oct 2011, 16:04:49 UTC |
Report deadline | 28 Jan 2012, 23:32:00 UTC |
Received | 30 Nov 2011, 8:51:43 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 193 (0x000000C1) EXIT_SIGNAL |
Computer ID | 1298787 |
Run time | 20 days 1 hours 37 min 39 sec |
CPU time | 17 days 1 hours 9 min 11 sec |
Validate state | Invalid |
Credit | 9,331.20 |
Device peak FLOPS | 2.63 GFLOPS |
Application version | UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-apple-darwin |
Stderr | <core_client_version>6.12.35</core_client_version> <![CDATA[ <message> process exited with code 193 (0xc1, -63) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 06:06:07 (27236): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 20:31:09 (70055): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:14:31 (7062): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:46:08 (92819): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 07:18:20 (27598): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:45:21 (63839): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(78986,0xacf652c0) malloc: *** error for object 0x972e04: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Signal 3 received, exiting... Called boinc_finish hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e004: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e000: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e004: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e000: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e004: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e000: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e004: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(62768,0xacf652c0) malloc: *** error for object 0x285e000: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 23:46:36 (60723): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:10:56 (507): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:10:57 (507): No heartbeat from core client for 30 sec - exiting 16:10:58 (507): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... hadcm3n_6.07_i686-apple-darwin(82466,0xacf652c0) malloc: *** error for object 0x81ee04: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82466,0xacf652c0) malloc: *** error for object 0x81ee00: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82466,0xacf652c0) malloc: *** error for object 0x1022404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82466,0xacf652c0) malloc: *** error for object 0x1022400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82466,0xacf652c0) malloc: *** error for object 0x1022404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82466,0xacf652c0) malloc: *** error for object 0x1022400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(82466,0xacf652c0) malloc: *** error for object 0x1022404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=82469, selfPID=82469, iMonCtr=1 hadcm3n_6.07_i686-apple-darwin(31259,0xacf652c0) malloc: *** error for object 0x70eba04: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31259,0xacf652c0) malloc: *** error for object 0x70eba00: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31259,0xacf652c0) malloc: *** error for object 0x70eba04: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31259,0xacf652c0) malloc: *** error for object 0x70eba00: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31259,0xacf652c0) malloc: *** error for object 0x70eba04: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31259,0xacf652c0) malloc: *** error for object 0x70eba00: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(31259,0xacf652c0) malloc: *** error for object 0x70eba04: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x229e400: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x2a99404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x2a9d404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug hadcm3n_6.07_i686-apple-darwin(7813,0xacf652c0) malloc: *** error for object 0x2aa1404: incorrect checksum for freed object - object was probably modified after being freed. *** set a breakpoint in malloc_error_break to debug SIGSEGV: segmentation violation Crashed executable name: hadcm3n_6.07_i686-apple-darwin built using BOINC library version 6.13.0 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.7.2 build 11C74 Wed Nov 30 06:01:46 2011 Thread 0 Crashed: Thread 1: atos cannot load symbols for the file ../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin for architecture i386. Thread 0 crashed with X86 Thread State (32-bit): eax: 0x00000000 ebx: 0x00000000 ecx: 0x00000000 edx: 0x00000000 edi: 0x00000000 esi: 0x00000000 ebp: 0xbff8f9c8 esp: 0x00000000 ss: 0x00000000 efl: 0x00000000 eip: 0x0007697e cs: 0x00000000 ds: 0x00000000 es: 0x00000000 fs: 0x00000000 gs: 0x00000000 Binary Images Description: 0x1000 - 0x93fff /Library/Application Support/BOINC Data/slots/30/../../projects/climateprediction.net/hadcm3n_6.07_i686-apple-darwin 0x90005000 - 0x90005fff /usr/lib/system/libkeymgr.dylib 0x903d9000 - 0x903defff /usr/lib/system/libmacho.dylib 0x92c25000 - 0x92c2dfff /usr/lib/system/libcopyfile.dylib 0x95061000 - 0x95065fff /usr/lib/system/libsystem_network.dylib 0x95066000 - 0x9506efff /usr/lib/system/libunwind.dylib 0x95538000 - 0x95546fff /usr/lib/libz.1.dylib 0x95f34000 - 0x95f35fff /usr/lib/system/libsystem_sandbox.dylib 0x96236000 - 0x9624cfff /usr/lib/system/libxpc.dylib 0x96539000 - 0x96567fff /usr/lib/libSystem.B.dylib 0x9674e000 - 0x96751fff /usr/lib/system/libcompiler_rt.dylib 0x9688e000 - 0x96890fff /usr/lib/system/libdyld.dylib 0x96ada000 - 0x96adbfff /usr/lib/system/libunc.dylib 0x96b09000 - 0x96b17fff /usr/lib/system/libdispatch.dylib 0x96e6f000 - 0x96e70fff /usr/lib/system/libremovefile.dylib 0x97799000 - 0x977fbfff /usr/lib/libstdc++.6.dylib 0x98282000 - 0x982a0fff /usr/lib/system/libsystem_kernel.dylib 0x98e51000 - 0x98e51fff /usr/lib/system/libdnsinfo.dylib 0x9903a000 - 0x9903bfff /usr/lib/system/libquarantine.dylib 0x996a2000 - 0x996a6fff /usr/lib/system/libcache.dylib 0x996bd000 - 0x996befff /usr/lib/system/libsystem_blocks.dylib 0x9986f000 - 0x9989efff /usr/lib/system/libsystem_info.dylib 0x999e3000 - 0x99aaefff /usr/lib/system/libsystem_c.dylib 0x9a569000 - 0x9a56cfff /usr/lib/system/libmathCommon.A.dylib 0x9a7f3000 - 0x9a7fafff /usr/lib/system/libsystem_dnssd.dylib 0x9b50e000 - 0x9b516fff /usr/lib/system/liblaunch.dylib 0x9c212000 - 0x9c219fff /usr/lib/system/libsystem_notify.dylib 0x9c905000 - 0x9c948fff /usr/lib/system/libcommonCrypto.dylib 0x9c9b3000 - 0x9c9bcfff /usr/lib/libc++abi.dylib Exiting... </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
29 Nov 2011 01:28:39 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 777,600 | 1,454,586 | 1.8706 |
28 Nov 2011 05:09:31 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 751,680 | 1,405,494 | 1.8698 |
25 Nov 2011 13:07:05 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 725,760 | 1,357,023 | 1.8698 |
24 Nov 2011 07:41:17 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 699,840 | 1,308,272 | 1.8694 |
23 Nov 2011 02:16:03 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 673,920 | 1,259,488 | 1.8689 |
21 Nov 2011 22:41:02 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 648,000 | 1,210,715 | 1.8684 |
20 Nov 2011 18:20:15 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 622,080 | 1,162,056 | 1.8680 |
19 Nov 2011 21:04:51 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 596,160 | 1,113,181 | 1.8673 |
18 Nov 2011 05:43:21 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 570,240 | 1,064,903 | 1.8675 |
16 Nov 2011 14:06:32 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 544,320 | 1,016,487 | 1.8674 |
15 Nov 2011 19:12:56 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 518,400 | 968,377 | 1.8680 |
15 Nov 2011 19:12:56 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 492,480 | 920,115 | 1.8683 |
15 Nov 2011 19:12:56 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 466,560 | 871,913 | 1.8688 |
15 Nov 2011 19:12:56 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 440,640 | 823,653 | 1.8692 |
15 Nov 2011 19:12:56 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 414,720 | 774,820 | 1.8683 |
15 Nov 2011 19:12:56 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 388,800 | 726,756 | 1.8692 |
09 Nov 2011 20:30:41 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 362,880 | 678,413 | 1.8695 |
08 Nov 2011 20:57:39 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 336,960 | 630,489 | 1.8711 |
08 Nov 2011 06:34:23 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 311,040 | 582,407 | 1.8725 |
07 Nov 2011 16:15:11 | 1174448 | 13558331 | hadcm3n_yf4w_1900_40_007526285_1 | 285,120 | 534,368 | 1.8742 |
©2024 cpdn.org