climateprediction.net home page
Task 12643850

Task 12643850

Name famous_vc8k_1799_200_006701626_6
Workunit 6904879
Created 8 Mar 2011, 9:48:00 UTC
Sent 8 Mar 2011, 9:58:12 UTC
Report deadline 7 Jun 2011, 17:25:23 UTC
Received 10 Mar 2011, 19:13:05 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1115458
Run time 1 days 7 hours 23 min 57 sec
CPU time 1 days 6 hours 54 min 44 sec
Validate state Invalid
Credit 1,420.64
Device peak FLOPS 2.46 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-apple-darwin
Stderr 
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Tue Mar  8 13:12:50 2011

  0  0x0048fdcf  PrintBacktrace (in famous_um_6.11_i686-apple-darwin) + 1221
  1  0x0049054a  boinc_catch_signal (in famous_um_6.11_i686-apple-darwin) + 474
  2  0x93f8546b  0x93f8546b
  3  0xffffffff  0xffffffff
  4  0x0134e6b3  0x134e6b3
  5  0x0134d44f  0x134d44f
  6  0x002eeac6  readcntl (in famous_um_6.11_i686-apple-darwin) + 2598
  7  0x003f55ea  um_setup (in famous_um_6.11_i686-apple-darwin) + 458
  8  0x00452abd  um_shell (in famous_um_6.11_i686-apple-darwin) + 11429
  9  0x004698c0  main (in famous_um_6.11_i686-apple-darwin) + 1112
 10  0x000026b6  _start (in famous_um_6.11_i686-apple-darwin) + 216
 11  0x000025dd  start (in famous_um_6.11_i686-apple-darwin) + 41

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x93f1f0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c
   ss: 0x0000001f efl: 0x00000206 eip: 0x93f1f0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libirc.dylib
0x93f1e000 - 0x940c5fff /usr/lib/libSystem.B.dylib
0x964da000 - 0x964e8fff /usr/lib/libz.1.dylib
0x9749a000 - 0x9749dfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=69952, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Tue Mar  8 22:31:11 2011

  0  0x0048fdcf  PrintBacktrace (in famous_um_6.11_i686-apple-darwin) + 1221
  1  0x0049054a  boinc_catch_signal (in famous_um_6.11_i686-apple-darwin) + 474
  2  0x93f8546b  0x93f8546b
  3  0xffffffff  0xffffffff
  4  0x0134e6b3  0x134e6b3
  5  0x0134d44f  0x134d44f
  6  0x002eeac6  readcntl (in famous_um_6.11_i686-apple-darwin) + 2598
  7  0x003f55ea  um_setup (in famous_um_6.11_i686-apple-darwin) + 458
  8  0x00452abd  um_shell (in famous_um_6.11_i686-apple-darwin) + 11429
  9  0x004698c0  main (in famous_um_6.11_i686-apple-darwin) + 1112
 10  0x000026b6  _start (in famous_um_6.11_i686-apple-darwin) + 216
 11  0x000025dd  start (in famous_um_6.11_i686-apple-darwin) + 41

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x93f1f0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c
   ss: 0x0000001f efl: 0x00000206 eip: 0x93f1f0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libirc.dylib
0x93f1e000 - 0x940c5fff /usr/lib/libSystem.B.dylib
0x964da000 - 0x964e8fff /usr/lib/libz.1.dylib
0x9749a000 - 0x9749dfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=74758, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Wed Mar  9 23:37:29 2011

  0  0x0048fdcf  PrintBacktrace (in famous_um_6.11_i686-apple-darwin) + 1221
  1  0x0049054a  boinc_catch_signal (in famous_um_6.11_i686-apple-darwin) + 474
  2  0x93f8546b  0x93f8546b
  3  0xffffffff  0xffffffff
  4  0x0134e6b3  0x134e6b3
  5  0x0134d44f  0x134d44f
  6  0x002eeac6  readcntl (in famous_um_6.11_i686-apple-darwin) + 2598
  7  0x003f55ea  um_setup (in famous_um_6.11_i686-apple-darwin) + 458
  8  0x00452abd  um_shell (in famous_um_6.11_i686-apple-darwin) + 11429
  9  0x004698c0  main (in famous_um_6.11_i686-apple-darwin) + 1112
 10  0x000026b6  _start (in famous_um_6.11_i686-apple-darwin) + 216
 11  0x000025dd  start (in famous_um_6.11_i686-apple-darwin) + 41

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x93f1f0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c
   ss: 0x0000001f efl: 0x00000206 eip: 0x93f1f0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libirc.dylib
0x93f1e000 - 0x940c5fff /usr/lib/libSystem.B.dylib
0x964da000 - 0x964e8fff /usr/lib/libz.1.dylib
0x9749a000 - 0x9749dfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=92090, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Thu Mar 10 00:30:07 2011

  0  0x0048fdcf  PrintBacktrace (in famous_um_6.11_i686-apple-darwin) + 1221
  1  0x0049054a  boinc_catch_signal (in famous_um_6.11_i686-apple-darwin) + 474
  2  0x93f8546b  0x93f8546b
  3  0xffffffff  0xffffffff
  4  0x0134e6b3  0x134e6b3
  5  0x0134d44f  0x134d44f
  6  0x002eeac6  readcntl (in famous_um_6.11_i686-apple-darwin) + 2598
  7  0x003f55ea  um_setup (in famous_um_6.11_i686-apple-darwin) + 458
  8  0x00452abd  um_shell (in famous_um_6.11_i686-apple-darwin) + 11429
  9  0x004698c0  main (in famous_um_6.11_i686-apple-darwin) + 1112
 10  0x000026b6  _start (in famous_um_6.11_i686-apple-darwin) + 216
 11  0x000025dd  start (in famous_um_6.11_i686-apple-darwin) + 41

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x93f1f0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c
   ss: 0x0000001f efl: 0x00000206 eip: 0x93f1f0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libirc.dylib
0x93f1e000 - 0x940c5fff /usr/lib/libSystem.B.dylib
0x964da000 - 0x964e8fff /usr/lib/libz.1.dylib
0x9749a000 - 0x9749dfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=92692, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Thu Mar 10 00:59:55 2011

  0  0x0048fdcf  PrintBacktrace (in famous_um_6.11_i686-apple-darwin) + 1221
  1  0x0049054a  boinc_catch_signal (in famous_um_6.11_i686-apple-darwin) + 474
  2  0x93f8546b  0x93f8546b
  3  0xffffffff  0xffffffff
  4  0x0134e6b3  0x134e6b3
  5  0x0134d44f  0x134d44f
  6  0x003f5826  um_submodel_init (in famous_um_6.11_i686-apple-darwin) + 406
  7  0x00450bd8  um_shell (in famous_um_6.11_i686-apple-darwin) + 3520
  8  0x004698c0  main (in famous_um_6.11_i686-apple-darwin) + 1112
  9  0x000026b6  _start (in famous_um_6.11_i686-apple-darwin) + 216
 10  0x000025dd  start (in famous_um_6.11_i686-apple-darwin) + 41

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd4ac edx: 0x93f1f0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4e8 esp: 0xbfffd4ac
   ss: 0x0000001f efl: 0x00000206 eip: 0x93f1f0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libirc.dylib
0x93f1e000 - 0x940c5fff /usr/lib/libSystem.B.dylib
0x964da000 - 0x964e8fff /usr/lib/libz.1.dylib
0x9749a000 - 0x9749dfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=92982, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Thu Mar 10 12:08:21 2011

  0  0x0048fdcf  PrintBacktrace (in famous_um_6.11_i686-apple-darwin) + 1221
  1  0x0049054a  boinc_catch_signal (in famous_um_6.11_i686-apple-darwin) + 474
  2  0x93f8546b  0x93f8546b
  3  0xffffffff  0xffffffff
  4  0x0134e6b3  0x134e6b3
  5  0x0134d44f  0x134d44f
  6  0x002eeac6  readcntl (in famous_um_6.11_i686-apple-darwin) + 2598
  7  0x003f55ea  um_setup (in famous_um_6.11_i686-apple-darwin) + 458
  8  0x00452abd  um_shell (in famous_um_6.11_i686-apple-darwin) + 11429
  9  0x004698c0  main (in famous_um_6.11_i686-apple-darwin) + 1112
 10  0x000026b6  _start (in famous_um_6.11_i686-apple-darwin) + 216
 11  0x000025dd  start (in famous_um_6.11_i686-apple-darwin) + 41

Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x93f1f0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c
   ss: 0x0000001f efl: 0x00000206 eip: 0x93f1f0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_vc8k_1799_200_006701626/lib/libirc.dylib
0x93f1e000 - 0x940c5fff /usr/lib/libSystem.B.dylib
0x964da000 - 0x964e8fff /usr/lib/libz.1.dylib
0x9749a000 - 0x9749dfff /usr/lib/system/libmathCommon.A.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=98585, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
 (7741): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
10 Mar 2011 18:20:20 1115458 12643850 famous_vc8k_1799_200_006701626_6 430,586 109,059 0.2533
10 Mar 2011 17:58:52 1115458 12643850 famous_vc8k_1799_200_006701626_6 421,226 106,650 0.2532
10 Mar 2011 10:24:39 1115458 12643850 famous_vc8k_1799_200_006701626_6 411,866 104,361 0.2534
10 Mar 2011 08:53:34 1115458 12643850 famous_vc8k_1799_200_006701626_6 402,506 101,995 0.2534
10 Mar 2011 08:17:46 1115458 12643850 famous_vc8k_1799_200_006701626_6 393,146 99,635 0.2534
10 Mar 2011 07:36:56 1115458 12643850 famous_vc8k_1799_200_006701626_6 383,786 97,278 0.2535
10 Mar 2011 06:56:00 1115458 12643850 famous_vc8k_1799_200_006701626_6 374,426 94,923 0.2535
10 Mar 2011 06:19:35 1115458 12643850 famous_vc8k_1799_200_006701626_6 365,066 92,564 0.2536
10 Mar 2011 05:36:59 1115458 12643850 famous_vc8k_1799_200_006701626_6 355,706 90,204 0.2536
10 Mar 2011 04:57:45 1115458 12643850 famous_vc8k_1799_200_006701626_6 346,346 87,811 0.2535
10 Mar 2011 04:14:05 1115458 12643850 famous_vc8k_1799_200_006701626_6 336,986 85,394 0.2534
10 Mar 2011 03:37:07 1115458 12643850 famous_vc8k_1799_200_006701626_6 327,626 83,075 0.2536
10 Mar 2011 02:59:12 1115458 12643850 famous_vc8k_1799_200_006701626_6 318,266 80,779 0.2538
10 Mar 2011 02:17:06 1115458 12643850 famous_vc8k_1799_200_006701626_6 308,906 78,417 0.2539
10 Mar 2011 00:55:12 1115458 12643850 famous_vc8k_1799_200_006701626_6 299,546 76,058 0.2539
09 Mar 2011 22:55:55 1115458 12643850 famous_vc8k_1799_200_006701626_6 290,186 73,712 0.2540
09 Mar 2011 20:33:16 1115458 12643850 famous_vc8k_1799_200_006701626_6 280,826 71,352 0.2541
09 Mar 2011 17:57:15 1115458 12643850 famous_vc8k_1799_200_006701626_6 271,466 69,007 0.2542
09 Mar 2011 15:04:29 1115458 12643850 famous_vc8k_1799_200_006701626_6 262,106 66,667 0.2544
09 Mar 2011 12:45:24 1115458 12643850 famous_vc8k_1799_200_006701626_6 252,746 64,384 0.2547


©2024 climateprediction.net