climateprediction.net home page
Task 11710888

Task 11710888

Name famous_v3ft_1399_200_006690223_4
Workunit 6893476
Created 26 Aug 2010, 15:53:59 UTC
Sent 5 Sep 2010, 22:22:39 UTC
Report deadline 6 Dec 2010, 5:49:50 UTC
Received 9 Sep 2010, 15:07:19 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1098845
Run time 22 hours 19 min 38 sec
CPU time 21 hours 11 min 18 sec
Validate state Invalid
Credit 1,729.46
Device peak FLOPS 3.58 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-apple-darwin
Stderr
<core_client_version>6.10.58</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Tue Sep  7 00:35:24 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x92a141fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x929ae0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c
   ss: 0x0000001f efl: 0x00000206 eip: 0x929ae0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90003000 - 0x90006fff /usr/lib/system/libmathCommon.A.dylib
0x929ad000 - 0x92b53fff /usr/lib/libSystem.B.dylib
0x92d4e000 - 0x92d5cfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3614, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Tue Sep  7 00:43:35 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x92a141fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x929ae0fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c
   ss: 0x0000001f efl: 0x00000206 eip: 0x929ae0fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90003000 - 0x90006fff /usr/lib/system/libmathCommon.A.dylib
0x929ad000 - 0x92b53fff /usr/lib/libSystem.B.dylib
0x92d4e000 - 0x92d5cfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3684, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Wed Sep  8 23:22:27 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x905971fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x905310fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c
   ss: 0x0000001f efl: 0x00000206 eip: 0x905310fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90397000 - 0x9039afff /usr/lib/system/libmathCommon.A.dylib
0x90530000 - 0x906d6fff /usr/lib/libSystem.B.dylib
0x9930d000 - 0x9931bfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1042, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Thu Sep  9 00:27:16 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x905971fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x905310fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c
   ss: 0x0000001f efl: 0x00000206 eip: 0x905310fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90397000 - 0x9039afff /usr/lib/system/libmathCommon.A.dylib
0x90530000 - 0x906d6fff /usr/lib/libSystem.B.dylib
0x9930d000 - 0x9931bfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1525, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Thu Sep  9 00:41:18 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x905971fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x905310fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c
   ss: 0x0000001f efl: 0x00000206 eip: 0x905310fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90397000 - 0x9039afff /usr/lib/system/libmathCommon.A.dylib
0x90530000 - 0x906d6fff /usr/lib/libSystem.B.dylib
0x9930d000 - 0x9931bfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1663, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...

Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED.                                                                                                                                                                                                                     tmp/pipe_dummy                                                                  

Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED.                                                                                                                                                                                                                     tmp/pipe_dummy                                                                  
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Thu Sep  9 17:05:33 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x905971fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x905310fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c
   ss: 0x0000001f efl: 0x00000206 eip: 0x905310fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90397000 - 0x9039afff /usr/lib/system/libmathCommon.A.dylib
0x90530000 - 0x906d6fff /usr/lib/libSystem.B.dylib
0x9930d000 - 0x9931bfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=214, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Thu Sep  9 17:05:35 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x905971fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd49c edx: 0x905310fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4d8 esp: 0xbfffd49c
   ss: 0x0000001f efl: 0x00000206 eip: 0x905310fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90397000 - 0x9039afff /usr/lib/system/libmathCommon.A.dylib
0x90530000 - 0x906d6fff /usr/lib/libSystem.B.dylib
0x9930d000 - 0x9931bfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=214, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.4 build 10F569
Thu Sep  9 17:05:37 2010

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x905971fb    3  0xffffffff    4  0x0134e6b3    5  0x0134d44f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc40c edx: 0x905310fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc448 esp: 0xbfffc40c
   ss: 0x0000001f efl: 0x00000206 eip: 0x905310fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12a9000 -  0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libsvml.dylib
 0x131c000 -  0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libifcoremt.dylib
 0x13fc000 -  0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libimf.dylib
 0x1669000 -  0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v3ft_1399_200_006690223/lib/libirc.dylib
0x90397000 - 0x9039afff /usr/lib/system/libmathCommon.A.dylib
0x90530000 - 0x906d6fff /usr/lib/libSystem.B.dylib
0x9930d000 - 0x9931bfff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=214, iMonCtr=1
Model crash detected, will try to restart...

Model crashed: P_TH_ADJ : NEGATIVE PRESSURE VALUE CREATED.                                                                                                                                                                                                                     tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
 (214): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
09 Sep 2010 15:00:58 1098845 11710888 famous_v3ft_1399_200_006690223_4 524,186 75,630 0.1443
09 Sep 2010 14:33:48 1098845 11710888 famous_v3ft_1399_200_006690223_4 514,826 74,286 0.1443
09 Sep 2010 14:12:42 1098845 11710888 famous_v3ft_1399_200_006690223_4 505,466 72,940 0.1443
09 Sep 2010 13:46:30 1098845 11710888 famous_v3ft_1399_200_006690223_4 496,106 71,598 0.1443
09 Sep 2010 13:25:00 1098845 11710888 famous_v3ft_1399_200_006690223_4 486,746 70,256 0.1443
09 Sep 2010 13:03:25 1098845 11710888 famous_v3ft_1399_200_006690223_4 477,386 68,913 0.1444
08 Sep 2010 20:11:27 1098845 11710888 famous_v3ft_1399_200_006690223_4 468,026 67,564 0.1444
08 Sep 2010 19:45:21 1098845 11710888 famous_v3ft_1399_200_006690223_4 458,666 66,215 0.1444
08 Sep 2010 19:18:37 1098845 11710888 famous_v3ft_1399_200_006690223_4 449,306 64,873 0.1444
08 Sep 2010 18:57:13 1098845 11710888 famous_v3ft_1399_200_006690223_4 439,946 63,532 0.1444
08 Sep 2010 18:30:39 1098845 11710888 famous_v3ft_1399_200_006690223_4 430,586 62,190 0.1444
08 Sep 2010 18:09:26 1098845 11710888 famous_v3ft_1399_200_006690223_4 421,226 60,847 0.1445
08 Sep 2010 17:48:14 1098845 11710888 famous_v3ft_1399_200_006690223_4 411,866 59,506 0.1445
08 Sep 2010 17:22:01 1098845 11710888 famous_v3ft_1399_200_006690223_4 402,506 58,162 0.1445
08 Sep 2010 17:00:41 1098845 11710888 famous_v3ft_1399_200_006690223_4 393,146 56,819 0.1445
08 Sep 2010 16:39:19 1098845 11710888 famous_v3ft_1399_200_006690223_4 383,786 55,478 0.1446
08 Sep 2010 16:12:58 1098845 11710888 famous_v3ft_1399_200_006690223_4 374,426 54,127 0.1446
08 Sep 2010 15:51:17 1098845 11710888 famous_v3ft_1399_200_006690223_4 365,066 52,775 0.1446
08 Sep 2010 15:25:00 1098845 11710888 famous_v3ft_1399_200_006690223_4 355,706 51,431 0.1446
07 Sep 2010 21:08:52 1098845 11710888 famous_v3ft_1399_200_006690223_4 346,346 50,076 0.1446


©2024 climateprediction.net