climateprediction.net home page
Task 12519182

Task 12519182

Name famous_wr71_1899_200_007122004_2
Workunit 7320364
Created 25 Jan 2011, 12:29:03 UTC
Sent 25 Jan 2011, 12:29:10 UTC
Report deadline 26 Apr 2011, 19:56:21 UTC
Received 5 Feb 2011, 19:43:42 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 942621
Run time 23 hours 55 min 53 sec
CPU time 21 hours 9 min 28 sec
Validate state Invalid
Credit 1,482.40
Device peak FLOPS 3.44 GFLOPS
Application version UK Met Office FAMOUS v6.11
i686-apple-darwin
Stderr 
<core_client_version>6.6.36</core_client_version>
<![CDATA[
<message>
process exited with code 22 (0x16, -234)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
 (17656): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
 (17656): No heartbeat from core client for 30 sec - exiting
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Tue Jan 25 16:37:32 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc47c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc4b8 esp: 0xbfffc47c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17699, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Tue Jan 25 17:08:02 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd50c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd548 esp: 0xbfffd50c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=17980, iMonCtr=1
Model crash detected, will try to restart...
 (17980): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
OPEN:  File Creation Failed: No space left on device
OPEN:  Unable to Open File dataout/wr71la#da0000019075g+ for Read/Write

Model crashed: DUMPCTL : Fail to open output dump - may already exist                                                                                                                                                                                                          tmp/pipe_dummy                                                                  
no start tag in app init data
 (18498): Can't parse init data file - running in standalone mode
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
 (19684): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Tue Jan 25 21:11:12 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc47c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc4b8 esp: 0xbfffc47c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20834, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 60 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 61 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 68 - Return code = 1

BUFFIN: Read Failed: No such file or directory
BUFFIN: C I/O Error feof - Unit 69 - Return code = 1
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Wed Jan 26 10:39:14 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd50c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd548 esp: 0xbfffd50c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=36396, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Wed Jan 26 10:39:17 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc47c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc4b8 esp: 0xbfffc47c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=36396, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Wed Jan 26 10:59:53 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd50c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd548 esp: 0xbfffd50c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=36713, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Wed Jan 26 15:11:46 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x003f5826    7  0x00450bd8    8  0x004698c0    9  0x000026b6   10  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd50c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd548 esp: 0xbfffd50c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=41095, iMonCtr=1
Model crash detected, will try to restart...
SIGSEGV: segmentation violation

Crashed executable name: famous_um_6.11_i686-apple-darwin
built using BOINC library version 6.11.1
Machine type Intel 80486 (32-bit executable)
System version: Macintosh OS 10.6.6 build 10J567
Wed Jan 26 15:11:48 2011

sh: /usr/bin/atos: No such file or directory
  0  0x0048fdcf    1  0x0049054a    2  0x900ae46b    3  0xffffffff    4  0x013596b3    5  0x0135844f    6  0x002eeac6    7  0x003f55ea    8  0x00452abd    9  0x004698c0   10  0x000026b6   11  0x000025dd  
Thread 0 crashed with X86 Thread State (32-bit):
  eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc47c edx: 0x900480fa
  edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc4b8 esp: 0xbfffc47c
   ss: 0x0000001f efl: 0x00000206 eip: 0x900480fa  cs: 0x00000007
   ds: 0x0000001f  es: 0x0000001f  fs: 0x00000000  gs: 0x00000037

Binary Images Description:
    0x1000 -   0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin
 0x12b4000 -  0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libsvml.dylib
 0x1327000 -  0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libifcoremt.dylib
 0x1407000 -  0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libimf.dylib
 0x1674000 -  0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_wr71_1899_200_007122004/lib/libirc.dylib
0x90047000 - 0x901eefff /usr/lib/libSystem.B.dylib
0x92764000 - 0x92767fff /usr/lib/system/libmathCommon.A.dylib
0x92a01000 - 0x92a0ffff /usr/lib/libz.1.dylib


Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=41095, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  
Sorry, too many model crashes! :-(
 (4532): called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
05 Feb 2011 19:22:13 942621 12519182 famous_wr71_1899_200_007122004_2 449,306 74,758 0.1664
05 Feb 2011 19:05:00 942621 12519182 famous_wr71_1899_200_007122004_2 439,946 73,079 0.1661
05 Feb 2011 18:58:22 942621 12519182 famous_wr71_1899_200_007122004_2 430,586 71,380 0.1658
05 Feb 2011 18:55:43 942621 12519182 famous_wr71_1899_200_007122004_2 421,226 69,689 0.1654
05 Feb 2011 18:51:02 942621 12519182 famous_wr71_1899_200_007122004_2 411,866 67,999 0.1651
05 Feb 2011 16:54:32 942621 12519182 famous_wr71_1899_200_007122004_2 402,506 66,307 0.1647
05 Feb 2011 16:06:52 942621 12519182 famous_wr71_1899_200_007122004_2 393,146 64,633 0.1644
05 Feb 2011 15:37:45 942621 12519182 famous_wr71_1899_200_007122004_2 383,786 62,937 0.1640
05 Feb 2011 15:11:19 942621 12519182 famous_wr71_1899_200_007122004_2 374,426 61,236 0.1635
05 Feb 2011 14:45:40 942621 12519182 famous_wr71_1899_200_007122004_2 365,066 59,527 0.1631
05 Feb 2011 14:08:39 942621 12519182 famous_wr71_1899_200_007122004_2 355,706 57,794 0.1625
26 Jan 2011 12:32:51 942621 12519182 famous_wr71_1899_200_007122004_2 346,346 55,940 0.1615
26 Jan 2011 08:54:52 942621 12519182 famous_wr71_1899_200_007122004_2 336,986 54,147 0.1607
26 Jan 2011 08:22:46 942621 12519182 famous_wr71_1899_200_007122004_2 327,626 52,474 0.1602
26 Jan 2011 07:57:50 942621 12519182 famous_wr71_1899_200_007122004_2 318,266 50,787 0.1596
26 Jan 2011 07:25:08 942621 12519182 famous_wr71_1899_200_007122004_2 308,906 49,108 0.1590
26 Jan 2011 07:00:51 942621 12519182 famous_wr71_1899_200_007122004_2 299,546 47,420 0.1583
26 Jan 2011 06:25:30 942621 12519182 famous_wr71_1899_200_007122004_2 290,186 45,710 0.1575
26 Jan 2011 05:57:02 942621 12519182 famous_wr71_1899_200_007122004_2 280,826 43,947 0.1565
26 Jan 2011 05:22:37 942621 12519182 famous_wr71_1899_200_007122004_2 271,466 42,198 0.1554


©2024 cpdn.org