climateprediction.net home page
Task 19954994

Task 19954994

Name wah2_sas50_ksog_199012_13_448_010710884_0
Workunit 10710884
Created 12 Sep 2016, 13:34:43 UTC
Sent 22 Sep 2016, 10:24:53 UTC
Report deadline 4 Sep 2017, 15:44:53 UTC
Received 11 Oct 2016, 2:47:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 0 (0x00000000)
Computer ID 1117652
Run time 16 days 5 hours 25 min 19 sec
CPU time 15 days 6 hours 24 min 50 sec
Validate state Invalid
Credit 9,898.90
Device peak FLOPS 2.26 GFLOPS
Application version Weather At Home 2 (wah2) (region independent) v8.12
windows_intelx86
Peak working set size 259.66 MB
Peak swap size 198.00 MB
Peak disk usage 90.81 MB
Stderr
<core_client_version>7.6.22</core_client_version>
<![CDATA[
<stderr_txt>
20:29:12 (6416): start_timer_thread(): CreateThread() failed, errno 0


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00463327 read attempt to address 0x008D73AC

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 09/22/16 21:24:09
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\slots\4;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.12_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 0000000077250000 000000000017b000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000076a80000 00000000000e0000 C:\WINDOWS\SYSTEM32\KERNEL32.DLL (6.2.10586.589) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074af0000 000000000017e000 C:\WINDOWS\SYSTEM32\KERNELBASE.dll (6.2.10586.589) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075230000 0000000000147000 C:\WINDOWS\SYSTEM32\USER32.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000076b60000 000000000014f000 C:\WINDOWS\SYSTEM32\GDI32.dll (6.2.10586.420) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.10586.420 (th2_release_sec.160527-1834)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.420

ModLoad: 0000000074fb0000 000000000007b000 C:\WINDOWS\SYSTEM32\ADVAPI32.dll (6.2.10586.63) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074050000 00000000000be000 C:\WINDOWS\SYSTEM32\msvcrt.dll (7.0.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.10586.0

ModLoad: 00000000769a0000 0000000000044000 C:\WINDOWS\SYSTEM32\sechost.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074110000 00000000000ad000 C:\WINDOWS\SYSTEM32\RPCRT4.dll (6.2.10586.306) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000073f60000 000000000001e000 C:\WINDOWS\SYSTEM32\SspiCli.dll (6.2.10586.589) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.10586.589 (th2_release.160906-1759)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.589

ModLoad: 0000000073f50000 000000000000a000 C:\WINDOWS\SYSTEM32\CRYPTBASE.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074c70000 0000000000058000 C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll (6.2.10586.589) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.10586.589 (th2_release.160906-1759)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.589

ModLoad: 00000000714f0000 0000000000028000 C:\WINDOWS\SYSTEM32\ntmarta.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntmarta.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 000000006c7f0000 0000000000071000 C:\WINDOWS\SYSTEM32\efswrt.dll (6.2.10586.494) (-exported- Symbols Loaded)
    Linked PDB Filename   : efswrt.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074ce0000 000000000008d000 C:\WINDOWS\SYSTEM32\SHCORE.dll (6.2.10586.589) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000076dc0000 00000000001bd000 C:\WINDOWS\SYSTEM32\combase.dll (6.2.10586.589) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074030000 000000000000c000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

Get Product Name Failed.
ModLoad: 0000000076cb0000 0000000000095000 C:\WINDOWS\SYSTEM32\OLEAUT32.dll (6.2.10586.589) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 10.0.10586.589
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 10.0.10586.589

ModLoad: 000000006b000000 00000000000c8000 C:\WINDOWS\SYSTEM32\wintypes.dll (6.2.10586.103) (-exported- Symbols Loaded)
    Linked PDB Filename   : WinTypes.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 000000006af00000 0000000000049000 C:\WINDOWS\SYSTEM32\edputil.dll (6.2.10586.494) (-exported- Symbols Loaded)
    Linked PDB Filename   : edputil.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 00000000751b0000 0000000000045000 C:\WINDOWS\SYSTEM32\shlwapi.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000071550000 000000000013f000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000072f00000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.10586.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 17751, Write: 0, Other 62693

- I/O Transfers Counters -
Read: 0, Write: 2730, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 198280, QuotaPeakPagedPoolUsage: 198856
QuotaNonPagedPoolUsage: 8544, QuotaPeakNonPagedPoolUsage: 9632

- Virtual Memory Usage -
VirtualSize: 107274240, PeakVirtualSize: 109965312

- Pagefile Usage -
PagefileUsage: 2732032, PeakPagefileUsage: 3227648

- Working Set Size -
WorkingSetSize: 79560704, PeakWorkingSetSize: 79970304, PageFaultCount: 20639

*** Dump of thread ID 8988 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 237408.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00463327 read attempt to address 0x008D73AC

- Registers -
eax=00000015 ebx=fffffffe ecx=00000000 edx=00000000 esi=00807020 edi=00000000
eip=00463327 esp=008ffe7c ebp=008ffe80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010216

- Callstack -
ChildEBP RetAddr  Args to Child
008ffe80 004484ff 00807020 00000000 00807020 fffffffe wah2_8.12_windows_intelx86!+0x0 
008ffe98 0044fd08 00807020 00000000 0051d6a8 008ffee8 wah2_8.12_windows_intelx86!+0x0 
008ffeac 00485b9c 00000015 0099f841 00000006 00485b0c wah2_8.12_windows_intelx86!+0x0 
008ffee8 74c0a8bf 00000006 7411605c 74c0a700 74c0a700 wah2_8.12_windows_intelx86!+0x0 
008fff80 76a938f4 80000006 76a938d0 76660e4d 008fffdc KERNELBASE!CtrlRoutine+0x0 
008fff94 772b5de3 80000006 77e09561 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
008fffdc 772b5dae ffffffff 772db7b0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
008fffec 00000000 74c0a700 80000006 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 5664 (state: Ready): ***

- Information -
Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 38750000.000000, User Time: 79843752.000000, Wait Time: 237362.000000

- Registers -
eax=00000000 ebx=0019efb4 ecx=00000000 edx=00000000 esi=00000000 edi=00000148
eip=772c6f1c esp=0019ef74 ebp=0019efd8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0019efd8 74baa6ff 000000c8 00000000 0019effc 004691c1 ntdll!ZwDelayExecution+0x0 
0019efe8 004691c1 000000c8 000000c8 00000000 0019f034 KERNELBASE!Sleep+0x0 
0019effc 00407f7d a0000000 027f9999 004d0c48 00000020 wah2_8.12_windows_intelx86!+0x0 
0019f034 00409542 00000000 000ffbe9 0048616e 00000000 wah2_8.12_windows_intelx86!+0x0 
0019fd40 00409731 0019fef0 004097af 004d1368 0019fd60 wah2_8.12_windows_intelx86!+0x0 
0019fd48 004097af 004d1368 0019fd60 006430a0 0019fd60 wah2_8.12_windows_intelx86!+0x0 
0019fef0 0048611b 00400000 00000000 006430de 0000000a wah2_8.12_windows_intelx86!+0x0 
0019ff80 76a938f4 00353000 76a938d0 76f00e4d 0019ffdc wah2_8.12_windows_intelx86!+0x0 
0019ff94 772b5de3 00353000 77769561 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 772b5dae ffffffff 772db7b0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
0019ffec 00000000 0048616e 00353000 00000000 78746341 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 5672 (state: Ready): ***

- Information -
Status: Base Priority: Unknown, Priority: Unknown, , Kernel Time: 312500.000000, User Time: 312500.000000, Wait Time: 237409.000000

- Registers -
eax=c0000022 ebx=011efd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=772c6f1c esp=011efcdc ebp=011efd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
011efd40 74baa6ff 00000064 00000000 011eff80 004790c4 ntdll!ZwDelayExecution+0x0 
011efd50 004790c4 00000064 004790a0 004790a0 00000000 KERNELBASE!Sleep+0x0 
011eff80 76a938f4 00000000 76a938d0 77f70e4d 011effdc wah2_8.12_windows_intelx86!+0x0 
011eff94 772b5de3 00000000 76719561 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
011effdc 772b5dae ffffffff 772db7b0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
011effec 00000000 004790a0 00000000 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6416, selfPID=6416, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=6416, selfPID=6516, iMonCtr=1
22:21:32 (4780): start_timer_thread(): CreateThread() failed, errno 0
22:21:34 (3168): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
18:38:10 (2880): start_timer_thread(): CreateThread() failed, errno 0
18:38:12 (4448): start_timer_thread(): CreateThread() failed, errno 0
17:41:34 (2316): Can't acquire lockfile (32) - waiting 35s
17:42:09 (2316): Can't acquire lockfile (32) - exiting
17:42:09 (2316): Error: The process cannot access the file because it is being used by another process.

 (0x20)
20:17:08 (7460): Can't acquire lockfile (32) - waiting 35s
20:17:43 (7460): Can't acquire lockfile (32) - exiting
20:17:43 (7460): Error: The process cannot access the file because it is being used by another process.

 (0x20)
20:30:56 (11276): Can't acquire lockfile (32) - waiting 35s
20:31:31 (11276): Can't acquire lockfile (32) - exiting
20:31:31 (11276): Error: The process cannot access the file because it is being used by another process.

 (0x20)
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2880, selfPID=2880, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2880, selfPID=4448, iMonCtr=1
20:51:09 (7096): start_timer_thread(): CreateThread() failed, errno 0
20:51:10 (4856): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
21:22:07 (2292): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
21:35:35 (2228): start_timer_thread(): CreateThread() failed, errno 0
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
CPDN Monitor - Quit request from BOINC...
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/wah2_se_8.12_windows_intelx86.dll after 11 attempts
21:56:02 (9072): start_timer_thread(): CreateThread() failed, errno 0
21:56:04 (8716): start_timer_thread(): CreateThread() failed, errno 0
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
CPDN Monitor - Quit request from BOINC...
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/wah2_se_8.12_windows_intelx86.dll after 11 attempts
23:12:01 (11132): start_timer_thread(): CreateThread() failed, errno 0
23:12:03 (11284): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/wah2_se_8.12_windows_intelx86.dll after 11 attempts
14:00:21 (10392): start_timer_thread(): CreateThread() failed, errno 0
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
CPDN Monitor - Quit request from BOINC...
Unable to load library wah2_se_8.12_windows_intelx86.dll
LoadLibrary error: 126
cpdnmonitor: cannot open input file C:\ProgramData\BOINC/projects/climateprediction.net/wah2_se_8.12_windows_intelx86.dll after 11 attempts
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Leaving CPDN_Main::Monitor...
12:59:47 (3048): called boinc_finish(0)

</stderr_txt>
<message>
upload failure: <file_xfer_error>
  <file_name>wah2_sas50_ksog_199012_13_448_010710884_0_r848718290_3.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>wah2_sas50_ksog_199012_13_448_010710884_0_r848718290_4.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>
<file_xfer_error>
  <file_name>wah2_sas50_ksog_199012_13_448_010710884_0_r848718290_5.zip</file_name>
  <error_code>-161 (not found)</error_code>
</file_xfer_error>

</message>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
11 Oct 2016 01:47:56 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 150,059 1,318,392 8.7858
08 Oct 2016 21:45:07 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 138,539 1,221,098 8.8141
07 Oct 2016 15:28:51 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 127,019 1,122,416 8.8366
06 Oct 2016 08:44:26 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 115,499 1,021,444 8.8437
05 Oct 2016 02:30:16 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 103,979 918,453 8.8331
03 Oct 2016 20:41:17 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 92,459 818,210 8.8494
02 Oct 2016 14:44:59 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 80,939 719,333 8.8873
01 Oct 2016 04:32:43 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 69,419 617,925 8.9014
29 Sep 2016 10:59:27 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 57,899 511,908 8.8414
28 Sep 2016 03:29:09 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 46,379 411,584 8.8744
26 Sep 2016 16:38:00 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 34,859 306,185 8.7835
25 Sep 2016 05:46:50 1117652 19954994 wah2_sas50_ksog_199012_13_448_010710884_0 23,339 207,043 8.8711


©2024 climateprediction.net