climateprediction.net home page
Task 19310546

Task 19310546

Name wah2_sas50_fic7_201412_13_348_010330860_0
Workunit 10330860
Created 23 Feb 2016, 14:48:06 UTC
Sent 29 Feb 2016, 16:05:54 UTC
Report deadline 10 Feb 2017, 21:25:54 UTC
Received 15 Mar 2016, 10:31:58 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 9 (0x00000009) Unknown error code
Computer ID 907613
Run time 2 days 14 hours 19 min 8 sec
CPU time 2 days 13 hours 27 min 14 sec
Validate state Invalid
Credit 7,619.09
Device peak FLOPS 5.11 GFLOPS
Application version Weather At Home 2 (wah2) (region independent) v7.08
windows_intelx86
Stderr
<core_client_version>7.6.22</core_client_version>
<![CDATA[
<message>
 - exit code 9 (0x9)
</message>
<stderr_txt>
03:23:48 (5936): start_timer_thread(): CreateThread() failed, errno 0
03:23:49 (18024): start_timer_thread(): CreateThread() failed, errno 0


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x004632F7 read attempt to address 0x0161E3AC

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.5.0


Dump Timestamp    : 03/03/16 08:01:18
Install Directory : 
Data Directory    : F:\BOINC
Project Symstore  : 
LoadLibraryA( F:\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( F:\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( F:\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( F:\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: F:\BOINC\slots\5;F:\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000150000 F:\BOINC\projects\climateprediction.net\wah2_7.08_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000773b0000 000000000017b000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.10586.20) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075c60000 00000000000e0000 C:\WINDOWS\SYSTEM32\KERNEL32.DLL (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000073f10000 000000000017e000 C:\WINDOWS\SYSTEM32\KERNELBASE.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000077210000 0000000000147000 C:\WINDOWS\SYSTEM32\USER32.dll (6.2.10586.20) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075a30000 000000000014f000 C:\WINDOWS\SYSTEM32\GDI32.dll (6.2.10586.63) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.10586.63 (th2_release.160104-1513)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.63

ModLoad: 0000000074b90000 000000000007b000 C:\WINDOWS\SYSTEM32\ADVAPI32.dll (6.2.10586.63) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074a90000 00000000000be000 C:\WINDOWS\SYSTEM32\msvcrt.dll (7.0.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 7.0.10586.0

ModLoad: 0000000077360000 0000000000044000 C:\WINDOWS\SYSTEM32\sechost.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074090000 00000000000ad000 C:\WINDOWS\SYSTEM32\RPCRT4.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000073ed0000 000000000001e000 C:\WINDOWS\SYSTEM32\SspiCli.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000073ec0000 000000000000a000 C:\WINDOWS\SYSTEM32\CRYPTBASE.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074ce0000 0000000000058000 C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000073df0000 0000000000028000 C:\WINDOWS\SYSTEM32\ntmarta.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntmarta.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 00000000719f0000 0000000000071000 C:\WINDOWS\SYSTEM32\efswrt.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : efswrt.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074920000 000000000008d000 C:\WINDOWS\SYSTEM32\SHCORE.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : shcore.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075430000 00000000001bd000 C:\WINDOWS\SYSTEM32\combase.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : combase.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000075b80000 000000000000c000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : Kernel.Appcore.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

Get Product Name Failed.
ModLoad: 0000000074c40000 0000000000092000 C:\WINDOWS\SYSTEM32\OLEAUT32.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 10.0.10586.0
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 10.0.10586.0

ModLoad: 00000000710a0000 00000000000c8000 C:\WINDOWS\SYSTEM32\wintypes.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : WinTypes.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 00000000719a0000 0000000000049000 C:\WINDOWS\SYSTEM32\edputil.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : edputil.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000074dc0000 0000000000045000 C:\WINDOWS\SYSTEM32\shlwapi.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : shlwapi.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000071c60000 000000000013f000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0

ModLoad: 0000000073eb0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.10586.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.10586.0 (th2_release.151029-1700)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 10.0.10586.0



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 18692, Write: 0, Other 65967

- I/O Transfers Counters -
Read: 0, Write: 3100, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 204960, QuotaPeakPagedPoolUsage: 205536
QuotaNonPagedPoolUsage: 8952, QuotaPeakNonPagedPoolUsage: 9648

- Virtual Memory Usage -
VirtualSize: 112005120, PeakVirtualSize: 113381376

- Pagefile Usage -
PagefileUsage: 3002368, PeakPagefileUsage: 3203072

- Working Set Size -
WorkingSetSize: 80056320, PeakWorkingSetSize: 80216064, PageFaultCount: 20726

*** Dump of thread ID 5480 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3227624.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x004632F7 read attempt to address 0x0161E3AC

- Registers -
eax=00000015 ebx=fffffffe ecx=00000000 edx=00000000 esi=0154e020 edi=00000000
eip=004632f7 esp=009dfe7c ebp=009dfe80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010216

- Callstack -
ChildEBP RetAddr  Args to Child
009dfe80 0044855f 0154e020 00000000 0154e020 fffffffe wah2_7.08_windows_intelx86!+0x0 
009dfe98 0044fd58 0154e020 00000000 0051b6a8 009dfee8 wah2_7.08_windows_intelx86!+0x0 
009dfeac 0048502c 00000015 ab04c301 00000006 00484f9c wah2_7.08_windows_intelx86!+0x0 
009dfee8 7402a9ff 00000006 83b39b90 7402a840 7402a840 wah2_7.08_windows_intelx86!+0x0 
009dff80 75c738f4 80000006 75c738d0 4c871e51 009dffdc KERNELBASE!CtrlRoutine+0x0 
009dff94 774156c3 80000006 01e9daa2 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009dffdc 7741568e ffffffff 7743b6e0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
009dffec 00000000 7402a840 80000006 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 632 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 5625000.000000, User Time: 37500000.000000, Wait Time: 3227621.000000

- Registers -
eax=00000000 ebx=0019efb4 ecx=00000000 edx=00000000 esi=00000000 edi=00002469
eip=77426e3c esp=0019ef74 ebp=0019efd8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0019efd8 73fca4ef 000000c8 00000000 0019effc 00469131 ntdll!NtDelayExecution+0x0 
0019efe8 00469131 000000c8 000000c8 00000000 0019f034 KERNELBASE!Sleep+0x0 
0019effc 00407f43 a0000000 027f9999 004d0c48 00000020 wah2_7.08_windows_intelx86!+0x0 
0019f034 00409508 00000000 ab80c0a9 004855fe 00000000 wah2_7.08_windows_intelx86!+0x0 
0019fd40 004096f7 0019fef0 00409775 004d1368 0019fd60 wah2_7.08_windows_intelx86!+0x0 
0019fd48 00409775 004d1368 0019fd60 006e23e0 0019fd60 wah2_7.08_windows_intelx86!+0x0 
0019fef0 004855ab 00400000 00000000 006e241e 0000000a wah2_7.08_windows_intelx86!+0x0 
0019ff80 75c738f4 003c9000 75c738d0 4c031e51 0019ffdc wah2_7.08_windows_intelx86!+0x0 
0019ff94 774156c3 003c9000 016ddaa2 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 7741568e ffffffff 7743b6e0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
0019ffec 00000000 004855fe 003c9000 00000000 78746341 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 18324 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3227621.000000

- Registers -
eax=c0000022 ebx=0153fd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001
eip=77426e3c esp=0153fcdc ebp=0153fd40
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0153fd40 73fca4ef 00000064 00000000 0153ff80 00478574 ntdll!NtDelayExecution+0x0 
0153fd50 00478574 00000064 00478550 00478550 00000000 KERNELBASE!Sleep+0x0 
0153ff80 75c738f4 00000000 75c738d0 4d491e51 0153ffdc wah2_7.08_windows_intelx86!+0x0 
0153ff94 774156c3 00000000 0027daa2 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0153ffdc 7741568e ffffffff 7743b6e0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
0153ffec 00000000 00478550 00000000 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 

*** Dump of thread ID 17228 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 3227507.000000

- Registers -
eax=00000000 ebx=006e1ee8 ecx=00000000 edx=00000000 esi=006e1d28 edi=006e2bb0
eip=7742868c esp=008dfdc8 ebp=008dff80
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
008dff80 75c738f4 006e2bb0 75c738d0 4c971e51 008dffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 
008dff94 774156c3 006e2bb0 01f9daa2 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
008dffdc 7741568e ffffffff 7743b6e0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 
008dffec 00000000 773f9fa0 006e2bb0 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5936, selfPID=5936, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5936, selfPID=18024, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13824, selfPID=13896, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13824, selfPID=13824, iMonCtr=1
21:20:41 (10636): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
00:13:42 (10280): start_timer_thread(): CreateThread() failed, errno 0
00:13:43 (10940): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
22:01:43 (10684): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12436, selfPID=12436, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12436, selfPID=10728, iMonCtr=1
16:11:30 (12268): start_timer_thread(): CreateThread() failed, errno 0
16:11:31 (12800): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12168, selfPID=3076, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12168, selfPID=12168, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
12:25:45 (12164): start_timer_thread(): CreateThread() failed, errno 0
12:25:46 (12408): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
18:06:11 (12008): start_timer_thread(): CreateThread() failed, errno 0
18:06:13 (13348): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12008, selfPID=12008, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12008, selfPID=13348, iMonCtr=1
00:20:58 (13076): start_timer_thread(): CreateThread() failed, errno 0
00:21:00 (13156): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
18:21:07 (11568): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11568, selfPID=11568, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=11568, selfPID=8896, iMonCtr=1
20:12:49 (13276): start_timer_thread(): CreateThread() failed, errno 0
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13276, selfPID=8080, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=13276, selfPID=13276, iMonCtr=1
18:53:54 (12656): start_timer_thread(): CreateThread() failed, errno 0
18:53:55 (12200): start_timer_thread(): CreateThread() failed, errno 0
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
23:50:23 (11292): start_timer_thread(): CreateThread() failed, errno 0
23:50:24 (11636): start_timer_thread(): CreateThread() failed, errno 0
CPDN Monitor - Quit request from BOINC...
19:04:49 (12752): start_timer_thread(): CreateThread() failed, errno 0
19:04:50 (9672): start_timer_thread(): CreateThread() failed, errno 0
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12752, selfPID=12752, iMonCtr=1
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=12752, selfPID=9672, iMonCtr=1

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
15 Mar 2016 10:37:42 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 115,499 203,738 1.7640
09 Mar 2016 15:27:26 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 103,979 183,365 1.7635
08 Mar 2016 14:02:54 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 92,459 162,935 1.7622
07 Mar 2016 13:59:22 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 80,939 142,997 1.7667
05 Mar 2016 15:23:44 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 69,419 122,687 1.7673
05 Mar 2016 08:16:08 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 57,899 102,143 1.7642
05 Mar 2016 02:02:52 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 46,379 81,433 1.7558
04 Mar 2016 20:11:58 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 34,859 60,861 1.7459
04 Mar 2016 14:30:31 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 23,339 40,556 1.7377
03 Mar 2016 12:36:26 907613 19310546 wah2_sas50_fic7_201412_13_348_010330860_0 11,819 21,001 1.7769


©2024 climateprediction.net