climateprediction.net home page
Task 21820679

Task 21820679

Name wah2_pnw25_a1oq_199412_13_854_011945501_0
Workunit 11945501
Created 2 Dec 2019, 12:26:59 UTC
Sent 2 Dec 2019, 14:24:50 UTC
Report deadline 13 Nov 2020, 19:44:50 UTC
Received 7 Dec 2019, 5:12:14 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 194 (0x000000C2) EXIT_ABORTED_BY_CLIENT
Computer ID 1495239
Run time 2 days 9 hours 41 min 6 sec
CPU time 2 days 4 hours 18 min 39 sec
Validate state Invalid
Credit 9,138.96
Device peak FLOPS 4.15 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 187.78 MB
Peak swap size 166.10 MB
Peak disk usage 21.45 MB
Stderr 
<core_client_version>7.14.2</core_client_version>
<![CDATA[
<message>
finish file present too long</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1276, selfPID=15928, iMonCtr=1
Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1276, selfPID=1276, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=9524, selfPID=19480, iMonCtr=1
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
22:28:10 (13008): start_timer_thread(): CreateThread() failed, errno 0
22:28:12 (14480): start_timer_thread(): CreateThread() failed, errno 0
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14480, selfPID=8968, iMonCtr=1
Model crash detected, will try to restart...
Leaving CPDN_ain::Monitor...
22:44:33 (8968): called boinc_finish(0)
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Abort request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0268312C

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 12/07/19 05:11:52
Install Directory : 
Data Directory    : F:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( F:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( F:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( F:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( F:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: F:\ProgramData\BOINC\projects\climateprediction.net;F:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000400000 0000000000152000 F:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 00000000770f0000 000000000019a000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.476

ModLoad: 0000000076990000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.18362.329) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.476

ModLoad: 0000000076c60000 00000000001fc000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.18362.476) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.476

ModLoad: 0000000076370000 0000000000197000 C:\WINDOWS\System32\USER32.dll (6.2.18362.476) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.476

ModLoad: 0000000076910000 0000000000017000 C:\WINDOWS\System32\win32u.dll (6.2.18362.476) (-exported- Symbols Loaded)
    Linked PDB Filename   : wwin32u.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.476

ModLoad: 0000000075090000 0000000000021000 C:\WINDOWS\System32\GDI32.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 00000000767b0000 000000000015a000 C:\WINDOWS\System32\gdi32full.dll (6.2.18362.476) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32full.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.476

ModLoad: 0000000075160000 000000000007c000 C:\WINDOWS\System32\msvcp_win.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 0000000075830000 000000000011f000 C:\WINDOWS\System32\ucrtbase.dll (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 0000000076b20000 0000000000079000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.18362.329) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.476

ModLoad: 0000000076ba0000 00000000000bf000 C:\WINDOWS\System32\msvcrt.dll (7.0.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.18362.1

ModLoad: 00000000748e0000 0000000000076000 C:\WINDOWS\System32\sechost.dll (6.2.18362.267) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000075a50000 00000000000bb000 C:\WINDOWS\System32\RPCRT4.dll (6.2.18362.476) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 10.0.18362.476 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Betriebssystem Microsoft&#174; Windows&#174;
    Product Version       : 10.0.18362.476

ModLoad: 00000000748c0000 0000000000020000 C:\WINDOWS\System32\SspiCli.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 00000000748b0000 000000000000a000 C:\WINDOWS\System32\CRYPTBASE.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000075c70000 000000000005f000 C:\WINDOWS\System32\bcryptPrimitives.dll (6.2.18362.295) (-exported- Symbols Loaded)
    Linked PDB Filename   : bcryptprimitives.pdb
    File Version          : 10.0.18362.295 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.295

ModLoad: 0000000075b30000 0000000000025000 C:\WINDOWS\System32\IMM32.DLL (6.2.18362.387) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 10.0.18362.387 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.387

ModLoad: 0000000065870000 000000000018f000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1

ModLoad: 0000000072c50000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.18362.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.18362.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.18362.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 7214, Write: 0, Other 43606

- I/O Transfers Counters -
Read: 0, Write: 1712, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 92824, QuotaPeakPagedPoolUsage: 92984
QuotaNonPagedPoolUsage: 9032, QuotaPeakNonPagedPoolUsage: 9320

- Virtual Memory Usage -
VirtualSize: 55959552, PeakVirtualSize: 55959552

- Pagefile Usage -
PagefileUsage: 3702784, PeakPagefileUsage: 3702784

- Working Set Size -
WorkingSetSize: 6262784, PeakWorkingSetSize: 6266880, PageFaultCount: 1867

*** Dump of thread ID 10688 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 8281250.000000, User Time: 15625000.000000, Wait Time: 2336179.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x0048EE46 write attempt to address 0x0268312C

- Registers -
eax=00000065 ebx=004b63b9 ecx=0019dc80 edx=0268312c esi=0019da38 edi=0019dcb0
eip=0048ee46 esp=0019d9cc ebp=0019dc5c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202

- Callstack -
ChildEBP RetAddr  Args to Child
0019dc5c 0048161c 0019dc80 004b63b8 00000000 0019dcb0 wah2_8.24_windows_intelx86!+0x0 
0019dca0 0044fa52 0268312c 004b63b8 004b6484 00000421 wah2_8.24_windows_intelx86!+0x0 
0019dcc0 00496e90 0000000b 00000000 00000000 00000000 wah2_8.24_windows_intelx86!+0x0 
0019dcd8 00485ca1 c0000005 0019dd0c 00490c54 00000000 wah2_8.24_windows_intelx86!+0x0 
0019ff70 769a6359 00218000 769a6340 0019ffdc 77157b74 wah2_8.24_windows_intelx86!+0x0 
0019ff80 77157b74 00218000 6080c476 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0019ffdc 77157b44 ffffffff 77178f28 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0019ffec 00000000 00485cce 00218000 00000000 78746341 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 14468 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2334517.000000

- Registers -
eax=00000000 ebx=005c6fc0 ecx=00000000 edx=00000000 esi=005c6fc0 edi=005c5aa8
eip=77163a4c esp=0088fdb8 ebp=0088ff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
0088ff70 769a6359 005c5aa8 769a6340 0088ffdc 77157b74 ntdll!NtWaitForWorkViaWorkerFactory+0x0 
0088ff80 77157b74 005c5aa8 6011c476 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
0088ffdc 77157b44 ffffffff 77178f28 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
0088ffec 00000000 771463b0 005c5aa8 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 11536 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2334517.000000

- Registers -
eax=00000000 ebx=005c7a48 ecx=00000000 edx=00000000 esi=005c7a48 edi=005c5aa8
eip=77163a4c esp=009cfdb8 ebp=009cff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
009cff70 769a6359 005c5aa8 769a6340 009cffdc 77157b74 ntdll!NtWaitForWorkViaWorkerFactory+0x0 
009cff80 77157b74 005c5aa8 6005c476 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
009cffdc 77157b44 ffffffff 77178f28 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
009cffec 00000000 771463b0 005c5aa8 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 13208 (state: Waiting): ***

- Information -
Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2334517.000000

- Registers -
eax=00000000 ebx=005cad60 ecx=00000000 edx=00000000 esi=005cad60 edi=005c5aa8
eip=77163a4c esp=00b0fdb8 ebp=00b0ff70
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
00b0ff70 769a6359 005c5aa8 769a6340 00b0ffdc 77157b74 ntdll!NtWaitForWorkViaWorkerFactory+0x0 
00b0ff80 77157b74 005c5aa8 6029c476 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
00b0ffdc 77157b44 ffffffff 77178f28 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
00b0ffec 00000000 771463b0 005c5aa8 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 

*** Dump of thread ID 14384 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 2336176.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=025bfd0c
eip=771620bc esp=025bfccc ebp=025bfd30
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
025bfd30 76d6e82f 00000064 00000000 025bff70 00478b44 ntdll!NtDelayExecution+0x0 
025bfd40 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!Sleep+0x0 
025bff70 769a6359 00000000 769a6340 025bffdc 77157b74 wah2_8.24_windows_intelx86!+0x0 
025bff80 77157b74 00000000 62c2c476 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
025bffdc 77157b44 ffffffff 77178f28 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 
025bffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlGetAppContainerNamedObjectPath+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
05 Dec 2019 21:45:53 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 138,539 175,248 1.2650
05 Dec 2019 17:14:53 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 127,019 160,676 1.2650
05 Dec 2019 12:53:16 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 115,499 146,359 1.2672
05 Dec 2019 08:27:24 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 103,979 132,075 1.2702
04 Dec 2019 17:42:32 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 92,459 117,417 1.2699
04 Dec 2019 13:25:28 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 80,939 103,114 1.2740
04 Dec 2019 08:54:25 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 69,419 88,687 1.2776
03 Dec 2019 20:20:30 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 57,899 73,939 1.2770
03 Dec 2019 15:50:00 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 46,379 58,938 1.2708
03 Dec 2019 11:45:44 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 34,859 44,339 1.2720
03 Dec 2019 07:36:50 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 23,339 29,831 1.2782
02 Dec 2019 19:42:29 1495239 21820679 wah2_pnw25_a1oq_199412_13_854_011945501_0 11,819 15,188 1.2850


©2024 climateprediction.net