climateprediction.net home page
Task 21177878

Task 21177878

Name wah2_nam50_p7j8_198812_13_735_011558378_0
Workunit 11558378
Created 17 Jul 2018, 11:56:26 UTC
Sent 17 Jul 2018, 18:10:29 UTC
Report deadline 29 Jun 2019, 23:30:29 UTC
Received 24 Jul 2018, 6:58:53 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1385685
Run time 5 days 21 hours 25 min 49 sec
CPU time 5 days 13 hours 26 min 14 sec
Validate state Valid
Credit 9,898.90
Device peak FLOPS 2.20 GFLOPS
Application version Weather At Home 2 (wah2) v8.24
windows_intelx86
Peak working set size 196.61 MB
Peak swap size 169.04 MB
Peak disk usage 98.39 MB
Stderr 
<core_client_version>7.12.1</core_client_version>
<![CDATA[
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x74E43502

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 7.7.0


Dump Timestamp    : 07/22/18 11:44:56
Install Directory : C:\Program Files\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\wah2_nam50_p7j8_198812_13_735_011558378;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 0000000000040000 00000000008f0000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2rm3m2t_um_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : F:\cpdn\cpdn\cpdnboinc\monitor\win_build\WAH2 Release\wah2rm3m2t.pdb

ModLoad: 00000000776f0000 000000000018f000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.17134.165) (-exported- Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 000000006ebc0000 000000000002f000 C:\Program Files\AVAST Software\Avast\aswhookx.dll (18.4.3.28536) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : r:\sources\2018_0423_093815\build\avgast\RU_NTDLL-lH-Y2A!Win32_vs140\bin\Release_Unicode_NTDLL_vs140\Win32\aswhookx.pdb
    File Version          : 18.4.3.28536
    Company Name          : AVAST Software
    Product Name          : Avast
    Product Version       : 18.4.3.28536

ModLoad: 0000000077270000 0000000000098000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074d30000 00000000001e6000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.17134.165) (-exported- Symbols Loaded)
    Linked PDB Filename   : kernelbase.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000766e0000 0000000000175000 C:\WINDOWS\System32\USER32.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 10.0.17134.165 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.165

ModLoad: 0000000074f20000 000000000001b000 C:\WINDOWS\System32\win32u.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : win32u.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000077240000 0000000000022000 C:\WINDOWS\System32\GDI32.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074230000 0000000000167000 C:\WINDOWS\System32\gdi32full.dll (6.2.17134.112) (-exported- Symbols Loaded)
    Linked PDB Filename   : gdi32full.pdb
    File Version          : 10.0.17134.112 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.112

ModLoad: 00000000743a0000 000000000007d000 C:\WINDOWS\System32\msvcp_win.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcp_win.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000749e0000 000000000011e000 C:\WINDOWS\System32\ucrtbase.dll (6.2.17134.165) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : ucrtbase.pdb
    File Version          : 10.0.17134.165 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.165

ModLoad: 0000000077170000 0000000000078000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.17134.1) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000077310000 00000000000bf000 C:\WINDOWS\System32\msvcrt.dll (7.0.17134.1) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.17134.1

ModLoad: 00000000771f0000 0000000000044000 C:\WINDOWS\System32\sechost.dll (6.2.17134.1) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 00000000774d0000 00000000000c7000 C:\WINDOWS\System32\RPCRT4.dll (6.2.17134.112) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000074fa0000 0000000000019000 C:\WINDOWS\System32\imagehlp.dll (6.2.17134.1) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : imagehlp.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 0000000077630000 0000000000026000 C:\WINDOWS\System32\IMM32.DLL (6.2.17134.1) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 000000006fac0000 0000000000182000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.17134.1) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1

ModLoad: 000000006ec50000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.17134.1) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 10.0.17134.1 (WinBuild.160101.0800)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 10.0.17134.1



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 7, Write: 0, Other 186

- I/O Transfers Counters -
Read: 0, Write: 91, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 223592, QuotaPeakPagedPoolUsage: 223704
QuotaNonPagedPoolUsage: 3640, QuotaPeakNonPagedPoolUsage: 3640

- Virtual Memory Usage -
VirtualSize: 1009913856, PeakVirtualSize: 1009913856

- Pagefile Usage -
PagefileUsage: 6922240, PeakPagefileUsage: 23240704

- Working Set Size -
WorkingSetSize: 4407296, PeakWorkingSetSize: 4407296, PageFaultCount: 1229

*** Dump of thread ID 11988 (state: Ready): ***

- Information -
Status: Base Priority: Above Normal, Priority: Highest, , Kernel Time: 156250.000000, User Time: 156250.000000, Wait Time: 48423868.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x74E43502

- Registers -
eax=1231f660 ebx=00b0d000 ecx=00000003 edx=00000000 esi=003d38a0 edi=004bd6ac
eip=74e43502 esp=1231f660 ebp=1231f6bc
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000216

- Callstack -
ChildEBP RetAddr  Args to Child
1231f6bc 00379707 e06d7363 00000001 00000003 1231f6e8 KERNELBASE!RaiseException+0x0 
1231f6f4 003762d1 1231f704 003f0274 003d38a0 003e8584 wah2rm3m2t_um_8.24_windows_inte!+0x0 
1231f714 00046512 01000000 000490b9 13390000 13393b56 wah2rm3m2t_um_8.24_windows_inte!+0x0 
1231fb40 003786ea 00040000 00000000 124f3006 0000000a wah2rm3m2t_um_8.24_windows_inte!+0x0 
1231fbd0 7728a1a4 00b0d000 7728a180 b6c6fd3e 1231fc2c wah2rm3m2t_um_8.24_windows_inte!+0x0 
1231fbe4 7775174e 00b0d000 b66b4e2c 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
1231fc2c 77751722 ffffffff 7779631f 00000000 00000000 ntdll!PssNtCaptureSnapshot+0x0 
1231fc3c 00000000 0037873d 00b0d000 00000000 00000000 ntdll!PssNtCaptureSnapshot+0x0 

*** Dump of thread ID 13952 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 48423864.000000

- Registers -
eax=00000151 ebx=00000000 ecx=3a7bf614 edx=77780750 esi=00000000 edi=00000001
eip=77780750 esp=3a7bf614 ebp=3a7bf67c
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000247

- Callstack -
ChildEBP RetAddr  Args to Child
3a7bf67c 74e43cef 00000064 00000000 3a7bf8bc 002fa6d4 ntdll!KiFastSystemCallRet+0x0 
3a7bf68c 002fa6d4 00000064 002fa6b0 002fa6b0 00000000 KERNELBASE!Sleep+0x0 
3a7bf8bc 7728a1a4 00000000 7728a180 9e8cfe0a 3a7bf918 wah2rm3m2t_um_8.24_windows_inte!+0x0 
3a7bf8d0 7775174e 00000000 9e214b18 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
3a7bf918 77751722 ffffffff 7779631f 00000000 00000000 ntdll!PssNtCaptureSnapshot+0x0 
3a7bf928 00000000 002fa6b0 00000000 00000000 00000000 ntdll!PssNtCaptureSnapshot+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3920, iMonCtr=2
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Signal 11 received: Segment violation
Signal 11 received: Software termination signal from kill 
Signal 11 received: Abnormal termination triggered by abort call
Signal 11 received, exiting...
11:45:09 (12496): called boinc_finish(193)
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Leaving CPDN_ain::Monitor...
18:41:13 (7188): called boinc_finish(0)

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
24 Jul 2018 07:00:20 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 150,059 480,170 3.1999
23 Jul 2018 19:48:16 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 138,539 444,185 3.2062
23 Jul 2018 04:22:39 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 127,019 406,900 3.2035
22 Jul 2018 17:14:50 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 115,499 369,005 3.1949
22 Jul 2018 03:34:50 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 103,979 331,098 3.1843
21 Jul 2018 12:18:17 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 92,459 293,358 3.1728
21 Jul 2018 01:04:35 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 80,939 255,840 3.1609
20 Jul 2018 14:05:38 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 69,419 219,237 3.1582
20 Jul 2018 02:54:42 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 57,899 182,598 3.1537
19 Jul 2018 16:50:40 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 46,379 146,234 3.1530
19 Jul 2018 04:45:45 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 34,859 109,955 3.1543
18 Jul 2018 18:08:11 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 23,339 73,820 3.1629
18 Jul 2018 05:30:59 1385685 21177878 wah2_nam50_p7j8_198812_13_735_011558378_0 11,819 37,620 3.1830


©2024 climateprediction.net