Name | hadam3p_pnw_pll5_2013_1_009975680_0 |
Workunit | 9982038 |
Created | 29 Jun 2015, 17:51:14 UTC |
Sent | 3 Jul 2015, 18:58:18 UTC |
Report deadline | 15 Jun 2016, 0:18:18 UTC |
Received | 15 Jul 2015, 21:08:45 UTC |
Server state | Over |
Outcome | Computation error |
Client state | Compute error |
Exit status | 9 (0x00000009) Unknown error code |
Computer ID | 1367205 |
Run time | 3 days 23 hours 20 min 56 sec |
CPU time | 3 days 18 hours 53 min 47 sec |
Validate state | Invalid |
Credit | 2,759.97 |
Device peak FLOPS | 2.92 GFLOPS |
Application version | UK Met Office HadAM3P-HadRM3P Pacific North West v7.27 windows_intelx86 |
Stderr | <core_client_version>7.4.42</core_client_version> <![CDATA[ <message> The storage control block address is invalid. (0x9) - exit code 9 (0x9) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x00D4E3AC Engaging BOINC Windows Runtime Debugger... ******************** BOINC Windows Runtime Debugger Version 7.5.1 Dump Timestamp : 07/04/15 00:25:23 Install Directory : Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\slots\2;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 0000000000400000 000000000018e000 C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_pnw_7.27_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 0000000077650000 000000000016e000 C:\windows\SYSTEM32\ntdll.dll (6.3.9600.17736) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 0000000074f10000 0000000000140000 C:\windows\SYSTEM32\KERNEL32.DLL (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 0000000075150000 00000000000d7000 C:\windows\SYSTEM32\KERNELBASE.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wkernelbase.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 000000006c110000 00000000000a0000 C:\windows\system32\apphelp.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : apphelp.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 0000000075230000 0000000000153000 C:\windows\SYSTEM32\USER32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wuser32.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 00000000773c0000 000000000007c000 C:\windows\SYSTEM32\ADVAPI32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 0000000076860000 000000000010e000 C:\windows\SYSTEM32\GDI32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wgdi32.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 0000000077090000 00000000000c3000 C:\windows\SYSTEM32\msvcrt.dll (7.0.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.9600.17415 ModLoad: 0000000077370000 0000000000041000 C:\windows\SYSTEM32\sechost.dll (6.3.9600.17734) (-exported- Symbols Loaded) Linked PDB Filename : sechost.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 0000000075050000 00000000000ba000 C:\windows\SYSTEM32\RPCRT4.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wrpcrt4.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 0000000074e50000 000000000001e000 C:\windows\SYSTEM32\SspiCli.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 0000000074e40000 000000000000a000 C:\windows\SYSTEM32\CRYPTBASE.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : cryptbase.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 0000000074de0000 0000000000054000 C:\windows\SYSTEM32\bcryptPrimitives.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : bcryptprimitives.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 0000000072050000 0000000000028000 C:\windows\SYSTEM32\ntmarta.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : ntmarta.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 0000000071f00000 0000000000141000 C:\windows\SYSTEM32\dbghelp.dll (6.3.9600.17787) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.3.9600.17787 (winblue_r10.150331-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17787 ModLoad: 0000000074c90000 0000000000008000 C:\windows\SYSTEM32\version.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 *** Dump of the Process Statistics: *** - I/O Operations Counters - Read: 18510, Write: 0, Other 68431 - I/O Transfers Counters - Read: 0, Write: 4060, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 93296, QuotaPeakPagedPoolUsage: 119104 QuotaNonPagedPoolUsage: 7664, QuotaPeakNonPagedPoolUsage: 7952 - Virtual Memory Usage - VirtualSize: 51601408, PeakVirtualSize: 62443520 - Pagefile Usage - PagefileUsage: 2449408, PeakPagefileUsage: 7319552 - Working Set Size - WorkingSetSize: 2392064, PeakWorkingSetSize: 31051776, PageFaultCount: 9892 *** Dump of thread ID 11724 (state: Waiting): *** - Information - Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 30601784.000000 - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x00D4E3AC - Registers - eax=00000015 ebx=fffffffe ecx=00000000 edx=00000000 esi=00c7e020 edi=00000000 eip=00462b87 esp=026dfe7c ebp=026dfe80 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010216 - Callstack - ChildEBP RetAddr Args to Child 026dfe80 00447d6f 00c7e020 00000000 00c7e020 fffffffe hadam3p_pnw_7.27_windows_intelx!+0x0 026dfe98 0044f568 00c7e020 00000000 005597a4 026dfee8 hadam3p_pnw_7.27_windows_intelx!+0x0 026dfeac 0048374c 00000015 0f5724bb 00000006 004836bc hadam3p_pnw_7.27_windows_intelx!+0x0 026dfee8 751e23c4 00000006 7a3af803 751e2200 751e2200 hadam3p_pnw_7.27_windows_intelx!+0x0 026dff80 74f27c04 80000006 74f27be0 7a1ac460 026dffdc KERNELBASE!CtrlRoutine+0x0 026dff94 776aad1f 80000006 786e105d 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 026dffdc 776aacea ffffffff 77690226 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 026dffec 00000000 751e2200 80000006 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 *** Dump of thread ID 9516 (state: Waiting): *** - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 17812500.000000, User Time: 88281248.000000, Wait Time: 30601772.000000 - Registers - eax=00000000 ebx=0018f9d0 ecx=00000000 edx=00000000 esi=00000000 edi=0018f9d0 eip=7768c7bc esp=0018f990 ebp=0018f9f4 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216 - Callstack - ChildEBP RetAddr Args to Child 0018f9f4 7515104f 000000c8 00000000 0018fa18 004689c1 ntdll!NtDelayExecution+0x0 0018fa04 004689c1 000000c8 000000c8 00000000 0018fa50 KERNELBASE!Sleep+0x0 0018fa18 00403ddc a0000000 027f9999 004cf858 004cfb88 hadam3p_pnw_7.27_windows_intelx!+0x0 0018fa50 00404d76 00000000 0d222713 00484e6e 00000000 hadam3p_pnw_7.27_windows_intelx!+0x0 0018fd40 00404f67 0018fef0 00404fe5 004d02a8 0018fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 0018fd48 00404fe5 004d02a8 0018fd60 002a2800 0018fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 0018fef0 00484e1b 00400000 00000000 002a2845 0000000a hadam3p_pnw_7.27_windows_intelx!+0x0 0018ff80 74f27c04 7ffde000 74f27be0 786fc460 0018ffdc hadam3p_pnw_7.27_windows_intelx!+0x0 0018ff94 776aad1f 7ffde000 7a1b105d 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 0018ffdc 776aacea ffffffff 77690226 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 0018ffec 00000000 00484e6e 7ffde000 00000000 78746341 ntdll!RtlInitializeExceptionChain+0x0 *** Dump of thread ID 7888 (state: Waiting): *** - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 1093750.000000, Wait Time: 30601782.000000 - Registers - eax=c0000022 ebx=00c6fd1c ecx=00000000 edx=00000000 esi=00000000 edi=00c6fd1c eip=7768c7bc esp=00c6fcdc ebp=00c6fd40 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 00c6fd40 7515104f 00000064 00000000 00c6ff80 00477e94 ntdll!NtDelayExecution+0x0 00c6fd50 00477e94 00000064 00477e70 00477e70 00000000 KERNELBASE!Sleep+0x0 00c6ff80 74f27c04 00000000 74f27be0 78b1c460 00c6ffdc hadam3p_pnw_7.27_windows_intelx!+0x0 00c6ff94 776aad1f 00000000 7ac5105d 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 00c6ffdc 776aacea ffffffff 77690226 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 00c6ffec 00000000 00477e70 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 *** Debug Message Dump **** *** Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7104, selfPID=4232, iMonCtr=1 No Process Handle Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7104, selfPID=7104, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5316, selfPID=5316, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5316, selfPID=4368, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3444, selfPID=3444, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3444, selfPID=3840, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2208, selfPID=2348, iMonCtr=1 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2208, selfPID=2208, iMonCtr=1 Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2312, selfPID=2312, iMonCtr=1 Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2312, selfPID=2356, iMonCtr=1 </stderr_txt> ]]> |
Latest Trickles Received | ||||||
---|---|---|---|---|---|---|
Time Sent (UTC) | Host ID | Result ID | Result Name | Timestep | CPU Time (sec) | Average (sec/TS) |
14 Jul 2015 23:49:06 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 127,019 | 318,411 | 2.5068 |
13 Jul 2015 02:17:26 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 115,499 | 292,931 | 2.5362 |
12 Jul 2015 15:49:54 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 103,979 | 259,870 | 2.4993 |
11 Jul 2015 23:17:28 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 92,459 | 229,297 | 2.4800 |
11 Jul 2015 13:29:57 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 80,939 | 198,448 | 2.4518 |
10 Jul 2015 19:57:13 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 69,419 | 171,415 | 2.4693 |
10 Jul 2015 04:16:09 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 57,899 | 142,403 | 2.4595 |
07 Jul 2015 00:02:27 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 46,379 | 111,975 | 2.4143 |
05 Jul 2015 22:34:57 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 34,859 | 86,704 | 2.4873 |
05 Jul 2015 02:49:01 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 23,339 | 62,993 | 2.6990 |
04 Jul 2015 19:35:54 | 1367205 | 18640594 | hadam3p_pnw_pll5_2013_1_009975680_0 | 11,819 | 38,030 | 3.2177 |
©2024 climateprediction.net