climateprediction.net home page
Task 10960933

Task 10960933

Name hadsm3dhet2_jkry_006589920_7
Workunit 6793293
Created 15 Mar 2010, 11:52:51 UTC
Sent 22 Oct 2010, 13:19:39 UTC
Report deadline 4 Oct 2011, 18:39:39 UTC
Received 5 Nov 2010, 21:47:09 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status -1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID 1076551
Run time 3 days 16 hours 28 min 32 sec
CPU time 5 days 4 hours 21 min 2 sec
Validate state Invalid
Credit 2,977.30
Device peak FLOPS 2.51 GFLOPS
Application version UK Met Office HadSM3 Slab Model v6.07
windows_intelx86
Stderr
<core_client_version>6.10.43</core_client_version>
<![CDATA[
<message>
 - exit code -1073741819 (0xc0000005)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
MainError:	06:29:06 PM	No files match the supplied pattern.
No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00405CC9 write attempt to address 0x000000C0

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.3.8


Dump Timestamp    : 11/06/10 05:15:56
Install Directory : C:\Program Files\BOINC\
Data Directory    : C:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
Loaded Library    : C:\Program Files\BOINC\\dbghelp.dll
Loaded Library    : C:\Program Files\BOINC\\symsrv.dll
Loaded Library    : C:\Program Files\BOINC\\srcsrv.dll
LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net;srv*C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\symbols*http://msdl.microsoft.com/download/symbols;srv*C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\symbols*http://boinc.berkeley.edu/symstore


ModLoad: 00400000 00109000 C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadsm3_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 7c920000 00096000 C:\WINDOWS\system32\ntdll.dll (5.1.2600.5755) (PDB Symbols Loaded)
    Linked PDB Filename   : ntdll.pdb
    File Version          : 5.1.2600.5755 (xpsp_sp3_gdr.090206-1234)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Windows(R) Operating System
    Product Version       : 5.1.2600.5755

ModLoad: 7c800000 0011e000 C:\WINDOWS\system32\kernel32.dll (5.1.2600.5781) (PDB Symbols Loaded)
    Linked PDB Filename   : kernel32.pdb
    File Version          : 5.1.2600.5781 (xpsp_sp3_gdr.090321-1317)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Windows(R) Operating System
    Product Version       : 5.1.2600.5781

ModLoad: 77d10000 00090000 C:\WINDOWS\system32\USER32.dll (5.1.2600.5512) (PDB Symbols Loaded)
    Linked PDB Filename   : user32.pdb
    File Version          : 5.1.2600.5512 (xpsp.080413-2105)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Windows(R) Operating System
    Product Version       : 5.1.2600.5512

ModLoad: 77ef0000 00049000 C:\WINDOWS\system32\GDI32.dll (5.1.2600.5698) (PDB Symbols Loaded)
    Linked PDB Filename   : gdi32.pdb
    File Version          : 5.1.2600.5698 (xpsp_sp3_gdr.081022-1932)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 5.1.2600.5698

ModLoad: 77da0000 000a9000 C:\WINDOWS\system32\ADVAPI32.dll (5.1.2600.5755) (PDB Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 5.1.2600.5755 (xpsp_sp3_gdr.090206-1234)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Windows(R) Operating System
    Product Version       : 5.1.2600.5755

ModLoad: 77e50000 00093000 C:\WINDOWS\system32\RPCRT4.dll (5.1.2600.6022) (PDB Symbols Loaded)
    Linked PDB Filename   : rpcrt4.pdb
    File Version          : 5.1.2600.6022 (xpsp_sp3_gdr.100813-1643)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 5.1.2600.6022

ModLoad: 77fc0000 00011000 C:\WINDOWS\system32\Secur32.dll (5.1.2600.5834) (PDB Symbols Loaded)
    Linked PDB Filename   : secur32.pdb
    File Version          : 5.1.2600.5834 (xpsp_sp3_qfe.090624-1332)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 5.1.2600.5834

ModLoad: 76300000 0001d000 C:\WINDOWS\system32\IMM32.DLL (5.1.2600.5512) (PDB Symbols Loaded)
    Linked PDB Filename   : imm32.pdb
    File Version          : 5.1.2600.5512 (xpsp.080413-2105)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 5.1.2600.5512

ModLoad: 62c20000 00009000 C:\WINDOWS\system32\LPK.DLL (5.1.2600.5512) (PDB Symbols Loaded)
    Linked PDB Filename   : lpk.pdb
    File Version          : 5.1.2600.5512 (xpsp.080413-2105)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 5.1.2600.5512

ModLoad: 73fa0000 0006b000 C:\WINDOWS\system32\USP10.dll (1.420.2600.5969) (PDB Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0420.2600.5969 (xpsp_sp3_gdr.100416-1716)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0420.2600.5969

ModLoad: 03000000 00115000 C:\Program Files\BOINC\dbghelp.dll (6.8.4.0) (PDB Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.8.0004.0 (debuggers(dbg).070515-1751)
    Company Name          : Microsoft Corporation
    Product Name          : Debugging Tools for Windows(R)
    Product Version       : 6.8.0004.0

ModLoad: 77be0000 00058000 C:\WINDOWS\system32\msvcrt.dll (7.0.2600.5512) (PDB Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.2600.5512 (xpsp.080413-2111)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 7.0.2600.5512

ModLoad: 01d00000 00048000 C:\Program Files\BOINC\symsrv.dll (6.8.4.0) (PDB Symbols Loaded)
    Linked PDB Filename   : symsrv.pdb
    File Version          : 6.8.0004.0 (debuggers(dbg).070515-1751)
    Company Name          : Microsoft Corporation
    Product Name          : Debugging Tools for Windows(R)
    Product Version       : 6.8.0004.0

ModLoad: 00df0000 0003b000 C:\Program Files\BOINC\srcsrv.dll (6.8.4.0) (PDB Symbols Loaded)
    Linked PDB Filename   : srcsrv.pdb
    File Version          : 6.8.0004.0 (debuggers(dbg).070515-1751)
    Company Name          : Microsoft Corporation
    Product Name          : Debugging Tools for Windows(R)
    Product Version       : 6.8.0004.0

ModLoad: 77bd0000 00008000 C:\WINDOWS\system32\version.dll (5.1.2600.5512) (PDB Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 5.1.2600.5512 (xpsp.080413-2105)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft? Windows? Operating System
    Product Version       : 5.1.2600.5512



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 8, Write: 0, Other 207

- I/O Transfers Counters -
Read: 0, Write: 475, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 30020, QuotaPeakPagedPoolUsage: 30020
QuotaNonPagedPoolUsage: 1792, QuotaPeakNonPagedPoolUsage: 2600

- Virtual Memory Usage -
VirtualSize: 18472960, PeakVirtualSize: 18472960

- Pagefile Usage -
PagefileUsage: 1036288, PeakPagefileUsage: 1036288

- Working Set Size -
WorkingSetSize: 1933312, PeakWorkingSetSize: 1933312, PageFaultCount: 467

*** Dump of thread ID 484 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 8208011.000000

- Unhandled Exception Record -
Reason: Access Violation (0xc0000005) at address 0x00405CC9 write attempt to address 0x000000C0

- Registers -
eax=00000000 ebx=00405c9f ecx=0000000b edx=003c4580 esi=003c1e90 edi=c0000005
eip=00405cc9 esp=0012f62c ebp=0012f650
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010297

- Callstack -
ChildEBP RetAddr  Args to Child
0012f650 0047f938 0000000b 00000000 00000000 00000000 hadsm3_6.07_windows_intelx86!+0x0 
0012f670 00473894 c0000005 0012f6a4 0047f560 00000001 hadsm3_6.07_windows_intelx86!+0x0 
0012ffc0 7c817077 00632ca8 00000000 7ffd5000 c0000005 hadsm3_6.07_windows_intelx86!+0x0 
0012fff0 00000000 004738d6 00000000 78746341 00000020 kernel32!_BaseProcessStart@4+0x0 

*** Dump of thread ID 3892 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 8208010.000000

- Registers -
eax=00bef890 ebx=00000000 ecx=00000006 edx=00000000 esi=00000000 edi=00beff70
eip=7c92e514 esp=00beff40 ebp=00beff98
cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
00beff3c 7c92d21a 7c8023f1 00000000 00beff70 00000000 ntdll!_KiFastSystemCallRet@0+0x0 FPO: [0,0,0] 
00beff40 7c8023f1 00000000 00beff70 00000000 00000000 ntdll!_NtDelayExecution@8+0x0 FPO: [2,0,0] 
00beff98 7c802455 00000064 00000000 00beffb4 00457694 kernel32!_SleepEx@8+0x0 
00beffa8 00457694 00000064 00beffec 7c80b729 00000000 kernel32!_Sleep@4+0x0 
00beffb4 7c80b729 00000000 00000000 00000000 00000000 hadsm3_6.07_windows_intelx86!+0x0 
00beffec 00000000 00457680 00000000 00000000 00000008 kernel32!_BaseThreadStart@8+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
05 Nov 2010 08:28:44 1076551 10960933 hadsm3dhet2_jkry_006589920_7 64,812 437,785 1.3509
04 Nov 2010 19:12:57 1076551 10960933 hadsm3dhet2_jkry_006589920_7 54,010 423,440 1.3517
04 Nov 2010 09:59:01 1076551 10960933 hadsm3dhet2_jkry_006589920_7 43,208 409,354 1.3534
03 Nov 2010 12:57:11 1076551 10960933 hadsm3dhet2_jkry_006589920_7 32,406 395,063 1.3546
02 Nov 2010 23:52:56 1076551 10960933 hadsm3dhet2_jkry_006589920_7 21,604 381,500 1.3584
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 10,802 367,509 1.3609
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 259,248 352,219 1.3586
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 248,446 336,858 1.3559
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 237,644 321,569 1.3532
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 226,842 306,262 1.3501
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 216,040 292,245 1.3527
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 205,238 276,980 1.3496
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 194,436 261,734 1.3461
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 183,634 246,610 1.3429
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 172,832 231,323 1.3384
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 162,030 217,735 1.3438
02 Nov 2010 00:00:13 1076551 10960933 hadsm3dhet2_jkry_006589920_7 151,228 203,936 1.3485
30 Oct 2010 03:54:03 1076551 10960933 hadsm3dhet2_jkry_006589920_7 140,426 190,329 1.3554
29 Oct 2010 19:09:08 1076551 10960933 hadsm3dhet2_jkry_006589920_7 129,624 176,174 1.3591
29 Oct 2010 11:44:01 1076551 10960933 hadsm3dhet2_jkry_006589920_7 118,822 163,180 1.3733


©2024 climateprediction.net