climateprediction.net home page
Task 15789827

Task 15789827

Name hadcm3n_z9il_1960_40_008363496_1
Workunit 8514355
Created 19 May 2013, 5:19:44 UTC
Sent 19 May 2013, 5:19:59 UTC
Report deadline 18 Aug 2013, 12:47:10 UTC
Received 27 May 2013, 5:31:23 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1279725
Run time 3 days 21 hours 32 min 5 sec
CPU time 3 days 21 hours 0 min 24 sec
Validate state Invalid
Credit 4,665.60
Device peak FLOPS 3.31 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
windows_intelx86
Stderr
<core_client_version>7.0.64</core_client_version>
<![CDATA[
<message>
The device does not recognize the command.
 (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4964, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 05/26/13 08:22:37
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_z9il_1960_40_008363496;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 77290000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.17725) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74f40000 00110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 750e0000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 751c0000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 752e0000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 75af0000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75bd0000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18009) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18009 (win7sp1_gdr.121121-1431)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18009

ModLoad: 74da0000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 75c70000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75b00000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 756d0000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74cf0000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.17940) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.17940 (win7sp1_gdr.120824-0334)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17940

ModLoad: 74ce0000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 70ca0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74270000 00022000 C:\Windows\system32\GLU32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : glu32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71570000 000e7000 C:\Windows\system32\DDRAW.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : ddraw.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 718e0000 00006000 C:\Windows\system32\DCIMAN32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dciman32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75d10000 0019d000 C:\Windows\syswow64\SETUPAPI.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75eb0000 00027000 C:\Windows\syswow64\CFGMGR32.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

Get Product Name Failed.
ModLoad: 75050000 0008f000 C:\Windows\syswow64\OLEAUT32.dll (6.1.7601.17676) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 6.1.7601.17676
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 6.1.7601.17676

ModLoad: 75400000 0015c000 C:\Windows\syswow64\ole32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75ba0000 00012000 C:\Windows\syswow64\DEVOBJ.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

ModLoad: 71550000 00013000 C:\Windows\system32\dwmapi.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dwmapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75370000 0002a000 C:\Windows\syswow64\imagehlp.dll (6.1.7601.17787) (-exported- Symbols Loaded)
    Linked PDB Filename   : imagehlp.pdb
    File Version          : 6.1.7601.17787 (win7sp1_gdr.120229-1502)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17787

ModLoad: 74ed0000 00060000 C:\Windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 76dc0000 000cc000 C:\Windows\syswow64\MSCTF.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71660000 000eb000 C:\Windows\system32\dbghelp.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 73460000 00009000 C:\Windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 3216 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

- Registers -
eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=750ec41f esp=1132f9c4 ebp=1132fa14
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 KERNELBASE!RaiseException+0x0 
1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 114b4759 1132faec 114b46e1 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 74f533aa 7efde000 1132ffd4 772c9ef2 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 772c9ef2 7efde000 7fcf8620 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
1132ffd4 772c9ec5 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 1bbc4072 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 6288 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=33ccff48 edi=00000000
eip=772afd71 esp=33ccff04 ebp=33ccff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
33ccff6c 750f4498 00000064 00000000 33ccff88 00749924 ntdll!ZwDelayExecution+0x0 
33ccff7c 00749924 00000064 33ccff94 74f533aa 00000000 KERNELBASE!Sleep+0x0 
33ccff88 74f533aa 00000000 33ccffd4 772c9ef2 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
33ccff94 772c9ef2 00000000 5d318620 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
33ccffd4 772c9ec5 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
33ccffec 00000000 00749910 00000000 00000000 341e0000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6172, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 05/26/13 08:22:41
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_z9il_1960_40_008363496;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 77290000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.17725) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74f40000 00110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 750e0000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 751c0000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 752e0000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 75af0000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75bd0000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18009) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18009 (win7sp1_gdr.121121-1431)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18009

ModLoad: 74da0000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 75c70000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75b00000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 756d0000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74cf0000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.17940) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.17940 (win7sp1_gdr.120824-0334)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17940

ModLoad: 74ce0000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 70ca0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74460000 00022000 C:\Windows\system32\GLU32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : glu32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71570000 000e7000 C:\Windows\system32\DDRAW.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : ddraw.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 718e0000 00006000 C:\Windows\system32\DCIMAN32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dciman32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75d10000 0019d000 C:\Windows\syswow64\SETUPAPI.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75eb0000 00027000 C:\Windows\syswow64\CFGMGR32.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

Get Product Name Failed.
ModLoad: 75050000 0008f000 C:\Windows\syswow64\OLEAUT32.dll (6.1.7601.17676) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 6.1.7601.17676
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 6.1.7601.17676

ModLoad: 75400000 0015c000 C:\Windows\syswow64\ole32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75ba0000 00012000 C:\Windows\syswow64\DEVOBJ.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

ModLoad: 71550000 00013000 C:\Windows\system32\dwmapi.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dwmapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75370000 0002a000 C:\Windows\syswow64\imagehlp.dll (6.1.7601.17787) (-exported- Symbols Loaded)
    Linked PDB Filename   : imagehlp.pdb
    File Version          : 6.1.7601.17787 (win7sp1_gdr.120229-1502)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17787

ModLoad: 74ed0000 00060000 C:\Windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 76dc0000 000cc000 C:\Windows\syswow64\MSCTF.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71660000 000eb000 C:\Windows\system32\dbghelp.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 73460000 00009000 C:\Windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 2896 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

- Registers -
eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=750ec41f esp=1132f9c4 ebp=1132fa14
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 KERNELBASE!RaiseException+0x0 
1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 00184759 1132faec 001846e1 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 74f533aa 7efde000 1132ffd4 772c9ef2 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 772c9ef2 7efde000 7fcfb4af 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
1132ffd4 772c9ec5 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 2056 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=33b2ff48 edi=00000000
eip=772afd71 esp=33b2ff04 ebp=33b2ff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
33b2ff6c 750f4498 00000064 00000000 33b2ff88 00749924 ntdll!ZwDelayExecution+0x0 
33b2ff7c 00749924 00000064 33b2ff94 74f533aa 00000000 KERNELBASE!Sleep+0x0 
33b2ff88 74f533aa 00000000 33b2ffd4 772c9ef2 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
33b2ff94 772c9ef2 00000000 5d4fb4af 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
33b2ffd4 772c9ec5 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
33b2ffec 00000000 00749910 00000000 00000000 34040000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6172, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6172, iMonCtr=1
Model crash detected, will try to restart...

Model crashed:                                                                                                                                                                                                                                                                 dataout/stdout_um.txt                                                           


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 05/26/13 08:22:56
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_z9il_1960_40_008363496;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 77290000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.17725) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74f40000 00110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 750e0000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 751c0000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 752e0000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 75af0000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75bd0000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18009) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18009 (win7sp1_gdr.121121-1431)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18009

ModLoad: 74da0000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 75c70000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75b00000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 756d0000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74cf0000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.17940) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.17940 (win7sp1_gdr.120824-0334)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17940

ModLoad: 74ce0000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 720b0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74460000 00022000 C:\Windows\system32\GLU32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : glu32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71570000 000e7000 C:\Windows\system32\DDRAW.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : ddraw.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 718e0000 00006000 C:\Windows\system32\DCIMAN32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dciman32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75d10000 0019d000 C:\Windows\syswow64\SETUPAPI.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75eb0000 00027000 C:\Windows\syswow64\CFGMGR32.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

Get Product Name Failed.
ModLoad: 75050000 0008f000 C:\Windows\syswow64\OLEAUT32.dll (6.1.7601.17676) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 6.1.7601.17676
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 6.1.7601.17676

ModLoad: 75400000 0015c000 C:\Windows\syswow64\ole32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75ba0000 00012000 C:\Windows\syswow64\DEVOBJ.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

ModLoad: 71550000 00013000 C:\Windows\system32\dwmapi.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dwmapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75370000 0002a000 C:\Windows\syswow64\imagehlp.dll (6.1.7601.17787) (-exported- Symbols Loaded)
    Linked PDB Filename   : imagehlp.pdb
    File Version          : 6.1.7601.17787 (win7sp1_gdr.120229-1502)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17787

ModLoad: 74ed0000 00060000 C:\Windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 76dc0000 000cc000 C:\Windows\syswow64\MSCTF.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71660000 000eb000 C:\Windows\system32\dbghelp.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 73460000 00009000 C:\Windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 6868 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

- Registers -
eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=750ec41f esp=1132f9c4 ebp=1132fa14
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 KERNELBASE!RaiseException+0x0 
1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 00184759 1132faec 001846e1 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 74f533aa 7efde000 1132ffd4 772c9ef2 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 772c9ef2 7efde000 7fcf7a47 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
1132ffd4 772c9ec5 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 3760 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=33c1ff48 edi=00000000
eip=772afd71 esp=33c1ff04 ebp=33c1ff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
33c1ff6c 750f4498 00000064 00000000 33c1ff88 00749924 ntdll!ZwDelayExecution+0x0 
33c1ff7c 00749924 00000064 33c1ff94 74f533aa 00000000 KERNELBASE!Sleep+0x0 
33c1ff88 74f533aa 00000000 33c1ffd4 772c9ef2 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
33c1ff94 772c9ef2 00000000 5d3c7a47 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
33c1ffd4 772c9ec5 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
33c1ffec 00000000 00749910 00000000 00000000 34130000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6172, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 05/26/13 08:23:00
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\Program Files (x86)\BOINC\\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_z9il_1960_40_008363496;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 77290000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.17725) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74f40000 00110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 750e0000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18015) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 751c0000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 752e0000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 75af0000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75bd0000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18009) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18009 (win7sp1_gdr.121121-1431)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18009

ModLoad: 74da0000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 75c70000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75b00000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 756d0000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74cf0000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.17940) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.17940 (win7sp1_gdr.120824-0334)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17940

ModLoad: 74ce0000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 720b0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74460000 00022000 C:\Windows\system32\GLU32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : glu32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71570000 000e7000 C:\Windows\system32\DDRAW.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : ddraw.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 718e0000 00006000 C:\Windows\system32\DCIMAN32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dciman32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75d10000 0019d000 C:\Windows\syswow64\SETUPAPI.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75eb0000 00027000 C:\Windows\syswow64\CFGMGR32.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

Get Product Name Failed.
ModLoad: 75050000 0008f000 C:\Windows\syswow64\OLEAUT32.dll (6.1.7601.17676) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 6.1.7601.17676
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 6.1.7601.17676

ModLoad: 75400000 0015c000 C:\Windows\syswow64\ole32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75ba0000 00012000 C:\Windows\syswow64\DEVOBJ.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

ModLoad: 71550000 00013000 C:\Windows\system32\dwmapi.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dwmapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75370000 0002a000 C:\Windows\syswow64\imagehlp.dll (6.1.7601.17787) (-exported- Symbols Loaded)
    Linked PDB Filename   : imagehlp.pdb
    File Version          : 6.1.7601.17787 (win7sp1_gdr.120229-1502)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17787

ModLoad: 74ed0000 00060000 C:\Windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 76dc0000 000cc000 C:\Windows\syswow64\MSCTF.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 71660000 000eb000 C:\Windows\system32\dbghelp.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 73460000 00009000 C:\Windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 4192 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x750EC41F

- Registers -
eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=750ec41f esp=1132f9c4 ebp=1132fa14
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 KERNELBASE!RaiseException+0x0 
1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 114a4759 1132faec 114a46e1 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 74f533aa 7efde000 1132ffd4 772c9ef2 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 772c9ef2 7efde000 7fcf6a7e 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
1132ffd4 772c9ec5 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 2d111dd2 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 6168 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=33e4ff48 edi=00000000
eip=772afd71 esp=33e4ff04 ebp=33e4ff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
33e4ff6c 750f4498 00000064 00000000 33e4ff88 00749924 ntdll!ZwDelayExecution+0x0 
33e4ff7c 00749924 00000064 33e4ff94 74f533aa 00000000 KERNELBASE!Sleep+0x0 
33e4ff88 74f533aa 00000000 33e4ffd4 772c9ef2 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
33e4ff94 772c9ef2 00000000 5d196a7e 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
33e4ffd4 772c9ec5 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
33e4ffec 00000000 00749910 00000000 00000000 34360000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6172, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
23 May 2013 01:15:25 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 388,800 321,949 0.8281
22 May 2013 19:15:11 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 362,880 300,496 0.8281
22 May 2013 13:50:14 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 336,960 279,039 0.8281
22 May 2013 07:12:42 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 311,040 257,626 0.8283
22 May 2013 01:10:38 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 285,120 236,004 0.8277
21 May 2013 19:05:39 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 259,200 214,505 0.8276
21 May 2013 14:05:01 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 233,280 193,071 0.8276
21 May 2013 07:07:24 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 207,360 171,751 0.8283
21 May 2013 01:05:08 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 181,440 150,144 0.8275
20 May 2013 19:05:25 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 155,520 128,731 0.8277
20 May 2013 13:07:20 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 129,600 107,272 0.8277
20 May 2013 07:10:22 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 103,680 85,790 0.8274
20 May 2013 01:13:14 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 77,760 64,339 0.8274
19 May 2013 19:13:18 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 51,840 42,909 0.8277
19 May 2013 13:13:42 1279725 15789827 hadcm3n_z9il_1960_40_008363496_1 25,920 21,508 0.8298


©2024 climateprediction.net