climateprediction.net home page
Task 16101081

Task 16101081

Name hadcm3n_o5w3_1940_40_008382016_2
Workunit 8532875
Created 30 Nov 2013, 18:28:01 UTC
Sent 30 Nov 2013, 18:28:05 UTC
Report deadline 2 Mar 2014, 1:55:16 UTC
Received 16 Dec 2013, 20:04:28 UTC
Server state Over
Outcome Computation error
Client state Compute error
Exit status 22 (0x00000016) Unknown error code
Computer ID 1303763
Run time 10 days 19 hours 45 min 29 sec
CPU time 10 days 3 hours 52 min 5 sec
Validate state Invalid
Credit 9,331.20
Device peak FLOPS 3.36 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
windows_intelx86
Stderr
<core_client_version>7.2.31</core_client_version>
<![CDATA[
<message>
The device does not recognize the command.
 (0x16) - exit code 22 (0x16)
</message>
<stderr_txt>
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
14:09:51 (6824): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
BUFFOUT: C I/O Error - Return code = 32

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1
Model crash detected, will try to restart...
BUFFOUT: C I/O Error - Return code = 32

Model crashed: WRITDUMP: BAD BUFFOUT OF DATA                                                                                                                                                                                                                                   tmp/pipe_dummy                                                                  2048    
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76A2C41F

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 12/13/13 23:37:29
Install Directory : C:\Program Files (x86)\BOINC\
Data Directory    : D:\BOINC
Project Symstore  : 
Loaded Library    : C:\Program Files (x86)\BOINC\\dbghelp.dll
Loaded Library    : C:\Program Files (x86)\BOINC\\symsrv.dll
Loaded Library    : C:\Program Files (x86)\BOINC\\srcsrv.dll
LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\BOINC\projects\climateprediction.net\hadcm3n_o5w3_1940_40_008382016;D:\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 D:\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 77dd0000 00180000 C:\windows\SysWOW64\ntdll.dll (6.1.7601.22436) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 77810000 00110000 C:\windows\syswow64\kernel32.dll (6.1.7601.18229) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 76a20000 00047000 C:\windows\syswow64\KERNELBASE.dll (6.1.7601.18229) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18015

ModLoad: 771e0000 00100000 C:\windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 77920000 00090000 C:\windows\syswow64\GDI32.dll (6.1.7601.18275) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.18275 (win7sp1_gdr.131002-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18275

ModLoad: 771d0000 0000a000 C:\windows\syswow64\LPK.dll (6.1.7601.18177) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7601.18177 (win7sp1_gdr.130605-1534)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18177

ModLoad: 76980000 0009d000 C:\windows\syswow64\USP10.dll (1.626.7601.18009) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18009 (win7sp1_gdr.121121-1431)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18009

ModLoad: 76cd0000 000ac000 C:\windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 770a0000 000a0000 C:\windows\syswow64\ADVAPI32.dll (6.1.7601.22436) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 77560000 00019000 C:\windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76f50000 000f0000 C:\windows\syswow64\RPCRT4.dll (6.1.7601.18205) (-exported- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 75930000 00060000 C:\windows\syswow64\SspiCli.dll (6.1.7601.18270) (-exported- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.18270 (win7sp1_gdr.130924-1532)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18270

ModLoad: 75920000 0000c000 C:\windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 691f0000 000c8000 C:\windows\system32\OPENGL32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 6ccb0000 00022000 C:\windows\system32\GLU32.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : glu32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 69100000 000e7000 C:\windows\system32\DDRAW.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : ddraw.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 6d8c0000 00006000 C:\windows\system32\DCIMAN32.dll (6.1.7601.18177) (-exported- Symbols Loaded)
    Linked PDB Filename   : dciman32.pdb
    File Version          : 6.1.7601.18177 (win7sp1_gdr.130605-1534)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18177

ModLoad: 76db0000 0019d000 C:\windows\syswow64\SETUPAPI.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : setupapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 77710000 00027000 C:\windows\syswow64\CFGMGR32.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : cfgmgr32.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

Get Product Name Failed.
ModLoad: 77140000 0008f000 C:\windows\syswow64\OLEAUT32.dll (6.1.7601.17676) (-exported- Symbols Loaded)
    Linked PDB Filename   : oleaut32.pdb
    File Version          : 6.1.7601.17676
    Company Name          : Microsoft Corporation
    Product Name          : 
    Product Version       : 6.1.7601.17676

ModLoad: 75990000 0015c000 C:\windows\syswow64\ole32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : ole32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76d90000 00012000 C:\windows\syswow64\DEVOBJ.dll (6.1.7601.17621) (-exported- Symbols Loaded)
    Linked PDB Filename   : devobj.pdb
    File Version          : 6.1.7601.17621 (win7sp1_gdr.110523-2108)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17621

ModLoad: 6f6c0000 00013000 C:\windows\system32\dwmapi.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : dwmapi.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 775e0000 0002b000 C:\windows\syswow64\imagehlp.dll (6.1.7601.18288) (-exported- Symbols Loaded)
    Linked PDB Filename   : imagehlp.pdb
    File Version          : 6.1.7601.18288 (win7sp1_gdr.131018-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18288

ModLoad: 77580000 00060000 C:\windows\system32\IMM32.DLL (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wimm32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 76740000 000cc000 C:\windows\syswow64\MSCTF.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : msctf.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 6c350000 00115000 C:\Program Files (x86)\BOINC\dbghelp.dll (6.8.4.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : dbghelp.pdb
    File Version          : 6.8.0004.0 (debuggers(dbg).070515-1751)
    Company Name          : Microsoft Corporation
    Product Name          : Debugging Tools for Windows(R)
    Product Version       : 6.8.0004.0

ModLoad: 6dcb0000 00048000 C:\Program Files (x86)\BOINC\symsrv.dll (6.8.4.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : symsrv.pdb
    File Version          : 6.8.0004.0 (debuggers(dbg).070515-1751)
    Company Name          : Microsoft Corporation
    Product Name          : Debugging Tools for Windows(R)
    Product Version       : 6.8.0004.0

ModLoad: 6dc30000 0003b000 C:\Program Files (x86)\BOINC\srcsrv.dll (6.8.4.0) (-exported- Symbols Loaded)
    Linked PDB Filename   : srcsrv.pdb
    File Version          : 6.8.0004.0 (debuggers(dbg).070515-1751)
    Company Name          : Microsoft Corporation
    Product Name          : Debugging Tools for Windows(R)
    Product Version       : 6.8.0004.0

ModLoad: 75850000 00009000 C:\windows\system32\version.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : version.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 6716 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76A2C41F

- Registers -
eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=76a2c41f esp=1132f9c4 ebp=1132fa14
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 KERNELBASE!RaiseException+0x0 
1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 114a3335 1132faec 114a32c9 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 7782336a 7efde000 1132ffd4 77e0bf32 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 77e0bf32 7efde000 6469551e 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
1132ffd4 77e0bf05 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 393412e9 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 9416 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00749910 ebx=00000000 ecx=00000000 edx=00000000 esi=33d1ff48 edi=00000000
eip=77defd91 esp=33d1ff04 ebp=33d1ff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
33d1ff6c 76a34498 00000064 00000000 33d1ff88 00749924 ntdll!NtDelayExecution+0x0 
33d1ff7c 00749924 00000064 33d1ff94 7782336a 00000000 KERNELBASE!Sleep+0x0 
33d1ff88 7782336a 00000000 33d1ffd4 77e0bf32 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
33d1ff94 77e0bf32 00000000 468a551e 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 
33d1ffd4 77e0bf05 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
33d1ffec 00000000 00749910 00000000 00000000 34230000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4340, iMonCtr=1
Model crash detected, will try to restart...
Sorry, too many model crashes! :-(
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
14 Dec 2013 00:48:09 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 777,600 903,799 1.1623
13 Dec 2013 15:20:48 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 751,680 873,080 1.1615
13 Dec 2013 06:27:42 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 725,760 842,075 1.1603
12 Dec 2013 15:08:42 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 699,840 811,400 1.1594
11 Dec 2013 01:55:47 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 673,920 780,655 1.1584
10 Dec 2013 17:21:31 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 648,000 749,938 1.1573
10 Dec 2013 09:11:35 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 622,080 719,147 1.1560
10 Dec 2013 00:03:13 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 596,160 688,414 1.1547
09 Dec 2013 14:56:07 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 570,240 656,983 1.1521
09 Dec 2013 05:53:40 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 544,320 625,797 1.1497
08 Dec 2013 19:48:44 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 518,400 595,456 1.1486
08 Dec 2013 11:21:01 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 492,480 564,729 1.1467
08 Dec 2013 02:47:17 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 466,560 533,254 1.1429
07 Dec 2013 17:57:22 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 440,640 502,289 1.1399
07 Dec 2013 08:10:12 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 414,720 470,451 1.1344
06 Dec 2013 22:57:05 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 388,800 438,916 1.1289
06 Dec 2013 13:09:38 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 362,880 408,392 1.1254
06 Dec 2013 03:45:41 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 336,960 378,821 1.1242
05 Dec 2013 19:26:47 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 311,040 349,139 1.1225
05 Dec 2013 11:08:09 1303763 16101081 hadcm3n_o5w3_1940_40_008382016_2 285,120 319,414 1.1203


©2024 cpdn.org