climateprediction.net home page
Task 16194419

Task 16194419

Name hadcm3n_801z_1980_40_008458010_2
Workunit 8608866
Created 1 Jan 2014, 18:12:37 UTC
Sent 1 Jan 2014, 18:12:55 UTC
Report deadline 3 Apr 2014, 1:40:06 UTC
Received 18 Jan 2014, 13:14:23 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1259513
Run time 15 days 8 hours 54 min 10 sec
CPU time 14 days 3 hours 47 min 40 sec
Validate state Valid
Credit 12,441.60
Device peak FLOPS 3.07 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
windows_intelx86
Stderr 
<core_client_version>7.0.28</core_client_version>
<![CDATA[
<stderr_txt>

Model crashed: ATM_DYN : INVALID THETA DETECTED.                                                                                                                                                                                                                               tmp/pipe_dummy                                                                  2048    
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
BUFFIN: C I/O Error feof - Unit 63 - Return code = 16
BUFFIN: C I/O Error feof - Unit 64 - Return code = 16
BUFFIN: C I/O Error feof - Unit 65 - Return code = 16
BUFFIN: C I/O Error feof - Unit 66 - Return code = 16
BUFFIN: C I/O Error feof - Unit 67 - Return code = 16
BUFFIN: C I/O Error feof - Unit 68 - Return code = 16
BUFFIN: C I/O Error feof - Unit 69 - Return code = 16
Error converting file to netcdf: dataout/801zko.pji2c10
Error converting file to netcdf: dataout/801zko.pii2c10
Error converting file to netcdf: dataout/801zko.pfi2c10
Error converting file to netcdf: dataout/801zka.phi2c10
Error converting file to netcdf: dataout/801zka.pgi2c10
Error converting file to netcdf: dataout/801zka.pei2c10
Error converting file to netcdf: dataout/801zka.pdi2c10
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2784, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2784, iMonCtr=1
Model crash detected, will try to restart...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2784, iMonCtr=1
Model crash detected, will try to restart...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 01/09/14 15:17:41
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_801z_1980_40_008458010;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 135126, Write: 0, Other 295228

- I/O Transfers Counters -
Read: 0, Write: 9502, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5664, QuotaPeakNonPagedPoolUsage: 5712

- Virtual Memory Usage -
VirtualSize: 1051312128, PeakVirtualSize: 1051312128

- Pagefile Usage -
PagefileUsage: 231759872, PeakPagefileUsage: 231759872

- Working Set Size -
WorkingSetSize: 226799616, PeakWorkingSetSize: 226799616, PageFaultCount: 63475

*** Dump of thread ID 3068 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 159375008.000000, User Time: 104857657344.000000, Wait Time: 696556.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40e51724 40e65a4a 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001324ef 1132fb24 00132451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 2852 (state: Waiting): ***

- Information -
Status: Wait Reason: ExecutionDelay, , Kernel Time: 156250.000000, User Time: 1093750.000000, Wait Time: 696552.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2220, iMonCtr=1
Model crash detected, will try to restart...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 01/09/14 15:30:34
Install Directory : 
Data Directory    : D:\Documents and Settings\All Users\Application Data\BOINC
Project Symstore  : 
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( D:\Documents and Settings\All Users\Application Data\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\hadcm3n_801z_1980_40_008458010;D:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net


SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 

SymGetModuleInfo(): GetLastError = 87
ModLoad: 00000000 00000000  ( Symbols Loaded)
    Linked PDB Filename   : 



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 10737, Write: 0, Other 28117

- I/O Transfers Counters -
Read: 0, Write: 949, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 124432, QuotaPeakPagedPoolUsage: 124432
QuotaNonPagedPoolUsage: 5584, QuotaPeakNonPagedPoolUsage: 5632

- Virtual Memory Usage -
VirtualSize: 1051308032, PeakVirtualSize: 1051308032

- Pagefile Usage -
PagefileUsage: 231628800, PeakPagefileUsage: 231628800

- Working Set Size -
WorkingSetSize: 226643968, PeakWorkingSetSize: 226643968, PageFaultCount: 55766

*** Dump of thread ID 1892 (state: Waiting): ***

- Information -
Status: Wait Reason: UserRequest, , Kernel Time: 12968750.000000, User Time: 7598125056.000000, Wait Time: 746044.000000

- Unhandled Exception Record -
Reason: Single Instruction Executed (0x80000004) at address 0x006C6D27

- Registers -
eax=00000007 ebx=0f270908 ecx=00000000 edx=10473da4 esi=0f52baa0 edi=0f253c20
eip=006c6d27 esp=0f104900 ebp=0f2708f8
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
0f2708f8 0064a92b 0f251ae0 00000488 0f2e2dcc 0064a92b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006c6d27'
0f2e2db8 00455431 0f52b618 00000122 0f525b78 0f4e2690 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0064a92b'
0f4b915c 0044f0de 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00455431'
0f6d5e78 0065501d 40e6edba 40e724f2 100cddf8 0065501d !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0044f0de'
100cdde8 005df71b 00000002 00000002 10332864 005df71b !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '0065501d'
10332858 005da7e1 10332d44 3de20020 005da7e1 1132f8dc !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005df71b'
10332d84 005dc121 1132f8dc 1132f8e0 1132f8e4 1132f8e8 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005da7e1'
10333188 007061a7 1031e8d0 10317b50 1132f7d8 007061a7 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '005dc121'
1132f7c8 006f3667 1132f8d8 1132f8fc 1132fab8 006f3667 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '007061a7'
1132faa8 004083f3 04600000 01000000 00000001 004083f3 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '006f3667'
1132fb10 00408130 1132fb24 001324ef 1132fb24 00132451 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '004083f3'
1132ffc0 7d4e7c9a 00000000 00000000 7efde000 80000004 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00408130'
1132fff0 00000000 007bde7f 00000000 00000000 00000000 !BaseProcessInitPostImport+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4e7c9a'

*** Dump of thread ID 1464 (state: Ready): ***

- Information -
Status: Base Priority: Unknown, Priority: Unknown, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 746001.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=00000000 edi=3404ff74
eip=7d61ccc6 esp=3404ff38 ebp=3404ff9c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000202

- Callstack -
ChildEBP RetAddr  Args to Child
3404ff9c 7d4d14ef 00000064 00000000 3404ffb8 00749924 !NtDelayExecution+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d61ccc6'
3404ffac 00749924 00000064 3404ffec 7d4dfb97 00000000 !Sleep+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4d14ef'
3404ffb8 7d4dfb97 00000000 00000000 00000000 00000000 !+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '00749924'
3404ffec 00000000 00749910 00000000 00000000 34560000 !FlsSetValue+0x0   SymGetModuleInfo(): GetLastError = '87' Address = '7d4dfb97'


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2220, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2276, iMonCtr=1
Model crash detected, will try to restart...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
18 Jan 2014 13:18:27 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 1,036,800 1,222,797 1.1794
18 Jan 2014 03:40:55 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 1,010,880 1,191,860 1.1790
17 Jan 2014 19:53:59 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 984,960 1,160,956 1.1787
17 Jan 2014 11:02:41 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 959,040 1,130,083 1.1783
17 Jan 2014 02:15:35 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 933,120 1,099,300 1.1781
16 Jan 2014 17:42:34 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 907,200 1,068,520 1.1778
16 Jan 2014 08:08:21 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 881,280 1,035,844 1.1754
15 Jan 2014 22:00:52 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 855,360 1,000,663 1.1699
15 Jan 2014 11:54:50 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 829,440 965,833 1.1644
15 Jan 2014 01:57:14 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 803,520 930,374 1.1579
14 Jan 2014 16:33:59 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 777,600 899,575 1.1569
14 Jan 2014 07:47:48 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 751,680 865,037 1.1508
14 Jan 2014 00:41:08 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 725,760 832,975 1.1477
13 Jan 2014 13:35:28 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 699,840 802,354 1.1465
13 Jan 2014 05:22:08 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 673,920 771,711 1.1451
12 Jan 2014 19:54:46 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 648,000 740,919 1.1434
12 Jan 2014 11:14:06 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 622,080 710,001 1.1413
12 Jan 2014 02:06:52 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 596,160 676,932 1.1355
11 Jan 2014 16:43:35 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 570,240 643,084 1.1277
11 Jan 2014 07:52:25 1259513 16194419 hadcm3n_801z_1980_40_008458010_2 544,320 610,394 1.1214


©2024 climateprediction.net