climateprediction.net home page
Task 16067648

Task 16067648

Name hadcm3n_855h_1980_40_008464633_1
Workunit 8615472
Created 14 Oct 2013, 20:05:19 UTC
Sent 14 Oct 2013, 20:05:30 UTC
Report deadline 14 Jan 2014, 3:32:41 UTC
Received 6 Nov 2013, 18:08:17 UTC
Server state Over
Outcome Success
Client state Done
Exit status 0 (0x00000000)
Computer ID 1275748
Run time 12 days 8 hours 50 min 31 sec
CPU time 12 days 6 hours 21 min 18 sec
Validate state Valid
Credit 12,441.60
Device peak FLOPS 3.48 GFLOPS
Application version UK Met Office Coupled Model Full Resolution Ocean v6.07
windows_intelx86
Stderr
<core_client_version>7.0.64</core_client_version>
<![CDATA[
<stderr_txt>
7:12:30 (5184): No heartbeat from core client for 30 sec - exiting
17:12:31 (5184): No heartbeat from core client for 30 sec - exiting
17:12:32 (5184): No heartbeat from core client for 30 sec - exiting
17:12:33 (5184): No heartbeat from core client for 30 sec - exiting
17:12:34 (5184): No heartbeat from core client for 30 sec - exiting
17:12:35 (5184): No heartbeat from core client for 30 sec - exiting
17:12:36 (5184): No heartbeat from core client for 30 sec - exiting
17:12:37 (5184): No heartbeat from core client for 30 sec - exiting
17:12:38 (5184): No heartbeat from core client for 30 sec - exiting
17:12:39 (5184): No heartbeat from core client for 30 sec - exiting
17:12:40 (5184): No heartbeat from core client for 30 sec - exiting
17:12:41 (5184): No heartbeat from core client for 30 sec - exiting
17:12:42 (5184): No heartbeat from core client for 30 sec - exiting
17:12:43 (5184): No heartbeat from core client for 30 sec - exiting
17:12:44 (5184): No heartbeat from core client for 30 sec - exiting
17:12:45 (5184): No heartbeat from core client for 30 sec - exiting
17:12:46 (5184): No heartbeat from core client for 30 sec - exiting
17:12:47 (5184): No heartbeat from core client for 30 sec - exiting
17:12:48 (5184): No heartbeat from core client for 30 sec - exiting
17:12:49 (5184): No heartbeat from core client for 30 sec - exiting
17:12:50 (5184): No heartbeat from core client for 30 sec - exiting
17:12:51 (5184): No heartbeat from core client for 30 sec - exiting
17:12:52 (5184): No heartbeat from core client for 30 sec - exiting
17:12:53 (5184): No heartbeat from core client for 30 sec - exiting
17:12:54 (5184): No heartbeat from core client for 30 sec - exiting
17:12:55 (5184): No heartbeat from core client for 30 sec - exiting
17:12:56 (5184): No heartbeat from core client for 30 sec - exiting
17:12:57 (5184): No heartbeat from core client for 30 sec - exiting
17:12:58 (5184): No heartbeat from core client for 30 sec - exiting
17:12:59 (5184): No heartbeat from core client for 30 sec - exiting
17:13:00 (5184): No heartbeat from core client for 30 sec - exiting
17:13:01 (5184): No heartbeat from core client for 30 sec - exiting
17:13:02 (5184): No heartbeat from core client for 30 sec - exiting
17:13:03 (5184): No heartbeat from core client for 30 sec - exiting
17:13:04 (5184): No heartbeat from core client for 30 sec - exiting
17:13:05 (5184): No heartbeat from core client for 30 sec - exiting
17:13:06 (5184): No heartbeat from core client for 30 sec - exiting
17:13:07 (5184): No heartbeat from core client for 30 sec - exiting
17:13:08 (5184): No heartbeat from core client for 30 sec - exiting
17:13:09 (5184): No heartbeat from core client for 30 sec - exiting
17:13:10 (5184): No heartbeat from core client for 30 sec - exiting
17:13:11 (5184): No heartbeat from core client for 30 sec - exiting
17:13:12 (5184): No heartbeat from core client for 30 sec - exiting
17:13:13 (5184): No heartbeat from core client for 30 sec - exiting
17:13:14 (5184): No heartbeat from core client for 30 sec - exiting
17:13:15 (5184): No heartbeat from core client for 30 sec - exiting
17:13:16 (5184): No heartbeat from core client for 30 sec - exiting
17:13:17 (5184): No heartbeat from core client for 30 sec - exiting
17:13:18 (5184): No heartbeat from core client for 30 sec - exiting
17:13:19 (5184): No heartbeat from core client for 30 sec - exiting
17:13:20 (5184): No heartbeat from core client for 30 sec - exiting
17:13:21 (5184): No heartbeat from core client for 30 sec - exiting
17:13:22 (5184): No heartbeat from core client for 30 sec - exiting
17:13:23 (5184): No heartbeat from core client for 30 sec - exiting
17:13:24 (5184): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:17:13 (4968): No heartbeat from core client for 30 sec - exiting
17:17:14 (4968): No heartbeat from core client for 30 sec - exiting
17:17:15 (4968): No heartbeat from core client for 30 sec - exiting
17:17:16 (4968): No heartbeat from core client for 30 sec - exiting
17:17:17 (4968): No heartbeat from core client for 30 sec - exiting
17:17:18 (4968): No heartbeat from core client for 30 sec - exiting
17:17:19 (4968): No heartbeat from core client for 30 sec - exiting
17:17:20 (4968): No heartbeat from core client for 30 sec - exiting
17:17:21 (4968): No heartbeat from core client for 30 sec - exiting
17:17:22 (4968): No heartbeat from core client for 30 sec - exiting
17:17:23 (4968): No heartbeat from core client for 30 sec - exiting
17:17:24 (4968): No heartbeat from core client for 30 sec - exiting
17:17:25 (4968): No heartbeat from core client for 30 sec - exiting
17:17:26 (4968): No heartbeat from core client for 30 sec - exiting
17:17:27 (4968): No heartbeat from core client for 30 sec - exiting
17:17:28 (4968): No heartbeat from core client for 30 sec - exiting
17:17:29 (4968): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7547C41F

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 10/23/13 17:20:11
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_855h_1980_40_008464633;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 77750000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.18247) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7600.16385

ModLoad: 732a0000 0003f000 C:\Program Files\AVAST Software\Avast\snxhk.dll (8.0.1497.376) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\Avast800\src\BUILDS\Release\x86\snxhk.pdb
    File Version          : 8.0.1497.376
    Company Name          : AVAST Software
    Product Name          : avast! Antivirus
    Product Version       : 8.0.1497.376

ModLoad: 757e0000 00110000 C:\Windows\syswow64\KERNEL32.dll (6.1.7601.18229) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7601.18015

ModLoad: 75470000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18229) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18015 (win7sp1_gdr.121129-1432)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7601.18015

ModLoad: 754c0000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7601.17514

ModLoad: 75750000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 770b0000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7601.18177) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7601.18177 (win7sp1_gdr.130605-1534)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18177

ModLoad: 75cb0000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18009) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18009 (win7sp1_gdr.121121-1431)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18009

ModLoad: 75b40000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-exported- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 75390000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.18247) (-exported- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7600.16385

ModLoad: 75d50000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-exported- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76e20000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.18205) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7600.16385

ModLoad: 751a0000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.17940) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.17940 (win7sp1_gdr.120824-0334)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17940

ModLoad: 75190000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 6cda0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 6488 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7547C41F

- Registers -
eax=1132f9cc ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=7547c41f esp=1132f9cc ebp=1132fa1c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000206

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa1c 007bfa90 e06d7363 00000001 00000003 1132fa48 KERNELBASE!RaiseException+0x0 
1132fa54 007bbbcb 1132fa64 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa70 00406e6e 00500000 004083ee 00000000 001d3fe0 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 001f4041 1132faec 001f3fc9 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 757f336a 7efde000 1132ffd4 77789f72 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 77789f72 7efde000 6030921b 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
1132ffd4 77789f45 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 05dedd21 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 6372 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=33a1ff48 edi=00000000
eip=7776fd91 esp=33a1ff04 ebp=33a1ff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
33a1ff6c 75484498 00000064 00000000 33a1ff88 00749924 ntdll!ZwDelayExecution+0x0 
33a1ff7c 00749924 00000064 33a1ff94 757f336a 00000000 KERNELBASE!Sleep+0x0 
33a1ff88 757f336a 00000000 33a1ffd4 77789f72 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
33a1ff94 77789f72 00000000 42a3921b 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
33a1ffd4 77789f45 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
33a1ffec 00000000 00749910 00000000 00000000 34040000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2140, iMonCtr=1
Model crash detected, will try to restart...
17:21:02 (2140): No heartbeat from core client for 30 sec - exiting
17:21:03 (2140): No heartbeat from core client for 30 sec - exiting
17:21:04 (2140): No heartbeat from core client for 30 sec - exiting
17:21:05 (2140): No heartbeat from core client for 30 sec - exiting
17:21:06 (2140): No heartbeat from core client for 30 sec - exiting
17:21:07 (2140): No heartbeat from core client for 30 sec - exiting
17:21:08 (2140): No heartbeat from core client for 30 sec - exiting
17:21:09 (2140): No heartbeat from core client for 30 sec - exiting
17:21:10 (2140): No heartbeat from core client for 30 sec - exiting
17:21:11 (2140): No heartbeat from core client for 30 sec - exiting
17:21:12 (2140): No heartbeat from core client for 30 sec - exiting
17:21:13 (2140): No heartbeat from core client for 30 sec - exiting
17:21:14 (2140): No heartbeat from core client for 30 sec - exiting
17:21:15 (2140): No heartbeat from core client for 30 sec - exiting
17:21:16 (2140): No heartbeat from core client for 30 sec - exiting
17:21:17 (2140): No heartbeat from core client for 30 sec - exiting
17:21:18 (2140): No heartbeat from core client for 30 sec - exiting
17:21:19 (2140): No heartbeat from core client for 30 sec - exiting
17:21:20 (2140): No heartbeat from core client for 30 sec - exiting
17:21:21 (2140): No heartbeat from core client for 30 sec - exiting
17:21:22 (2140): No heartbeat from core client for 30 sec - exiting
17:21:23 (2140): No heartbeat from core client for 30 sec - exiting
17:21:24 (2140): No heartbeat from core client for 30 sec - exiting
17:21:25 (2140): No heartbeat from core client for 30 sec - exiting
17:21:26 (2140): No heartbeat from core client for 30 sec - exiting
17:21:27 (2140): No heartbeat from core client for 30 sec - exiting
17:21:28 (2140): No heartbeat from core client for 30 sec - exiting
17:21:29 (2140): No heartbeat from core client for 30 sec - exiting
17:21:30 (2140): No heartbeat from core client for 30 sec - exiting
17:21:31 (2140): No heartbeat from core client for 30 sec - exiting
17:21:32 (2140): No heartbeat from core client for 30 sec - exiting
17:21:33 (2140): No heartbeat from core client for 30 sec - exiting
17:21:34 (2140): No heartbeat from core client for 30 sec - exiting
17:21:35 (2140): No heartbeat from core client for 30 sec - exiting
17:21:36 (2140): No heartbeat from core client for 30 sec - exiting
17:21:37 (2140): No heartbeat from core client for 30 sec - exiting
17:21:38 (2140): No heartbeat from core client for 30 sec - exiting
17:21:39 (2140): No heartbeat from core client for 30 sec - exiting
17:21:40 (2140): No heartbeat from core client for 30 sec - exiting
17:21:41 (2140): No heartbeat from core client for 30 sec - exiting
17:21:42 (2140): No heartbeat from core client for 30 sec - exiting
17:21:43 (2140): No heartbeat from core client for 30 sec - exiting
17:21:44 (2140): No heartbeat from core client for 30 sec - exiting
17:21:45 (2140): No heartbeat from core client for 30 sec - exiting
17:21:46 (2140): No heartbeat from core client for 30 sec - exiting
17:21:47 (2140): No heartbeat from core client for 30 sec - exiting
17:21:48 (2140): No heartbeat from core client for 30 sec - exiting
17:21:49 (2140): No heartbeat from core client for 30 sec - exiting
17:21:50 (2140): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:22:56 (4876): No heartbeat from core client for 30 sec - exiting
17:22:58 (4876): No heartbeat from core client for 30 sec - exiting
17:22:59 (4876): No heartbeat from core client for 30 sec - exiting
17:23:00 (4876): No heartbeat from core client for 30 sec - exiting
17:23:01 (4876): No heartbeat from core client for 30 sec - exiting
17:23:02 (4876): No heartbeat from core client for 30 sec - exiting
17:23:03 (4876): No heartbeat from core client for 30 sec - exiting
17:23:04 (4876): No heartbeat from core client for 30 sec - exiting
17:23:05 (4876): No heartbeat from core client for 30 sec - exiting
17:23:06 (4876): No heartbeat from core client for 30 sec - exiting
17:23:07 (4876): No heartbeat from core client for 30 sec - exiting
17:23:08 (4876): No heartbeat from core client for 30 sec - exiting
17:23:09 (4876): No heartbeat from core client for 30 sec - exiting
17:23:10 (4876): No heartbeat from core client for 30 sec - exiting
17:23:11 (4876): No heartbeat from core client for 30 sec - exiting
17:23:12 (4876): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:23:59 (2744): No heartbeat from core client for 30 sec - exiting
17:24:00 (2744): No heartbeat from core client for 30 sec - exiting
17:24:01 (2744): No heartbeat from core client for 30 sec - exiting
17:24:02 (2744): No heartbeat from core client for 30 sec - exiting
17:24:03 (2744): No heartbeat from core client for 30 sec - exiting
17:24:04 (2744): No heartbeat from core client for 30 sec - exiting
17:24:05 (2744): No heartbeat from core client for 30 sec - exiting
17:24:06 (2744): No heartbeat from core client for 30 sec - exiting
17:24:07 (2744): No heartbeat from core client for 30 sec - exiting
17:24:08 (2744): No heartbeat from core client for 30 sec - exiting
17:24:09 (2744): No heartbeat from core client for 30 sec - exiting
17:24:10 (2744): No heartbeat from core client for 30 sec - exiting
17:24:11 (2744): No heartbeat from core client for 30 sec - exiting
17:24:12 (2744): No heartbeat from core client for 30 sec - exiting
17:24:13 (2744): No heartbeat from core client for 30 sec - exiting
17:24:14 (2744): No heartbeat from core client for 30 sec - exiting
17:24:15 (2744): No heartbeat from core client for 30 sec - exiting
17:24:16 (2744): No heartbeat from core client for 30 sec - exiting
17:24:17 (2744): No heartbeat from core client for 30 sec - exiting
17:24:18 (2744): No heartbeat from core client for 30 sec - exiting
17:24:19 (2744): No heartbeat from core client for 30 sec - exiting
17:24:20 (2744): No heartbeat from core client for 30 sec - exiting
17:24:21 (2744): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:25:09 (1896): No heartbeat from core client for 30 sec - exiting
17:25:10 (1896): No heartbeat from core client for 30 sec - exiting
17:25:11 (1896): No heartbeat from core client for 30 sec - exiting
17:25:12 (1896): No heartbeat from core client for 30 sec - exiting
17:25:13 (1896): No heartbeat from core client for 30 sec - exiting
17:25:14 (1896): No heartbeat from core client for 30 sec - exiting
17:25:15 (1896): No heartbeat from core client for 30 sec - exiting
17:25:16 (1896): No heartbeat from core client for 30 sec - exiting
17:25:17 (1896): No heartbeat from core client for 30 sec - exiting
17:25:18 (1896): No heartbeat from core client for 30 sec - exiting
17:25:19 (1896): No heartbeat from core client for 30 sec - exiting
17:25:20 (1896): No heartbeat from core client for 30 sec - exiting
17:25:21 (1896): No heartbeat from core client for 30 sec - exiting
17:25:22 (1896): No heartbeat from core client for 30 sec - exiting
17:25:23 (1896): No heartbeat from core client for 30 sec - exiting
17:25:24 (1896): No heartbeat from core client for 30 sec - exiting
17:25:25 (1896): No heartbeat from core client for 30 sec - exiting
17:25:26 (1896): No heartbeat from core client for 30 sec - exiting
17:25:27 (1896): No heartbeat from core client for 30 sec - exiting
17:25:28 (1896): No heartbeat from core client for 30 sec - exiting
17:25:29 (1896): No heartbeat from core client for 30 sec - exiting
17:25:30 (1896): No heartbeat from core client for 30 sec - exiting
17:25:31 (1896): No heartbeat from core client for 30 sec - exiting
17:25:32 (1896): No heartbeat from core client for 30 sec - exiting
17:25:33 (1896): No heartbeat from core client for 30 sec - exiting
17:25:34 (1896): No heartbeat from core client for 30 sec - exiting
17:25:35 (1896): No heartbeat from core client for 30 sec - exiting
17:25:36 (1896): No heartbeat from core client for 30 sec - exiting
17:25:37 (1896): No heartbeat from core client for 30 sec - exiting
17:25:38 (1896): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:25:39 (1896): No heartbeat from core client for 30 sec - exiting
17:26:36 (6236): No heartbeat from core client for 30 sec - exiting
17:26:37 (6236): No heartbeat from core client for 30 sec - exiting
17:26:38 (6236): No heartbeat from core client for 30 sec - exiting
17:26:39 (6236): No heartbeat from core client for 30 sec - exiting
17:26:40 (6236): No heartbeat from core client for 30 sec - exiting
17:26:41 (6236): No heartbeat from core client for 30 sec - exiting
17:26:42 (6236): No heartbeat from core client for 30 sec - exiting
17:26:43 (6236): No heartbeat from core client for 30 sec - exiting
17:26:44 (6236): No heartbeat from core client for 30 sec - exiting
17:26:45 (6236): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:27:27 (4376): No heartbeat from core client for 30 sec - exiting
17:27:28 (4376): No heartbeat from core client for 30 sec - exiting
17:27:29 (4376): No heartbeat from core client for 30 sec - exiting
17:27:30 (4376): No heartbeat from core client for 30 sec - exiting
17:27:31 (4376): No heartbeat from core client for 30 sec - exiting
17:27:32 (4376): No heartbeat from core client for 30 sec - exiting
17:27:33 (4376): No heartbeat from core client for 30 sec - exiting
17:27:34 (4376): No heartbeat from core client for 30 sec - exiting
17:27:35 (4376): No heartbeat from core client for 30 sec - exiting
17:27:36 (4376): No heartbeat from core client for 30 sec - exiting
17:27:37 (4376): No heartbeat from core client for 30 sec - exiting
17:27:38 (4376): No heartbeat from core client for 30 sec - exiting
17:27:39 (4376): No heartbeat from core client for 30 sec - exiting
17:27:40 (4376): No heartbeat from core client for 30 sec - exiting
17:27:41 (4376): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:28:19 (5584): No heartbeat from core client for 30 sec - exiting
17:28:20 (5584): No heartbeat from core client for 30 sec - exiting
17:28:21 (5584): No heartbeat from core client for 30 sec - exiting
17:28:22 (5584): No heartbeat from core client for 30 sec - exiting
17:28:23 (5584): No heartbeat from core client for 30 sec - exiting
17:28:24 (5584): No heartbeat from core client for 30 sec - exiting
17:28:25 (5584): No heartbeat from core client for 30 sec - exiting
17:28:26 (5584): No heartbeat from core client for 30 sec - exiting
17:28:27 (5584): No heartbeat from core client for 30 sec - exiting
17:28:28 (5584): No heartbeat from core client for 30 sec - exiting
17:28:29 (5584): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:29:07 (2132): No heartbeat from core client for 30 sec - exiting
17:29:09 (2132): No heartbeat from core client for 30 sec - exiting
17:29:10 (2132): No heartbeat from core client for 30 sec - exiting
17:29:11 (2132): No heartbeat from core client for 30 sec - exiting
17:29:12 (2132): No heartbeat from core client for 30 sec - exiting
17:29:13 (2132): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:29:14 (2132): No heartbeat from core client for 30 sec - exiting
17:29:52 (5032): No heartbeat from core client for 30 sec - exiting
17:29:53 (5032): No heartbeat from core client for 30 sec - exiting
17:29:54 (5032): No heartbeat from core client for 30 sec - exiting
17:29:55 (5032): No heartbeat from core client for 30 sec - exiting
17:29:56 (5032): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:30:35 (2664): No heartbeat from core client for 30 sec - exiting
17:30:36 (2664): No heartbeat from core client for 30 sec - exiting
17:30:37 (2664): No heartbeat from core client for 30 sec - exiting
17:30:38 (2664): No heartbeat from core client for 30 sec - exiting
17:30:39 (2664): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:30:40 (2664): No heartbeat from core client for 30 sec - exiting
17:31:41 (5508): No heartbeat from core client for 30 sec - exiting
17:31:42 (5508): No heartbeat from core client for 30 sec - exiting
17:31:43 (5508): No heartbeat from core client for 30 sec - exiting
17:31:44 (5508): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Signal 11 received, exiting...
Called boinc_finish
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5096, iMonCtr=1
Model crash detected, will try to restart...
17:32:47 (5096): No heartbeat from core client for 30 sec - exiting
17:32:48 (5096): No heartbeat from core client for 30 sec - exiting
17:32:49 (5096): No heartbeat from core client for 30 sec - exiting
17:32:50 (5096): No heartbeat from core client for 30 sec - exiting
17:32:51 (5096): No heartbeat from core client for 30 sec - exiting
17:32:52 (5096): No heartbeat from core client for 30 sec - exiting
17:32:53 (5096): No heartbeat from core client for 30 sec - exiting
17:32:54 (5096): No heartbeat from core client for 30 sec - exiting
17:32:55 (5096): No heartbeat from core client for 30 sec - exiting
17:32:56 (5096): No heartbeat from core client for 30 sec - exiting
17:32:57 (5096): No heartbeat from core client for 30 sec - exiting
17:32:58 (5096): No heartbeat from core client for 30 sec - exiting
17:32:59 (5096): No heartbeat from core client for 30 sec - exiting
17:33:00 (5096): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:34:12 (6764): No heartbeat from core client for 30 sec - exiting
17:34:13 (6764): No heartbeat from core client for 30 sec - exiting
17:34:14 (6764): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:34:15 (6764): No heartbeat from core client for 30 sec - exiting
17:34:53 (6196): No heartbeat from core client for 30 sec - exiting
17:34:54 (6196): No heartbeat from core client for 30 sec - exiting
17:34:55 (6196): No heartbeat from core client for 30 sec - exiting
17:34:56 (6196): No heartbeat from core client for 30 sec - exiting
17:34:57 (6196): No heartbeat from core client for 30 sec - exiting
17:34:58 (6196): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:35:35 (5636): No heartbeat from core client for 30 sec - exiting
17:35:36 (5636): No heartbeat from core client for 30 sec - exiting
17:35:37 (5636): No heartbeat from core client for 30 sec - exiting
17:35:38 (5636): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:36:20 (4468): No heartbeat from core client for 30 sec - exiting
17:36:21 (4468): No heartbeat from core client for 30 sec - exiting
17:36:22 (4468): No heartbeat from core client for 30 sec - exiting
17:36:23 (4468): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:36:24 (4468): No heartbeat from core client for 30 sec - exiting


Unhandled Exception Detected...

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7547C41F

Engaging BOINC Windows Runtime Debugger...



********************


BOINC Windows Runtime Debugger Version 6.13.0


Dump Timestamp    : 10/23/13 17:37:24
Install Directory : 
Data Directory    : C:\ProgramData\BOINC
Project Symstore  : 
LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126
Loaded Library    : dbghelp.dll
LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126
LoadLibraryA( symsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126
LoadLibraryA( srcsrv.dll ): GetLastError = 126
LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126
Loaded Library    : version.dll
Debugger Engine   : 4.0.5.0
Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_855h_1980_40_008464633;C:\ProgramData\BOINC\projects\climateprediction.net


ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : 

ModLoad: 77750000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.18247) (-exported- Symbols Loaded)
    Linked PDB Filename   : wntdll.pdb
    File Version          : 6.1.7601.18247 (win7sp1_gdr.130828-1532)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18247

ModLoad: 732a0000 0003f000 C:\Program Files\AVAST Software\Avast\snxhk.dll (8.0.1497.376) (-exported- Symbols Loaded)
    Linked PDB Filename   : d:\Avast800\src\BUILDS\Release\x86\snxhk.pdb
    File Version          : 8.0.1497.376
    Company Name          : AVAST Software
    Product Name          : avast! Antivirus
    Product Version       : 8.0.1497.376

ModLoad: 757e0000 00110000 C:\Windows\syswow64\KERNEL32.dll (6.1.7601.18229) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernel32.pdb
    File Version          : 6.1.7601.18229 (win7sp1_gdr.130801-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18229

ModLoad: 75470000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18229) (-exported- Symbols Loaded)
    Linked PDB Filename   : wkernelbase.pdb
    File Version          : 6.1.7601.18229 (win7sp1_gdr.130801-1533)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18229

ModLoad: 754c0000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wuser32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7601.17514

ModLoad: 75750000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.17514) (-exported- Symbols Loaded)
    Linked PDB Filename   : wgdi32.pdb
    File Version          : 6.1.7601.17514 (win7sp1_rtm.101119-1850)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17514

ModLoad: 770b0000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7601.18177) (-exported- Symbols Loaded)
    Linked PDB Filename   : wlpk.pdb
    File Version          : 6.1.7601.18177 (win7sp1_gdr.130605-1534)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18177

ModLoad: 75cb0000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18009) (-exported- Symbols Loaded)
    Linked PDB Filename   : usp10.pdb
    File Version          : 1.0626.7601.18009 (win7sp1_gdr.121121-1431)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft(R) Uniscribe Unicode script processor
    Product Version       : 1.0626.7601.18009

ModLoad: 75b40000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : msvcrt.pdb
    File Version          : 7.0.7601.17744 (win7sp1_gdr.111215-1535)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 7.0.7601.17744

ModLoad: 75390000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.18247) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : advapi32.pdb
    File Version          : 6.1.7601.18247 (win7sp1_gdr.130828-1532)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.18247

ModLoad: 75d50000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : sechost.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 76e20000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.18205) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : wrpcrt4.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7600.16385

ModLoad: 751a0000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.17940) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : wsspicli.pdb
    File Version          : 6.1.7601.17940 (win7sp1_gdr.120824-0334)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7601.17940

ModLoad: 75190000 0000c000 C:\Windows\syswow64\CRYPTBASE.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : cryptbase.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 6cda0000 000c8000 C:\Windows\system32\OPENGL32.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : opengl32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Microsoft&#174; Windows&#174; Operating System
    Product Version       : 6.1.7600.16385

ModLoad: 74050000 00022000 C:\Windows\system32\GLU32.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded)
    Linked PDB Filename   : glu32.pdb
    File Version          : 6.1.7600.16385 (win7_rtm.090713-1255)
    Company Name          : Microsoft Corporation
    Product Name          : Sistema operativo Microsoft&#174; Windows&#174;
    Product Version       : 6.1.7600.16385



*** Dump of the Process Statistics: ***

- I/O Operations Counters -
Read: 0, Write: 0, Other 0

- I/O Transfers Counters -
Read: 0, Write: 0, Other 0

- Paged Pool Usage -
QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0
QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0

- Virtual Memory Usage -
VirtualSize: 0, PeakVirtualSize: 0

- Pagefile Usage -
PagefileUsage: 0, PeakPagefileUsage: 0

- Working Set Size -
WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0

*** Dump of thread ID 5524 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Unhandled Exception Record -
Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7547C41F

- Registers -
eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe
eip=7547c41f esp=1132f9c4 ebp=1132fa14
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000212

- Callstack -
ChildEBP RetAddr  Args to Child
1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 KERNELBASE!RaiseException+0x0 
1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 
1132fad8 00408130 1132faec 11474041 1132faec 11473fc9 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff88 757f336a 7efde000 1132ffd4 77789f72 7efde000 hadcm3n_um_6.07_windows_intelx8!+0x0 
1132ff94 77789f72 7efde000 6020fc50 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
1132ffd4 77789f45 007bde7f 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
1132ffec 00000000 007bde7f 7efde000 00000000 5c1882b0 ntdll!RtlInitializeExceptionChain+0x0 

*** Dump of thread ID 6668 (state: Initialized): ***

- Information -
Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000

- Registers -
eax=00000000 ebx=00000000 ecx=00000000 edx=00000000 esi=3395ff48 edi=00000000
eip=7776fd91 esp=3395ff04 ebp=3395ff6c
cs=0023  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00000246

- Callstack -
ChildEBP RetAddr  Args to Child
3395ff6c 75484498 00000064 00000000 3395ff88 00749924 ntdll!ZwDelayExecution+0x0 
3395ff7c 00749924 00000064 3395ff94 757f336a 00000000 KERNELBASE!Sleep+0x0 
3395ff88 757f336a 00000000 3395ffd4 77789f72 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 
3395ff94 77789f72 00000000 4287fc50 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 
3395ffd4 77789f45 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 
3395ffec 00000000 00749910 00000000 00000000 33e70000 ntdll!RtlInitializeExceptionChain+0x0 


*** Debug Message Dump ****


*** Foreground Window Data ***
    Window Name      : 
    Window Class     : 
    Window Process ID: 0
    Window Thread ID : 0

Exiting...
Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4644, iMonCtr=1
Model crash detected, will try to restart...
17:38:11 (4644): No heartbeat from core client for 30 sec - exiting
17:38:12 (4644): No heartbeat from core client for 30 sec - exiting
17:38:13 (4644): No heartbeat from core client for 30 sec - exiting
17:38:14 (4644): No heartbeat from core client for 30 sec - exiting
17:38:15 (4644): No heartbeat from core client for 30 sec - exiting
17:38:16 (4644): No heartbeat from core client for 30 sec - exiting
17:38:17 (4644): No heartbeat from core client for 30 sec - exiting
17:38:18 (4644): No heartbeat from core client for 30 sec - exiting
17:38:19 (4644): No heartbeat from core client for 30 sec - exiting
17:38:20 (4644): No heartbeat from core client for 30 sec - exiting
17:38:21 (4644): No heartbeat from core client for 30 sec - exiting
17:38:22 (4644): No heartbeat from core client for 30 sec - exiting
17:38:23 (4644): No heartbeat from core client for 30 sec - exiting
17:38:24 (4644): No heartbeat from core client for 30 sec - exiting
17:38:25 (4644): No heartbeat from core client for 30 sec - exiting
17:38:26 (4644): No heartbeat from core client for 30 sec - exiting
17:38:27 (4644): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:39:42 (5560): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:40:18 (5944): No heartbeat from core client for 30 sec - exiting
17:40:20 (5944): No heartbeat from core client for 30 sec - exiting
17:40:21 (5944): No heartbeat from core client for 30 sec - exiting
17:40:22 (5944): No heartbeat from core client for 30 sec - exiting
17:40:23 (5944): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:41:01 (6284): No heartbeat from core client for 30 sec - exiting
17:41:02 (6284): No heartbeat from core client for 30 sec - exiting
17:41:03 (6284): No heartbeat from core client for 30 sec - exiting
17:41:04 (6284): No heartbeat from core client for 30 sec - exiting
17:41:05 (6284): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:41:44 (2120): No heartbeat from core client for 30 sec - exiting
17:41:45 (2120): No heartbeat from core client for 30 sec - exiting
17:41:46 (2120): No heartbeat from core client for 30 sec - exiting
17:41:47 (2120): No heartbeat from core client for 30 sec - exiting
17:41:48 (2120): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:42:39 (3896): No heartbeat from core client for 30 sec - exiting
17:42:40 (3896): No heartbeat from core client for 30 sec - exiting
17:42:41 (3896): No heartbeat from core client for 30 sec - exiting
17:42:42 (3896): No heartbeat from core client for 30 sec - exiting
17:42:43 (3896): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:43:21 (1100): No heartbeat from core client for 30 sec - exiting
17:43:22 (1100): No heartbeat from core client for 30 sec - exiting
17:43:23 (1100): No heartbeat from core client for 30 sec - exiting
17:43:24 (1100): No heartbeat from core client for 30 sec - exiting
17:43:25 (1100): No heartbeat from core client for 30 sec - exiting
17:43:26 (1100): No heartbeat from core client for 30 sec - exiting
17:43:27 (1100): No heartbeat from core client for 30 sec - exiting
17:43:28 (1100): No heartbeat from core client for 30 sec - exiting
17:43:29 (1100): No heartbeat from core client for 30 sec - exiting
17:43:30 (1100): No heartbeat from core client for 30 sec - exiting
17:43:31 (1100): No heartbeat from core client for 30 sec - exiting
17:43:32 (1100): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:44:09 (4000): No heartbeat from core client for 30 sec - exiting
17:44:10 (4000): No heartbeat from core client for 30 sec - exiting
17:44:11 (4000): No heartbeat from core client for 30 sec - exiting
17:44:12 (4000): No heartbeat from core client for 30 sec - exiting
17:44:13 (4000): No heartbeat from core client for 30 sec - exiting
17:44:14 (4000): No heartbeat from core client for 30 sec - exiting
17:44:15 (4000): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:44:54 (6432): No heartbeat from core client for 30 sec - exiting
17:44:55 (6432): No heartbeat from core client for 30 sec - exiting
17:44:56 (6432): No heartbeat from core client for 30 sec - exiting
17:44:57 (6432): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:45:35 (4340): No heartbeat from core client for 30 sec - exiting
17:45:36 (4340): No heartbeat from core client for 30 sec - exiting
17:45:37 (4340): No heartbeat from core client for 30 sec - exiting
17:45:38 (4340): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:46:17 (6916): No heartbeat from core client for 30 sec - exiting
17:46:18 (6916): No heartbeat from core client for 30 sec - exiting
17:46:19 (6916): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:46:55 (6480): No heartbeat from core client for 30 sec - exiting
17:46:56 (6480): No heartbeat from core client for 30 sec - exiting
17:46:57 (6480): No heartbeat from core client for 30 sec - exiting
17:46:58 (6480): No heartbeat from core client for 30 sec - exiting
17:46:59 (6480): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:47:00 (6480): No heartbeat from core client for 30 sec - exiting
17:47:59 (1808): No heartbeat from core client for 30 sec - exiting
17:48:00 (1808): No heartbeat from core client for 30 sec - exiting
17:48:01 (1808): No heartbeat from core client for 30 sec - exiting
17:48:02 (1808): No heartbeat from core client for 30 sec - exiting
17:48:03 (1808): No heartbeat from core client for 30 sec - exiting
17:48:04 (1808): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:48:40 (1076): No heartbeat from core client for 30 sec - exiting
17:48:41 (1076): No heartbeat from core client for 30 sec - exiting
17:48:42 (1076): No heartbeat from core client for 30 sec - exiting
17:48:43 (1076): No heartbeat from core client for 30 sec - exiting
17:48:44 (1076): No heartbeat from core client for 30 sec - exiting
17:48:45 (1076): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:48:46 (1076): No heartbeat from core client for 30 sec - exiting
17:49:26 (5132): No heartbeat from core client for 30 sec - exiting
17:49:27 (5132): No heartbeat from core client for 30 sec - exiting
17:49:28 (5132): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:50:01 (6296): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:50:57 (7064): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:51:37 (6128): No heartbeat from core client for 30 sec - exiting
17:51:39 (6128): No heartbeat from core client for 30 sec - exiting
17:51:40 (6128): No heartbeat from core client for 30 sec - exiting
17:51:41 (6128): No heartbeat from core client for 30 sec - exiting
17:51:42 (6128): No heartbeat from core client for 30 sec - exiting
17:51:43 (6128): No heartbeat from core client for 30 sec - exiting
17:51:44 (6128): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:52:24 (7132): No heartbeat from core client for 30 sec - exiting
17:52:25 (7132): No heartbeat from core client for 30 sec - exiting
17:52:26 (7132): No heartbeat from core client for 30 sec - exiting
17:52:27 (7132): No heartbeat from core client for 30 sec - exiting
17:52:28 (7132): No heartbeat from core client for 30 sec - exiting
17:52:29 (7132): No heartbeat from core client for 30 sec - exiting
17:52:30 (7132): No heartbeat from core client for 30 sec - exiting
17:52:31 (7132): No heartbeat from core client for 30 sec - exiting
17:52:32 (7132): No heartbeat from core client for 30 sec - exiting
17:52:33 (7132): No heartbeat from core client for 30 sec - exiting
17:52:34 (7132): No heartbeat from core client for 30 sec - exiting
17:52:35 (7132): No heartbeat from core client for 30 sec - exiting
17:52:36 (7132): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:53:50 (3264): No heartbeat from core client for 30 sec - exiting
17:53:51 (3264): No heartbeat from core client for 30 sec - exiting
17:53:52 (3264): No heartbeat from core client for 30 sec - exiting
17:53:53 (3264): No heartbeat from core client for 30 sec - exiting
17:53:54 (3264): No heartbeat from core client for 30 sec - exiting
17:53:55 (3264): No heartbeat from core client for 30 sec - exiting
17:53:56 (3264): No heartbeat from core client for 30 sec - exiting
17:53:57 (3264): No heartbeat from core client for 30 sec - exiting
17:53:58 (3264): No heartbeat from core client for 30 sec - exiting
17:53:59 (3264): No heartbeat from core client for 30 sec - exiting
17:54:00 (3264): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:54:44 (6436): No heartbeat from core client for 30 sec - exiting
17:54:46 (6436): No heartbeat from core client for 30 sec - exiting
17:54:47 (6436): No heartbeat from core client for 30 sec - exiting
17:54:48 (6436): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:55:24 (5640): No heartbeat from core client for 30 sec - exiting
17:55:25 (5640): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:56:05 (6856): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:57:10 (6744): No heartbeat from core client for 30 sec - exiting
17:57:11 (6744): No heartbeat from core client for 30 sec - exiting
17:57:12 (6744): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:57:49 (1004): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:58:35 (6536): No heartbeat from core client for 30 sec - exiting
17:58:36 (6536): No heartbeat from core client for 30 sec - exiting
17:58:37 (6536): No heartbeat from core client for 30 sec - exiting
17:58:38 (6536): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:59:18 (6780): No heartbeat from core client for 30 sec - exiting
17:59:19 (6780): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
17:59:54 (1128): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:00:44 (3476): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:01:25 (6416): No heartbeat from core client for 30 sec - exiting
18:01:26 (6416): No heartbeat from core client for 30 sec - exiting
18:01:27 (6416): No heartbeat from core client for 30 sec - exiting
18:01:28 (6416): No heartbeat from core client for 30 sec - exiting
18:01:29 (6416): No heartbeat from core client for 30 sec - exiting
18:01:30 (6416): No heartbeat from core client for 30 sec - exiting
18:01:31 (6416): No heartbeat from core client for 30 sec - exiting
18:01:32 (6416): No heartbeat from core client for 30 sec - exiting
18:01:33 (6416): No heartbeat from core client for 30 sec - exiting
18:01:34 (6416): No heartbeat from core client for 30 sec - exiting
18:01:35 (6416): No heartbeat from core client for 30 sec - exiting
18:01:36 (6416): No heartbeat from core client for 30 sec - exiting
18:01:37 (6416): No heartbeat from core client for 30 sec - exiting
18:01:38 (6416): No heartbeat from core client for 30 sec - exiting
18:01:39 (6416): No heartbeat from core client for 30 sec - exiting
18:01:40 (6416): No heartbeat from core client for 30 sec - exiting
18:01:41 (6416): No heartbeat from core client for 30 sec - exiting
18:01:42 (6416): No heartbeat from core client for 30 sec - exiting
18:01:43 (6416): No heartbeat from core client for 30 sec - exiting
18:01:44 (6416): No heartbeat from core client for 30 sec - exiting
18:01:45 (6416): No heartbeat from core client for 30 sec - exiting
18:01:46 (6416): No heartbeat from core client for 30 sec - exiting
18:01:47 (6416): No heartbeat from core client for 30 sec - exiting
18:01:48 (6416): No heartbeat from core client for 30 sec - exiting
18:01:49 (6416): No heartbeat from core client for 30 sec - exiting
18:01:50 (6416): No heartbeat from core client for 30 sec - exiting
18:01:51 (6416): No heartbeat from core client for 30 sec - exiting
18:01:52 (6416): No heartbeat from core client for 30 sec - exiting
18:01:53 (6416): No heartbeat from core client for 30 sec - exiting
18:01:54 (6416): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:02:35 (6032): No heartbeat from core client for 30 sec - exiting
18:02:37 (6032): No heartbeat from core client for 30 sec - exiting
18:02:38 (6032): No heartbeat from core client for 30 sec - exiting
18:02:39 (6032): No heartbeat from core client for 30 sec - exiting
18:02:40 (6032): No heartbeat from core client for 30 sec - exiting
18:02:41 (6032): No heartbeat from core client for 30 sec - exiting
18:02:42 (6032): No heartbeat from core client for 30 sec - exiting
18:02:43 (6032): No heartbeat from core client for 30 sec - exiting
18:02:44 (6032): No heartbeat from core client for 30 sec - exiting
18:02:45 (6032): No heartbeat from core client for 30 sec - exiting
18:02:46 (6032): No heartbeat from core client for 30 sec - exiting
18:02:47 (6032): No heartbeat from core client for 30 sec - exiting
18:02:48 (6032): No heartbeat from core client for 30 sec - exiting
18:02:49 (6032): No heartbeat from core client for 30 sec - exiting
18:02:50 (6032): No heartbeat from core client for 30 sec - exiting
18:02:51 (6032): No heartbeat from core client for 30 sec - exiting
18:02:52 (6032): No heartbeat from core client for 30 sec - exiting
18:02:53 (6032): No heartbeat from core client for 30 sec - exiting
18:02:54 (6032): No heartbeat from core client for 30 sec - exiting
18:02:55 (6032): No heartbeat from core client for 30 sec - exiting
18:02:56 (6032): No heartbeat from core client for 30 sec - exiting
18:02:57 (6032): No heartbeat from core client for 30 sec - exiting
18:02:58 (6032): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:03:59 (6136): No heartbeat from core client for 30 sec - exiting
18:04:00 (6136): No heartbeat from core client for 30 sec - exiting
18:04:01 (6136): No heartbeat from core client for 30 sec - exiting
18:04:02 (6136): No heartbeat from core client for 30 sec - exiting
18:04:03 (6136): No heartbeat from core client for 30 sec - exiting
18:04:04 (6136): No heartbeat from core client for 30 sec - exiting
18:04:05 (6136): No heartbeat from core client for 30 sec - exiting
18:04:06 (6136): No heartbeat from core client for 30 sec - exiting
18:04:07 (6136): No heartbeat from core client for 30 sec - exiting
18:04:08 (6136): No heartbeat from core client for 30 sec - exiting
18:04:09 (6136): No heartbeat from core client for 30 sec - exiting
18:04:10 (6136): No heartbeat from core client for 30 sec - exiting
18:04:11 (6136): No heartbeat from core client for 30 sec - exiting
18:04:12 (6136): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:04:13 (6136): No heartbeat from core client for 30 sec - exiting
18:05:40 (7108): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:06:33 (6664): No heartbeat from core client for 30 sec - exiting
18:06:35 (6664): No heartbeat from core client for 30 sec - exiting
18:06:36 (6664): No heartbeat from core client for 30 sec - exiting
18:06:37 (6664): No heartbeat from core client for 30 sec - exiting
18:06:38 (6664): No heartbeat from core client for 30 sec - exiting
18:06:39 (6664): No heartbeat from core client for 30 sec - exiting
18:06:40 (6664): No heartbeat from core client for 30 sec - exiting
18:06:41 (6664): No heartbeat from core client for 30 sec - exiting
18:06:42 (6664): No heartbeat from core client for 30 sec - exiting
18:06:43 (6664): No heartbeat from core client for 30 sec - exiting
18:06:44 (6664): No heartbeat from core client for 30 sec - exiting
18:06:45 (6664): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:07:32 (6740): No heartbeat from core client for 30 sec - exiting
18:07:33 (6740): No heartbeat from core client for 30 sec - exiting
18:07:34 (6740): No heartbeat from core client for 30 sec - exiting
18:07:35 (6740): No heartbeat from core client for 30 sec - exiting
18:07:36 (6740): No heartbeat from core client for 30 sec - exiting
18:07:37 (6740): No heartbeat from core client for 30 sec - exiting
18:07:38 (6740): No heartbeat from core client for 30 sec - exiting
18:07:39 (6740): No heartbeat from core client for 30 sec - exiting
18:07:40 (6740): No heartbeat from core client for 30 sec - exiting
18:07:41 (6740): No heartbeat from core client for 30 sec - exiting
18:07:42 (6740): No heartbeat from core client for 30 sec - exiting
18:07:43 (6740): No heartbeat from core client for 30 sec - exiting
18:07:44 (6740): No heartbeat from core client for 30 sec - exiting
18:07:45 (6740): No heartbeat from core client for 30 sec - exiting
18:07:46 (6740): No heartbeat from core client for 30 sec - exiting
18:07:47 (6740): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:08:27 (7032): No heartbeat from core client for 30 sec - exiting
18:08:28 (7032): No heartbeat from core client for 30 sec - exiting
18:08:29 (7032): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:09:22 (2752): No heartbeat from core client for 30 sec - exiting
18:09:23 (2752): No heartbeat from core client for 30 sec - exiting
18:09:24 (2752): No heartbeat from core client for 30 sec - exiting
18:09:25 (2752): No heartbeat from core client for 30 sec - exiting
18:09:26 (2752): No heartbeat from core client for 30 sec - exiting
18:09:27 (2752): No heartbeat from core client for 30 sec - exiting
18:09:28 (2752): No heartbeat from core client for 30 sec - exiting
18:09:29 (2752): No heartbeat from core client for 30 sec - exiting
18:09:30 (2752): No heartbeat from core client for 30 sec - exiting
18:09:31 (2752): No heartbeat from core client for 30 sec - exiting
18:09:32 (2752): No heartbeat from core client for 30 sec - exiting
18:09:33 (2752): No heartbeat from core client for 30 sec - exiting
18:09:34 (2752): No heartbeat from core client for 30 sec - exiting
18:09:35 (2752): No heartbeat from core client for 30 sec - exiting
18:09:36 (2752): No heartbeat from core client for 30 sec - exiting
18:09:37 (2752): No heartbeat from core client for 30 sec - exiting
18:09:38 (2752): No heartbeat from core client for 30 sec - exiting
18:09:39 (2752): No heartbeat from core client for 30 sec - exiting
18:09:40 (2752): No heartbeat from core client for 30 sec - exiting
18:09:41 (2752): No heartbeat from core client for 30 sec - exiting
18:09:42 (2752): No heartbeat from core client for 30 sec - exiting
18:09:43 (2752): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:10:26 (3748): No heartbeat from core client for 30 sec - exiting
18:10:27 (3748): No heartbeat from core client for 30 sec - exiting
18:10:28 (3748): No heartbeat from core client for 30 sec - exiting
18:10:29 (3748): No heartbeat from core client for 30 sec - exiting
18:10:30 (3748): No heartbeat from core client for 30 sec - exiting
18:10:31 (3748): No heartbeat from core client for 30 sec - exiting
18:10:32 (3748): No heartbeat from core client for 30 sec - exiting
18:10:33 (3748): No heartbeat from core client for 30 sec - exiting
18:10:34 (3748): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
18:14:31 (6764): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
11:08:59 (2560): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
16:27:21 (4780): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:27:22 (4780): No heartbeat from core client for 30 sec - exiting
16:27:23 (4780): No heartbeat from core client for 30 sec - exiting
16:32:09 (4724): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
16:32:10 (4724): No heartbeat from core client for 30 sec - exiting
16:32:11 (4724): No heartbeat from core client for 30 sec - exiting
16:32:12 (4724): No heartbeat from core client for 30 sec - exiting
16:32:13 (4724): No heartbeat from core client for 30 sec - exiting
16:32:14 (4724): No heartbeat from core client for 30 sec - exiting
16:32:15 (4724): No heartbeat from core client for 30 sec - exiting
16:32:16 (4724): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - Quit request from BOINC...
10:58:54 (4824): No heartbeat from core client for 30 sec - exiting
CPDN Monitor - No 'heartbeat' from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
CPDN Monitor - Quit request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Suspended CPDN Monitor - Suspend request from BOINC...
Called boinc_finish

</stderr_txt>
]]>
Latest Trickles Received
Time Sent (UTC) Host ID Result ID Result Name Timestep CPU Time (sec) Average (sec/TS)
06 Nov 2013 17:04:37 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 1,036,800 1,059,268 1.0217
06 Nov 2013 09:47:57 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 1,010,880 1,033,271 1.0222
06 Nov 2013 02:30:33 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 984,960 1,007,330 1.0227
05 Nov 2013 19:02:53 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 959,040 980,807 1.0227
05 Nov 2013 16:51:23 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 933,120 954,891 1.0233
05 Nov 2013 03:51:41 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 907,200 928,895 1.0239
04 Nov 2013 20:33:26 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 881,280 902,708 1.0243
04 Nov 2013 13:10:17 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 855,360 876,572 1.0248
31 Oct 2013 03:07:26 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 829,440 851,229 1.0263
30 Oct 2013 19:48:55 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 803,520 824,880 1.0266
30 Oct 2013 12:31:02 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 777,600 798,874 1.0274
29 Oct 2013 01:24:34 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 751,680 772,840 1.0282
28 Oct 2013 18:04:32 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 725,760 746,818 1.0290
28 Oct 2013 10:52:03 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 699,840 721,015 1.0303
28 Oct 2013 02:27:50 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 673,920 694,727 1.0309
27 Oct 2013 19:07:29 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 648,000 668,289 1.0313
27 Oct 2013 11:39:48 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 622,080 641,814 1.0317
27 Oct 2013 04:16:44 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 596,160 615,414 1.0323
26 Oct 2013 20:48:35 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 570,240 588,621 1.0322
26 Oct 2013 13:19:23 1275748 16067648 hadcm3n_855h_1980_40_008464633_1 544,320 561,859 1.0322


©2024 climateprediction.net