Task 15757829

Name	hadcm3n_3lbj_1940_40_008260115_2
Workunit	8415239
Created	1 May 2013, 9:26:55 UTC
Sent	1 May 2013, 9:27:10 UTC
Report deadline	31 Jul 2013, 16:54:21 UTC
Received	10 May 2013, 20:28:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-2 (0xFFFFFFFE) Unknown error code
Computer ID	969347
Run time	7 days 19 hours 18 min 45 sec
CPU time	7 days 4 hours 9 min 5 sec
Validate state	Invalid
Credit	4,043.52
Device peak FLOPS	2.34 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -2 (0xfffffffe) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:26:12 (90016): No heartbeat from core client for 30 sec - exiting 14:26:13 (90016): No heartbeat from core client for 30 sec - exiting 14:26:14 (90016): No heartbeat from core client for 30 sec - exiting 14:26:15 (90016): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C8FC16 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 05/10/13 14:26:28 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll CPDN Monitor - No 'heartbeat' from BOINC... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=88992, iMonCtr=1 Model crash detected, will try to restart... 14:27:28 (88992): No heartbeat from core client for 30 sec - exiting 14:27:29 (88992): No heartbeat from core client for 30 sec - exiting 14:27:30 (88992): No heartbeat from core client for 30 sec - exiting 14:27:31 (88992): No heartbeat from core client for 30 sec - exiting 14:27:32 (88992): No heartbeat from core client for 30 sec - exiting 14:27:33 (88992): No heartbeat from core client for 30 sec - exiting 14:27:34 (88992): No heartbeat from core client for 30 sec - exiting 14:27:35 (88992): No heartbeat from core client for 30 sec - exiting 14:27:36 (88992): No heartbeat from core client for 30 sec - exiting 14:27:37 (88992): No heartbeat from core client for 30 sec - exiting 14:27:38 (88992): No heartbeat from core client for 30 sec - exiting 14:27:39 (88992): No heartbeat from core client for 30 sec - exiting 14:27:40 (88992): No heartbeat from core client for 30 sec - exiting 14:27:41 (88992): No heartbeat from core client for 30 sec - exiting 14:27:42 (88992): No heartbeat from core client for 30 sec - exiting 14:27:43 (88992): No heartbeat from core client for 30 sec - exiting 14:27:44 (88992): No heartbeat from core client for 30 sec - exiting 14:27:45 (88992): No heartbeat from core client for 30 sec - exiting 14:27:46 (88992): No heartbeat from core client for 30 sec - exiting 14:27:47 (88992): No heartbeat from core client for 30 sec - exiting 14:27:48 (88992): No heartbeat from core client for 30 sec - exiting 14:27:49 (88992): No heartbeat from core client for 30 sec - exiting 14:27:50 (88992): No heartbeat from core client for 30 sec - exiting 14:27:51 (88992): No heartbeat from core client for 30 sec - exiting 14:27:52 (88992): No heartbeat from core client for 30 sec - exiting 14:27:53 (88992): No heartbeat from core client for 30 sec - exiting 14:27:54 (88992): No heartbeat from core client for 30 sec - exiting 14:27:55 (88992): No heartbeat from core client for 30 sec - exiting 14:27:56 (88992): No heartbeat from core client for 30 sec - exiting 14:27:57 (88992): No heartbeat from core client for 30 sec - exiting 14:27:58 (88992): No heartbeat from core client for 30 sec - exiting 14:27:59 (88992): No heartbeat from core client for 30 sec - exiting 14:28:00 (88992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:28:01 (88992): No heartbeat from core client for 30 sec - exiting Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C8FC16 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 05/10/13 14:29:56 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_3lbj_1940_40_008260115;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77770000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18541) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 76c50000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18704) (-exported- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 76e30000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 75c60000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76d60000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76af0000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-exported- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 59a60000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 75f20000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18551) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18551 (vistasp2_gdr.111214-0301) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.6002.18551 ModLoad: 59a30000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 57b60000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5a930000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 ModLoad: 75ad0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 Get Product Name Failed. ModLoad: 759e0000 0008d000 C:\Windows\system32\OLEAUT32.dll (6.0.6002.18508) (-exported- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 6.0.6002.18508 Company Name : Microsoft Corporation Product Name : Product Version : 6.0.6002.18508 ModLoad: 75dd0000 00145000 C:\Windows\system32\ole32.dll (6.0.6002.18277) (-exported- Symbols Loaded) Linked PDB Filename : ole32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 73650000 0000c000 C:\Windows\system32\dwmapi.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : dwmapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 76d30000 0002a000 C:\Windows\system32\imagehlp.dll (6.0.6002.18592) (-exported- Symbols Loaded) Linked PDB Filename : imagehlp.pdb File Version : 6.0.6002.18592 (vistasp2_gdr.120229-0238) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18592 ModLoad: 76f50000 0001e000 C:\Windows\system32\IMM32.DLL (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : imm32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 75d00000 000c8000 C:\Windows\system32\MSCTF.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : msctf.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 778b0000 00009000 C:\Windows\system32\LPK.DLL (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : lpk.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 ModLoad: 75840000 0007d000 C:\Windows\system32\USP10.dll (1.626.6002.18244) (-exported- Symbols Loaded) Linked PDB Filename : usp10.pdb File Version : 1.0626.6002.18244 (vistasp2_gdr.100416-0345) Company Name : Microsoft Corporation Product Name : Microsoft(R) Uniscribe Unicode script processor Product Version : 1.0626.6002.18244 ModLoad: 59be0000 00115000 C:\Program Files\BOINC\dbghelp.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 60e40000 00048000 C:\Program Files\BOINC\symsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : symsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 62870000 0003b000 C:\Program Files\BOINC\srcsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : srcsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 74890000 00008000 C:\Windows\system32\version.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 386, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 126704, QuotaPeakPagedPoolUsage: 126704 QuotaNonPagedPoolUsage: 2968, QuotaPeakNonPagedPoolUsage: 2968 - Virtual Memory Usage - VirtualSize: 906211328, PeakVirtualSize: 906211328 - Pagefile Usage - PagefileUsage: 33865728, PeakPagefileUsage: 107085824 - Working Set Size - WorkingSetSize: 4120576, PeakWorkingSetSize: 4120576, PageFaultCount: 1009 * Dump of thread ID 89348 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 312002.000000, User Time: 0.000000, Wait Time: 53768512.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C8FC16 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=76c8fc16 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!RaiseException+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 000e2a41 1132faec 000e29c9 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 76c9d2e9 7ffd7000 1132ffd4 777b1603 7ffd7000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 777b1603 7ffd7000 574f546f 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 1132ffd4 777b15d6 007bde7f 7ffd7000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffd7000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 89388 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 53768508.000000 - Registers - eax=00156c69 ebx=00000000 ecx=00000000 edx=00000000 esi=3452ff48 edi=00000000 eip=777d5cd4 esp=3452ff04 ebp=3452ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3452ff6c 76c51c6c 00000064 00000000 3452ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3452ff7c 00749924 00000064 3452ff94 76c9d2e9 00000000 kernel32!Sleep+0x0 3452ff88 76c9d2e9 00000000 3452ffd4 777b1603 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3452ff94 777b1603 00000000 722f546f 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 3452ffd4 777b15d6 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3452ffec 00000000 00749910 00000000 00000000 34a40000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=88184, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C8FC16 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 05/10/13 14:31:26 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_3lbj_1940_40_008260115;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 00400000 00b28000 C:\ProgramData\BOINC\projects\climateprediction.net\hadcm3n_um_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77770000 00128000 C:\Windows\system32\ntdll.dll (6.0.6002.18541) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 76c50000 000dc000 C:\Windows\system32\kernel32.dll (6.0.6002.18704) (-exported- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 76e30000 0009d000 C:\Windows\system32\USER32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 75c60000 0004b000 C:\Windows\system32\GDI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76d60000 000c6000 C:\Windows\system32\ADVAPI32.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 76af0000 000c3000 C:\Windows\system32\RPCRT4.dll (6.0.6002.18024) (-exported- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 59a60000 000cb000 C:\Windows\system32\OPENGL32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 75f20000 000aa000 C:\Windows\system32\msvcrt.dll (7.0.6002.18551) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.6002.18551 (vistasp2_gdr.111214-0301) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.6002.18551 ModLoad: 59a30000 00023000 C:\Windows\system32\GLU32.dll (6.0.6000.16386) (-exported- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 57b60000 000e5000 C:\Windows\system32\DDRAW.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 5a930000 00006000 C:\Windows\system32\DCIMAN32.dll (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 ModLoad: 75ad0000 0018a000 C:\Windows\system32\SETUPAPI.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : setupapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 Get Product Name Failed. ModLoad: 759e0000 0008d000 C:\Windows\system32\OLEAUT32.dll (6.0.6002.18508) (-exported- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 6.0.6002.18508 Company Name : Microsoft Corporation Product Name : Product Version : 6.0.6002.18508 ModLoad: 75dd0000 00145000 C:\Windows\system32\ole32.dll (6.0.6002.18277) (-exported- Symbols Loaded) Linked PDB Filename : ole32.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 73650000 0000c000 C:\Windows\system32\dwmapi.dll (6.0.6001.18000) (-exported- Symbols Loaded) Linked PDB Filename : dwmapi.pdb File Version : 6.0.6001.18000 (longhorn_rtm.080118-1840) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6001.18000 ModLoad: 76d30000 0002a000 C:\Windows\system32\imagehlp.dll (6.0.6002.18592) (-exported- Symbols Loaded) Linked PDB Filename : imagehlp.pdb File Version : 6.0.6002.18592 (vistasp2_gdr.120229-0238) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18592 ModLoad: 76f50000 0001e000 C:\Windows\system32\IMM32.DLL (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : imm32.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 ModLoad: 75d00000 000c8000 C:\Windows\system32\MSCTF.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : msctf.pdb File Version : 6.0.6000.16386 (vista_rtm.061101-2205) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6000.16386 ModLoad: 778b0000 00009000 C:\Windows\system32\LPK.DLL (6.0.6002.18051) (-exported- Symbols Loaded) Linked PDB Filename : lpk.pdb File Version : 6.0.6002.18051 (vistasp2_gdr.090615-0258) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18051 ModLoad: 75840000 0007d000 C:\Windows\system32\USP10.dll (1.626.6002.18244) (-exported- Symbols Loaded) Linked PDB Filename : usp10.pdb File Version : 1.0626.6002.18244 (vistasp2_gdr.100416-0345) Company Name : Microsoft Corporation Product Name : Microsoft(R) Uniscribe Unicode script processor Product Version : 1.0626.6002.18244 ModLoad: 59be0000 00115000 C:\Program Files\BOINC\dbghelp.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 68750000 00048000 C:\Program Files\BOINC\symsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : symsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 680c0000 0003b000 C:\Program Files\BOINC\srcsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : srcsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 74890000 00008000 C:\Windows\system32\version.dll (6.0.6002.18005) (-exported- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.0.6002.18005 (lh_sp2rtm.090410-1830) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.0.6002.18005 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 207, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 126704, QuotaPeakPagedPoolUsage: 126704 QuotaNonPagedPoolUsage: 2968, QuotaPeakNonPagedPoolUsage: 2968 - Virtual Memory Usage - VirtualSize: 906211328, PeakVirtualSize: 906211328 - Pagefile Usage - PagefileUsage: 33853440, PeakPagefileUsage: 107077632 - Working Set Size - WorkingSetSize: 4112384, PeakWorkingSetSize: 4112384, PageFaultCount: 1007 * Dump of thread ID 89100 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156001.000000, User Time: 156001.000000, Wait Time: 53774256.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C8FC16 - Registers - eax=1132f9c4 ebx=00000001 ecx=00000003 edx=00000000 esi=009fca88 edi=fffffffe eip=76c8fc16 esp=1132f9c4 ebp=1132fa14 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1132fa14 007bfa90 e06d7363 00000001 00000003 1132fa40 kernel32!RaiseException+0x0 1132fa4c 007bbbcb 1132fa5c 00860978 0082f1b4 0085d354 hadcm3n_um_6.07_windows_intelx8!+0x0 1132fa68 00406e8c 04600000 01000000 00500000 004083ee hadcm3n_um_6.07_windows_intelx8!+0x0 1132fad8 00408130 1132faec 00072a41 1132faec 000729c9 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff88 76c9d2e9 7ffdc000 1132ffd4 777b1603 7ffdc000 hadcm3n_um_6.07_windows_intelx8!+0x0 1132ff94 777b1603 7ffdc000 54b1e2f9 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 1132ffd4 777b15d6 007bde7f 7ffdc000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1132ffec 00000000 007bde7f 7ffdc000 00000000 00000030 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 84688 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 53774256.000000 - Registers - eax=0000003c ebx=00000000 ecx=00000069 edx=00000001 esi=3426ff48 edi=00000000 eip=777d5cd4 esp=3426ff04 ebp=3426ff6c cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3426ff6c 76c51c6c 00000064 00000000 3426ff88 00749924 ntdll!KiFastSystemCallRet+0x0 3426ff7c 00749924 00000064 3426ff94 76c9d2e9 00000000 kernel32!Sleep+0x0 3426ff88 76c9d2e9 00000000 3426ffd4 777b1603 00000000 hadcm3n_um_6.07_windows_intelx8!+0x0 3426ff94 777b1603 00000000 71a5e2f9 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 3426ffd4 777b15d6 00749910 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3426ffec 00000000 00749910 00000000 00000000 34780000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=88184, iMonCtr=1 Model crash detected, will try to restart... Could not launch model process. Last Error=1455 Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 May 2013 05:40:17	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	336,960	600,864	1.7832
09 May 2013 15:25:53	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	311,040	556,040	1.7877
09 May 2013 00:08:36	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	285,120	511,456	1.7938
08 May 2013 10:09:35	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	259,200	466,725	1.8006
07 May 2013 21:25:22	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	233,280	422,160	1.8097
07 May 2013 05:45:54	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	207,360	375,697	1.8118
06 May 2013 14:55:32	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	181,440	328,754	1.8119
06 May 2013 01:41:35	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	155,520	282,392	1.8158
04 May 2013 06:00:29	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	129,600	235,282	1.8154
03 May 2013 17:06:38	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	103,680	188,255	1.8157
03 May 2013 02:46:52	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	77,760	141,503	1.8197
02 May 2013 13:02:42	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	51,840	93,574	1.8051
01 May 2013 23:28:11	969347	15757829	hadcm3n_3lbj_1940_40_008260115_2	25,920	48,235	1.8609