Task 17508727

Name	hadam3p_eu_z34a_2013_1_009237985_0
Workunit	9350811
Created	27 Nov 2014, 17:59:01 UTC
Sent	27 Nov 2014, 18:02:45 UTC
Report deadline	9 Nov 2015, 23:22:45 UTC
Received	29 Nov 2014, 14:24:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1346657
Run time	22 hours 19 min 22 sec
CPU time	21 hours 54 min 42 sec
Validate state	Invalid
Credit	796.57
Device peak FLOPS	4.96 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>7.4.27</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8672, selfPID=8672, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2280, selfPID=2280, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2596, selfPID=2596, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7100, selfPID=7100, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2564, selfPID=2564, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7612C42D Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 11/29/14 08:06:02 Install Directory : Data Directory : D:\boinc Project Symstore : LoadLibraryA( D:\boinc\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( D:\boinc\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( D:\boinc\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( D:\boinc\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: D:\boinc\projects\climateprediction.net\hadam3p_eu_z34a_2013_1_009237985;D:\boinc\projects\climateprediction.net ModLoad: 00b60000 008d3000 D:\boinc\projects\climateprediction.net\hadrm3p_eu_um_6.09_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : d:\cpdn\cpdnboinc\cpdnprecis\test\projects\reqs.comlab.ox.ac.uk_cpdn\hadrm3p_eu_um_6.09_windows_intelx86.pdb ModLoad: 777a0000 00180000 C:\Windows\SysWOW64\ntdll.dll (6.1.7601.18247) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 76270000 00110000 C:\Windows\syswow64\kernel32.dll (6.1.7601.18409) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Version : 6.1.7601.18409 (win7sp1_gdr.140303-2144) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.18409 ModLoad: 76120000 00047000 C:\Windows\syswow64\KERNELBASE.dll (6.1.7601.18409) (-exported- Symbols Loaded) Linked PDB Filename : wkernelbase.pdb File Version : 6.1.7601.18409 (win7sp1_gdr.140303-2144) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.18409 ModLoad: 76170000 00100000 C:\Windows\syswow64\USER32.dll (6.1.7601.17514) (-exported- Symbols Loaded) Linked PDB Filename : wuser32.pdb File Version : 6.1.7601.17514 (win7sp1_rtm.101119-1850) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.17514 ModLoad: 75e20000 00090000 C:\Windows\syswow64\GDI32.dll (6.1.7601.18577) (-nosymbols- Symbols Loaded) Linked PDB Filename : wgdi32.pdb File Version : 6.1.7601.18577 (win7sp1_gdr.140822-1508) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.18577 ModLoad: 75640000 0000a000 C:\Windows\syswow64\LPK.dll (6.1.7601.18177) (-nosymbols- Symbols Loaded) Linked PDB Filename : wlpk.pdb File Version : 6.1.7601.18177 (win7sp1_gdr.130605-1534) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.18177 ModLoad: 75690000 0009d000 C:\Windows\syswow64\USP10.dll (1.626.7601.18454) (-nosymbols- Symbols Loaded) Linked PDB Filename : usp10.pdb File Version : 1.0626.7601.18454 (win7sp1_gdr.140424-1533) Company Name : Microsoft Corporation Product Name : Microsoft(R) Uniscribe Unicode script processor Product Version : 1.0626.7601.18454 ModLoad: 75730000 000ac000 C:\Windows\syswow64\msvcrt.dll (7.0.7601.17744) (-nosymbols- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.7601.17744 (win7sp1_gdr.111215-1535) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.7601.17744 ModLoad: 75990000 000a0000 C:\Windows\syswow64\ADVAPI32.dll (6.1.7601.18247) (-nosymbols- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 758d0000 00019000 C:\Windows\SysWOW64\sechost.dll (6.1.7600.16385) (-nosymbols- Symbols Loaded) Linked PDB Filename : sechost.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 765a0000 000f0000 C:\Windows\syswow64\RPCRT4.dll (6.1.7601.18532) (-nosymbols- Symbols Loaded) Linked PDB Filename : wrpcrt4.pdb File Version : 6.1.7600.16385 (win7_rtm.090713-1255) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7600.16385 ModLoad: 75140000 00060000 C:\Windows\syswow64\SspiCli.dll (6.1.7601.18637) (-nosymbols- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 6.1.7601.18637 (win7sp1_gdr.141013-1517) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.1.7601.18637 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 2776 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7612C42D - Registers - eax=118afb8c ebx=7efde000 ecx=00000003 edx=00000000 esi=00fe4e48 edi=00000000 eip=7612c42d esp=118afb8c ebp=118afbdc cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 118afbdc 00e81793 e06d7363 00000001 00000003 118afc08 KERNELBASE!RaiseException+0x0 118afc14 00e7e9b5 118afc24 00f1cee0 00eee0dc 00f19214 hadrm3p_eu_um_6.09_windows_inte!+0x0 118afc30 00b68322 01000000 00b6985b 00000000 118afc30 hadrm3p_eu_um_6.09_windows_inte!+0x0 118afc80 00e80893 00000002 00531a20 00531aa8 af92f5f7 hadrm3p_eu_um_6.09_windows_inte!+0x0 118afcc8 7628338a 7efde000 118afd14 777d9f72 7efde000 hadrm3p_eu_um_6.09_windows_inte!+0x0 118afcd4 777d9f72 7efde000 65e6a95a 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 118afd14 777d9f45 00e808ea 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 118afd2c 00000000 00e808ea 7efde000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 2416 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Registers - eax=00e0b2e0 ebx=00000000 ecx=00000000 edx=00000000 esi=34ccfcdc edi=00000000 eip=777bfd91 esp=34ccfc98 ebp=34ccfd00 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246 - Callstack - ChildEBP RetAddr Args to Child 34ccfd00 761344a5 00000064 00000000 34ccfd1c 00e0b2f4 ntdll!ZwDelayExecution+0x0 34ccfd10 00e0b2f4 00000064 34ccfd28 7628338a 00000000 KERNELBASE!Sleep+0x0 34ccfd1c 7628338a 00000000 34ccfd68 777d9f72 00000000 hadrm3p_eu_um_6.09_windows_inte!+0x0 34ccfd28 777d9f72 00000000 40a0a926 00000000 00000000 kernel32!BaseThreadInitThunk+0x0 34ccfd68 777d9f45 00e0b2e0 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 34ccfd80 00000000 00e0b2e0 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Signal 11 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6648, iMonCtr=2 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_5.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_6.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_7.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_z34a_2013_1_009237985_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
29 Nov 2014 08:39:12	1346657	17508727	hadam3p_eu_z34a_2013_1_009237985_0	46,176	63,846	1.3827
29 Nov 2014 03:36:00	1346657	17508727	hadam3p_eu_z34a_2013_1_009237985_0	34,656	48,156	1.3895
28 Nov 2014 21:30:53	1346657	17508727	hadam3p_eu_z34a_2013_1_009237985_0	23,136	32,101	1.3875
28 Nov 2014 07:28:54	1346657	17508727	hadam3p_eu_z34a_2013_1_009237985_0	11,616	16,069	1.3834