Task 20151495

Name	wah2_afr50_t5yg_201412_13_502_010849833_0
Workunit	10849833
Created	25 Jan 2017, 16:05:51 UTC
Sent	26 Jan 2017, 2:39:21 UTC
Report deadline	8 Jan 2018, 7:59:21 UTC
Received	31 Jan 2017, 1:30:06 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-529697949 (0xE06D7363) Unknown error code
Computer ID	1372066
Run time	4 days 21 hours 48 min 53 sec
CPU time	4 days 18 hours 42 min 46 sec
Validate state	Invalid
Credit	9,138.96
Device peak FLOPS	3.45 GFLOPS
Application version	Weather At Home 2 (wah2) v8.24 windows_intelx86
Peak working set size	215.80 MB
Peak swap size	183.71 MB
Peak disk usage	63.18 MB
Stderr	<core_client_version>7.6.6</core_client_version> <![CDATA[ <message> (unknown error) - exit code -529697949 (0xe06d7363) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C9A832 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 7.7.0 Dump Timestamp : 01/30/17 16:30:00 Install Directory : Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\wah2_afr50_t5yg_201412_13_502_010849833;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 0000000000400000 0000000000152000 C:\ProgramData\BOINC\projects\climateprediction.net\wah2_8.24_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 0000000077190000 0000000000183000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.14393.479) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb File Version : 10.0.14393.206 (rs1_release.160915-0644) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.206 ModLoad: 0000000076590000 00000000000e0000 C:\WINDOWS\System32\KERNEL32.DLL (6.2.14393.0) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Version : 10.0.14393.206 (rs1_release.160915-0644) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.206 ModLoad: 0000000076bc0000 00000000001a1000 C:\WINDOWS\System32\KERNELBASE.dll (6.2.14393.479) (-nosymbols- Symbols Loaded) Linked PDB Filename : wkernelbase.pdb File Version : 10.0.14393.206 (rs1_release.160915-0644) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.206 ModLoad: 0000000076670000 000000000015f000 C:\WINDOWS\System32\USER32.dll (6.2.14393.576) (-exported- Symbols Loaded) Linked PDB Filename : wuser32.pdb File Version : 10.0.14393.0 (rs1_release.160715-1616) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.0 ModLoad: 0000000073c50000 0000000000015000 C:\WINDOWS\System32\win32u.dll (6.2.14393.51) (-exported- Symbols Loaded) Linked PDB Filename : wwin32u.pdb File Version : 10.0.14393.51 (rs1_release_inmarket.160801-1836) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.51 ModLoad: 00000000754a0000 000000000002b000 C:\WINDOWS\System32\GDI32.dll (6.2.14393.206) (-exported- Symbols Loaded) Linked PDB Filename : wgdi32.pdb File Version : 10.0.14393.206 (rs1_release.160915-0644) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.206 ModLoad: 0000000076f90000 000000000015b000 C:\WINDOWS\System32\gdi32full.dll (6.2.14393.576) (-nosymbols- Symbols Loaded) Linked PDB Filename : wgdi32full.pdb File Version : 10.0.14393.576 (rs1_release_inmarket.161208-2252) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.576 ModLoad: 0000000076830000 0000000000077000 C:\WINDOWS\System32\ADVAPI32.dll (6.2.14393.0) (-nosymbols- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 10.0.14393.0 (rs1_release.160715-1616) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.0 ModLoad: 00000000760c0000 00000000000be000 C:\WINDOWS\System32\msvcrt.dll (7.0.14393.0) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.14393.0 (rs1_release.160715-1616) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.14393.0 ModLoad: 0000000076d70000 0000000000041000 C:\WINDOWS\System32\sechost.dll (6.2.14393.0) (-exported- Symbols Loaded) Linked PDB Filename : sechost.pdb File Version : 10.0.14393.0 (rs1_release.160715-1616) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.0 ModLoad: 0000000075ff0000 00000000000c1000 C:\WINDOWS\System32\RPCRT4.dll (6.2.14393.82) (-exported- Symbols Loaded) Linked PDB Filename : wrpcrt4.pdb File Version : 10.0.14393.0 (rs1_release.160715-1616) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.0 ModLoad: 0000000073c30000 000000000001e000 C:\WINDOWS\System32\SspiCli.dll (6.2.14393.576) (-nosymbols- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 10.0.14393.576 (rs1_release_inmarket.161208-2252) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 10.0.14393.576 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 193310, Write: 0, Other 325805 - I/O Transfers Counters - Read: 0, Write: 18503, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 238936, QuotaPeakPagedPoolUsage: 238936 QuotaNonPagedPoolUsage: 8952, QuotaPeakNonPagedPoolUsage: 9352 - Virtual Memory Usage - VirtualSize: 131067904, PeakVirtualSize: 144945152 - Pagefile Usage - PagefileUsage: 3215360, PeakPagefileUsage: 17678336 - Working Set Size - WorkingSetSize: 77434880, PeakWorkingSetSize: 95236096, PageFaultCount: 139550 * Dump of thread ID 5832 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 92500000.000000, User Time: 952812480.000000, Wait Time: 44501984.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C9A832 - Registers - eax=0019e690 ebx=06d70048 ecx=00000003 edx=00000000 esi=5de1922c edi=5de2c450 eip=76c9a832 esp=0019e690 ebp=0019e6ec cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000216 - Callstack - ChildEBP RetAddr Args to Child 0019e6ec 5ddfcaf9 e06d7363 00000001 00000003 0019e718 KERNELBASE!+0x0 0019e724 5ddf9ebc 0019e734 5de259d8 5de1922c 5de1b27c KERNELBASE!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '5ddfcaf9' 0019e744 5ddbe42b 002ce004 06d70048 004e4d99 06f36c00 KERNELBASE!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '5ddf9ebc' 0019e760 5ddb29b1 0019e798 0019e884 0019e86c 0019e860 KERNELBASE!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '5ddbe42b' 0019e9a8 5ddbefac 06f32740 0019ea3d 00000000 0019eb44 KERNELBASE!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '5ddb29b1' 0019eb44 0040bdf2 004e4d98 004e4ed0 004e4c90 00000000 KERNELBASE!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '5ddbefac' 0019eb7c 00401f39 02680ab8 02680af0 02680af0 00000000 wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '0040bdf2' 0019ebd0 004021ad 000007df 0000000c 00000000 004cefe0 wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '00401f39' 0019ed98 00402703 000007df 0000000c 02780040 0019edf8 wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '004021ad' 0019edac 0040724e 000007df 02780040 004cefe0 004a56b0 wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '00402703' 0019edf8 004092c5 00000000 9f9c05b1 00485cce 00000000 wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '0040724e' 0019fd40 004094b7 0019fef0 00409535 004d1370 0019fd60 wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '004092c5' 0019fd48 00409535 004d1370 0019fd60 005f3b38 0019fd60 wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '004094b7' 0019fef0 00485c7b 00400000 00000000 005f3b76 0000000a wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '00409535' 0019ff80 765a62c4 0021a000 765a62a0 e9b4f780 0019ffdc wah2_8.24_windows_intelx86!+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '00485c7b' 0019ff94 771f0fd9 0021a000 e8fa888f 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '765a62c4' 0019ffdc 771f0fa4 ffffffff 77212f06 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '771f0fd9' 0019ffec 00000000 00485cce 0021a000 00000000 78746341 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 SymFromAddr(): GetLastError = '126' SymGetLineFromAddr(): GetLastError = '126' SymGetModuleInfo(): GetLastError = '126' Address = '771f0fa4' * Dump of thread ID 1672 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 1093750.000000, User Time: 312500.000000, Wait Time: 44501980.000000 - Registers - eax=00000000 ebx=0259fd1c ecx=00000000 edx=00000000 esi=00000000 edi=0259fd1c eip=771fe91c esp=0259fcdc ebp=0259fd40 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 0259fd40 76c92bcf 00000064 00000000 0259ff80 00478b44 ntdll!NtDelayExecution+0x0 0259fd50 00478b44 00000064 00478b20 00478b20 00000000 KERNELBASE!+0x0 0259ff80 765a62c4 00000000 765a62a0 ebf4f780 0259ffdc wah2_8.24_windows_intelx86!+0x0 0259ff94 771f0fd9 00000000 eaba888f 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 0259ffdc 771f0fa4 ffffffff 77212f06 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 0259ffec 00000000 00478b20 00000000 00000000 00000000 ntdll!RtlSubscribeWnfStateChangeNotification+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14964, selfPID=14964, iMonCtr=1 No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=14964, selfPID=12020, iMonCtr=1 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
30 Jan 2017 15:08:43	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	138,539	379,862	2.7419
30 Jan 2017 05:47:40	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	127,019	346,646	2.7291
29 Jan 2017 20:14:49	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	115,499	313,362	2.7131
29 Jan 2017 10:55:01	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	103,979	280,426	2.6969
29 Jan 2017 02:12:41	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	92,459	249,708	2.7007
28 Jan 2017 16:53:05	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	80,939	216,919	2.6800
28 Jan 2017 07:40:58	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	69,419	184,286	2.6547
27 Jan 2017 22:54:50	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	57,899	153,373	2.6490
27 Jan 2017 13:33:10	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	46,379	122,694	2.6455
27 Jan 2017 04:58:28	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	34,859	92,268	2.6469
26 Jan 2017 20:19:22	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	23,339	61,810	2.6484
26 Jan 2017 11:36:10	1372066	20151495	wah2_afr50_t5yg_201412_13_502_010849833_0	11,819	31,397	2.6565