Task 19170189

Name	hadam3p_pnw_xgc7_200912_12_010211319_2
Workunit	10211319
Created	20 Dec 2015, 13:47:18 UTC
Sent	20 Dec 2015, 13:47:58 UTC
Report deadline	1 Dec 2016, 19:07:58 UTC
Received	4 Jan 2016, 13:12:51 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	9 (0x00000009) Unknown error code
Computer ID	1303097
Run time	1 days 19 hours 33 min 41 sec
CPU time	1 days 16 hours 12 min 36 sec
Validate state	Invalid
Credit	1,508.39
Device peak FLOPS	4.17 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Pacific North West v7.27 windows_intelx86
Stderr	<core_client_version>7.6.9</core_client_version> <![CDATA[ <message> - exit code 9 (0x9) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x0168B3AC Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 7.5.1 Dump Timestamp : 12/25/15 17:59:29 Install Directory : Data Directory : C:\ProgramData\BOINC Project Symstore : LoadLibraryA( C:\ProgramData\BOINC\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( C:\ProgramData\BOINC\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( C:\ProgramData\BOINC\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\slots\2;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 0000000000400000 000000000018e000 C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_pnw_7.27_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 00000000775e0000 000000000017b000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.2.10586.20) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000077320000 00000000000e0000 C:\WINDOWS\SYSTEM32\KERNEL32.DLL (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 00000000766b0000 000000000017e000 C:\WINDOWS\SYSTEM32\KERNELBASE.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : wkernelbase.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000074070000 0000000000092000 C:\WINDOWS\system32\apphelp.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : apphelp.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000076260000 0000000000147000 C:\WINDOWS\SYSTEM32\USER32.dll (6.2.10586.20) (-exported- Symbols Loaded) Linked PDB Filename : wuser32.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000076980000 000000000014f000 C:\WINDOWS\SYSTEM32\GDI32.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : wgdi32.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000075860000 000000000007b000 C:\WINDOWS\SYSTEM32\ADVAPI32.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000076420000 00000000000be000 C:\WINDOWS\SYSTEM32\msvcrt.dll (7.0.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 7.0.10586.0 ModLoad: 0000000076830000 0000000000044000 C:\WINDOWS\SYSTEM32\sechost.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : sechost.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000076880000 00000000000ad000 C:\WINDOWS\SYSTEM32\RPCRT4.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : wrpcrt4.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000074310000 000000000001e000 C:\WINDOWS\SYSTEM32\SspiCli.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000074300000 000000000000a000 C:\WINDOWS\SYSTEM32\CRYPTBASE.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : cryptbase.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000075e10000 0000000000058000 C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : bcryptprimitives.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000074200000 0000000000028000 C:\WINDOWS\SYSTEM32\ntmarta.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : ntmarta.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 000000006a0c0000 0000000000071000 C:\WINDOWS\SYSTEM32\efswrt.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : efswrt.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000076b80000 000000000008d000 C:\WINDOWS\SYSTEM32\SHCORE.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : shcore.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 00000000764f0000 00000000001bd000 C:\WINDOWS\SYSTEM32\combase.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : combase.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 00000000764e0000 000000000000c000 C:\WINDOWS\SYSTEM32\kernel.appcore.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : Kernel.Appcore.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 Get Product Name Failed. ModLoad: 00000000757c0000 0000000000092000 C:\WINDOWS\SYSTEM32\OLEAUT32.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : oleaut32.pdb File Version : 10.0.10586.0 Company Name : Microsoft Corporation Product Name : Product Version : 10.0.10586.0 ModLoad: 0000000069ff0000 00000000000c8000 C:\WINDOWS\SYSTEM32\wintypes.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : WinTypes.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000074020000 0000000000049000 C:\WINDOWS\SYSTEM32\edputil.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : edputil.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000076930000 0000000000045000 C:\WINDOWS\SYSTEM32\shlwapi.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : shlwapi.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 0000000069e30000 000000000013f000 C:\WINDOWS\SYSTEM32\dbghelp.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 ModLoad: 00000000742f0000 0000000000008000 C:\WINDOWS\SYSTEM32\version.dll (6.2.10586.0) (-exported- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 10.0.10586.0 (th2_release.151029-1700) Company Name : Microsoft Corporation Product Name : MicrosoftR WindowsR Operating System Product Version : 10.0.10586.0 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 18090, Write: 0, Other 63148 - I/O Transfers Counters - Read: 0, Write: 3423, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 111592, QuotaPeakPagedPoolUsage: 127200 QuotaNonPagedPoolUsage: 9088, QuotaPeakNonPagedPoolUsage: 9904 - Virtual Memory Usage - VirtualSize: 64200704, PeakVirtualSize: 72454144 - Pagefile Usage - PagefileUsage: 2969600, PeakPagefileUsage: 3428352 - Working Set Size - WorkingSetSize: 31424512, PeakWorkingSetSize: 31735808, PageFaultCount: 9942 * Dump of thread ID 7776 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 26777830.000000 - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x00462B87 read attempt to address 0x0168B3AC - Registers - eax=00000015 ebx=fffffffe ecx=00000000 edx=00000000 esi=015bb020 edi=00000000 eip=00462b87 esp=009ffe7c ebp=009ffe80 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010216 - Callstack - ChildEBP RetAddr Args to Child 009ffe80 00447d6f 015bb020 00000000 015bb020 fffffffe hadam3p_pnw_7.27_windows_intelx!+0x0 009ffe98 0044f568 015bb020 00000000 005597a4 009ffee8 hadam3p_pnw_7.27_windows_intelx!+0x0 009ffeac 0048374c 00000015 2d521505 00000006 004836bc hadam3p_pnw_7.27_windows_intelx!+0x0 009ffee8 767ca9ff 00000006 a662943c 767ca840 767ca840 hadam3p_pnw_7.27_windows_intelx!+0x0 009fff80 773338f4 80000006 773338d0 1bbd3849 009fffdc KERNELBASE!CtrlRoutine+0x0 009fff94 776456c3 80000006 278ac9ad 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 009fffdc 7764568e ffffffff 7766b6dd 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 009fffec 00000000 767ca840 80000006 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 * Dump of thread ID 3048 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 15468750.000000, User Time: 39531248.000000, Wait Time: 26777820.000000 - Registers - eax=00000000 ebx=0019f988 ecx=00000000 edx=00000000 esi=00000000 edi=00001219 eip=77656e3c esp=0019f948 ebp=0019f9ac cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 0019f9ac 7676a4ef 000000c8 00000000 0019f9d0 004689c1 ntdll!NtDelayExecution+0x0 0019f9bc 004689c1 000000c8 000000c8 00000000 0019fa1c KERNELBASE!Sleep+0x0 0019f9d0 00403970 a0000000 027f9999 00000000 004cf841 hadam3p_pnw_7.27_windows_intelx!+0x0 0019fa1c 00403dcd 00000001 004cf858 004cfb88 00000000 hadam3p_pnw_7.27_windows_intelx!+0x0 0019fa50 00404d76 00000000 2dd416ad 00484e6e 00000000 hadam3p_pnw_7.27_windows_intelx!+0x0 0019fd40 00404f67 0019fef0 00404fe5 004d02a8 0019fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 0019fd48 00404fe5 004d02a8 0019fd60 00591fe8 0019fd60 hadam3p_pnw_7.27_windows_intelx!+0x0 0019fef0 00484e1b 00400000 00000000 0059202d 0000000a hadam3p_pnw_7.27_windows_intelx!+0x0 0019ff80 773338f4 0027d000 773338d0 1b3b3849 0019ffdc hadam3p_pnw_7.27_windows_intelx!+0x0 0019ff94 776456c3 0027d000 270cc9ad 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 0019ffdc 7764568e ffffffff 7766b6dd 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 0019ffec 00000000 00484e6e 0027d000 00000000 78746341 ntdll!RtlUnicodeStringToInteger+0x0 * Dump of thread ID 6784 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 26777824.000000 - Registers - eax=c0000022 ebx=015afd1c ecx=00000000 edx=00000000 esi=00000000 edi=00000001 eip=77656e3c esp=015afcdc ebp=015afd40 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 015afd40 7676a4ef 00000064 00000000 015aff80 00477e94 ntdll!NtDelayExecution+0x0 015afd50 00477e94 00000064 00477e70 00477e70 00000000 KERNELBASE!Sleep+0x0 015aff80 773338f4 00000000 773338d0 1a783849 015affdc hadam3p_pnw_7.27_windows_intelx!+0x0 015aff94 776456c3 00000000 264fc9ad 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 015affdc 7764568e ffffffff 7766b6dd 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 015affec 00000000 00477e70 00000000 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 * Dump of thread ID 2044 (state: Waiting): * - Information - Status: Wait Reason: EventPairLow, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 26777816.000000 - Registers - eax=00000000 ebx=0059e0c0 ecx=00000000 edx=00000000 esi=0059df00 edi=00592798 eip=7765868c esp=008ffdc8 ebp=008fff80 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 008fff80 773338f4 00592798 773338d0 1bad3849 008fffdc ntdll!ZwWaitForWorkViaWorkerFactory+0x0 008fff94 776456c3 00592798 279ac9ad 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 008fffdc 7764568e ffffffff 7766b6dd 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 008fffec 00000000 77629fa0 00592798 00000000 00000000 ntdll!RtlUnicodeStringToInteger+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:09:55 (3336): start_timer_thread(): CreateThread() failed, errno 0 01:09:56 (3276): start_timer_thread(): CreateThread() failed, errno 0 CPDN Monitor - Quit request from BOINC... 12:41:14 (8640): start_timer_thread(): CreateThread() failed, errno 0 12:41:15 (8504): start_timer_thread(): CreateThread() failed, errno 0 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Global Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8640, selfPID=8640, iMonCtr=1 No Process Handle Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=8640, selfPID=8504, iMonCtr=1 </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
03 Jan 2016 07:25:28	1303097	19170189	hadam3p_pnw_xgc7_200912_12_010211319_2	69,419	126,353	1.8202
02 Jan 2016 14:03:01	1303097	19170189	hadam3p_pnw_xgc7_200912_12_010211319_2	57,899	105,553	1.8231
01 Jan 2016 23:36:04	1303097	19170189	hadam3p_pnw_xgc7_200912_12_010211319_2	46,379	84,112	1.8136
30 Dec 2015 16:33:08	1303097	19170189	hadam3p_pnw_xgc7_200912_12_010211319_2	34,859	62,673	1.7979
30 Dec 2015 07:51:27	1303097	19170189	hadam3p_pnw_xgc7_200912_12_010211319_2	23,339	41,632	1.7838
27 Dec 2015 12:46:53	1303097	19170189	hadam3p_pnw_xgc7_200912_12_010211319_2	11,819	20,454	1.7306