Task 15955613

Name	hadcm3n_n2f3_1920_40_008378184_1
Workunit	8529043
Created	29 Aug 2013, 22:53:49 UTC
Sent	29 Aug 2013, 22:56:27 UTC
Report deadline	29 Nov 2013, 6:23:38 UTC
Received	27 Sep 2013, 3:07:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	-1073741819 (0xC0000005) STATUS_ACCESS_VIOLATION
Computer ID	903597
Run time	16 days 2 hours 11 min 5 sec
CPU time	12 days 17 hours 53 min 30 sec
Validate state	Invalid
Credit	6,220.80
Device peak FLOPS	2.27 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> (unknown error) - exit code -1073741819 (0xc0000005) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:30:24 (4344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6932, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:33:51 (3836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 19:14:02 (4452): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:27 (5648): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:29:52 (5268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:30:45 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:30:46 (5108): No heartbeat from core client for 30 sec - exiting 01:30:47 (5108): No heartbeat from core client for 30 sec - exiting 01:30:48 (5108): No heartbeat from core client for 30 sec - exiting 01:30:49 (5108): No heartbeat from core client for 30 sec - exiting 01:30:50 (5108): No heartbeat from core client for 30 sec - exiting 01:30:51 (5108): No heartbeat from core client for 30 sec - exiting 01:30:52 (5108): No heartbeat from core client for 30 sec - exiting 01:30:53 (5108): No heartbeat from core client for 30 sec - exiting 01:30:55 (5108): No heartbeat from core client for 30 sec - exiting 01:30:56 (5108): No heartbeat from core client for 30 sec - exiting 01:30:57 (5108): No heartbeat from core client for 30 sec - exiting 01:30:58 (5108): No heartbeat from core client for 30 sec - exiting 18:25:25 (3784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:00:40 (7984): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7776AFDD read attempt to address 0x408E69F2 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 09/27/13 05:06:01 Install Directory : Data Directory : D:\Programme\BOINCdaten Project Symstore : LoadLibraryA( D:\Programme\BOINCdaten\dbghelp.dll ): GetLastError = 126 Loaded Library : dbghelp.dll LoadLibraryA( D:\Programme\BOINCdaten\symsrv.dll ): GetLastError = 126 LoadLibraryA( symsrv.dll ): GetLastError = 126 LoadLibraryA( D:\Programme\BOINCdaten\srcsrv.dll ): GetLastError = 126 LoadLibraryA( srcsrv.dll ): GetLastError = 126 LoadLibraryA( D:\Programme\BOINCdaten\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: D:\Programme\BOINCdaten\projects\climateprediction.net\hadcm3n_n2f3_1920_40_008378184;D:\Programme\BOINCdaten\projects\climateprediction.net ModLoad: 00400000 00111000 D:\Programme\BOINCdaten\projects\climateprediction.net\hadcm3n_6.07_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 77740000 00157000 C:\Windows\SYSTEM32\ntdll.dll (6.2.9200.16578) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 76740000 00130000 C:\Windows\SYSTEM32\KERNEL32.DLL (6.2.9200.16627) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 75130000 000a6000 C:\Windows\SYSTEM32\KERNELBASE.dll (6.2.9200.16451) (-exported- Symbols Loaded) Linked PDB Filename : wkernelbase.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 751e0000 00116000 C:\Windows\SYSTEM32\USER32.dll (6.2.9200.16420) (-exported- Symbols Loaded) Linked PDB Filename : wuser32.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 76870000 000ae000 C:\Windows\SYSTEM32\ADVAPI32.dll (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 76bf0000 000fd000 C:\Windows\SYSTEM32\GDI32.dll (6.2.9200.16654) (-exported- Symbols Loaded) Linked PDB Filename : wgdi32.pdb File Version : 6.2.9200.16654 (win8_gdr.130628-1505) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16654 ModLoad: 75070000 000b1000 C:\Windows\SYSTEM32\msvcrt.dll (7.0.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.9200.16384 ModLoad: 76940000 00034000 C:\Windows\SYSTEM32\sechost.dll (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : sechost.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16384 ModLoad: 75500000 000ac000 C:\Windows\SYSTEM32\RPCRT4.dll (6.2.9200.16622) (-exported- Symbols Loaded) Linked PDB Filename : wrpcrt4.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 74d70000 0001c000 C:\Windows\SYSTEM32\SspiCli.dll (6.2.9200.16420) (-exported- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 6.2.9200.16420 (win8_gdr.120919-1813) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16420 ModLoad: 74d60000 00009000 C:\Windows\SYSTEM32\CRYPTBASE.dll (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : cryptbase.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16384 ModLoad: 74d00000 00051000 C:\Windows\SYSTEM32\bcryptPrimitives.dll (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : bcryptprimitives.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16384 ModLoad: 76920000 00020000 C:\Windows\system32\IMM32.DLL (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : wimm32.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16384 ModLoad: 77460000 000de000 C:\Windows\SYSTEM32\MSCTF.dll (6.2.9200.16578) (-exported- Symbols Loaded) Linked PDB Filename : msctf.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 73c10000 00026000 C:\Windows\SYSTEM32\ntmarta.dll (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : ntmarta.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Betriebssystem Microsoft® Windows® Product Version : 6.2.9200.16384 ModLoad: 726c0000 00129000 C:\Windows\SYSTEM32\dbghelp.dll (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16384 ModLoad: 74b00000 00008000 C:\Windows\SYSTEM32\version.dll (6.2.9200.16384) (-exported- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.2.9200.16384 (win8_rtm.120725-1247) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.2.9200.16384 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 6972 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Access Violation (0xc0000005) at address 0x7776AFDD read attempt to address 0x408E69F2 - Registers - eax=00002281 ebx=002e08f0 ecx=02520ce8 edx=408e6996 esi=02520ce8 edi=02520d08 eip=7776afdd esp=001226a4 ebp=001226a4 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010206 - Callstack - ChildEBP RetAddr Args to Child 001226a4 7781a628 408e6996 02520ce8 00122714 777d11bc ntdll!RtlGetUserPreferredUILanguages+0x0 001226b4 777d11bc 408e6996 02520ce8 00000004 01de0000 ntdll!RtlpNtMakeTemporaryKey+0x0 00122714 77783711 00000004 00000000 77783655 00000004 ntdll!RtlCreateUserThread+0x0 001227a0 00474314 01de0000 00000000 00000004 00122834 ntdll!RtlAllocateHeap+0x0 001227c0 0047482a 00000004 ffffffff 001229ac 00000000 hadcm3n_6.07_windows_intelx86!+0x0 001227d8 00401e06 00000004 00122834 001229b0 00122afc hadcm3n_6.07_windows_intelx86!+0x0 00000000 00000000 00000000 00000000 00000000 00000000 hadcm3n_6.07_windows_intelx86!+0x0 * Dump of thread ID 7476 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Registers - eax=00000000 ebx=01f2ff44 ecx=00000000 edx=00000000 esi=01f2ff44 edi=00000000 eip=7776df34 esp=01f2ff04 ebp=01f2ff68 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 01f2ff68 7513119c 00000064 00000000 01f2ff84 00413da4 ntdll!ZwDelayExecution+0x0 01f2ff78 00413da4 00000064 01f2ff90 7676850d 00000000 KERNELBASE!Sleep+0x0 01f2ff84 7676850d 00000000 01f2ffd4 7779bf39 00000000 hadcm3n_6.07_windows_intelx86!+0x0 01f2ff90 7779bf39 00000000 a2cc841e 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 01f2ffd4 7779bf0c 00413d90 00000000 ffffffff 77811df7 ntdll!RtlInitializeExceptionChain+0x0 01f2ffec 00000000 00413d90 00000000 00000000 63646168 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Sep 2013 03:10:34	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	518,400	1,101,204	2.1242
25 Sep 2013 21:42:46	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	492,480	1,044,449	2.1208
23 Sep 2013 18:01:58	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	466,560	991,439	2.1250
21 Sep 2013 20:21:13	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	440,640	939,485	2.1321
21 Sep 2013 01:53:13	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	414,720	888,635	2.1427
19 Sep 2013 21:19:48	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	388,800	838,552	2.1568
18 Sep 2013 18:02:36	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	362,880	787,654	2.1706
15 Sep 2013 23:18:08	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	336,960	733,607	2.1771
15 Sep 2013 17:41:37	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	311,040	687,140	2.2092
14 Sep 2013 12:29:00	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	285,120	635,969	2.2305
11 Sep 2013 23:25:48	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	259,200	584,850	2.2564
10 Sep 2013 20:37:33	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	233,280	532,470	2.2825
08 Sep 2013 20:50:18	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	207,360	475,339	2.2923
07 Sep 2013 21:22:58	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	181,440	410,287	2.2613
06 Sep 2013 23:26:47	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	155,520	345,426	2.2211
04 Sep 2013 18:04:33	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	129,600	280,522	2.1645
03 Sep 2013 01:37:36	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	103,680	222,690	2.1479
01 Sep 2013 21:44:44	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	77,760	166,033	2.1352
01 Sep 2013 01:06:41	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	51,840	111,898	2.1585
31 Aug 2013 05:28:26	903597	15955613	hadcm3n_n2f3_1920_40_008378184_1	25,920	56,383	2.1753