Task 18661935

Name	hadam3p_anz_u2bz_2013_1_010000719_0
Workunit	9999077
Created	3 Jul 2015, 10:34:42 UTC
Sent	7 Jul 2015, 10:20:56 UTC
Report deadline	18 Jun 2016, 15:40:56 UTC
Received	27 Jul 2015, 22:01:16 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	1254278
Run time	8 days 9 hours 53 min 6 sec
CPU time	5 days 21 hours 29 min 45 sec
Validate state	Invalid
Credit	3,490.64
Device peak FLOPS	1.56 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Australia New Zealand v6.10 windows_intelx86
Stderr	<core_client_version>7.2.47</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:00:02 (10744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:00:03 (10744): No heartbeat from core client for 30 sec - exiting 12:00:04 (10744): No heartbeat from core client for 30 sec - exiting 12:05:01 (11644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 00:00:36 (6440): No heartbeat from core client for 30 sec - exiting 00:00:37 (6440): No heartbeat from core client for 30 sec - exiting 00:00:38 (6440): No heartbeat from core client for 30 sec - exiting 00:00:39 (6440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 02:00:09 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:00:13 (6060): No heartbeat from core client for 30 sec - exiting 02:00:14 (6060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:24:48 (7784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:26:03 (15260): No heartbeat from core client for 30 sec - exiting 14:26:05 (15260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C44598 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 07/27/15 17:57:23 Install Directory : C:\Program Files (x86)\BOINC\ Data Directory : C:\ProgramData\BOINC Project Symstore : Loaded Library : C:\Program Files (x86)\BOINC\\dbghelp.dll Loaded Library : C:\Program Files (x86)\BOINC\\symsrv.dll Loaded Library : C:\Program Files (x86)\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files (x86)\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_anz_u2bz_2013_1_010000719;C:\ProgramData\BOINC\projects\climateprediction.net ModLoad: 000c0000 00904000 C:\ProgramData\BOINC\projects\climateprediction.net\hadam3p_anz_um_6.10_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : d:\cpdn\cpdnboinc\cpdnprecis\test\projects\reqs.comlab.ox.ac.uk_cpdn\hadam3p_anz_um_6.10_windows_intelx86.pdb ModLoad: 76f20000 0016e000 C:\WINDOWS\SYSTEM32\ntdll.dll (6.3.9600.17736) (-exported- Symbols Loaded) Linked PDB Filename : wntdll.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 74b20000 00140000 C:\WINDOWS\SYSTEM32\KERNEL32.DLL (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wkernel32.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 76c30000 000d7000 C:\WINDOWS\SYSTEM32\KERNELBASE.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wkernelbase.pdb File Version : 6.3.9600.17031 (winblue_gdr.140221-1952) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17031 ModLoad: 74d30000 00153000 C:\WINDOWS\SYSTEM32\USER32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wuser32.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 747a0000 0010f000 C:\WINDOWS\SYSTEM32\GDI32.dll (6.3.9600.17902) (-exported- Symbols Loaded) Linked PDB Filename : wgdi32.pdb File Version : 6.3.9600.17902 (winblue_ltsb.150609-2200) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17902 ModLoad: 74aa0000 0007c000 C:\WINDOWS\SYSTEM32\ADVAPI32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 65b10000 000de000 C:\WINDOWS\SYSTEM32\OPENGL32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : opengl32.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 68050000 00025000 C:\WINDOWS\SYSTEM32\GLU32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : glu32.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 74a70000 00014000 C:\WINDOWS\SYSTEM32\imagehlp.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : imagehlp.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 74c60000 000c3000 C:\WINDOWS\SYSTEM32\msvcrt.dll (7.0.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.9600.17415 ModLoad: 76590000 00041000 C:\WINDOWS\SYSTEM32\sechost.dll (6.3.9600.17734) (-exported- Symbols Loaded) Linked PDB Filename : sechost.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 74e90000 000ba000 C:\WINDOWS\SYSTEM32\RPCRT4.dll (6.3.9600.17919) (-exported- Symbols Loaded) Linked PDB Filename : wrpcrt4.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 65530000 000ec000 C:\WINDOWS\SYSTEM32\DDRAW.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : ddraw.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 74720000 0001e000 C:\WINDOWS\SYSTEM32\SspiCli.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wsspicli.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 691c0000 00007000 C:\WINDOWS\SYSTEM32\DCIMAN32.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : dciman32.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 74710000 0000a000 C:\WINDOWS\SYSTEM32\CRYPTBASE.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : cryptbase.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 746b0000 00054000 C:\WINDOWS\SYSTEM32\bcryptPrimitives.dll (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : bcryptprimitives.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 74a40000 00027000 C:\WINDOWS\system32\IMM32.DLL (6.3.9600.17415) (-exported- Symbols Loaded) Linked PDB Filename : wimm32.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 ModLoad: 76410000 00112000 C:\WINDOWS\SYSTEM32\MSCTF.dll (6.3.9600.17706) (-exported- Symbols Loaded) Linked PDB Filename : msctf.pdb File Version : 6.3.9600.16384 (winblue_rtm.130821-1623) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.16384 ModLoad: 62a30000 00115000 C:\Program Files (x86)\BOINC\dbghelp.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 666f0000 00048000 C:\Program Files (x86)\BOINC\symsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : symsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 666b0000 0003b000 C:\Program Files (x86)\BOINC\srcsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : srcsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 739d0000 00008000 C:\WINDOWS\SYSTEM32\version.dll (6.3.9600.17415) (-nosymbols- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 6.3.9600.17415 (winblue_r4.141028-1500) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 6.3.9600.17415 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 0, Write: 0, Other 0 - I/O Transfers Counters - Read: 0, Write: 0, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 0, QuotaPeakPagedPoolUsage: 0 QuotaNonPagedPoolUsage: 0, QuotaPeakNonPagedPoolUsage: 0 - Virtual Memory Usage - VirtualSize: 0, PeakVirtualSize: 0 - Pagefile Usage - PagefileUsage: 0, PeakPagefileUsage: 0 - Working Set Size - WorkingSetSize: 0, PeakWorkingSetSize: 0, PageFaultCount: 0 * Dump of thread ID 20812 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76C44598 - Registers - eax=1216f2c8 ebx=00000000 ecx=00000003 edx=00000000 esi=00572ac8 edi=1216f41c eip=76c44598 esp=1216f2c8 ebp=1216f320 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000212 - Callstack - ChildEBP RetAddr Args to Child 1216f320 0040f393 e06d7363 00000001 00000003 1216f34c KERNELBASE!RaiseException+0x0 1216f358 0040c595 1216f368 004a5c98 0047b7bc 004a1d74 hadam3p_anz_um_6.10_windows_int!+0x0 1216f374 000c9be3 01000000 fffffffe 00000000 00000001 hadam3p_anz_um_6.10_windows_int!+0x0 1216f3e8 000c987f 124432af 1216f3f8 12443191 124431ef hadam3p_anz_um_6.10_windows_int!+0x0 1216f894 74b37c04 7fa65000 74b37be0 a7624f27 1216f8f0 hadam3p_anz_um_6.10_windows_int!+0x0 1216f8a8 76f7ad1f 7fa65000 a4a4657c 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 1216f8f0 76f7acea ffffffff 76f6024e 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 1216f900 00000000 0040e41b 7fa65000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 * Dump of thread ID 20968 (state: Initialized): * - Information - Status: Base Priority: Normal, Priority: Normal, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 0.000000 - Registers - eax=003992f0 ebx=3881f99c ecx=00000000 edx=00000000 esi=00000000 edi=3881f99c eip=76f5c7bc esp=3881f95c ebp=3881f9c0 cs=0023 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 3881f9c0 76c3104f 00000064 00000000 3881f9dc 00399304 ntdll!NtDelayExecution+0x0 3881f9d0 00399304 00000064 3881f9f0 74b37c04 00000000 KERNELBASE!Sleep+0x0 3881f9dc 74b37c04 00000000 74b37be0 8df54e7f 3881fa38 hadam3p_anz_um_6.10_windows_int!+0x0 3881f9f0 76f7ad1f 00000000 8e3367b4 00000000 00000000 KERNEL32!BaseThreadInitThunk+0x0 3881fa38 76f7acea ffffffff 76f6024e 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 3881fa48 00000000 003992f0 00000000 00000000 00000000 ntdll!RtlInitializeExceptionChain+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=19040, selfPID=19040, iMonCtr=2 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=19040, selfPID=20744, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> upload failure: <file_xfer_error> <file_name>hadam3p_anz_u2bz_2013_1_010000719_0_8.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u2bz_2013_1_010000719_0_9.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u2bz_2013_1_010000719_0_10.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u2bz_2013_1_010000719_0_11.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_anz_u2bz_2013_1_010000719_0_12.zip</file_name> <error_code>-161 (not found)</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Jul 2015 00:26:06	1254278	18661935	hadam3p_anz_u2bz_2013_1_010000719_0	80,939	449,950	5.5591
21 Jul 2015 07:15:55	1254278	18661935	hadam3p_anz_u2bz_2013_1_010000719_0	69,419	386,855	5.5728
19 Jul 2015 10:41:51	1254278	18661935	hadam3p_anz_u2bz_2013_1_010000719_0	57,899	321,980	5.5611
16 Jul 2015 23:43:43	1254278	18661935	hadam3p_anz_u2bz_2013_1_010000719_0	46,379	256,475	5.5300
14 Jul 2015 09:10:05	1254278	18661935	hadam3p_anz_u2bz_2013_1_010000719_0	34,859	192,358	5.5182
11 Jul 2015 20:16:36	1254278	18661935	hadam3p_anz_u2bz_2013_1_010000719_0	23,339	127,625	5.4683
10 Jul 2015 10:34:36	1254278	18661935	hadam3p_anz_u2bz_2013_1_010000719_0	11,819	64,066	5.4206