Task 14309179

Name	hadam3p_eu_40o1_2001_1_007253226_1
Workunit	7451466
Created	23 Mar 2012, 16:51:11 UTC
Sent	23 Mar 2012, 16:51:15 UTC
Report deadline	5 Mar 2013, 22:11:15 UTC
Received	27 Mar 2012, 11:59:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	0 (0x00000000)
Computer ID	892733
Run time	19 hours 12 min 37 sec
CPU time	19 hours 12 min 37 sec
Validate state	Invalid
Credit	200.38
Device peak FLOPS	1.66 GFLOPS
Application version	UK Met Office HadAM3P-HadRM3P Europe v6.09 windows_intelx86
Stderr	<core_client_version>5.10.45</core_client_version> <![CDATA[ <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5816, selfPID=5816, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=1288, selfPID=1288, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4264, selfPID=4264, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5756, selfPID=5756, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5800, selfPID=5800, iMonCtr=2 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=3868, selfPID=3868, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4840, selfPID=4840, iMonCtr=2 CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5244, selfPID=5244, iMonCtr=2 CPDN Monitor - Quit request from BOINC... 00:32:26 (156): No heartbeat from core client for 30 sec - exiting 00:32:27 (156): No heartbeat from core client for 30 sec - exiting 00:32:28 (156): No heartbeat from core client for 30 sec - exiting 00:32:29 (156): No heartbeat from core client for 30 sec - exiting 00:32:30 (156): No heartbeat from core client for 30 sec - exiting 00:32:31 (156): No heartbeat from core client for 30 sec - exiting 00:32:32 (156): No heartbeat from core client for 30 sec - exiting 00:32:33 (156): No heartbeat from core client for 30 sec - exiting 00:32:34 (156): No heartbeat from core client for 30 sec - exiting 00:32:35 (156): No heartbeat from core client for 30 sec - exiting 00:32:37 (156): No heartbeat from core client for 30 sec - exiting 00:32:38 (156): No heartbeat from core client for 30 sec - exiting 00:32:39 (156): No heartbeat from core client for 30 sec - exiting 00:32:40 (156): No heartbeat from core client for 30 sec - exiting 00:32:41 (156): No heartbeat from core client for 30 sec - exiting 00:32:42 (156): No heartbeat from core client for 30 sec - exiting 00:32:44 (156): No heartbeat from core client for 30 sec - exiting 00:32:45 (156): No heartbeat from core client for 30 sec - exiting 00:32:46 (156): No heartbeat from core client for 30 sec - exiting 00:32:47 (156): No heartbeat from core client for 30 sec - exiting 00:32:49 (156): No heartbeat from core client for 30 sec - exiting 00:32:50 (156): No heartbeat from core client for 30 sec - exiting 00:32:51 (156): No heartbeat from core client for 30 sec - exiting 00:32:52 (156): No heartbeat from core client for 30 sec - exiting 00:32:54 (156): No heartbeat from core client for 30 sec - exiting 00:32:55 (156): No heartbeat from core client for 30 sec - exiting 00:32:56 (156): No heartbeat from core client for 30 sec - exiting 00:32:57 (156): No heartbeat from core client for 30 sec - exiting 00:32:59 (156): No heartbeat from core client for 30 sec - exiting 00:33:00 (156): No heartbeat from core client for 30 sec - exiting 00:33:01 (156): No heartbeat from core client for 30 sec - exiting 00:33:03 (156): No heartbeat from core client for 30 sec - exiting 00:33:04 (156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Regional Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5040, selfPID=5040, iMonCtr=2 12:54:41 (4832): No heartbeat from core client for 30 sec - exiting 12:54:42 (4832): No heartbeat from core client for 30 sec - exiting 12:54:43 (4832): No heartbeat from core client for 30 sec - exiting 12:54:44 (4832): No heartbeat from core client for 30 sec - exiting 12:54:45 (4832): No heartbeat from core client for 30 sec - exiting 12:54:46 (4832): No heartbeat from core client for 30 sec - exiting 12:54:47 (4832): No heartbeat from core client for 30 sec - exiting 12:54:48 (4832): No heartbeat from core client for 30 sec - exiting 12:54:49 (4832): No heartbeat from core client for 30 sec - exiting 12:54:50 (4832): No heartbeat from core client for 30 sec - exiting 12:54:51 (4832): No heartbeat from core client for 30 sec - exiting 12:54:52 (4832): No heartbeat from core client for 30 sec - exiting 12:54:53 (4832): No heartbeat from core client for 30 sec - exiting 12:54:54 (4832): No heartbeat from core client for 30 sec - exiting 12:54:55 (4832): No heartbeat from core client for 30 sec - exiting 12:54:56 (4832): No heartbeat from core client for 30 sec - exiting 12:54:57 (4832): No heartbeat from core client for 30 sec - exiting 12:54:58 (4832): No heartbeat from core client for 30 sec - exiting 12:54:59 (4832): No heartbeat from core client for 30 sec - exiting 12:55:00 (4832): No heartbeat from core client for 30 sec - exiting 12:55:01 (4832): No heartbeat from core client for 30 sec - exiting 12:55:02 (4832): No heartbeat from core client for 30 sec - exiting 12:55:03 (4832): No heartbeat from core client for 30 sec - exiting 12:55:04 (4832): No heartbeat from core client for 30 sec - exiting 12:55:05 (4832): No heartbeat from core client for 30 sec - exiting 12:55:06 (4832): No heartbeat from core client for 30 sec - exiting 12:55:07 (4832): No heartbeat from core client for 30 sec - exiting 12:55:08 (4832): No heartbeat from core client for 30 sec - exiting 12:55:09 (4832): No heartbeat from core client for 30 sec - exiting 12:55:11 (4832): No heartbeat from core client for 30 sec - exiting 12:55:12 (4832): No heartbeat from core client for 30 sec - exiting 12:55:13 (4832): No heartbeat from core client for 30 sec - exiting 12:55:14 (4832): No heartbeat from core client for 30 sec - exiting 12:55:15 (4832): No heartbeat from core client for 30 sec - exiting 12:55:16 (4832): No heartbeat from core client for 30 sec - exiting 12:55:17 (4832): No heartbeat from core client for 30 sec - exiting 12:55:18 (4832): No heartbeat from core client for 30 sec - exiting 12:55:19 (4832): No heartbeat from core client for 30 sec - exiting 12:55:20 (4832): No heartbeat from core client for 30 sec - exiting 12:55:21 (4832): No heartbeat from core client for 30 sec - exiting 12:55:22 (4832): No heartbeat from core client for 30 sec - exiting 12:55:23 (4832): No heartbeat from core client for 30 sec - exiting 12:55:24 (4832): No heartbeat from core client for 30 sec - exiting 12:55:25 (4832): No heartbeat from core client for 30 sec - exiting 12:55:26 (4832): No heartbeat from core client for 30 sec - exiting 12:55:27 (4832): No heartbeat from core client for 30 sec - exiting 12:55:28 (4832): No heartbeat from core client for 30 sec - exiting 12:55:29 (4832): No heartbeat from core client for 30 sec - exiting 12:55:30 (4832): No heartbeat from core client for 30 sec - exiting 12:55:31 (4832): No heartbeat from core client for 30 sec - exiting 12:55:32 (4832): No heartbeat from core client for 30 sec - exiting 12:55:33 (4832): No heartbeat from core client for 30 sec - exiting 12:55:34 (4832): No heartbeat from core client for 30 sec - exiting 12:55:35 (4832): No heartbeat from core client for 30 sec - exiting 12:55:36 (4832): No heartbeat from core client for 30 sec - exiting 12:55:37 (4832): No heartbeat from core client for 30 sec - exiting 12:55:38 (4832): No heartbeat from core client for 30 sec - exiting 12:55:39 (4832): No heartbeat from core client for 30 sec - exiting 12:55:40 (4832): No heartbeat from core client for 30 sec - exiting 12:55:41 (4832): No heartbeat from core client for 30 sec - exiting 12:55:42 (4832): No heartbeat from core client for 30 sec - exiting 12:55:43 (4832): No heartbeat from core client for 30 sec - exiting 12:55:44 (4832): No heartbeat from core client for 30 sec - exiting 12:55:45 (4832): No heartbeat from core client for 30 sec - exiting 12:55:46 (4832): No heartbeat from core client for 30 sec - exiting 12:55:47 (4832): No heartbeat from core client for 30 sec - exiting 12:55:48 (4832): No heartbeat from core client for 30 sec - exiting 12:55:49 (4832): No heartbeat from core client for 30 sec - exiting 12:55:50 (4832): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 2:58:38 (5924): start_timer_threa d (): CreateThread() failed, errn o 0 Unhandled Exception Detected... - Unhandled Exception Record - Rea s on: Out Of Memory (C++ Exception) (0xe06d73 6 3) at address 0x76F54503 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 03/27/12 12:58:38 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Program Files\BOINC Project Symstore : **************** BOINC Windows Runtime Debugger Version 6.13.0 Dump Timestamp : 03/27/12 12:58:38 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Program Files\BOINC Project Symstore : LoaadLiLibbaryryA(:\C:\ProamraFi Fesles\BINC\\\dbgghellp.ddll ): GeetLaastEErroor == 11144 LLadadLibaryA( dbgheelp.dll ): GGetLastErrorr = 11114 * DumpofDump of thes Statess Sts:titi - I/O** - I/O s Counters - Reaters - W Read:: 0, Other 58 - I/O 1ra8 - I/O ounterfers Cead: 0, Write: 226: 0, Wer 0 2 PageOt Per l Usage - QuotaPagedPoolUsage: 163624, QuotaPeakPagedPoolUsage: 163624 QuotaNonPagedPoolUsage: 2960, QuotaPeakNonPagedPoolUsage: 2960 - Virtual Memory Usage - VirtualSize: 630718464, PeakVirtualSize: 630718464 - Page0il e Usage - Pagefisage - Q: otaP961ge, PeakPagef:le164016, Q5ot37568 -g WorkiUsage: 1640 6 Q uorkaNngSPagedPe: 421age: 3 P12ak QuotaPgSetSize: 421edPoolUsage: 311Cou n t:Virtua MemoDump of thr Vd ID 5968ze: 98te:922aiti Pg)akVirtu a lSize: 9ation 288 -tuPa: efait RUsason User- Paequest, , Kernel722304: 156kPagefi0000, User Time:12 -00Working Set SizeTi- Wo 307ng7524.00e000 - Unh,ndled ExcepngioenSiRecord - Rea22on, OageFaultCoory (C1074 cptionump of thread IDat4680 rests tex7 WF545i3 D e b Information um S tatus : * Foreground WrRndoequest, , Kernel Tme : Windo.0000e 0, User T i Win.00000lass : m Window 4.roce00s -: 0 nd Wedndoxcephion R ecord Exiting... - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x76F54503 Dump of thread ID 2484 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 30787520.000000 * Debug Message Dump * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5924, selfPID=4768, iMonCtr=1 Model crash detected, will try to restart... Leaving CPDN_Main::Monitor... Called boinc_finish </stderr_txt> <message> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_2.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_3.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_4.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_5.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_6.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_7.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_8.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_9.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_10.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_11.zip</file_name> <error_code>-161</error_code> </file_xfer_error> <file_xfer_error> <file_name>hadam3p_eu_40o1_2001_1_007253226_1_12.zip</file_name> <error_code>-161</error_code> </file_xfer_error> </message> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Mar 2012 07:39:59	892733	14309179	hadam3p_eu_40o1_2001_1_007253226_1	11,616	47,327	4.0743