Task 12414870

Name	famous_xgk6_1199_200_007083293_0
Workunit	7286593
Created	18 Dec 2010, 15:23:47 UTC
Sent	29 Dec 2010, 11:19:17 UTC
Report deadline	30 Mar 2011, 18:46:28 UTC
Received	24 Nov 2011, 4:35:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1125388
Run time	7 days 9 hours 3 min 58 sec
CPU time	5 days 19 hours 27 min 2 sec
Validate state	Invalid
Credit	1,235.35
Device peak FLOPS	1.33 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 12/29/10 14:04:29 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1455 LoadLibraryA( dbghelp.dll ): GetLastError = 1455 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 2923, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37308, QuotaPeakPagedPoolUsage: 37308 QuotaNonPagedPoolUsage: 1872, QuotaPeakNonPagedPoolUsage: 2904 - Virtual Memory Usage - VirtualSize: 858943488, PeakVirtualSize: 858943488 - Pagefile Usage - PagefileUsage: 23396352, PeakPagefileUsage: 40046592 - Working Set Size - WorkingSetSize: 2048000, PeakWorkingSetSize: 2048000, PageFaultCount: 499 * Dump of thread ID 4312 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 1406250.000000, User Time: 0.000000, Wait Time: 4617474.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB * Dump of thread ID 4948 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 4617473.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4064, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 12/29/10 14:04:31 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll Loaded Library : C:\Program Files\BOINC\\symsrv.dll Loaded Library : C:\Program Files\BOINC\\srcsrv.dll LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll Debugger Engine : 4.0.5.0 Symbol Search Path: C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\famous_xgk6_1199_200_007083293;C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net ModLoad: 00400000 0143c000 C:\Documents and Settings\All Users\Application Data\BOINC\projects\climateprediction.net\famous_um_6.11_windows_intelx86.exe (-nosymbols- Symbols Loaded) Linked PDB Filename : ModLoad: 7c900000 000b2000 C:\WINDOWS\system32\ntdll.dll (5.1.2600.5755) (-exported- Symbols Loaded) Linked PDB Filename : ntdll.pdb File Version : 5.1.2600.5755 (xpsp_sp3_gdr.090206-1234) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5755 ModLoad: 7c800000 000f6000 C:\WINDOWS\system32\kernel32.dll (5.1.2600.5781) (-exported- Symbols Loaded) Linked PDB Filename : kernel32.pdb File Version : 5.1.2600.5781 (xpsp_sp3_gdr.090321-1317) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5781 ModLoad: 7e410000 00091000 C:\WINDOWS\system32\USER32.dll (5.1.2600.5512) (-exported- Symbols Loaded) Linked PDB Filename : user32.pdb File Version : 5.1.2600.5512 (xpsp.080413-2105) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5512 ModLoad: 77f10000 00049000 C:\WINDOWS\system32\GDI32.dll (5.1.2600.5698) (-exported- Symbols Loaded) Linked PDB Filename : gdi32.pdb File Version : 5.1.2600.5698 (xpsp_sp3_gdr.081022-1932) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5698 ModLoad: 77dd0000 0009b000 C:\WINDOWS\system32\ADVAPI32.dll (5.1.2600.5755) (-exported- Symbols Loaded) Linked PDB Filename : advapi32.pdb File Version : 5.1.2600.5755 (xpsp_sp3_gdr.090206-1234) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5755 ModLoad: 77e70000 00093000 C:\WINDOWS\system32\RPCRT4.dll (5.1.2600.6022) (-exported- Symbols Loaded) Linked PDB Filename : rpcrt4.pdb File Version : 5.1.2600.6022 (xpsp_sp3_gdr.100813-1643) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.6022 ModLoad: 77fe0000 00011000 C:\WINDOWS\system32\Secur32.dll (5.1.2600.5834) (-exported- Symbols Loaded) Linked PDB Filename : secur32.pdb File Version : 5.1.2600.5834 (xpsp_sp3_gdr.090624-1305) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5834 ModLoad: 76c90000 00028000 C:\WINDOWS\system32\imagehlp.dll (5.1.2600.5512) (-exported- Symbols Loaded) Linked PDB Filename : imagehlp.pdb File Version : 5.1.2600.5512 (xpsp.080413-2105) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5512 ModLoad: 77c10000 00058000 C:\WINDOWS\system32\msvcrt.dll (7.0.2600.5512) (-exported- Symbols Loaded) Linked PDB Filename : msvcrt.pdb File Version : 7.0.2600.5512 (xpsp.080413-2111) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 7.0.2600.5512 ModLoad: 76390000 0001d000 C:\WINDOWS\system32\IMM32.DLL (5.1.2600.5512) (-exported- Symbols Loaded) Linked PDB Filename : imm32.pdb File Version : 5.1.2600.5512 (xpsp.080413-2105) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5512 ModLoad: 32f20000 00115000 C:\Program Files\BOINC\dbghelp.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : dbghelp.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 00370000 00048000 C:\Program Files\BOINC\symsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : symsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 003c0000 0003b000 C:\Program Files\BOINC\srcsrv.dll (6.8.4.0) (-exported- Symbols Loaded) Linked PDB Filename : srcsrv.pdb File Version : 6.8.0004.0 (debuggers(dbg).070515-1751) Company Name : Microsoft Corporation Product Name : Debugging Tools for Windows(R) Product Version : 6.8.0004.0 ModLoad: 77c00000 00008000 C:\WINDOWS\system32\version.dll (5.1.2600.5512) (-exported- Symbols Loaded) Linked PDB Filename : version.pdb File Version : 5.1.2600.5512 (xpsp.080413-2105) Company Name : Microsoft Corporation Product Name : Microsoft® Windows® Operating System Product Version : 5.1.2600.5512 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 5, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 539, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37308, QuotaPeakPagedPoolUsage: 37308 QuotaNonPagedPoolUsage: 1872, QuotaPeakNonPagedPoolUsage: 2904 - Virtual Memory Usage - VirtualSize: 858943488, PeakVirtualSize: 858943488 - Pagefile Usage - PagefileUsage: 23396352, PeakPagefileUsage: 40046592 - Working Set Size - WorkingSetSize: 2043904, PeakWorkingSetSize: 2043904, PageFaultCount: 498 * Dump of thread ID 704 (state: Ready): * - Information - Status: Base Priority: Above Normal, Priority: Above Normal, , Kernel Time: 312500.000000, User Time: 0.000000, Wait Time: 4617600.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x7C812AFB - Registers - eax=11c3feb4 ebx=7ffdf000 ecx=00000000 edx=00cdbd70 esi=11c3ff3c edi=004b0590 eip=7c812afb esp=11c3feb0 ebp=11c3ff04 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206 - Callstack - ChildEBP RetAddr Args to Child 11c3ff04 00853626 e06d7363 00000001 00000003 11c3ff30 kernel32!RaiseException+0x0 11c3ff3c 008535dd 11c3ff4c 009358e0 00908514 00907d38 famous_um_6.11_windows_intelx86!+0x0 11c3ff58 007c57be 01000000 00500000 007ca0f1 12150000 famous_um_6.11_windows_intelx86!+0x0 11c3ffc0 7c817077 004b0590 00000664 7ffdf000 e06d7363 famous_um_6.11_windows_intelx86!+0x0 11c3fff0 00000000 00856ecc 00000000 00000000 00000000 kernel32!RegisterWaitForInputIdle+0x0 * Dump of thread ID 4380 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 4617599.000000 - Registers - eax=007eb5f0 ebx=00000000 ecx=7c8109ff edx=11c3fc88 esi=00000000 edi=32a0ff70 eip=7c90e514 esp=32a0ff40 ebp=32a0ff98 cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202 - Callstack - ChildEBP RetAddr Args to Child 32a0ff98 7c802455 00000064 00000000 32a0ffb4 007eb604 ntdll!KiFastSystemCallRet+0x0 32a0ffa8 007eb604 00000064 32a0ffec 7c80b729 00000000 kernel32!Sleep+0x0 32a0ffb4 7c80b729 00000000 625f6d68 636e696f 00000000 famous_um_6.11_windows_intelx86!+0x0 32a0ffec 00000000 007eb5f0 00000000 00000000 00350050 kernel32!GetModuleFileNameA+0x0 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4064, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 14:05:02 (4064): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: Result too large BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 22:07:57 (3920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 13:29:58 (3048): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:30:00 (3048): No heartbeat from core client for 30 sec - exiting 10:02:48 (5788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:02:50 (5788): No heartbeat from core client for 30 sec - exiting 23:27:01 (3720): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4796, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 23:31:03 (4796): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
23 Nov 2011 11:27:45	1125388	12414870	famous_xgk6_1199_200_007083293_0	374,426	492,281	1.3148
23 Nov 2011 06:02:02	1125388	12414870	famous_xgk6_1199_200_007083293_0	365,066	479,871	1.3145
23 Nov 2011 00:45:37	1125388	12414870	famous_xgk6_1199_200_007083293_0	355,706	467,421	1.3141
22 Nov 2011 20:08:50	1125388	12414870	famous_xgk6_1199_200_007083293_0	346,346	455,127	1.3141
22 Nov 2011 15:00:28	1125388	12414870	famous_xgk6_1199_200_007083293_0	336,986	442,690	1.3137
22 Nov 2011 10:34:13	1125388	12414870	famous_xgk6_1199_200_007083293_0	327,626	430,217	1.3131
22 Nov 2011 06:28:10	1125388	12414870	famous_xgk6_1199_200_007083293_0	318,266	417,846	1.3129
22 Nov 2011 00:11:32	1125388	12414870	famous_xgk6_1199_200_007083293_0	308,906	405,478	1.3126
21 Nov 2011 19:54:52	1125388	12414870	famous_xgk6_1199_200_007083293_0	299,546	392,949	1.3118
21 Nov 2011 15:35:18	1125388	12414870	famous_xgk6_1199_200_007083293_0	290,186	380,328	1.3106
21 Nov 2011 11:04:07	1125388	12414870	famous_xgk6_1199_200_007083293_0	280,826	367,718	1.3094
21 Nov 2011 06:07:47	1125388	12414870	famous_xgk6_1199_200_007083293_0	271,466	355,074	1.3080
21 Nov 2011 01:26:42	1125388	12414870	famous_xgk6_1199_200_007083293_0	262,106	342,343	1.3061
20 Nov 2011 21:00:17	1125388	12414870	famous_xgk6_1199_200_007083293_0	252,746	329,940	1.3054
20 Nov 2011 14:49:09	1125388	12414870	famous_xgk6_1199_200_007083293_0	243,386	318,352	1.3080
20 Nov 2011 09:37:59	1125388	12414870	famous_xgk6_1199_200_007083293_0	234,026	306,190	1.3084
20 Nov 2011 04:36:55	1125388	12414870	famous_xgk6_1199_200_007083293_0	224,666	293,962	1.3084
20 Nov 2011 00:05:50	1125388	12414870	famous_xgk6_1199_200_007083293_0	215,306	281,396	1.3070
19 Nov 2011 19:34:06	1125388	12414870	famous_xgk6_1199_200_007083293_0	205,946	268,749	1.3049
19 Nov 2011 15:25:30	1125388	12414870	famous_xgk6_1199_200_007083293_0	196,586	256,155	1.3030