Task 11628268

Name	famous_u6t1_1799_200_006641664_5
Workunit	6845036
Created	31 Jul 2010, 4:21:36 UTC
Sent	31 Jul 2010, 5:20:02 UTC
Report deadline	30 Oct 2010, 12:47:13 UTC
Received	9 Oct 2010, 22:02:21 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	986925
Run time	19 days 0 hours 47 min 44 sec
CPU time	18 days 12 hours 57 min 37 sec
Validate state	Invalid
Credit	5,867.60
Device peak FLOPS	1.55 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 Engaging BOINC Windows Runtime Debugger... ****************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 10/08/10 18:48:15 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 31, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37532, QuotaPeakPagedPoolUsage: 37532 QuotaNonPagedPoolUsage: 1832, QuotaPeakNonPagedPoolUsage: 1832 - Virtual Memory Usage - VirtualSize: 853819392, PeakVirtualSize: 853819392 - Pagefile Usage - PagefileUsage: 18182144, PeakPagefileUsage: 23244800 - Working Set Size - WorkingSetSize: 2166784, PeakWorkingSetSize: 2166784, PageFaultCount: 529 * Dump of thread ID 5876 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 312500.000000, User Time: 0.000000, Wait Time: 73014304.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 * Dump of thread ID 5320 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73014304.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 10/08/10 18:48:18 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 31, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37532, QuotaPeakPagedPoolUsage: 37532 QuotaNonPagedPoolUsage: 1832, QuotaPeakNonPagedPoolUsage: 1832 - Virtual Memory Usage - VirtualSize: 853819392, PeakVirtualSize: 853819392 - Pagefile Usage - PagefileUsage: 18182144, PeakPagefileUsage: 23244800 - Working Set Size - WorkingSetSize: 2166784, PeakWorkingSetSize: 2166784, PageFaultCount: 529 * Dump of thread ID 1812 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73014504.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 * Dump of thread ID 5812 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73014504.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 10/08/10 18:48:20 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 1455 LoadLibraryA( symsrv.dll ): GetLastError = 1455 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 1455 LoadLibraryA( srcsrv.dll ): GetLastError = 1455 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll SymInitialize(): GetLastError = 8 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 31, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37532, QuotaPeakPagedPoolUsage: 37532 QuotaNonPagedPoolUsage: 1832, QuotaPeakNonPagedPoolUsage: 1832 - Virtual Memory Usage - VirtualSize: 853819392, PeakVirtualSize: 853819392 - Pagefile Usage - PagefileUsage: 18182144, PeakPagefileUsage: 23244800 - Working Set Size - WorkingSetSize: 2166784, PeakWorkingSetSize: 2166784, PageFaultCount: 529 * Dump of thread ID 5440 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 156250.000000, Wait Time: 73014632.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 * Dump of thread ID 4904 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73014632.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 10/08/10 18:48:23 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : Loaded Library : C:\Program Files\BOINC\\dbghelp.dll LoadLibraryA( C:\Program Files\BOINC\\symsrv.dll ): GetLastError = 1455 LoadLibraryA( symsrv.dll ): GetLastError = 1455 LoadLibraryA( C:\Program Files\BOINC\\srcsrv.dll ): GetLastError = 1455 LoadLibraryA( srcsrv.dll ): GetLastError = 1455 LoadLibraryA( C:\Program Files\BOINC\\version.dll ): GetLastError = 126 Loaded Library : version.dll SymInitialize(): GetLastError = 0 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 31, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37532, QuotaPeakPagedPoolUsage: 37532 QuotaNonPagedPoolUsage: 1832, QuotaPeakNonPagedPoolUsage: 1832 - Virtual Memory Usage - VirtualSize: 853819392, PeakVirtualSize: 853819392 - Pagefile Usage - PagefileUsage: 18182144, PeakPagefileUsage: 23244800 - Working Set Size - WorkingSetSize: 2166784, PeakWorkingSetSize: 2166784, PageFaultCount: 529 * Dump of thread ID 5228 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 0.000000, Wait Time: 73014808.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 * Dump of thread ID 5688 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73014808.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 10/08/10 18:48:26 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 31, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37532, QuotaPeakPagedPoolUsage: 37532 QuotaNonPagedPoolUsage: 1832, QuotaPeakNonPagedPoolUsage: 1832 - Virtual Memory Usage - VirtualSize: 853819392, PeakVirtualSize: 853819392 - Pagefile Usage - PagefileUsage: 18182144, PeakPagefileUsage: 23244800 - Working Set Size - WorkingSetSize: 2170880, PeakWorkingSetSize: 2170880, PageFaultCount: 530 * Dump of thread ID 4216 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73014984.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 * Dump of thread ID 744 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73014984.000000 * Debug Message Dump ** * Foreground Window Data * Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Unhandled Exception Detected... - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 Engaging BOINC Windows Runtime Debugger... **************** BOINC Windows Runtime Debugger Version 6.11.1 Dump Timestamp : 10/08/10 18:48:28 Install Directory : C:\Program Files\BOINC\ Data Directory : C:\Documents and Settings\All Users\Application Data\BOINC Project Symstore : LoadLibraryA( C:\Program Files\BOINC\\dbghelp.dll ): GetLastError = 1114 LoadLibraryA( dbghelp.dll ): GetLastError = 1114 * Dump of the Process Statistics: * - I/O Operations Counters - Read: 4, Write: 0, Other 186 - I/O Transfers Counters - Read: 0, Write: 31, Other 0 - Paged Pool Usage - QuotaPagedPoolUsage: 37532, QuotaPeakPagedPoolUsage: 37532 QuotaNonPagedPoolUsage: 1832, QuotaPeakNonPagedPoolUsage: 1832 - Virtual Memory Usage - VirtualSize: 853819392, PeakVirtualSize: 853819392 - Pagefile Usage - PagefileUsage: 18182144, PeakPagefileUsage: 23244800 - Working Set Size - WorkingSetSize: 2170880, PeakWorkingSetSize: 2170880, PageFaultCount: 530 * Dump of thread ID 2532 (state: Waiting): * - Information - Status: Wait Reason: UserRequest, , Kernel Time: 156250.000000, User Time: 156250.000000, Wait Time: 73015152.000000 - Unhandled Exception Record - Reason: Out Of Memory (C++ Exception) (0xe06d7363) at address 0x77E4BEE7 * Dump of thread ID 4136 (state: Waiting): * - Information - Status: Wait Reason: ExecutionDelay, , Kernel Time: 0.000000, User Time: 0.000000, Wait Time: 73015152.000000 * Debug Message Dump ** * Foreground Window Data *** Window Name : Window Class : Window Process ID: 0 Window Thread ID : 0 Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4756, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( 18:48:30 (4756): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Oct 2010 13:04:55	986925	11628268	famous_u6t1_1799_200_006641664_5	1,778,426	1,597,592	0.8983
06 Oct 2010 10:41:58	986925	11628268	famous_u6t1_1799_200_006641664_5	1,769,066	1,589,220	0.8983
06 Oct 2010 03:51:26	986925	11628268	famous_u6t1_1799_200_006641664_5	1,759,706	1,580,607	0.8982
05 Oct 2010 22:23:00	986925	11628268	famous_u6t1_1799_200_006641664_5	1,750,346	1,571,780	0.8980
05 Oct 2010 14:45:53	986925	11628268	famous_u6t1_1799_200_006641664_5	1,740,986	1,563,943	0.8983
05 Oct 2010 03:59:37	986925	11628268	famous_u6t1_1799_200_006641664_5	1,731,626	1,556,573	0.8989
04 Oct 2010 22:35:49	986925	11628268	famous_u6t1_1799_200_006641664_5	1,722,266	1,548,684	0.8992
04 Oct 2010 15:30:11	986925	11628268	famous_u6t1_1799_200_006641664_5	1,712,906	1,541,773	0.9001
04 Oct 2010 13:32:37	986925	11628268	famous_u6t1_1799_200_006641664_5	1,703,546	1,534,873	0.9010
04 Oct 2010 11:34:53	986925	11628268	famous_u6t1_1799_200_006641664_5	1,694,186	1,527,967	0.9019
04 Oct 2010 09:41:56	986925	11628268	famous_u6t1_1799_200_006641664_5	1,684,826	1,521,061	0.9028
04 Oct 2010 07:48:11	986925	11628268	famous_u6t1_1799_200_006641664_5	1,675,466	1,514,155	0.9037
04 Oct 2010 06:27:30	986925	11628268	famous_u6t1_1799_200_006641664_5	1,666,106	1,507,252	0.9047
04 Oct 2010 04:01:28	986925	11628268	famous_u6t1_1799_200_006641664_5	1,656,746	1,500,337	0.9056
04 Oct 2010 04:01:28	986925	11628268	famous_u6t1_1799_200_006641664_5	1,647,386	1,493,359	0.9065
04 Oct 2010 02:59:32	986925	11628268	famous_u6t1_1799_200_006641664_5	1,638,026	1,486,461	0.9075
03 Oct 2010 22:10:45	986925	11628268	famous_u6t1_1799_200_006641664_5	1,628,666	1,479,599	0.9085
03 Oct 2010 20:13:29	986925	11628268	famous_u6t1_1799_200_006641664_5	1,619,306	1,472,718	0.9095
03 Oct 2010 18:20:45	986925	11628268	famous_u6t1_1799_200_006641664_5	1,609,946	1,465,715	0.9104
03 Oct 2010 16:23:14	986925	11628268	famous_u6t1_1799_200_006641664_5	1,600,586	1,458,755	0.9114