Task 11740797

Name	famous_v81z_1599_200_006696205_3
Workunit	6899458
Created	26 Aug 2010, 16:26:03 UTC
Sent	10 Dec 2010, 12:16:24 UTC
Report deadline	11 Mar 2011, 19:43:35 UTC
Received	13 Dec 2010, 15:22:53 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1122393
Run time	1 days 21 hours 25 min 17 sec
CPU time	1 days 16 hours 56 min 27 sec
Validate state	Invalid
Credit	3,335.30
Device peak FLOPS	3.41 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-apple-darwin
Stderr	<core_client_version>6.10.43</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Sat Dec 11 15:04:33 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x96e8646b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc31c edx: 0x96e200fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc358 esp: 0xbfffc31c ss: 0x0000001f efl: 0x00000206 eip: 0x96e200fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libirc.dylib 0x9000d000 - 0x90010fff /usr/lib/system/libmathCommon.A.dylib 0x9673a000 - 0x96748fff /usr/lib/libz.1.dylib 0x96e1f000 - 0x96fc6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7533, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Sat Dec 11 20:16:14 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x96e8646b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc31c edx: 0x96e200fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc358 esp: 0xbfffc31c ss: 0x0000001f efl: 0x00000206 eip: 0x96e200fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libirc.dylib 0x9000d000 - 0x90010fff /usr/lib/system/libmathCommon.A.dylib 0x9673a000 - 0x96748fff /usr/lib/libz.1.dylib 0x96e1f000 - 0x96fc6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10355, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Sat Dec 11 20:56:54 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x96e8646b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc31c edx: 0x96e200fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc358 esp: 0xbfffc31c ss: 0x0000001f efl: 0x00000206 eip: 0x96e200fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libirc.dylib 0x9000d000 - 0x90010fff /usr/lib/system/libmathCommon.A.dylib 0x9673a000 - 0x96748fff /usr/lib/libz.1.dylib 0x96e1f000 - 0x96fc6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10788, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.5 build 10H574 Sun Dec 12 18:48:03 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x96e8646b 3 0xffffffff 4 0x0134e6b3 5 0x0134d44f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc31c edx: 0x96e200fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc358 esp: 0xbfffc31c ss: 0x0000001f efl: 0x00000206 eip: 0x96e200fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12a9000 - 0x12c4fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libsvml.dylib 0x131c000 - 0x13b3fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libifcoremt.dylib 0x13fc000 - 0x1556fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libimf.dylib 0x1669000 - 0x1697fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_v81z_1599_200_006696205/lib/libirc.dylib 0x9000d000 - 0x90010fff /usr/lib/system/libmathCommon.A.dylib 0x9673a000 - 0x96748fff /usr/lib/libz.1.dylib 0x96e1f000 - 0x96fc6fff /usr/lib/libSystem.B.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=20840, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: Inappropriate ioctl for device BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( (24504): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Dec 2010 15:24:40	1122393	11740797	famous_v81z_1599_200_006696205_3	1,010,906	146,695	0.1451
13 Dec 2010 15:24:40	1122393	11740797	famous_v81z_1599_200_006696205_3	1,001,546	145,336	0.1451
13 Dec 2010 00:49:37	1122393	11740797	famous_v81z_1599_200_006696205_3	992,186	143,982	0.1451
13 Dec 2010 00:28:33	1122393	11740797	famous_v81z_1599_200_006696205_3	982,826	142,631	0.1451
13 Dec 2010 00:02:12	1122393	11740797	famous_v81z_1599_200_006696205_3	973,466	141,280	0.1451
12 Dec 2010 23:40:56	1122393	11740797	famous_v81z_1599_200_006696205_3	964,106	139,931	0.1451
12 Dec 2010 23:19:16	1122393	11740797	famous_v81z_1599_200_006696205_3	954,746	138,582	0.1452
12 Dec 2010 22:52:21	1122393	11740797	famous_v81z_1599_200_006696205_3	945,386	137,233	0.1452
12 Dec 2010 22:29:11	1122393	11740797	famous_v81z_1599_200_006696205_3	936,026	135,879	0.1452
12 Dec 2010 22:06:14	1122393	11740797	famous_v81z_1599_200_006696205_3	926,666	134,523	0.1452
12 Dec 2010 21:43:44	1122393	11740797	famous_v81z_1599_200_006696205_3	917,306	133,168	0.1452
12 Dec 2010 21:11:28	1122393	11740797	famous_v81z_1599_200_006696205_3	907,946	131,810	0.1452
12 Dec 2010 20:46:07	1122393	11740797	famous_v81z_1599_200_006696205_3	898,586	130,445	0.1452
12 Dec 2010 20:18:34	1122393	11740797	famous_v81z_1599_200_006696205_3	889,226	129,094	0.1452
12 Dec 2010 19:56:09	1122393	11740797	famous_v81z_1599_200_006696205_3	879,866	127,746	0.1452
12 Dec 2010 19:33:43	1122393	11740797	famous_v81z_1599_200_006696205_3	870,506	126,397	0.1452
12 Dec 2010 19:12:40	1122393	11740797	famous_v81z_1599_200_006696205_3	861,146	125,049	0.1452
12 Dec 2010 18:20:04	1122393	11740797	famous_v81z_1599_200_006696205_3	851,786	123,699	0.1452
12 Dec 2010 17:58:45	1122393	11740797	famous_v81z_1599_200_006696205_3	842,426	122,350	0.1452
12 Dec 2010 17:32:05	1122393	11740797	famous_v81z_1599_200_006696205_3	833,066	120,989	0.1452