Task 11916244

Name	famous_ufn1_1199_200_006720463_5
Workunit	6923715
Created	23 Sep 2010, 18:45:27 UTC
Sent	24 Sep 2010, 8:11:04 UTC
Report deadline	24 Dec 2010, 15:38:15 UTC
Received	17 Oct 2010, 6:44:04 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	766777
Run time	3 days 8 hours 55 min 13 sec
CPU time	2 days 19 hours 10 min 2 sec
Validate state	Invalid
Credit	2,625.03
Device peak FLOPS	3.42 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-apple-darwin
Stderr	<core_client_version>6.10.21</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Thu Sep 30 04:02:10 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9165c1fb 3 0xffffffff 4 0x013596b3 5 0x0135844f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x915f60fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c ss: 0x0000001f efl: 0x00000206 eip: 0x915f60fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12b4000 - 0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libsvml.dylib 0x1327000 - 0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libifcoremt.dylib 0x1407000 - 0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libimf.dylib 0x1674000 - 0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libirc.dylib 0x915f5000 - 0x9179bfff /usr/lib/libSystem.B.dylib 0x93da8000 - 0x93dabfff /usr/lib/system/libmathCommon.A.dylib 0x940d5000 - 0x940e3fff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=49731, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (56415): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (57139): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (57405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (57479): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (57593): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (57926): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (58744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (58766): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (58788): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (58872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (59081): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (59153): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (59237): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (59405): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (59493): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (59529): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (59621): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (60156): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (60485): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (60835): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (60978): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (61237): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (61261): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62069): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62119): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62248): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62339): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62411): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62574): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62690): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (62838): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 (63078): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63420): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63548): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63681): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63726): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63786): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (63976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (64159): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (64264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (64558): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Sat Oct 2 07:48:52 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9165c1fb 3 0xffffffff 4 0x013596b3 5 0x0135844f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x915f60fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c ss: 0x0000001f efl: 0x00000206 eip: 0x915f60fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12b4000 - 0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libsvml.dylib 0x1327000 - 0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libifcoremt.dylib 0x1407000 - 0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libimf.dylib 0x1674000 - 0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libirc.dylib 0x915f5000 - 0x9179bfff /usr/lib/libSystem.B.dylib 0x93da8000 - 0x93dabfff /usr/lib/system/libmathCommon.A.dylib 0x940d5000 - 0x940e3fff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11746, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Sat Oct 2 08:03:24 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9165c1fb 3 0xffffffff 4 0x013596b3 5 0x0135844f 6 0x002eeac6 7 0x003f55ea 8 0x00452abd 9 0x004698c0 10 0x000026b6 11 0x000025dd 12 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffc41c edx: 0x915f60fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffc458 esp: 0xbfffc41c ss: 0x0000001f efl: 0x00000206 eip: 0x915f60fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12b4000 - 0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libsvml.dylib 0x1327000 - 0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libifcoremt.dylib 0x1407000 - 0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libimf.dylib 0x1674000 - 0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libirc.dylib 0x915f5000 - 0x9179bfff /usr/lib/libSystem.B.dylib 0x93da8000 - 0x93dabfff /usr/lib/system/libmathCommon.A.dylib 0x940d5000 - 0x940e3fff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=11918, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... SIGSEGV: segmentation violation Crashed executable name: famous_um_6.11_i686-apple-darwin built using BOINC library version 6.11.1 Machine type Intel 80486 (32-bit executable) System version: Macintosh OS 10.6.4 build 10F569 Sat Oct 2 17:45:39 2010 sh: /usr/bin/atos: No such file or directory 0 0x0048fdcf 1 0x0049054a 2 0x9165c1fb 3 0xffffffff 4 0x013596b3 5 0x0135844f 6 0x003f5826 7 0x00450bd8 8 0x004698c0 9 0x000026b6 10 0x000025dd 11 0x00000003 Thread 0 crashed with X86 Thread State (32-bit): eax: 0xffffffe1 ebx: 0x00000003 ecx: 0xbfffd4ac edx: 0x915f60fa edi: 0x00000000 esi: 0x00000000 ebp: 0xbfffd4e8 esp: 0xbfffd4ac ss: 0x0000001f efl: 0x00000206 eip: 0x915f60fa cs: 0x00000007 ds: 0x0000001f es: 0x0000001f fs: 0x00000000 gs: 0x00000037 Binary Images Description: 0x1000 - 0x517fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_um_6.11_i686-apple-darwin 0x12b4000 - 0x12cffff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libsvml.dylib 0x1327000 - 0x13befff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libifcoremt.dylib 0x1407000 - 0x1561fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libimf.dylib 0x1674000 - 0x16a2fff /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/lib/libirc.dylib 0x915f5000 - 0x9179bfff /usr/lib/libSystem.B.dylib 0x93da8000 - 0x93dabfff /usr/lib/system/libmathCommon.A.dylib 0x940d5000 - 0x940e3fff /usr/lib/libz.1.dylib Exiting... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5296, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STASH : Unsupported MPP submodel tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... no start tag in app init data (19297): Can't parse init data file - running in standalone mode no start tag in app init data (19297): Can't parse init data file - running in standalone mode Could not change to project directory (19297): called boinc_finish BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /Library/Application Support/BOINC Data/projects/climateprediction.net/famous_ufn1_1199_200_006720463/dataout/stdout_um.txt Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3938, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Write ERROR on history file for namelistNLCFILES tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Model crashed: READHIST: End of file in READ from history file for namelist NLIHISTO tmp/pipe_dummy Sorry, too many model crashes! :-( (2627): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Oct 2010 05:17:44	766777	11916244	famous_ufn1_1199_200_006720463_5	795,626	239,852	0.3015
17 Oct 2010 03:52:29	766777	11916244	famous_ufn1_1199_200_006720463_5	786,266	237,838	0.3025
16 Oct 2010 16:39:13	766777	11916244	famous_ufn1_1199_200_006720463_5	776,906	235,715	0.3034
16 Oct 2010 14:29:32	766777	11916244	famous_ufn1_1199_200_006720463_5	767,546	233,732	0.3045
16 Oct 2010 14:29:32	766777	11916244	famous_ufn1_1199_200_006720463_5	758,186	231,740	0.3057
16 Oct 2010 14:29:32	766777	11916244	famous_ufn1_1199_200_006720463_5	748,826	229,743	0.3068
15 Oct 2010 21:28:09	766777	11916244	famous_ufn1_1199_200_006720463_5	739,466	227,712	0.3079
15 Oct 2010 20:48:21	766777	11916244	famous_ufn1_1199_200_006720463_5	730,106	225,666	0.3091
14 Oct 2010 19:09:45	766777	11916244	famous_ufn1_1199_200_006720463_5	720,746	223,635	0.3103
14 Oct 2010 18:08:04	766777	11916244	famous_ufn1_1199_200_006720463_5	711,386	221,683	0.3116
14 Oct 2010 17:11:23	766777	11916244	famous_ufn1_1199_200_006720463_5	702,026	219,768	0.3130
14 Oct 2010 13:36:26	766777	11916244	famous_ufn1_1199_200_006720463_5	692,666	217,833	0.3145
14 Oct 2010 12:58:16	766777	11916244	famous_ufn1_1199_200_006720463_5	683,306	215,899	0.3160
14 Oct 2010 12:26:32	766777	11916244	famous_ufn1_1199_200_006720463_5	673,946	213,969	0.3175
14 Oct 2010 11:49:56	766777	11916244	famous_ufn1_1199_200_006720463_5	664,586	212,016	0.3190
14 Oct 2010 11:12:29	766777	11916244	famous_ufn1_1199_200_006720463_5	655,226	210,052	0.3206
14 Oct 2010 10:35:46	766777	11916244	famous_ufn1_1199_200_006720463_5	645,866	208,047	0.3221
14 Oct 2010 09:56:09	766777	11916244	famous_ufn1_1199_200_006720463_5	636,506	206,035	0.3237
14 Oct 2010 09:19:30	766777	11916244	famous_ufn1_1199_200_006720463_5	627,146	204,044	0.3254
14 Oct 2010 08:44:45	766777	11916244	famous_ufn1_1199_200_006720463_5	617,786	202,068	0.3271