Task 11512135

Name	famous_uezc_1599_200_006652259_3
Workunit	6855631
Created	10 Jun 2010, 13:49:51 UTC
Sent	18 Aug 2010, 22:47:40 UTC
Report deadline	18 Nov 2010, 6:14:51 UTC
Received	28 Aug 2010, 16:05:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1094755
Run time	7 days 6 hours 37 min 45 sec
CPU time	6 days 18 hours 2 min 42 sec
Validate state	Invalid
Credit	3,952.94
Device peak FLOPS	2.69 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 15 received, exiting... (12312): called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_uezc_1599_200_006652259/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E03A1 Unknown Unknown Unknown famous_um_6.11_i6 08267D05 Unknown Unknown Unknown famous_um_6.11_i6 082EFC2C Unknown Unknown Unknown famous_um_6.11_i6 08328714 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F755ABD6 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5068, iMonCtr=1 Model crash detected, will try to restart... no start tag in app init data (3809): Can't parse init data file - running in standalone mode forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_uezc_1599_200_006652259/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E03A1 Unknown Unknown Unknown famous_um_6.11_i6 083DFAFC Unknown Unknown Unknown famous_um_6.11_i6 08133B55 Unknown Unknown Unknown famous_um_6.11_i6 082AEFA7 Unknown Unknown Unknown famous_um_6.11_i6 0833098C Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F75EDBD6 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5068, iMonCtr=1 Model crash detected, will try to restart... no start tag in app init data (5068): Can't parse init data file - running in standalone mode no start tag in app init data (3820): Can't parse init data file - running in standalone mode forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_uezc_1599_200_006652259/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous (5068): called boinc_finish BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy (5628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (5628): No heartbeat from core client for 30 sec - exiting (6760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (7202): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (7571): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting (7571): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy forrtl: No sController:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9137, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_uezc_1599_200_006652259/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083DEAA4 Unknown Unknown Unknown famous_um_6.11_i6 08222DC4 Unknown Unknown Unknown famous_um_6.11_i6 0824527D Unknown Unknown Unknown famous_um_6.11_i6 08163FDC Unknown Unknown Unknown famous_um_6.11_i6 08163460 Unknown Unknown Unknown famous_um_6.11_i6 081777BF Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F761ABD6 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9137, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_uezc_1599_200_006652259/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083DEAA4 Unknown Unknown Unknown famous_um_6.11_i6 083DDD57 Unknown Unknown Unknown famous_um_6.11_i6 0822276F Unknown Unknown Unknown famous_um_6.11_i6 0824527D Unknown Unknown Unknown famous_um_6.11_i6 08160816 Unknown Unknown Unknown famous_um_6.11_i6 08163229 Unknown Unknown Unknown famous_um_6.11_i6 081777BF Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F757FBD6 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=9137, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_uezc_1599_200_006652259/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E03A1 Unknown Unknown Unknown famous_um_6.11_i6 080EADAA Unknown Unknown Unknown famous_um_6.11_i6 08202CBF Unknown Unknown Unknown famous_um_6.11_i6 08173370 Unknown Unknown Unknown famous_um_6.11_i6 0817613B Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown UnknowSorry, too many model crashes! :-( (9137): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Aug 2010 00:20:30	1094755	11512135	famous_uezc_1599_200_006652259_3	1,198,106	449,478	0.3752
24 Aug 2010 22:23:10	1094755	11512135	famous_uezc_1599_200_006652259_3	1,188,746	445,888	0.3751
24 Aug 2010 21:18:24	1094755	11512135	famous_uezc_1599_200_006652259_3	1,179,386	442,300	0.3750
24 Aug 2010 20:19:34	1094755	11512135	famous_uezc_1599_200_006652259_3	1,170,026	438,747	0.3750
24 Aug 2010 19:10:11	1094755	11512135	famous_uezc_1599_200_006652259_3	1,160,666	435,204	0.3750
24 Aug 2010 17:53:21	1094755	11512135	famous_uezc_1599_200_006652259_3	1,151,306	431,616	0.3749
24 Aug 2010 16:50:18	1094755	11512135	famous_uezc_1599_200_006652259_3	1,141,946	428,069	0.3749
24 Aug 2010 15:45:35	1094755	11512135	famous_uezc_1599_200_006652259_3	1,132,586	424,498	0.3748
24 Aug 2010 14:41:33	1094755	11512135	famous_uezc_1599_200_006652259_3	1,123,226	420,972	0.3748
24 Aug 2010 13:38:03	1094755	11512135	famous_uezc_1599_200_006652259_3	1,113,866	417,438	0.3748
24 Aug 2010 12:37:58	1094755	11512135	famous_uezc_1599_200_006652259_3	1,104,506	413,928	0.3748
24 Aug 2010 11:33:48	1094755	11512135	famous_uezc_1599_200_006652259_3	1,095,146	410,458	0.3748
24 Aug 2010 10:30:28	1094755	11512135	famous_uezc_1599_200_006652259_3	1,085,786	406,950	0.3748
24 Aug 2010 09:27:02	1094755	11512135	famous_uezc_1599_200_006652259_3	1,076,426	403,390	0.3747
24 Aug 2010 08:22:56	1094755	11512135	famous_uezc_1599_200_006652259_3	1,067,066	399,786	0.3747
24 Aug 2010 07:22:42	1094755	11512135	famous_uezc_1599_200_006652259_3	1,057,706	396,279	0.3747
24 Aug 2010 06:16:09	1094755	11512135	famous_uezc_1599_200_006652259_3	1,048,346	392,735	0.3746
24 Aug 2010 05:12:16	1094755	11512135	famous_uezc_1599_200_006652259_3	1,038,986	389,125	0.3745
24 Aug 2010 04:04:16	1094755	11512135	famous_uezc_1599_200_006652259_3	1,029,626	385,550	0.3745
24 Aug 2010 03:01:14	1094755	11512135	famous_uezc_1599_200_006652259_3	1,020,266	381,994	0.3744