Task 12603883

Name	famous_s3ce_799_200_006671673_2
Workunit	6874926
Created	19 Feb 2011, 4:14:35 UTC
Sent	19 Feb 2011, 13:59:26 UTC
Report deadline	21 May 2011, 21:26:37 UTC
Received	8 Mar 2011, 10:08:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1362654
Run time	1 days 7 hours 55 min 48 sec
CPU time	12 hours 5 min 4 sec
Validate state	Invalid
Credit	154.49
Device peak FLOPS	1.40 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Signal 15 received, exiting... (14702): called boinc_finish Signal 15 received, exiting... (21084): called boinc_finish Signal 15 received, exiting... (6508): called boinc_finish CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: TEMPHIST: Failed in OPEN of history file tmp/pipe_dummy cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_se_6.11_i686-pc-linux-gnu.so after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.namelists after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/ocean_restart.day after 11 attempts Could not change to slots dir /boinc/BOINC/slots/3 execl(/boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu, 127945) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5679, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_se_6.11_i686-pc-linux-gnu.so after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.namelists after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/ocean_restart.day after 11 attempts Could not change to slots dir /boinc/BOINC/slots/3 execl(/boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu, 127945) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5679, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_se_6.11_i686-pc-linux-gnu.so after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.namelists after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/ocean_restart.day after 11 attempts Could not change to slots dir /boinc/BOINC/slots/3 execl(/boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu, 127945) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5679, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_se_6.11_i686-pc-linux-gnu.so after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.namelists after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/ocean_restart.day after 11 attempts Could not change to slots dir /boinc/BOINC/slots/3 execl(/boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu, 127945) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5679, iMonCtr=1 Model crash detected, will try to restart... cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_se_6.11_i686-pc-linux-gnu.so after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.ihist after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/jobs/afyel.namelists after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/atmos_restart.day after 11 attempts cpdnmonitor: cannot open input file /boinc/BOINC/projects/climateprediction.net/famous_s3ce_799_200_006671673/dataout/ocean_restart.day after 11 attempts Could not change to slots dir /boinc/BOINC/slots/3 execl(/boinc/BOINC/projects/climateprediction.net/famous_um_6.11_i686-pc-linux-gnu, 127945) failed! Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5679, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Could not change to slots directory /boinc/BOINC/slots/3 (5679): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Feb 2011 23:04:45	1129992	12603883	famous_s3ce_799_200_006671673_2	46,826	39,880	0.8517
24 Feb 2011 19:10:30	1129992	12603883	famous_s3ce_799_200_006671673_2	37,466	31,885	0.8510
24 Feb 2011 14:19:16	1129992	12603883	famous_s3ce_799_200_006671673_2	28,106	23,973	0.8529
23 Feb 2011 16:47:48	1129992	12603883	famous_s3ce_799_200_006671673_2	18,746	15,953	0.8510
22 Feb 2011 20:09:17	1129992	12603883	famous_s3ce_799_200_006671673_2	9,386	7,928	0.8447