Task 11436784

Name	famous_u3cw_1799_200_006637195_2
Workunit	6840567
Created	10 Jun 2010, 11:38:46 UTC
Sent	21 Jul 2010, 5:04:41 UTC
Report deadline	20 Oct 2010, 12:31:52 UTC
Received	4 Sep 2010, 2:49:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1070329
Run time	10 days 13 hours 0 min 51 sec
CPU time	7 days 6 hours 41 min 9 sec
Validate state	Invalid
Credit	4,416.16
Device peak FLOPS	2.47 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... (14429): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (14429): No heartbeat from core client for 30 sec - exiting (14429): No heartbeat from core client for 30 sec - exiting (14429): No heartbeat from core client for 30 sec - exiting (14975): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (14975): No heartbeat from core client for 30 sec - exiting (14975): No heartbeat from core client for 30 sec - exiting (14975): No heartbeat from core client for 30 sec - exiting (15045): No heartbeat from core client for 30 sec - exiting (15045): No heartbeat from core client for 30 sec - exiting (15045): No heartbeat from core client for 30 sec - exiting (15045): No heartbeat from core client for 30 sec - exiting (15045): No heartbeat from core client for 30 sec - exiting (15045): No heartbeat from core client for 30 sec - exiting (15045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... (15045): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/famous_u3cw_1799_200_006637195/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083E03A1 Unknown Unknown Unknown famous_um_6.11_i6 08267EB8 Unknown Unknown Unknown famous_um_6.11_i6 082EFC2C Unknown Unknown Unknown famous_um_6.11_i6 08328714 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F75B6C76 Unknown Unknown Unknown famous_um_6.11_i6 0804C041 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2450, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Sorry, too many model crashes! :-( (12199): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
04 Sep 2010 02:54:46	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,338,506	625,350	0.4672
04 Sep 2010 02:54:47	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,329,146	621,018	0.4672
04 Sep 2010 02:54:46	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,319,786	616,688	0.4673
04 Sep 2010 02:54:46	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,310,426	612,284	0.4672
04 Sep 2010 02:54:47	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,301,066	607,895	0.4672
04 Sep 2010 02:54:47	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,291,706	603,525	0.4672
04 Sep 2010 02:54:46	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,282,346	599,115	0.4672
04 Sep 2010 02:54:47	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,272,986	594,743	0.4672
04 Sep 2010 02:54:47	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,263,626	590,303	0.4672
24 Aug 2010 04:40:59	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,254,266	585,884	0.4671
24 Aug 2010 01:36:52	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,244,906	581,390	0.4670
23 Aug 2010 10:30:21	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,235,546	576,916	0.4669
23 Aug 2010 08:23:06	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,226,186	572,479	0.4669
23 Aug 2010 06:35:29	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,216,826	568,057	0.4668
23 Aug 2010 04:34:36	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,207,466	563,612	0.4668
23 Aug 2010 02:33:18	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,198,106	559,196	0.4667
23 Aug 2010 00:44:25	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,188,746	554,806	0.4667
22 Aug 2010 10:28:08	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,179,386	550,389	0.4667
22 Aug 2010 08:37:10	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,170,026	546,000	0.4667
22 Aug 2010 06:46:24	1070329	11436784	famous_u3cw_1799_200_006637195_2	1,160,666	541,595	0.4666