Task 12574453

Name	famous_x3ax_1499_200_007066112_1
Workunit	7269412
Created	12 Feb 2011, 0:12:35 UTC
Sent	12 Feb 2011, 0:12:42 UTC
Report deadline	14 May 2011, 7:39:53 UTC
Received	16 Feb 2011, 11:56:02 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1408374
Run time	1 days 0 hours 41 min 21 sec
CPU time	22 hours 29 min 4 sec
Validate state	Invalid
Credit	432.43
Device peak FLOPS	2.72 GFLOPS
Application version	UK Met Office FAMOUS v6.11 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 1 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Signal 1 received, exiting... Signal 15 received, exiting... (6262): called boinc_finish Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/jed/BOINC/projects/climateprediction.net/famous_x3ax_1499_200_007066112/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA00Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16553, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/jed/BOINC/projects/climateprediction.net/famous_x3ax_1499_200_007066112/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083DEAA4 Unknown Unknown Unknown famous_um_6.11_i6 083DDD57 Unknown Unknown Unknown famous_um_6.11_i6 080F5BD7 Unknown Unknown Unknown famous_um_6.11_i6 0816C400 Unknown Unknown Unknown famous_um_6.11_i6 081756AB Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F75DDC0E Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16553, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/jed/BOINC/projects/climateprediction.net/famous_x3ax_1499_200_007066112/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083DF6B0 Unknown Unknown Unknown famous_um_6.11_i6 080F5531 Unknown Unknown Unknown famous_um_6.11_i6 0816C400 Unknown Unknown Unknown famous_um_6.11_i6 081756AB Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F7617C0E Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16553, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /home/jed/BOINC/projects/climateprediction.net/famous_x3ax_1499_200_007066112/dataout/stdout_um.txt Image PC Routine Line Source famous_um_6.11_i6 0840BA88 Unknown Unknown Unknown famous_um_6.11_i6 0840A221 Unknown Unknown Unknown famous_um_6.11_i6 083ED787 Unknown Unknown Unknown famous_um_6.11_i6 083BA000 Unknown Unknown Unknown famous_um_6.11_i6 083DF6B0 Unknown Unknown Unknown famous_um_6.11_i6 083DDD57 Unknown Unknown Unknown famous_um_6.11_i6 080F5508 Unknown Unknown Unknown famous_um_6.11_i6 0816C400 Unknown Unknown Unknown famous_um_6.11_i6 081756AB Unknown Unknown Unknown famous_um_6.11_i6 08325C74 Unknown Unknown Unknown famous_um_6.11_i6 08330BDF Unknown Unknown Unknown famous_um_6.11_i6 083415D2 Unknown Unknown Unknown libc.so.6 F75CBC0E Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=16553, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( (16553): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
16 Feb 2011 06:41:36	1007907	12574453	famous_x3ax_1499_200_007066112_1	131,066	80,274	0.6125
16 Feb 2011 04:52:38	1007907	12574453	famous_x3ax_1499_200_007066112_1	121,706	74,591	0.6129
16 Feb 2011 02:49:31	1007907	12574453	famous_x3ax_1499_200_007066112_1	112,346	68,340	0.6083
16 Feb 2011 00:50:53	1007907	12574453	famous_x3ax_1499_200_007066112_1	102,986	62,420	0.6061
15 Feb 2011 22:58:34	1007907	12574453	famous_x3ax_1499_200_007066112_1	93,626	56,505	0.6035
12 Feb 2011 15:22:51	1007907	12574453	famous_x3ax_1499_200_007066112_1	84,266	50,930	0.6044
12 Feb 2011 13:37:20	1007907	12574453	famous_x3ax_1499_200_007066112_1	74,906	45,254	0.6041
12 Feb 2011 12:06:00	1007907	12574453	famous_x3ax_1499_200_007066112_1	65,546	39,516	0.6029
12 Feb 2011 10:32:29	1007907	12574453	famous_x3ax_1499_200_007066112_1	56,186	34,037	0.6058
12 Feb 2011 08:55:28	1007907	12574453	famous_x3ax_1499_200_007066112_1	46,826	28,526	0.6092
12 Feb 2011 07:07:00	1007907	12574453	famous_x3ax_1499_200_007066112_1	37,466	22,799	0.6085
12 Feb 2011 05:36:59	1007907	12574453	famous_x3ax_1499_200_007066112_1	28,106	17,314	0.6160
12 Feb 2011 03:49:48	1007907	12574453	famous_x3ax_1499_200_007066112_1	18,746	11,620	0.6199
12 Feb 2011 02:07:11	1007907	12574453	famous_x3ax_1499_200_007066112_1	9,386	5,738	0.6113