Task 13559773

Name	hadcm3n_yinw_1900_40_007527000_0
Workunit	7724475
Created	28 Oct 2011, 13:48:46 UTC
Sent	29 Oct 2011, 3:58:16 UTC
Report deadline	28 Jan 2012, 11:25:27 UTC
Received	2 Dec 2011, 10:17:56 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1148818
Run time	27 days 0 hours 25 min 44 sec
CPU time	24 days 15 hours 12 min 59 sec
Validate state	Invalid
Credit	9,020.16
Device peak FLOPS	2.29 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 23:59:51 (25717): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10545, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_yinw_1900_40_007527000/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0824EBA6 Unknown Unknown Unknown hadcm3n_um_6.07_i 082712DC Unknown Unknown Unknown hadcm3n_um_6.07_i 08177076 Unknown Unknown Unknown hadcm3n_um_6.07_i 081795D4 Unknown Unknown Unknown hadcm3n_um_6.07_i 081908AF Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F762FBD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=10545, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
01 Dec 2011 00:12:42	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	751,680	2,083,454	2.7717
29 Nov 2011 21:02:39	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	725,760	2,010,951	2.7708
28 Nov 2011 14:17:05	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	699,840	1,937,192	2.7680
27 Nov 2011 16:32:45	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	673,920	1,866,189	2.7692
25 Nov 2011 06:20:18	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	648,000	1,798,838	2.7760
24 Nov 2011 10:27:00	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	622,080	1,731,243	2.7830
23 Nov 2011 09:02:53	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	596,160	1,659,751	2.7841
22 Nov 2011 02:22:17	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	570,240	1,587,991	2.7848
20 Nov 2011 15:29:21	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	544,320	1,514,632	2.7826
19 Nov 2011 17:21:00	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	518,400	1,444,656	2.7868
18 Nov 2011 18:32:23	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	492,480	1,371,911	2.7857
17 Nov 2011 10:38:52	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	466,560	1,296,818	2.7795
16 Nov 2011 07:49:28	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	440,640	1,225,959	2.7822
15 Nov 2011 18:12:38	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	414,720	1,159,059	2.7948
15 Nov 2011 18:12:38	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	388,800	1,088,998	2.8009
15 Nov 2011 18:12:38	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	362,880	1,017,687	2.8045
15 Nov 2011 18:12:38	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	336,960	948,338	2.8144
15 Nov 2011 18:12:38	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	311,040	875,520	2.8148
15 Nov 2011 18:12:38	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	285,120	803,449	2.8179
09 Nov 2011 22:19:11	1148818	13559773	hadcm3n_yinw_1900_40_007527000_0	259,200	729,515	2.8145