Task 17564841

Name	hadcm3n_sa1p_1980_40_009270471_1
Workunit	9363387
Created	9 Dec 2014, 23:25:52 UTC
Sent	9 Dec 2014, 23:26:00 UTC
Report deadline	11 Mar 2015, 6:53:11 UTC
Received	20 Jan 2015, 14:43:14 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1031234
Run time	19 days 22 hours 46 min 30 sec
CPU time	18 days 12 hours 12 min 12 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	1.60 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.2.33</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: STWORK : Error in PP_FILE BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_sa1p_1980_40_009270471/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08455448 Unknown Unknown Unknown hadcm3n_um_6.07_i 0829D502 Unknown Unknown Unknown hadcm3n_um_6.07_i 08340D81 Unknown Unknown Unknown hadcm3n_um_6.07_i 08194301 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 001BFD36 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2601, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc/projects/climateprediction.net/hadcm3n_sa1p_1980_40_009270471/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F9Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2601, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
20 Jan 2015 08:00:53	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	933,120	1,606,275	1.7214
19 Jan 2015 13:14:33	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	907,200	1,560,765	1.7204
18 Jan 2015 22:05:12	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	881,280	1,514,947	1.7190
18 Jan 2015 06:57:54	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	855,360	1,469,310	1.7178
17 Jan 2015 08:55:34	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	829,440	1,424,064	1.7169
16 Jan 2015 15:18:39	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	803,520	1,378,479	1.7156
15 Jan 2015 22:51:18	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	777,600	1,333,563	1.7150
15 Jan 2015 09:26:56	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	751,680	1,289,783	1.7159
14 Jan 2015 20:29:54	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	725,760	1,245,894	1.7167
14 Jan 2015 07:26:45	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	699,840	1,202,053	1.7176
13 Jan 2015 15:58:32	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	673,920	1,158,216	1.7186
13 Jan 2015 02:33:34	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	648,000	1,114,291	1.7196
12 Jan 2015 14:19:12	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	622,080	1,070,423	1.7207
11 Jan 2015 21:20:39	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	596,160	1,026,311	1.7215
11 Jan 2015 08:13:44	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	570,240	981,705	1.7216
25 Dec 2014 11:13:31	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	544,320	937,838	1.7230
24 Dec 2014 16:02:59	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	518,400	893,711	1.7240
23 Dec 2014 21:56:05	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	492,480	849,729	1.7254
23 Dec 2014 09:13:40	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	466,560	806,032	1.7276
22 Dec 2014 16:18:15	1031234	17564841	hadcm3n_sa1p_1980_40_009270471_1	440,640	761,894	1.7291