Task 15548183

Name	hadcm3n_3b20_1940_40_008263609_1
Workunit	8418733
Created	17 Jan 2013, 9:17:41 UTC
Sent	17 Jan 2013, 9:18:04 UTC
Report deadline	18 Apr 2013, 16:45:15 UTC
Received	22 Feb 2013, 14:08:37 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1171203
Run time	15 days 2 hours 38 min 39 sec
CPU time	13 days 23 hours 34 min 42 sec
Validate state	Invalid
Credit	8,087.04
Device peak FLOPS	3.08 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>6.10.17</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 01:48:36 (12406): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:54:50 (12911): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:56:40 (13039): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:58:30 (13088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 05:09:09 (7890): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:09:15 (7890): No heartbeat from core client for 30 sec - exiting 05:09:16 (7890): No heartbeat from core client for 30 sec - exiting 05:09:17 (7890):13:15:28 (3678): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:15:29 (3678): No heartbeat from core client for 30 sec - exiting 13:15:30 (3678): No heartbeat from core client for 30 sec - exiting 13:15:31 (3678): No heartbeat from core client for 30 sec - exiting 13:15:32 (3678): No heartbeat from core client for 30 sec - exiting 13:15:33 (3678): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 SETPOS: Seek Failed: No space left on device SETPOS: Unit 22 to Word Address 292864 Failed with Error Code -1 Model crashed: SETPOS: Unit 22 to Word Address 292864 Fail BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_3b20_1940_40_008263609/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08451069 Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0836D10D Unknown Unknown Unknown hadcm3n_um_6.07_i 082EB086 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F66D Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F754EBD6 Unknown Unknown Unknown hadcm3n_um_6.07_i 0804CB11 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=23621, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Feb 2013 08:06:51	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	673,920	1,226,213	1.8195
21 Feb 2013 13:40:22	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	648,000	1,195,084	1.8443
21 Feb 2013 05:03:52	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	622,080	1,161,630	1.8673
20 Feb 2013 09:43:17	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	596,160	1,127,406	1.8911
19 Feb 2013 13:05:13	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	570,240	1,093,208	1.9171
19 Feb 2013 04:02:13	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	544,320	1,059,198	1.9459
18 Feb 2013 08:58:22	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	518,400	1,024,088	1.9755
17 Feb 2013 14:10:21	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	492,480	982,803	1.9956
16 Feb 2013 22:19:41	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	466,560	933,492	2.0008
16 Feb 2013 06:03:24	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	440,640	883,356	2.0047
15 Feb 2013 08:22:04	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	414,720	836,393	2.0168
14 Feb 2013 07:25:26	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	388,800	783,745	2.0158
13 Feb 2013 09:16:27	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	362,880	736,558	2.0298
12 Feb 2013 08:19:30	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	336,960	687,739	2.0410
11 Feb 2013 07:15:52	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	311,040	638,184	2.0518
10 Feb 2013 18:06:49	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	285,120	588,987	2.0658
09 Feb 2013 18:21:43	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	259,200	540,423	2.0850
09 Feb 2013 04:20:47	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	233,280	491,823	2.1083
07 Feb 2013 13:45:49	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	207,360	443,743	2.1400
05 Feb 2013 13:30:20	1171203	15548183	hadcm3n_3b20_1940_40_008263609_1	181,440	394,093	2.1720