Task 15655944

Name	hadcm3n_3khm_1940_40_008259820_3
Workunit	8414944
Created	10 Mar 2013, 14:32:19 UTC
Sent	10 Mar 2013, 14:32:27 UTC
Report deadline	9 Jun 2013, 21:59:38 UTC
Received	13 Apr 2013, 15:15:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1271162
Run time	7 days 3 hours 4 min 48 sec
CPU time	7 days 0 hours 13 min 17 sec
Validate state	Invalid
Credit	7,776.00
Device peak FLOPS	4.06 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 i686-pc-linux-gnu
Stderr	<core_client_version>7.0.27</core_client_version> <![CDATA[ <message> process exited with code 22 (0x16, -234) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 21:29:22 (1939): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:30:47 (22873): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:31:31 (22910): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:32:12 (22946): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:34:57 (23314): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:47:00 (23362): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 21:48:51 (24138): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 21:57:28 (3641): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: Write Failed: No space left on device BUFFOUT: C I/O Error - Return code = 1 Model crashed: WRITHEAD: I/O error tmp/pipe_dummy 2048 forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_3khm_1940_40_008259820/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822D68F Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F75164D3 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28325, iMonCtr=1 Model crash detected, will try to restart... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_3khm_1940_40_008259820/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown Unknown hadcm3n_um_6.07_i 0822D868 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818A767 Unknown Unknown Unknown hadcm3n_um_6.07_i 0818D749 Unknown Unknown Unknown hadcm3n_um_6.07_i 08391957 Unknown Unknown Unknown hadcm3n_um_6.07_i 0838F8B7 Unknown Unknown Unknown hadcm3n_um_6.07_i 0839BDF8 Unknown Unknown Unknown libc.so.6 F74E74D3 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=28325, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... forrtl: No space left on device forrtl: severe (38): error during write, unit 6, file /var/lib/boinc-client/projects/climateprediction.net/hadcm3n_3khm_1940_40_008259820/dataout/stdout_um.txt Image PC Routine Line Source hadcm3n_um_6.07_i 0848EB7D Unknown Unknown Unknown hadcm3n_um_6.07_i 0848D975 Unknown Unknown Unknown hadcm3n_um_6.07_i 0845F3CF Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F90D Unknown Unknown Unknown hadcm3n_um_6.07_i 0841F257 Unknown Unknown Unknown hadcm3n_um_6.07_i 08450E2C Unknown Unknown Unknown hadcm3n_um_6.07_i 0844E937 Unknown Unknown UnkCalled boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Apr 2013 11:40:23	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	648,000	588,419	0.9081
12 Apr 2013 05:03:03	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	622,080	564,736	0.9078
11 Apr 2013 21:09:39	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	596,160	540,765	0.9071
11 Apr 2013 07:38:41	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	570,240	516,808	0.9063
10 Apr 2013 23:56:10	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	544,320	492,786	0.9053
10 Apr 2013 16:31:01	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	518,400	468,722	0.9042
10 Apr 2013 09:53:37	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	492,480	445,236	0.9041
10 Apr 2013 03:11:05	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	466,560	421,592	0.9036
24 Mar 2013 00:09:11	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	440,640	397,922	0.9031
23 Mar 2013 17:11:05	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	414,720	374,199	0.9023
23 Mar 2013 10:18:54	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	388,800	350,252	0.9009
23 Mar 2013 03:41:23	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	362,880	326,686	0.9003
22 Mar 2013 20:55:52	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	336,960	303,138	0.8996
17 Mar 2013 02:22:35	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	311,040	279,741	0.8994
16 Mar 2013 19:54:03	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	285,120	256,512	0.8997
16 Mar 2013 13:20:06	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	259,200	233,180	0.8996
16 Mar 2013 06:47:34	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	233,280	209,970	0.9001
16 Mar 2013 00:15:15	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	207,360	186,650	0.9001
15 Mar 2013 17:46:43	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	181,440	163,392	0.9005
15 Mar 2013 11:19:10	1271162	15655944	hadcm3n_3khm_1940_40_008259820_3	155,520	140,373	0.9026