Task 16229986

Name	hadcm3n_7js2_1980_40_008436917_1
Workunit	8587773
Created	14 Jan 2014, 6:17:29 UTC
Sent	14 Jan 2014, 6:20:18 UTC
Report deadline	15 Apr 2014, 13:47:29 UTC
Received	27 Apr 2014, 18:13:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	871213
Run time	15 days 14 hours 8 min 3 sec
CPU time	12 days 1 hours 14 min 59 sec
Validate state	Invalid
Credit	7,464.96
Device peak FLOPS	2.59 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.2.42</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 02:35:14 (2368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 03:10:34 (1860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:11:23 (4352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 04:10:36 (4112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:11:40 (2628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 05:10:42 (4928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:11:24 (4816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 06:27:53 (3552): No heartbeat from core client for 30 sec - exiting 06:27:55 (3552): No heartbeat from core client for 30 sec - exiting 06:27:56 (3552): No heartbeat from core client for 30 sec - exiting 06:27:57 (3552): No heartbeat from core client for 30 sec - exiting 06:27:58 (3552): No heartbeat from core client for 30 sec - exiting 06:27:59 (3552): No heartbeat from core client for 30 sec - exiting 06:28:00 (3552): No heartbeat from core client for 30 sec - exiting 06:28:01 (3552): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:10:49 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 23:12:47 (2936): No heartbeat from core client for 30 sec - exiting 23:12:49 (2936): No heartbeat from core client for 30 sec - exiting 23:12:50 (2936): No heartbeat from core client for 30 sec - exiting 23:12:51 (2936): No heartbeat from core client for 30 sec - exiting 23:12:52 (2936): No heartbeat from core client for 30 sec - exiting 23:12:53 (2936): No heartbeat from core client for 30 sec - exiting 23:12:54 (2936): No heartbeat from core client for 30 sec - exiting 23:12:55 (2936): No heartbeat from core client for 30 sec - exiting 23:12:56 (2936): No heartbeat from core client for 30 sec - exiting 23:12:57 (2936): No heartbeat from core client for 30 sec - exiting 23:12:58 (2936): No heartbeat from core client for 30 sec - exiting 23:12:59 (2936): No heartbeat from core client for 30 sec - exiting 23:13:00 (2936): No heartbeat from core client for 30 sec - exiting 23:13:01 (2936): No heartbeat from core client for 30 sec - exiting 23:13:02 (2936): No heartbeat from core client for 30 sec - exiting 23:13:03 (2936): No heartbeat from core client for 30 sec - exiting 23:13:04 (2936): No heartbeat from core client for 30 sec - exiting 23:13:05 (2936): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:16:23 (2012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 forrtl: There is not enough space on the disk. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4800, iMonCtr=1 Model crash detected, will try to restart... BUFFOUT: C I/O Error - Return code = 32 Model crashed: WRITDUMP: BAD BUFFOUT OF DATA tmp/pipe_dummy 2048 CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:06:24 (4404): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 00:31:28 (6104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:43:34 (4392): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:12:19 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:08:20 (2148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:09:13 (4416): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:01:12 (3276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:03 (4520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:57:53 (1476): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:41:07 (1928): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 15:28:11 (3268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3528, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3528, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3528, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5064, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=676, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=676, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Apr 2014 21:18:17	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	622,080	1,017,440	1.6355
22 Apr 2014 06:50:22	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	596,160	973,826	1.6335
21 Apr 2014 16:40:54	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	570,240	926,860	1.6254
21 Apr 2014 02:24:03	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	544,320	882,295	1.6209
20 Apr 2014 10:13:21	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	518,400	830,922	1.6029
19 Apr 2014 20:24:52	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	492,480	787,434	1.5989
19 Apr 2014 06:29:14	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	466,560	744,470	1.5957
18 Apr 2014 15:54:21	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	440,640	699,992	1.5886
17 Apr 2014 23:30:19	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	414,720	649,854	1.5670
17 Apr 2014 06:58:56	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	388,800	599,975	1.5431
16 Apr 2014 17:05:19	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	362,880	556,268	1.5329
16 Apr 2014 02:04:27	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	336,960	509,995	1.5135
15 Apr 2014 10:29:18	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	311,040	464,434	1.4932
14 Apr 2014 17:58:54	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	285,120	418,230	1.4669
14 Apr 2014 02:19:19	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	259,200	490,426	1.8921
13 Apr 2014 12:11:50	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	233,280	445,690	1.9105
12 Apr 2014 22:04:16	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	207,360	398,996	1.9242
12 Apr 2014 07:40:26	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	181,440	353,153	1.9464
11 Apr 2014 17:41:49	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	155,520	308,260	1.9821
11 Apr 2014 01:46:55	871213	16229986	hadcm3n_7js2_1980_40_008436917_1	129,600	258,548	1.9950