Task 14784076

Name	hadcm3n_o06x_2100_40_008000092_0
Workunit	8155206
Created	10 Jun 2012, 23:47:26 UTC
Sent	10 Jun 2012, 23:47:36 UTC
Report deadline	10 Sep 2012, 7:14:47 UTC
Received	23 Jun 2012, 22:56:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	459505
Run time	8 days 8 hours 39 min 31 sec
CPU time	7 days 6 hours 54 min 26 sec
Validate state	Invalid
Credit	1,866.24
Device peak FLOPS	1.38 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 06:01:40 (2508): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:05:10 (5300): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:06:07 (4396): No heartbeat from core client for 30 sec - exiting 06:06:08 (4396): No heartbeat from core client for 30 sec - exiting 06:06:09 (4396): No heartbeat from core client for 30 sec - exiting 06:06:11 (4396): No heartbeat from core client for 30 sec - exiting 06:06:12 (4396): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:09 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:07:10 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 07:49:54 (520): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:49:55 (520): No heartbeat from core client for 30 sec - exiting 07:49:56 (520): No heartbeat from core client for 30 sec - exiting 07:51:24 (4664): No heartbeat from core client for 30 sec - exiting 07:51:26 (4664): No heartbeat from core client for 30 sec - exiting 07:51:27 (4664): No heartbeat from core client for 30 sec - exiting 07:51:28 (4664): No heartbeat from core client for 30 sec - exiting 07:51:29 (4664): No heartbeat from core client for 30 sec - exiting 07:51:30 (4664): No heartbeat from core client for 30 sec - exiting 07:51:31 (4664): No heartbeat from core client for 30 sec - exiting 07:51:32 (4664): No heartbeat from core client for 30 sec - exiting 07:51:33 (4664): No heartbeat from core client for 30 sec - exiting 07:51:34 (4664): No heartbeat from core client for 30 sec - exiting 07:51:35 (4664): No heartbeat from core client for 30 sec - exiting 07:51:36 (4664): No heartbeat from core client for 30 sec - exiting 07:51:37 (4664): No heartbeat from core client for 30 sec - exiting 07:51:38 (4664): No heartbeat from core client for 30 sec - exiting 07:51:39 (4664): No heartbeat from core client for 30 sec - exiting 07:51:40 (4664): No heartbeat from core client for 30 sec - exiting 07:51:41 (4664): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:51 (2840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:57:52 (2840): No heartbeat from core client for 30 sec - exiting 07:59:15 (4972): No heartbeat from core client for 30 sec - exiting 07:59:16 (4972): No heartbeat from core client for 30 sec - exiting 07:59:17 (4972): No heartbeat from core client for 30 sec - exiting 07:59:18 (4972): No heartbeat from core client for 30 sec - exiting 07:59:19 (4972): No heartbeat from core client for 30 sec - exiting 07:59:20 (4972): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:45:20 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:45:22 (4872): No heartbeat from core client for 30 sec - exiting 03:45:23 (4872): No heartbeat from core client for 30 sec - exiting 03:45:24 (4872): No heartbeat from core client for 30 sec - exiting 03:50:03 (5304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:50:04 (5304): No heartbeat from core client for 30 sec - exiting 03:51:24 (6848): No heartbeat from core client for 30 sec - exiting 03:51:25 (6848): No heartbeat from core client for 30 sec - exiting 03:51:26 (6848): No heartbeat from core client for 30 sec - exiting 03:51:27 (6848): No heartbeat from core client for 30 sec - exiting 03:51:28 (6848): No heartbeat from core client for 30 sec - exiting 03:51:29 (6848): No heartbeat from core client for 30 sec - exiting 03:51:30 (6848): No heartbeat from core client for 30 sec - exiting 03:51:31 (6848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:51:32 (6848): No heartbeat from core client for 30 sec - exiting 03:51:33 (6848): No heartbeat from core client for 30 sec - exiting 03:51:34 (6848): No heartbeat from core client for 30 sec - exiting 03:51:35 (6848): No heartbeat from core client for 30 sec - exiting 03:51:36 (6848): No heartbeat from core client for 30 sec - exiting 03:51:37 (6848): No heartbeat from core client for 30 sec - exiting 03:51:38 (6848): No heartbeat from core client for 30 sec - exiting 03:51:39 (6848): No heartbeat from core client for 30 sec - exiting 03:51:40 (6848): No heartbeat from core client for 30 sec - exiting 03:51:41 (6848): No heartbeat from core client for 30 sec - exiting 03:54:57 (9908): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:54:58 (9908): No heartbeat from core client for 30 sec - exiting 04:04:53 (5852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:04:54 (5852): No heartbeat from core client for 30 sec - exiting 04:25:18 (5836): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:14:50 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 10:14:51 (4084): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:21:02 (6516): No heartbeat from core client for 30 sec - exiting 17:21:03 (6516): No heartbeat from core client for 30 sec - exiting 17:21:04 (6516): No heartbeat from core client for 30 sec - exiting 17:21:05 (6516): No heartbeat from core client for 30 sec - exiting 17:21:06 (6516): No heartbeat from core client for 30 sec - exiting 17:21:07 (6516): No heartbeat from core client for 30 sec - exiting 17:21:08 (6516): No heartbeat from core client for 30 sec - exiting 17:21:09 (6516): No heartbeat from core client for 30 sec - exiting 17:21:10 (6516): No heartbeat from core client for 30 sec - exiting 17:21:11 (6516): No heartbeat from core client for 30 sec - exiting 17:21:12 (6516): No heartbeat from core client for 30 sec - exiting 17:21:13 (6516): No heartbeat from core client for 30 sec - exiting 17:21:14 (6516): No heartbeat from core client for 30 sec - exiting 17:21:15 (6516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:21:16 (6516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:06:27 (2680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:06:29 (2680): No heartbeat from core client for 30 sec - exiting 13:11:51 (5748): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:11:52 (5748): No heartbeat from core client for 30 sec - exiting 13:17:17 (256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:17:18 (256): No heartbeat from core client for 30 sec - exiting 13:20:16 (4576): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:21:28 (4864): No heartbeat from core client for 30 sec - exiting 13:21:30 (4864): No heartbeat from core client for 30 sec - exiting 13:21:31 (4864): No heartbeat from core client for 30 sec - exiting 13:21:32 (4864): No heartbeat from core client for 30 sec - exiting 13:21:33 (4864): No heartbeat from core client for 30 sec - exiting 13:21:34 (4864): No heartbeat from core client for 30 sec - exiting 13:21:35 (4864): No heartbeat from core client for 30 sec - exiting 13:21:36 (4864): No heartbeat from core client for 30 sec - exiting 13:21:37 (4864): No heartbeat from core client for 30 sec - exiting 13:21:38 (4864): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:25:00 (5352): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:25:02 (5352): No heartbeat from core client for 30 sec - exiting 13:25:03 (5352): No heartbeat from core client for 30 sec - exiting 13:26:33 (6088): No heartbeat from core client for 30 sec - exiting 13:26:34 (6088): No heartbeat from core client for 30 sec - exiting 13:26:35 (6088): No heartbeat from core client for 30 sec - exiting 13:26:36 (6088): No heartbeat from core client for 30 sec - exiting 13:26:37 (6088): No heartbeat from core client for 30 sec - exiting 13:26:38 (6088): No heartbeat from core client for 30 sec - exiting 13:26:40 (6088): No heartbeat from core client for 30 sec - exiting 13:26:41 (6088): No heartbeat from core client for 30 sec - exiting 13:26:42 (6088): No heartbeat from core client for 30 sec - exiting 13:26:43 (6088): No heartbeat from core client for 30 sec - exiting 13:26:44 (6088): No heartbeat from core client for 30 sec - exiting 13:26:45 (6088): No heartbeat from core client for 30 sec - exiting 13:26:46 (6088): No heartbeat from core client for 30 sec - exiting 13:26:47 (6088): No heartbeat from core client for 30 sec - exiting 13:26:48 (6088): No heartbeat from core client for 30 sec - exiting 13:26:49 (6088): No heartbeat from core client for 30 sec - exiting 13:26:51 (6088): No heartbeat from core client for 30 sec - exiting 13:26:52 (6088): No heartbeat from core client for 30 sec - exiting 13:26:53 (6088): No heartbeat from core client for 30 sec - exiting 13:26:54 (6088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:44:06 (3760): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 23:32:16 (4612): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:32:18 (4612): No heartbeat from core client for 30 sec - exiting 23:32:19 (4612): No heartbeat from core client for 30 sec - exiting 23:32:20 (4612): No heartbeat from core client for 30 sec - exiting 23:32:21 (4612): No heartbeat from core client for 30 sec - exiting 23:34:27 (852): No heartbeat from core client for 30 sec - exiting 23:34:28 (852): No heartbeat from core client for 30 sec - exiting 23:34:29 (852): No heartbeat from core client for 30 sec - exiting 23:34:31 (852): No heartbeat from core client for 30 sec - exiting 23:34:32 (852): No heartbeat from core client for 30 sec - exiting 23:34:33 (852): No heartbeat from core client for 30 sec - exiting 23:34:34 (852): No heartbeat from core client for 30 sec - exiting 23:34:35 (852): No heartbeat from core client for 30 sec - exiting 23:34:36 (852): No heartbeat from core client for 30 sec - exiting 23:34:37 (852): No heartbeat from core client for 30 sec - exiting 23:34:38 (852): No heartbeat from core client for 30 sec - exiting 23:34:39 (852): No heartbeat from core client for 30 sec - exiting 23:34:40 (852): No heartbeat from core client for 30 sec - exiting 23:34:41 (852): No heartbeat from core client for 30 sec - exiting 23:34:42 (852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:39:52 (2164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:39:53 (2164): No heartbeat from core client for 30 sec - exiting 23:39:54 (2164): No heartbeat from core client for 30 sec - exiting 23:41:56 (5772): No heartbeat from core client for 30 sec - exiting 23:41:57 (5772): No heartbeat from core client for 30 sec - exiting 23:41:58 (5772): No heartbeat from core client for 30 sec - exiting 23:41:59 (5772): No heartbeat from core client for 30 sec - exiting 23:42:00 (5772): No heartbeat from core client for 30 sec - exiting 23:42:01 (5772): No heartbeat from core client for 30 sec - exiting 23:42:02 (5772): No heartbeat from core client for 30 sec - exiting 23:42:03 (5772): No heartbeat from core client for 30 sec - exiting 23:42:04 (5772): No heartbeat from core client for 30 sec - exiting 23:42:05 (5772): No heartbeat from core client for 30 sec - exiting 23:42:06 (5772): No heartbeat from core client for 30 sec - exiting 23:42:07 (5772): No heartbeat from core client for 30 sec - exiting 23:42:08 (5772): No heartbeat from core client for 30 sec - exiting 23:42:09 (5772): No heartbeat from core client for 30 sec - exiting 23:42:10 (5772): No heartbeat from core client for 30 sec - exiting 23:42:11 (5772): No heartbeat from core client for 30 sec - exiting 23:42:12 (5772): No heartbeat from core client for 30 sec - exiting 23:42:13 (5772): No heartbeat from core client for 30 sec - exiting 23:42:14 (5772): No heartbeat from core client for 30 sec - exiting 23:42:15 (5772): No heartbeat from core client for 30 sec - exiting 23:42:16 (5772): No heartbeat from core client for 30 sec - exiting 23:42:17 (5772): No heartbeat from core client for 30 sec - exiting 23:42:18 (5772): No heartbeat from core client for 30 sec - exiting 23:42:19 (5772): No heartbeat from core client for 30 sec - exiting 23:42:20 (5772): No heartbeat from core client for 30 sec - exiting 23:42:21 (5772): No heartbeat from core client for 30 sec - exiting 23:42:22 (5772): No heartbeat from core client for 30 sec - exiting 23:42:23 (5772): No heartbeat from core client for 30 sec - exiting 23:42:24 (5772): No heartbeat from core client for 30 sec - exiting 23:42:25 (5772): No heartbeat from core client for 30 sec - exiting 23:42:26 (5772): No heartbeat from core client for 30 sec - exiting 23:42:27 (5772): No heartbeat from core client for 30 sec - exiting 23:42:28 (5772): No heartbeat from core client for 30 sec - exiting 23:42:29 (5772): No heartbeat from core client for 30 sec - exiting 23:42:30 (5772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:46:05 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:46:08 (2968): No heartbeat from core client for 30 sec - exiting 23:47:28 (3268): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:10 (4388): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:52:11 (4388): No heartbeat from core client for 30 sec - exiting 23:52:12 (4388): No heartbeat from core client for 30 sec - exiting 23:52:13 (4388): No heartbeat from core client for 30 sec - exiting 23:54:12 (280): No heartbeat from core client for 30 sec - exiting 23:54:13 (280): No heartbeat from core client for 30 sec - exiting 23:54:14 (280): No heartbeat from core client for 30 sec - exiting 23:54:15 (280): No heartbeat from core client for 30 sec - exiting 23:54:16 (280): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:27 (4604): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:59:28 (4604): No heartbeat from core client for 30 sec - exiting 00:01:20 (5644): No heartbeat from core client for 30 sec - exiting 00:01:22 (5644): No heartbeat from core client for 30 sec - exiting 00:01:23 (5644): No heartbeat from core client for 30 sec - exiting 00:01:24 (5644): No heartbeat from core client for 30 sec - exiting 00:01:25 (5644): No heartbeat from core client for 30 sec - exiting 00:01:26 (5644): No heartbeat from core client for 30 sec - exiting 00:01:27 (5644): No heartbeat from core client for 30 sec - exiting 00:01:28 (5644): No heartbeat from core client for 30 sec - exiting 00:01:29 (5644): No heartbeat from core client for 30 sec - exiting 00:01:30 (5644): No heartbeat from core client for 30 sec - exiting 00:01:31 (5644): No heartbeat from core client for 30 sec - exiting 00:01:32 (5644): No heartbeat from core client for 30 sec - exiting 00:01:33 (5644): No heartbeat from core client for 30 sec - exiting 00:01:34 (5644): No heartbeat from core client for 30 sec - exiting 00:01:35 (5644): No heartbeat from core client for 30 sec - exiting 00:01:36 (5644): No heartbeat from core client for 30 sec - exiting 00:01:37 (5644): No heartbeat from core client for 30 sec - exiting 00:01:38 (5644): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:40 (3200): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:10:42 (3200): No heartbeat from core client for 30 sec - exiting 00:11:48 (4724): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3184, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=980, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3708, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
22 Jun 2012 04:35:41	459505	14784076	hadcm3n_o06x_2100_40_008000092_0	155,520	546,437	3.5136
20 Jun 2012 15:30:54	459505	14784076	hadcm3n_o06x_2100_40_008000092_0	129,600	455,791	3.5169
19 Jun 2012 15:28:30	459505	14784076	hadcm3n_o06x_2100_40_008000092_0	103,680	364,548	3.5161
17 Jun 2012 22:56:32	459505	14784076	hadcm3n_o06x_2100_40_008000092_0	77,760	272,419	3.5033
16 Jun 2012 12:54:31	459505	14784076	hadcm3n_o06x_2100_40_008000092_0	51,840	181,682	3.5047
14 Jun 2012 19:36:54	459505	14784076	hadcm3n_o06x_2100_40_008000092_0	25,920	90,661	3.4977