Task 14781787

Name	hadcm3n_o1br_2060_40_007998831_4
Workunit	8153945
Created	8 Jun 2012, 18:24:59 UTC
Sent	8 Jun 2012, 18:25:18 UTC
Report deadline	8 Sep 2012, 1:52:29 UTC
Received	6 Aug 2012, 4:59:30 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1111322
Run time	20 days 22 hours 57 min 6 sec
CPU time	18 days 21 hours 30 min 56 sec
Validate state	Invalid
Credit	4,665.60
Device peak FLOPS	1.54 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 07:44:11 (5620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:46:12 (5180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 07:48:38 (3108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 07:42:37 (4168): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 07:43:07 (2820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5316, iMonCtr=1 Model crash detected, will try to restart... 09:21:25 (5764): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:21:26 (5764): No heartbeat from core client for 30 sec - exiting 09:22:18 (1040): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:22:20 (1040): No heartbeat from core client for 30 sec - exiting 00:13:34 (5024): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:13:44 (5024): No heartbeat from core client for 30 sec - exiting 08:49:53 (1632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:39 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:09:02 (4840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:20:57 (516): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:16:29 (3868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:24:02 (4148): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 08:12:16 (3488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:17:10 (3456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:24:28 (9456): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: Access is denied. 08:32:27 (4852): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:14:20 (4380): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 09:14:21 (4380): No heartbeat from core client for 30 sec - exiting 08:55:50 (2792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:55:51 (2792): No heartbeat from core client for 30 sec - exiting 08:55:52 (2792): No heartbeat from core client for 30 sec - exiting 08:55:53 (2792): No heartbeat from core client for 30 sec - exiting 08:55:54 (2792): No heartbeat from core client for 30 sec - exiting 08:55:55 (2792): No heartbeat from core client for 30 sec - exiting 08:55:56 (2792): No heartbeat from core client for 30 sec - exiting 08:55:57 (2792): No heartbeat from core client for 30 sec - exiting 08:55:58 (2792): No heartbeat from core client for 30 sec - exiting 08:56:00 (2792): No heartbeat from core client for 30 sec - exiting 08:56:01 (2792): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:18:03 (7600): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:09:33 (14180): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:02:58 (13564): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:06:59 (9488): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:17 (12628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:15:18 (12628): No heartbeat from core client for 30 sec - exiting 09:26:06 (7120): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 16:53:23 (11680): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:33 (8424): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:51:49 (10608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:46:49 (5448): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:45 (3324): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:48:59 (5108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:49:00 (5108): No heartbeat from core client for 30 sec - exiting 23:49:02 (5108): No heartbeat from core client for 30 sec - exiting 23:49:03 (5108): No heartbeat from core client for 30 sec - exiting 23:49:04 (5108): No heartbeat from core client for 30 sec - exiting 23:49:05 (5108): No heartbeat from core client for 30 sec - exiting 23:49:06 (5108): No heartbeat from core client for 30 sec - exiting 23:49:07 (5108): No heartbeat from core client for 30 sec - exiting 23:49:08 (5108): No heartbeat from core client for 30 sec - exiting 23:49:09 (5108): No heartbeat from core client for 30 sec - exiting 23:49:10 (5108): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. 23:58:07 (7620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:21:56 (5976): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:32 (4868): No heartbeat from core client for 30 sec - exiting 02:51:33 (4868): No heartbeat from core client for 30 sec - exiting 02:51:35 (4868): No heartbeat from core client for 30 sec - exiting 02:51:36 (4868): No heartbeat from core client for 30 sec - exiting 02:51:37 (4868): No heartbeat from core client for 30 sec - exiting 02:51:39 (4868): No heartbeat from core client for 30 sec - exiting 02:51:40 (4868): No heartbeat from core client for 30 sec - exiting 02:51:42 (4868): No heartbeat from core client for 30 sec - exiting 02:51:44 (4868): No heartbeat from core client for 30 sec - exiting 02:51:45 (4868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:51:46 (4868): No heartbeat from core client for 30 sec - exiting 04:52:06 (5844): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:53:27 (4772): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5600, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5600, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 02:51:04 (5808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 06:55:20 (9848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:49:30 (3800): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 04:58:55 (276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4632, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5640, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
14 Jul 2012 00:28:24	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	388,800	1,558,490	4.0085
12 Jul 2012 12:07:27	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	362,880	1,438,482	3.9641
09 Jul 2012 18:35:21	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	336,960	1,328,158	3.9416
08 Jul 2012 08:44:55	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	311,040	1,220,179	3.9229
04 Jul 2012 12:12:33	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	285,120	1,107,772	3.8853
03 Jul 2012 08:33:02	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	259,200	1,012,353	3.9057
28 Jun 2012 22:34:04	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	233,280	909,548	3.8990
27 Jun 2012 14:23:21	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	207,360	801,104	3.8633
26 Jun 2012 07:03:08	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	181,440	698,584	3.8502
16 Jun 2012 18:45:45	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	155,520	593,566	3.8167
15 Jun 2012 13:24:29	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	129,600	494,764	3.8176
14 Jun 2012 09:37:58	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	103,680	401,483	3.8723
12 Jun 2012 21:55:21	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	77,760	309,079	3.9748
11 Jun 2012 16:44:21	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	51,840	226,997	4.3788
10 Jun 2012 09:18:28	1111322	14781787	hadcm3n_o1br_2060_40_007998831_4	25,920	123,467	4.7634