Task 13390920

Name	hadcm3n_p2c1_1940_40_007420013_2
Workunit	7617648
Created	16 Sep 2011, 3:16:33 UTC
Sent	16 Sep 2011, 3:19:09 UTC
Report deadline	16 Dec 2011, 10:46:20 UTC
Received	24 Dec 2011, 8:04:01 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1055935
Run time	34 days 18 hours 35 min 4 sec
CPU time	33 days 5 hours 46 min 34 sec
Validate state	Invalid
Credit	11,819.52
Device peak FLOPS	1.67 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.6.36</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4232, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4776, iMonCtr=1 Model crash detected, will try to restart... 15:08:22 (504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:08:24 (504): No heartbeat from core client for 30 sec - exiting 15:08:26 (504): No heartbeat from core client for 30 sec - exiting 15:08:27 (504): No heartbeat from core client for 30 sec - exiting 15:08:28 (504): No heartbeat from core client for 30 sec - exiting 15:08:29 (504): No heartbeat from core client for 30 sec - exiting 15:08:30 (504): No heartbeat from core client for 30 sec - exiting 15:08:31 (504): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. 15:11:21 (2968): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 15:11:23 (2968): No heartbeat from core client for 30 sec - exiting 15:11:24 (2968): No heartbeat from core client for 30 sec - exiting 15:11:25 (2968): No heartbeat from core client for 30 sec - exiting 15:11:26 (2968): No heartbeat from core client for 30 sec - exiting 15:11:27 (2968): No heartbeat from core client for 30 sec - exiting 15:11:28 (2968): No heartbeat from core client for 30 sec - exiting 15:11:29 (2968): No heartbeat from core client for 30 sec - exiting 15:11:30 (2968): No heartbeat from core client for 30 sec - exiting 15:11:31 (2968): No heartbeat from core client for 30 sec - exiting 15:11:32 (2968): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2356, selfPID=2356, iMonCtr=1 00:21:47 (2784): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:21:49 (2784): No heartbeat from core client for 30 sec - exiting 00:21:50 (2784): No heartbeat from core client for 30 sec - exiting 00:21:51 (2784): No heartbeat from core client for 30 sec - exiting 00:21:53 (2784): No heartbeat from core client for 30 sec - exiting 00:21:54 (2784): No heartbeat from core client for 30 sec - exiting 00:21:55 (2784): No heartbeat from core client for 30 sec - exiting 00:21:56 (2784): No heartbeat from core client for 30 sec - exiting 00:21:57 (2784): No heartbeat from core client for 30 sec - exiting 00:21:58 (2784): No heartbeat from core client for 30 sec - exiting 00:21:59 (2784): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2816, selfPID=2816, iMonCtr=1 00:23:37 (2344): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:23:39 (2344): No heartbeat from core client for 30 sec - exiting 00:23:40 (2344): No heartbeat from core client for 30 sec - exiting 00:23:41 (2344): No heartbeat from core client for 30 sec - exiting 00:23:42 (2344): No heartbeat from core client for 30 sec - exiting 00:23:43 (2344): No heartbeat from core client for 30 sec - exiting 00:23:44 (2344): No heartbeat from core client for 30 sec - exiting 00:23:45 (2344): No heartbeat from core client for 30 sec - exiting 00:23:46 (2344): No heartbeat from core client for 30 sec - exiting 00:23:47 (2344): No heartbeat from core client for 30 sec - exiting 00:23:48 (2344): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4176, selfPID=4176, iMonCtr=1 00:25:28 (4872): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:25:29 (4872): No heartbeat from core client for 30 sec - exiting 00:25:30 (4872): No heartbeat from core client for 30 sec - exiting 00:25:31 (4872): No heartbeat from core client for 30 sec - exiting 00:25:32 (4872): No heartbeat from core client for 30 sec - exiting 00:25:33 (4872): No heartbeat from core client for 30 sec - exiting 00:25:34 (4872): No heartbeat from core client for 30 sec - exiting 00:25:35 (4872): No heartbeat from core client for 30 sec - exiting 00:25:36 (4872): No heartbeat from core client for 30 sec - exiting 00:25:37 (4872): No heartbeat from core client for 30 sec - exiting 00:25:39 (4872): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1528, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1528, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1528, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. 00:27:20 (1528): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 00:27:21 (1528): No heartbeat from core client for 30 sec - exiting 00:27:22 (1528): No heartbeat from core client for 30 sec - exiting 00:27:23 (1528): No heartbeat from core client for 30 sec - exiting 00:27:24 (1528): No heartbeat from core client for 30 sec - exiting 00:27:26 (1528): No heartbeat from core client for 30 sec - exiting 00:27:27 (1528): No heartbeat from core client for 30 sec - exiting 00:27:29 (1528): No heartbeat from core client for 30 sec - exiting 00:27:30 (1528): No heartbeat from core client for 30 sec - exiting 00:27:32 (1528): No heartbeat from core client for 30 sec - exiting 00:27:33 (1528): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2816, selfPID=2816, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 01:36:58 (4568): No heartbeat from core client for 30 sec - exiting 01:36:59 (4568): No heartbeat from core client for 30 sec - exiting 01:37:01 (4568): No heartbeat from core client for 30 sec - exiting 01:37:02 (4568): No heartbeat from core client for 30 sec - exiting 01:37:03 (4568): No heartbeat from core client for 30 sec - exiting 01:37:04 (4568): No heartbeat from core client for 30 sec - exiting 01:37:05 (4568): No heartbeat from core client for 30 sec - exiting 01:37:06 (4568): No heartbeat from core client for 30 sec - exiting 01:37:07 (4568): No heartbeat from core client for 30 sec - exiting 01:37:08 (4568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Worker:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4344, iMonCtr=1 Model crash detected, will try to restart... 01:25:38 (5868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:25:39 (5868): No heartbeat from core client for 30 sec - exiting 01:25:40 (5868): No heartbeat from core client for 30 sec - exiting 01:25:41 (5868): No heartbeat from core client for 30 sec - exiting 01:25:43 (5868): No heartbeat from core client for 30 sec - exiting 01:25:44 (5868): No heartbeat from core client for 30 sec - exiting 01:25:45 (5868): No heartbeat from core client for 30 sec - exiting 01:25:46 (5868): No heartbeat from core client for 30 sec - exiting 01:25:47 (5868): No heartbeat from core client for 30 sec - exiting 01:25:48 (5868): No heartbeat from core client for 30 sec - exiting 01:25:49 (5868): No heartbeat from core client for 30 sec - exiting No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5280, selfPID=5280, iMonCtr=1 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 01:57:22 (4260): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:57:23 (4260): No heartbeat from core client for 30 sec - exiting 01:57:24 (4260): No heartbeat from core client for 30 sec - exiting 01:57:25 (4260): No heartbeat from core client for 30 sec - exiting 01:57:26 (4260): No heartbeat from core client for 30 sec - exiting 01:57:27 (4260): No heartbeat from core client for 30 sec - exiting 01:57:28 (4260): No heartbeat from core client for 30 sec - exiting 01:57:29 (4260): No heartbeat from core client for 30 sec - exiting 01:57:30 (4260): No heartbeat from core client for 30 sec - exiting 01:57:31 (4260): No heartbeat from core client for 30 sec - exiting 01:57:32 (4260): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=4268, selfPID=4268, iMonCtr=1 01:59:37 (5884): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 01:59:38 (5884): No heartbeat from core client for 30 sec - exiting 01:59:39 (5884): No heartbeat from core client for 30 sec - exiting 01:59:40 (5884): No heartbeat from core client for 30 sec - exiting 01:59:41 (5884): No heartbeat from core client for 30 sec - exiting 01:59:42 (5884): No heartbeat from core client for 30 sec - exiting 01:59:43 (5884): No heartbeat from core client for 30 sec - exiting 01:59:44 (5884): No heartbeat from core client for 30 sec - exiting 01:59:45 (5884): No heartbeat from core client for 30 sec - exiting 01:59:46 (5884): No heartbeat from core client for 30 sec - exiting 01:59:47 (5884): No heartbeat from core client for 30 sec - exiting Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=5936, selfPID=5936, iMonCtr=1 02:01:33 (5676): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:01:34 (5676): No heartbeat from core client for 30 sec - exiting 02:01:35 (5676): No heartbeat from core client for 30 sec - exiting 02:01:36 (5676): No heartbeat from core client for 30 sec - exiting 02:01:37 (5676): No heartbeat from core client for 30 sec - exiting 02:01:38 (5676): No heartbeat from core client for 30 sec - exiting 02:01:39 (5676): No heartbeat from core client for 30 sec - exiting 02:01:40 (5676): No heartbeat from core client for 30 sec - exiting 02:01:42 (5676): No heartbeat from core client for 30 sec - exiting 02:01:43 (5676): No heartbeat from core client for 30 sec - exiting 02:01:44 (5676): No heartbeat from core client for 30 sec - exiting forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... forrtl: Access is denied. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5320, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Dec 2011 07:04:39	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	984,960	2,868,654	2.9125
24 Dec 2011 07:04:39	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	959,040	2,731,651	2.8483
17 Dec 2011 04:52:58	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	933,120	2,613,462	2.8008
17 Dec 2011 04:52:58	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	907,200	2,538,306	2.7980
17 Dec 2011 04:52:58	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	881,280	2,465,468	2.7976
17 Dec 2011 04:52:58	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	855,360	2,396,287	2.8015
17 Dec 2011 04:52:58	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	829,440	2,328,645	2.8075
17 Dec 2011 04:52:58	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	803,520	2,262,885	2.8162
11 Dec 2011 15:18:26	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	777,600	2,180,963	2.8047
10 Dec 2011 06:05:31	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	751,680	2,091,680	2.7827
10 Dec 2011 06:05:31	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	725,760	2,023,324	2.7879
03 Dec 2011 03:30:06	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	699,840	1,938,396	2.7698
25 Nov 2011 23:47:19	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	673,920	1,876,782	2.7849
25 Nov 2011 06:45:23	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	648,000	1,802,321	2.7814
25 Nov 2011 06:45:23	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	622,080	1,750,273	2.8136
25 Nov 2011 06:45:23	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	596,160	1,697,934	2.8481
25 Nov 2011 06:45:23	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	570,240	1,644,682	2.8842
25 Nov 2011 06:45:23	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	544,320	1,592,181	2.9251
25 Nov 2011 06:45:23	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	518,400	1,539,682	2.9701
25 Nov 2011 06:45:23	1055935	13390920	hadcm3n_p2c1_1940_40_007420013_2	492,480	1,486,699	3.0188