Task 14666563

Name	hadcm3n_o21e_1980_40_007959067_2
Workunit	8114179
Created	14 May 2012, 16:30:12 UTC
Sent	14 May 2012, 19:12:08 UTC
Report deadline	14 Aug 2012, 2:39:19 UTC
Received	10 Aug 2012, 19:22:39 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	901376
Run time	39 days 21 hours 9 min 12 sec
CPU time	35 days 0 hours 11 min 13 sec
Validate state	Invalid
Credit	11,197.44
Device peak FLOPS	1.42 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:46:59 (2152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:47:02 (2152): No heartbeat from core client for 30 sec - exiting 12:47:03 (2152): No heartbeat from core client for 30 sec - exiting 12:47:05 (2152): No heartbeat from core client for 30 sec - exiting 12:47:06 (2152): No heartbeat from core client for 30 sec - exiting 12:47:07 (2152): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - Suspend request from BOINC... 08:49:51 (3840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 08:49:52 (3840): No heartbeat from core client for 30 sec - exiting 08:49:53 (3840): No heartbeat from core client for 30 sec - exiting 08:49:54 (3840): No heartbeat from core client for 30 sec - exiting 08:49:55 (3840): No heartbeat from core client for 30 sec - exiting 08:49:56 (3840): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 22:05:48 (556): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 16:08:52 (1592): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:36:19 (3128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:47:05 (252): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:56:31 (952): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 22:58:01 (1920): No heartbeat from core client for 30 sec - exiting 22:58:02 (1920): No heartbeat from core client for 30 sec - exiting 22:58:03 (1920): No heartbeat from core client for 30 sec - exiting 22:58:04 (1920): No heartbeat from core client for 30 sec - exiting 22:58:06 (1920): No heartbeat from core client for 30 sec - exiting 22:58:07 (1920): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/o21eko.pjj4c10 Error converting file to netcdf: dataout/o21eko.pij4c10 Error converting file to netcdf: dataout/o21eko.pfj4c10 Error converting file to netcdf: dataout/o21eka.phj4c10 Error converting file to netcdf: dataout/o21eka.pgj4c10 Error converting file to netcdf: dataout/o21eka.pej4c10 Error converting file to netcdf: dataout/o21eka.pdj4c10 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:05:02 (860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:42:23 (3608): No heartbeat from core client for 30 sec - exiting 12:42:24 (3608): No heartbeat from core client for 30 sec - exiting 12:42:26 (3608): No heartbeat from core client for 30 sec - exiting 12:42:27 (3608): No heartbeat from core client for 30 sec - exiting 12:42:28 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:42:29 (3608): No heartbeat from core client for 30 sec - exiting 12:42:31 (3608): No heartbeat from core client for 30 sec - exiting 12:42:32 (3608): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:57:22 (1628): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:57 (3304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:25:09 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:38:14 (3700): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 09:59:18 (4092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:53:21 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:53:23 (6080): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 15:19:19 (5544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1932, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1932, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1932, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1932, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1932, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=1932, iMonCtr=1 Model crash detected, will try to restart... 15:22:01 (1932): No heartbeat from core client for 30 sec - exiting Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
13 Jul 2012 12:05:10	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	933,120	2,975,699	3.1890
05 Jul 2012 02:57:25	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	907,200	2,892,361	3.1882
04 Jul 2012 00:20:07	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	881,280	2,808,808	3.1872
02 Jul 2012 21:56:01	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	855,360	2,725,308	3.1862
02 Jul 2012 13:58:06	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	829,440	2,641,059	3.1841
02 Jul 2012 13:58:06	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	803,520	2,558,442	3.1840
29 Jun 2012 14:48:31	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	777,600	2,476,262	3.1845
28 Jun 2012 12:37:35	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	751,680	2,394,101	3.1850
27 Jun 2012 10:19:27	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	725,760	2,311,517	3.1850
26 Jun 2012 08:35:24	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	699,840	2,230,095	3.1866
25 Jun 2012 06:52:49	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	673,920	2,148,598	3.1882
24 Jun 2012 04:45:35	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	648,000	2,065,799	3.1880
23 Jun 2012 02:41:42	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	622,080	1,982,865	3.1875
22 Jun 2012 01:17:09	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	596,160	1,899,954	3.1870
20 Jun 2012 21:28:36	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	570,240	1,817,985	3.1881
06 Jun 2012 18:50:18	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	544,320	1,735,442	3.1883
05 Jun 2012 14:59:17	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	518,400	1,652,753	3.1882
04 Jun 2012 12:35:05	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	492,480	1,569,921	3.1878
03 Jun 2012 10:19:06	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	466,560	1,487,416	3.1880
02 Jun 2012 08:12:35	901376	14666563	hadcm3n_o21e_1980_40_007959067_2	440,640	1,403,926	3.1861