Task 13635381

Name	hadcm3n_ybcx_1900_40_007520139_3
Workunit	7717614
Created	15 Nov 2011, 19:13:57 UTC
Sent	18 Nov 2011, 13:58:23 UTC
Report deadline	17 Feb 2012, 21:25:34 UTC
Received	27 Dec 2011, 16:21:09 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1121021
Run time	10 days 5 hours 34 min 41 sec
CPU time	7 days 22 hours 9 min 5 sec
Validate state	Invalid
Credit	3,732.48
Device peak FLOPS	3.03 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:45:07 (2256): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:47:40 (2152): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:25:36 (932): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 08:03:32 (1636): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:30:13 (4012): No heartbeat from core client for 30 sec - exiting BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ybcxko.pja5c10 Error converting file to netcdf: dataout/ybcxko.pia5c10 Error converting file to netcdf: dataout/ybcxko.pfa5c10 Error converting file to netcdf: dataout/ybcxka.pha5c10 Error converting file to netcdf: dataout/ybcxka.pga5c10 Error converting file to netcdf: dataout/ybcxka.pea5c10 Error converting file to netcdf: dataout/ybcxka.pda5c10 CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/ybcxko.pja5c10 Error converting file to netcdf: dataout/ybcxko.pia5c10 Error converting file to netcdf: dataout/ybcxko.pfa5c10 Error converting file to netcdf: dataout/ybcxka.pha5c10 Error converting file to netcdf: dataout/ybcxka.pga5c10 Error converting file to netcdf: dataout/ybcxka.pea5c10 Error converting file to netcdf: dataout/ybcxka.pda5c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 08:29:59 (204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 13:07:59 (2184): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 08:59:35 (3708): No heartbeat from core client for 30 sec - exiting 08:59:37 (3708): No heartbeat from core client for 30 sec - exiting 08:59:38 (3708): No heartbeat from core client for 30 sec - exiting 08:59:39 (3708): No heartbeat from core client for 30 sec - exiting 08:59:40 (3708): No heartbeat from core client for 30 sec - exiting 08:59:41 (3708): No heartbeat from core client for 30 sec - exiting 08:59:42 (3708): No heartbeat from core client for 30 sec - exiting 08:59:43 (3708): No heartbeat from core client for 30 sec - exiting 08:59:44 (3708): No heartbeat from core client for 30 sec - exiting 08:59:45 (3708): No heartbeat from core client for 30 sec - exiting 08:59:46 (3708): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 17:54:37 (2752): No heartbeat from core client for 30 sec - exiting 17:54:38 (2752): No heartbeat from core client for 30 sec - exiting 17:54:39 (2752): No heartbeat from core client for 30 sec - exiting 17:54:40 (2752): No heartbeat from core client for 30 sec - exiting 17:54:41 (2752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 15:00:15 (2264): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 14:45:00 (2580): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 11:39:07 (2240): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 18:32:35 (2820): No heartbeat from core client for 30 sec - exiting 18:32:36 (2820): No heartbeat from core client for 30 sec - exiting 18:32:37 (2820): No heartbeat from core client for 30 sec - exiting 18:32:38 (2820): No heartbeat from core client for 30 sec - exiting 18:32:39 (2820): No heartbeat from core client for 30 sec - exiting 18:32:40 (2820): No heartbeat from core client for 30 sec - exiting 18:32:41 (2820): No heartbeat from core client for 30 sec - exiting 18:32:42 (2820): No heartbeat from core client for 30 sec - exiting 18:32:44 (2820): No heartbeat from core client for 30 sec - exiting 18:32:45 (2820): No heartbeat from core client for 30 sec - exiting 18:32:46 (2820): No heartbeat from core client for 30 sec - exiting 18:32:47 (2820): No heartbeat from core client for 30 sec - exiting 18:32:48 (2820): No heartbeat from core client for 30 sec - exiting 18:32:49 (2820): No heartbeat from core client for 30 sec - exiting 18:32:50 (2820): No heartbeat from core client for 30 sec - exiting 18:32:51 (2820): No heartbeat from core client for 30 sec - exiting 18:32:52 (2820): No heartbeat from core client for 30 sec - exiting 18:32:53 (2820): No heartbeat from core client for 30 sec - exiting 18:32:54 (2820): No heartbeat from core client for 30 sec - exiting 18:32:56 (2820): No heartbeat from core client for 30 sec - exiting 18:32:57 (2820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:32:58 (2820): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... 17:31:17 (2104): No heartbeat from core client for 30 sec - exiting 17:31:18 (2104): No heartbeat from core client for 30 sec - exiting 17:31:19 (2104): No heartbeat from core client for 30 sec - exiting 17:31:20 (2104): No heartbeat from core client for 30 sec - exiting 17:31:21 (2104): No heartbeat from core client for 30 sec - exiting 17:31:22 (2104): No heartbeat from core client for 30 sec - exiting 17:31:23 (2104): No heartbeat from core client for 30 sec - exiting 17:31:24 (2104): No heartbeat from core client for 30 sec - exiting 17:31:25 (2104): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:57:42 (3912): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 16:25:51 (2140): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 14:47:42 (2012): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2396, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish CPDN Monitor - Quit request from BOINC... 14:37:37 (1816): No heartbeat from core client for 30 sec - exiting 14:37:38 (1816): No heartbeat from core client for 30 sec - exiting 14:37:39 (1816): No heartbeat from core client for 30 sec - exiting 14:37:40 (1816): No heartbeat from core client for 30 sec - exiting 14:37:41 (1816): No heartbeat from core client for 30 sec - exiting 14:37:42 (1816): No heartbeat from core client for 30 sec - exiting 14:37:43 (1816): No heartbeat from core client for 30 sec - exiting 14:37:44 (1816): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:37:45 (1816): No heartbeat from core client for 30 sec - exiting Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2988, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
21 Dec 2011 16:29:52	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	311,040	651,424	2.0943
18 Dec 2011 13:43:30	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	285,120	596,888	2.0935
17 Dec 2011 10:54:21	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	259,200	541,472	2.0890
14 Dec 2011 11:43:37	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	233,280	487,183	2.0884
10 Dec 2011 19:37:50	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	207,360	432,912	2.0877
05 Dec 2011 17:18:54	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	181,440	377,706	2.0817
02 Dec 2011 11:57:54	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	155,520	323,387	2.0794
28 Nov 2011 14:47:13	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	129,600	268,697	2.0733
26 Nov 2011 21:21:57	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	103,680	214,131	2.0653
25 Nov 2011 14:32:26	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	77,760	161,050	2.0711
20 Nov 2011 19:54:43	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	51,840	106,993	2.0639
19 Nov 2011 18:16:38	1121021	13635381	hadcm3n_ybcx_1900_40_007520139_3	25,920	53,729	2.0729