Task 13757877

Name	hadcm3n_t142_1940_40_007446933_3
Workunit	7644436
Created	10 Dec 2011, 0:25:26 UTC
Sent	10 Dec 2011, 0:25:37 UTC
Report deadline	10 Mar 2012, 7:52:48 UTC
Received	27 Dec 2011, 2:40:22 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1184174
Run time	11 days 6 hours 37 min 13 sec
CPU time	9 days 17 hours 2 min 51 sec
Validate state	Invalid
Credit	5,287.68
Device peak FLOPS	2.03 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.34</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 03:48:35 (4780): No heartbeat from core client for 30 sec - exiting 03:48:36 (4780): No heartbeat from core client for 30 sec - exiting 03:48:37 (4780): No heartbeat from core client for 30 sec - exiting 03:48:38 (4780): No heartbeat from core client for 30 sec - exiting 03:48:39 (4780): No heartbeat from core client for 30 sec - exiting 03:48:40 (4780): No heartbeat from core client for 30 sec - exiting 03:48:41 (4780): No heartbeat from core client for 30 sec - exiting 03:48:42 (4780): No heartbeat from core client for 30 sec - exiting 03:48:43 (4780): No heartbeat from core client for 30 sec - exiting 03:48:44 (4780): No heartbeat from core client for 30 sec - exiting 03:48:45 (4780): No heartbeat from core client for 30 sec - exiting 03:48:46 (4780): No heartbeat from core client for 30 sec - exiting 03:48:47 (4780): No heartbeat from core client for 30 sec - exiting 03:48:48 (4780): No heartbeat from core client for 30 sec - exiting 03:48:49 (4780): No heartbeat from core client for 30 sec - exiting 03:48:50 (4780): No heartbeat from core client for 30 sec - exiting 03:48:51 (4780): No heartbeat from core client for 30 sec - exiting 03:48:52 (4780): No heartbeat from core client for 30 sec - exiting 03:48:53 (4780): No heartbeat from core client for 30 sec - exiting 03:48:54 (4780): No heartbeat from core client for 30 sec - exiting 03:48:55 (4780): No heartbeat from core client for 30 sec - exiting 03:48:56 (4780): No heartbeat from core client for 30 sec - exiting 03:48:57 (4780): No heartbeat from core client for 30 sec - exiting 03:48:58 (4780): No heartbeat from core client for 30 sec - exiting 03:48:59 (4780): No heartbeat from core client for 30 sec - exiting 03:49:00 (4780): No heartbeat from core client for 30 sec - exiting 03:49:01 (4780): No heartbeat from core client for 30 sec - exiting 03:49:02 (4780): No heartbeat from core client for 30 sec - exiting 03:49:03 (4780): No heartbeat from core client for 30 sec - exiting 03:49:04 (4780): No heartbeat from core client for 30 sec - exiting 03:49:05 (4780): No heartbeat from core client for 30 sec - exiting 03:49:06 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 03:49:07 (4780): No heartbeat from core client for 30 sec - exiting 03:49:08 (4780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 18:12:21 (3192): No heartbeat from core client for 30 sec - exiting 18:12:22 (3192): No heartbeat from core client for 30 sec - exiting 18:12:23 (3192): No heartbeat from core client for 30 sec - exiting 18:12:24 (3192): No heartbeat from core client for 30 sec - exiting 18:12:25 (3192): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=7032, selfPID=7032, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/t142ko.pjf0c10 Error converting file to netcdf: dataout/t142ko.pif0c10 Error converting file to netcdf: dataout/t142ko.pff0c10 Error converting file to netcdf: dataout/t142ko.pcf0c10 Error converting file to netcdf: dataout/t142ko.pbf0c10 Error converting file to netcdf: dataout/t142ko.paf0c10 Error converting file to netcdf: dataout/t142ka.phf0c10 Error converting file to netcdf: dataout/t142ka.pgf0c10 Error converting file to netcdf: dataout/t142ka.pef0c10 Error converting file to netcdf: dataout/t142ka.pdf0c10 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5992, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
27 Dec 2011 00:45:32	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	440,640	838,302	1.9025
25 Dec 2011 22:48:08	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	414,720	788,190	1.9005
24 Dec 2011 00:13:17	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	388,800	739,716	1.9026
23 Dec 2011 02:24:34	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	362,880	690,965	1.9041
22 Dec 2011 02:31:33	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	336,960	642,200	1.9059
21 Dec 2011 07:02:29	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	311,040	594,371	1.9109
20 Dec 2011 11:09:56	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	285,120	543,631	1.9067
19 Dec 2011 15:14:14	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	259,200	491,254	1.8953
18 Dec 2011 11:33:03	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	233,280	442,890	1.8985
17 Dec 2011 15:20:22	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	207,360	393,359	1.8970
16 Dec 2011 19:54:31	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	181,440	344,289	1.8975
15 Dec 2011 21:06:58	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	155,520	288,965	1.8581
15 Dec 2011 01:47:28	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	129,600	241,357	1.8623
14 Dec 2011 08:17:35	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	103,680	193,681	1.8681
13 Dec 2011 14:27:58	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	77,760	145,801	1.8750
12 Dec 2011 19:44:24	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	51,840	97,590	1.8825
11 Dec 2011 13:48:05	1184174	13757877	hadcm3n_t142_1940_40_007446933_3	25,920	49,389	1.9054