Task 16049449

Name	hadcm3n_oe9h_1900_40_008473608_2
Workunit	8624447
Created	29 Sep 2013, 13:05:22 UTC
Sent	29 Sep 2013, 13:15:08 UTC
Report deadline	29 Dec 2013, 20:42:19 UTC
Received	11 Oct 2013, 0:24:46 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1295325
Run time	3 days 18 hours 53 min 40 sec
CPU time	3 days 11 hours 35 min 19 sec
Validate state	Invalid
Credit	2,177.28
Device peak FLOPS	2.37 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.64</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/oe9hko.pja7c10 Error converting file to netcdf: dataout/oe9hko.pia7c10 Error converting file to netcdf: dataout/oe9hko.pfa7c10 Error converting file to netcdf: dataout/oe9hka.pha7c10 Error converting file to netcdf: dataout/oe9hka.pga7c10 Error converting file to netcdf: dataout/oe9hka.pea7c10 Error converting file to netcdf: dataout/oe9hka.pda7c10 CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6024, iMonCtr=1 Model crash detected, will try to restart... 18:46:09 (5292): Can't acquire lockfile (32) - waiting 35s 18:46:15 (1828): Can't acquire lockfile (32) - waiting 35s Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5812, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish 18:46:35 (5812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 18:46:44 (5292): Can't set up shared mem: -1. Will run in standalone mode. 18:46:45 (4136): Can't set up shared mem: -1. Will run in standalone mode. Signal 22 received, exiting... 18:46:50 (1828): Can't acquire lockfile (32) - exiting 18:46:50 (1828): Error: The process cannot access the file because it is being used by another process. (0x20) CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7024, iMonCtr=1 Model crash detected, will try to restart... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7024, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Signal 22 received, exiting... Called boinc_finish Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3452, iMonCtr=1 Model crash detected, will try to restart... Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
10 Oct 2013 17:06:57	1295325	16049449	hadcm3n_oe9h_1900_40_008473608_2	181,440	296,143	1.6322
10 Oct 2013 02:52:55	1295325	16049449	hadcm3n_oe9h_1900_40_008473608_2	155,520	253,887	1.6325
08 Oct 2013 13:09:51	1295325	16049449	hadcm3n_oe9h_1900_40_008473608_2	129,600	214,804	1.6574
06 Oct 2013 04:22:12	1295325	16049449	hadcm3n_oe9h_1900_40_008473608_2	103,680	177,416	1.7112
02 Oct 2013 13:59:50	1295325	16049449	hadcm3n_oe9h_1900_40_008473608_2	77,760	137,096	1.7631
01 Oct 2013 18:01:45	1295325	16049449	hadcm3n_oe9h_1900_40_008473608_2	51,840	95,547	1.8431
30 Sep 2013 17:02:17	1295325	16049449	hadcm3n_oe9h_1900_40_008473608_2	25,920	54,537	2.1041