Task 17905346

Name	hadcm3n_xcce_1940_40_009152292_1
Workunit	9282628
Created	6 Feb 2015, 16:38:44 UTC
Sent	6 Feb 2015, 18:56:13 UTC
Report deadline	9 May 2015, 2:23:24 UTC
Received	3 Mar 2015, 1:42:31 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1319477
Run time	20 days 2 hours 52 min 38 sec
CPU time	18 days 12 hours 43 min 24 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	2.64 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.4.36</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3236, iMonCtr=1 Model crash detected, will try to restart... BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: C I/O Error feof - Unit 62 - Return code = 16 BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/xcceko.pjh0c10 Error converting file to netcdf: dataout/xcceko.pih0c10 Error converting file to netcdf: dataout/xcceko.pfh0c10 Error converting file to netcdf: dataout/xcceko.pch0c10 Error converting file to netcdf: dataout/xcceko.pbh0c10 Error converting file to netcdf: dataout/xcceko.pah0c10 Error converting file to netcdf: dataout/xcceka.phh0c10 Error converting file to netcdf: dataout/xcceka.pgh0c10 Error converting file to netcdf: dataout/xcceka.peh0c10 Error converting file to netcdf: dataout/xcceka.pdh0c10 CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=572, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Model crashed: ATM_DYN : INVALID THETA DETECTED. tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
02 Mar 2015 23:19:54	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	1,010,880	1,666,728	1.6488
02 Mar 2015 09:51:41	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	984,960	1,621,832	1.6466
01 Mar 2015 21:35:51	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	959,040	1,576,590	1.6439
01 Mar 2015 07:33:05	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	933,120	1,531,394	1.6412
28 Feb 2015 19:52:51	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	907,200	1,486,761	1.6388
28 Feb 2015 05:49:15	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	881,280	1,441,827	1.6361
27 Feb 2015 17:02:09	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	855,360	1,396,233	1.6323
27 Feb 2015 01:33:42	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	829,440	1,350,909	1.6287
26 Feb 2015 12:55:38	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	803,520	1,306,038	1.6254
25 Feb 2015 22:45:14	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	777,600	1,263,526	1.6249
24 Feb 2015 21:17:14	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	751,680	1,218,246	1.6207
24 Feb 2015 08:44:16	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	725,760	1,173,567	1.6170
23 Feb 2015 19:03:14	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	699,840	1,128,924	1.6131
23 Feb 2015 06:09:49	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	673,920	1,084,104	1.6087
22 Feb 2015 17:22:36	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	648,000	1,040,249	1.6053
22 Feb 2015 04:43:15	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	622,080	997,577	1.6036
21 Feb 2015 15:33:20	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	596,160	955,437	1.6027
21 Feb 2015 03:13:24	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	570,240	914,075	1.6030
20 Feb 2015 15:34:10	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	544,320	872,726	1.6033
20 Feb 2015 03:34:08	1319477	17905346	hadcm3n_xcce_1940_40_009152292_1	518,400	831,420	1.6038