Task 15110207

Name	hadcm3n_zmsk_1880_40_008026519_2
Workunit	8181633
Created	13 Aug 2012, 13:56:20 UTC
Sent	13 Aug 2012, 14:05:39 UTC
Report deadline	12 Nov 2012, 21:32:50 UTC
Received	13 Oct 2012, 2:35:43 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1088309
Run time	28 days 11 hours 54 min 53 sec
CPU time	22 days 4 hours 12 min 59 sec
Validate state	Invalid
Credit	12,130.56
Device peak FLOPS	2.47 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3448, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3700, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3984, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3652, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2868, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3508, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zmskko.pj96c10 Error converting file to netcdf: dataout/zmskko.pi96c10 Error converting file to netcdf: dataout/zmskko.pf96c10 Error converting file to netcdf: dataout/zmskka.ph96c10 Error converting file to netcdf: dataout/zmskka.pg96c10 Error converting file to netcdf: dataout/zmskka.pe96c10 Error converting file to netcdf: dataout/zmskka.pd96c10 Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3148, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2144, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3700, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4072, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3784, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: C I/O Error feof - Unit 63 - Return code = 16 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 65 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Error converting file to netcdf: dataout/zmskko.pjb1c10 Error converting file to netcdf: dataout/zmskko.pib1c10 Error converting file to netcdf: dataout/zmskko.pfb1c10 Error converting file to netcdf: dataout/zmskka.phb1c10 Error converting file to netcdf: dataout/zmskka.pgb1c10 Error converting file to netcdf: dataout/zmskka.peb1c10 Error converting file to netcdf: dataout/zmskka.pdb1c10 Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3688, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3508, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4228, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4168, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4080, iMonCtr=1 Model crash detected, will try to restart... Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Model crashed: REPLANCA: PP HEADERS ON ANCILLARY FILE DO NOT MATCH tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
12 Oct 2012 18:48:02	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	1,010,880	1,920,150	1.8995
10 Oct 2012 19:16:10	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	984,960	1,870,982	1.8996
08 Oct 2012 17:32:52	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	959,040	1,822,818	1.9007
06 Oct 2012 22:21:59	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	933,120	1,774,166	1.9013
06 Oct 2012 00:15:20	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	907,200	1,725,316	1.9018
04 Oct 2012 20:13:07	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	881,280	1,676,606	1.9025
02 Oct 2012 19:01:11	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	855,360	1,628,308	1.9037
30 Sep 2012 16:28:15	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	829,440	1,579,364	1.9041
29 Sep 2012 11:26:22	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	803,520	1,530,474	1.9047
25 Sep 2012 20:28:48	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	777,600	1,480,228	1.9036
24 Sep 2012 01:43:20	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	751,680	1,431,088	1.9039
22 Sep 2012 21:28:13	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	725,760	1,382,698	1.9052
21 Sep 2012 20:03:59	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	699,840	1,334,808	1.9073
20 Sep 2012 15:18:17	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	673,920	1,286,715	1.9093
19 Sep 2012 01:01:06	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	648,000	1,238,082	1.9106
17 Sep 2012 22:32:56	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	622,080	1,188,900	1.9112
16 Sep 2012 14:46:00	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	596,160	1,140,444	1.9130
15 Sep 2012 14:03:46	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	570,240	1,091,652	1.9144
13 Sep 2012 21:34:16	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	544,320	1,040,165	1.9109
12 Sep 2012 00:32:39	1088309	15110207	hadcm3n_zmsk_1880_40_008026519_2	518,400	991,166	1.9120