Task 16032596

Name	hadcm3n_o7dd_2140_40_008270372_3
Workunit	8425496
Created	24 Sep 2013, 1:07:51 UTC
Sent	24 Sep 2013, 1:07:58 UTC
Report deadline	24 Dec 2013, 8:35:09 UTC
Received	10 Oct 2013, 2:39:47 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1164011
Run time	9 days 13 hours 29 min 12 sec
CPU time	8 days 23 hours 54 min 59 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.24 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>6.12.33</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 11:55:46 PM No files match the supplied pattern. MainError: 11:55:46 PM No files match the supplied pattern. MainError: 06:11:12 AM No files match the supplied pattern. MainError: 06:11:12 AM No files match the supplied pattern. MainError: 12:51:42 AM No files match the supplied pattern. MainError: 12:51:42 AM No files match the supplied pattern. MainError: 07:36:52 PM No files match the supplied pattern. MainError: 07:36:52 PM No files match the supplied pattern. MainError: 02:20:31 AM No files match the supplied pattern. MainError: 02:20:31 AM No files match the supplied pattern. MainError: 09:01:45 AM No files match the supplied pattern. MainError: 09:01:45 AM No files match the supplied pattern. MainError: 03:44:22 PM No files match the supplied pattern. MainError: 03:44:22 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 10:30:23 PM No files match the supplied pattern. MainError: 10:30:23 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 05:16:28 AM No files match the supplied pattern. MainError: 05:16:28 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2828, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2828, iMonCtr=1 Model crash detected, will try to restart... MainError: 04:46:54 PM No files match the supplied pattern. MainError: 04:46:54 PM No files match the supplied pattern. Error converting file to netcdf: dataout/o7ddka.ph11c10 Error converting file to netcdf: dataout/o7ddka.pg11c10 Error converting file to netcdf: dataout/o7ddka.pe11c10 MainError: 11:35:05 PM No files match the supplied pattern. MainError: 11:35:05 PM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
09 Oct 2013 23:36:00	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	777,600	818,216	1.0522
09 Oct 2013 16:50:42	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	751,680	793,770	1.0560
09 Oct 2013 05:21:06	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	725,760	769,492	1.0603
08 Oct 2013 22:32:47	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	699,840	745,391	1.0651
08 Oct 2013 15:47:59	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	673,920	721,122	1.0700
08 Oct 2013 09:04:14	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	648,000	696,998	1.0756
08 Oct 2013 02:25:19	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	622,080	672,941	1.0818
07 Oct 2013 19:40:37	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	596,160	648,780	1.0883
07 Oct 2013 12:53:43	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	570,240	624,529	1.0952
07 Oct 2013 06:38:39	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	544,320	600,514	1.1032
07 Oct 2013 00:34:23	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	518,400	578,003	1.1150
06 Oct 2013 16:27:43	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	492,480	551,264	1.1194
06 Oct 2013 08:19:39	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	466,560	522,167	1.1192
06 Oct 2013 00:10:00	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	440,640	493,091	1.1190
05 Oct 2013 16:04:51	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	414,720	464,078	1.1190
05 Oct 2013 07:53:37	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	388,800	435,012	1.1189
04 Oct 2013 23:44:56	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	362,880	405,926	1.1186
04 Oct 2013 15:38:09	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	336,960	376,850	1.1184
04 Oct 2013 07:30:42	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	311,040	347,836	1.1183
03 Oct 2013 23:16:23	1164011	16032596	hadcm3n_o7dd_2140_40_008270372_3	285,120	318,745	1.1179