Task 15523627

Name	hadcm3n_o2af_2140_40_008269865_2
Workunit	8424989
Created	5 Jan 2013, 17:58:18 UTC
Sent	5 Jan 2013, 17:58:21 UTC
Report deadline	7 Apr 2013, 1:25:32 UTC
Received	24 Feb 2013, 12:23:23 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1262174
Run time	28 days 21 hours 5 min 10 sec
CPU time	10 days 16 hours 10 min 23 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.09 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=6444, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4872, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 07:44:14 AM No files match the supplied pattern. MainError: 07:44:14 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 08:35:08 PM No files match the supplied pattern. MainError: 08:35:08 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 05:29:05 AM No files match the supplied pattern. MainError: 05:29:05 AM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 06:00:38 AM No files match the supplied pattern. MainError: 06:00:38 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 06:56:47 PM No files match the supplied pattern. MainError: 06:56:47 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 02:17:39 PM No files match the supplied pattern. MainError: 02:17:39 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5200, iMonCtr=1 Model crash detected, will try to restart... MainError: 09:17:45 AM No files match the supplied pattern. MainError: 09:17:45 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 07:10:27 AM No files match the supplied pattern. MainError: 07:10:27 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 11:52:52 AM No files match the supplied pattern. MainError: 11:52:52 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... MainError: 10:10:20 PM No files match the supplied pattern. MainError: 10:10:20 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Error converting file to netcdf: dataout/o2afka.ph11c10 Error converting file to netcdf: dataout/o2afka.pg11c10 Error converting file to netcdf: dataout/o2afka.pe11c10 MainError: 09:51:46 AM No files match the supplied pattern. MainError: 09:51:46 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
24 Feb 2013 09:55:51	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	777,600	935,548	1.2031
14 Feb 2013 22:14:42	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	751,680	903,236	1.2016
13 Feb 2013 12:17:05	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	725,760	869,419	1.1979
12 Feb 2013 07:14:17	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	699,840	837,178	1.1962
11 Feb 2013 09:21:18	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	673,920	806,737	1.1971
08 Feb 2013 14:20:30	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	648,000	774,140	1.1947
06 Feb 2013 19:05:55	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	622,080	741,119	1.1914
03 Feb 2013 06:02:03	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	596,160	710,173	1.1912
30 Jan 2013 05:32:44	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	570,240	680,509	1.1934
28 Jan 2013 20:36:24	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	544,320	649,898	1.1940
27 Jan 2013 08:05:53	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	518,400	618,748	1.1936
26 Jan 2013 06:09:30	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	492,480	587,874	1.1937
25 Jan 2013 07:35:07	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	466,560	556,892	1.1936
24 Jan 2013 01:24:47	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	440,640	525,736	1.1931
21 Jan 2013 20:32:02	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	414,720	495,411	1.1946
20 Jan 2013 18:03:11	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	388,800	464,935	1.1958
19 Jan 2013 14:05:23	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	362,880	434,731	1.1980
18 Jan 2013 17:59:17	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	336,960	405,392	1.2031
17 Jan 2013 17:10:56	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	311,040	376,493	1.2104
16 Jan 2013 18:32:33	1262174	15523627	hadcm3n_o2af_2140_40_008269865_2	285,120	347,344	1.2182