Task 15521352

Name	hadcm3n_o0mh_2140_40_008270132_4
Workunit	8425256
Created	2 Jan 2013, 9:56:24 UTC
Sent	2 Jan 2013, 9:57:02 UTC
Report deadline	3 Apr 2013, 17:24:13 UTC
Received	17 Jan 2013, 3:23:42 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1208755
Run time	8 days 21 hours 23 min 37 sec
CPU time	7 days 2 hours 10 min 22 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	3.22 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.25</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4040, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4040, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4040, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3536, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3536, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3536, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3536, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... MainError: 06:03:44 PM No files match the supplied pattern. MainError: 06:03:44 PM No files match the supplied pattern. MainError: 01:04:55 AM No files match the supplied pattern. MainError: 01:04:55 AM No files match the supplied pattern. MainError: 08:17:19 AM No files match the supplied pattern. MainError: 08:17:19 AM No files match the supplied pattern. MainError: 03:51:23 PM No files match the supplied pattern. MainError: 03:51:23 PM No files match the supplied pattern. MainError: 11:36:36 PM No files match the supplied pattern. MainError: 11:36:36 PM No files match the supplied pattern. MainError: 07:11:24 AM No files match the supplied pattern. MainError: 07:11:24 AM No files match the supplied pattern. MainError: 02:55:35 PM No files match the supplied pattern. MainError: 02:55:35 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 04:13:00 AM No files match the supplied pattern. MainError: 04:13:00 AM No files match the supplied pattern. MainError: 11:43:59 AM No files match the supplied pattern. MainError: 11:43:59 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 06:49:49 PM No files match the supplied pattern. MainError: 06:49:49 PM No files match the supplied pattern. Error converting file to netcdf: dataout/o0mhka.ph11c10 Error converting file to netcdf: dataout/o0mhka.pg11c10 Error converting file to netcdf: dataout/o0mhka.pe11c10 MainError: 01:52:54 AM No files match the supplied pattern. MainError: 01:52:54 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
17 Jan 2013 01:53:07	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	777,600	698,327	0.8981
16 Jan 2013 19:54:18	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	751,680	674,772	0.8977
16 Jan 2013 12:40:11	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	725,760	652,295	0.8988
16 Jan 2013 04:16:33	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	699,840	628,448	0.8980
15 Jan 2013 14:55:49	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	673,920	603,861	0.8960
15 Jan 2013 07:13:09	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	648,000	578,364	0.8925
14 Jan 2013 23:41:31	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	622,080	553,008	0.8890
14 Jan 2013 15:56:02	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	596,160	527,824	0.8854
14 Jan 2013 08:18:41	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	570,240	502,703	0.8816
14 Jan 2013 01:37:00	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	544,320	478,154	0.8784
13 Jan 2013 18:06:09	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	518,400	454,852	0.8774
13 Jan 2013 11:09:47	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	492,480	431,739	0.8767
12 Jan 2013 20:25:53	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	466,560	408,922	0.8765
12 Jan 2013 01:27:57	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	440,640	386,447	0.8770
11 Jan 2013 19:34:40	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	414,720	364,096	0.8779
11 Jan 2013 11:49:49	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	388,800	340,895	0.8768
10 Jan 2013 15:45:46	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	362,880	317,119	0.8739
10 Jan 2013 07:53:30	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	336,960	293,307	0.8705
10 Jan 2013 01:12:21	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	311,040	270,838	0.8707
09 Jan 2013 18:16:27	1208755	15521352	hadcm3n_o0mh_2140_40_008270132_4	285,120	248,436	0.8713