Task 15502350

Name	hadcm3n_o2nt_2140_40_008269509_0
Workunit	8424633
Created	23 Dec 2012, 23:14:05 UTC
Sent	23 Dec 2012, 23:15:56 UTC
Report deadline	25 Mar 2013, 6:43:07 UTC
Received	15 Jan 2013, 15:16:41 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1260009
Run time	15 days 1 hours 0 min 50 sec
CPU time	14 days 8 hours 57 min 29 sec
Validate state	Invalid
Credit	9,331.20
Device peak FLOPS	2.42 GFLOPS
Application version	UK Met Office Coupled Model Full Resolution Ocean v6.07 windows_intelx86
Stderr	<core_client_version>7.0.28</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=4356, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 00:39:26 (1632): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=3388, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... MainError: 10:08:14 PM No files match the supplied pattern. MainError: 10:08:14 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 08:23:24 AM No files match the supplied pattern. MainError: 08:23:24 AM No files match the supplied pattern. MainError: 05:50:04 PM No files match the supplied pattern. MainError: 05:50:04 PM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... MainError: 03:42:14 AM No files match the supplied pattern. MainError: 03:42:14 AM No files match the supplied pattern. MainError: 01:31:51 PM No files match the supplied pattern. MainError: 01:31:51 PM No files match the supplied pattern. MainError: 11:18:01 PM No files match the supplied pattern. MainError: 11:18:01 PM No files match the supplied pattern. CPDN Monitor - Quit request from BOINC... MainError: 09:06:33 AM No files match the supplied pattern. MainError: 09:06:33 AM No files match the supplied pattern. Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... MainError: 06:47:36 PM No files match the supplied pattern. MainError: 06:47:36 PM No files match the supplied pattern. MainError: 10:07:39 AM No files match the supplied pattern. MainError: 10:07:39 AM No files match the supplied pattern. MainError: 11:31:43 PM No files match the supplied pattern. MainError: 11:31:43 PM No files match the supplied pattern. Error converting file to netcdf: dataout/o2ntka.ph11c10 Error converting file to netcdf: dataout/o2ntka.pg11c10 Error converting file to netcdf: dataout/o2ntka.pe11c10 MainError: 12:22:59 AM No files match the supplied pattern. MainError: 12:22:59 AM No files match the supplied pattern. BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 BUFFIN: C I/O Error feof - Unit 64 - Return code = 16 BUFFIN: C I/O Error feof - Unit 66 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: C I/O Error feof - Unit 67 - Return code = 16 Model crashed: STWORK : I/O error - PP fixed length header tmp/pipe_dummy 2048 Sorry, too many model crashes! :-( Called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
15 Jan 2013 12:50:11	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	777,600	1,275,352	1.6401
14 Jan 2013 23:36:17	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	751,680	1,229,412	1.6356
14 Jan 2013 10:24:02	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	725,760	1,181,270	1.6276
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	699,840	1,137,504	1.6254
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	673,920	1,102,918	1.6366
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	648,000	1,067,830	1.6479
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	622,080	1,032,726	1.6601
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	596,160	997,447	1.6731
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	570,240	962,175	1.6873
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	544,320	928,188	1.7052
13 Jan 2013 22:19:53	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	518,400	891,426	1.7196
10 Jan 2013 11:44:10	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	492,480	853,908	1.7339
10 Jan 2013 00:37:15	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	466,560	814,391	1.7455
09 Jan 2013 14:10:23	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	440,640	770,455	1.7485
09 Jan 2013 01:43:33	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	414,720	727,478	1.7541
08 Jan 2013 03:40:55	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	388,800	684,669	1.7610
07 Jan 2013 17:57:42	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	362,880	643,605	1.7736
07 Jan 2013 07:00:37	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	336,960	603,154	1.7900
06 Jan 2013 11:16:40	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	311,040	562,309	1.8078
05 Jan 2013 21:17:18	1260009	15502350	hadcm3n_o2nt_2140_40_008269509_0	285,120	521,166	1.8279