Task 11526285

Name	famous_uh5x_1599_200_006655088_4
Workunit	6858460
Created	10 Jun 2010, 14:14:30 UTC
Sent	24 Aug 2010, 7:35:37 UTC
Report deadline	23 Nov 2010, 15:02:48 UTC
Received	6 Sep 2010, 16:55:29 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1057648
Run time	5 days 0 hours 38 min 24 sec
CPU time	4 days 23 hours 8 min 55 sec
Validate state	Invalid
Credit	4,694.09
Device peak FLOPS	3.05 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.56</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> 12:37:23 (4336): No heartbeat from core client for 30 sec - exiting 12:37:24 (4336): No heartbeat from core client for 30 sec - exiting 12:37:25 (4336): No heartbeat from core client for 30 sec - exiting 12:37:26 (4336): No heartbeat from core client for 30 sec - exiting 12:37:27 (4336): No heartbeat from core client for 30 sec - exiting 12:37:28 (4336): No heartbeat from core client for 30 sec - exiting 12:37:29 (4336): No heartbeat from core client for 30 sec - exiting 12:37:30 (4336): No heartbeat from core client for 30 sec - exiting 12:37:31 (4336): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... 19:42:30 (3112): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:37 (3112): No heartbeat from core client for 30 sec - exiting 19:43:01 (2624): Can't acquire lockfile (32) - waiting 35s 19:44:32 (4400): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:38 (4400): No heartbeat from core client for 30 sec - exiting 19:44:39 (4400): No heartbeat from core client for 30 sec - exiting 19:44:40 (4400): No heartbeat from core client for 30 sec - exiting 19:44:41 (4400): No heartbeat from core client for 30 sec - exiting 19:44:42 (4400): No heartbeat from core client for 30 sec - exiting 19:44:43 (4400): No heartbeat from core client for 30 sec - exiting 19:44:44 (4400): No heartbeat from core client for 30 sec - exiting 19:44:45 (4400): No heartbeat from core client for 30 sec - exiting 19:44:46 (4400): No heartbeat from core client for 30 sec - exiting 19:44:47 (4400): No heartbeat from core client for 30 sec - exiting 19:45:31 (4796): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: The requested operation cannot be performed on a file with a user-mapped section open. forrtl: severe (38): error during write, unit 6, file D:\BOINC\Data\projects\climateprediction.net\famous_uh5x_1599_200_006655088\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=532, iMonCtr=1 Model crash detected, will try to restart... 19:51:30 (532): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:31:50 (4276): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:32:36 (3752): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:33:20 (4100): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 22:34:10 (4732): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:26:41 (1372): No heartbeat from core client for 30 sec - exiting 17:26:49 (1372): No heartbeat from core client for 30 sec - exiting 17:26:50 (1372): No heartbeat from core client for 30 sec - exiting 17:26:51 (1372): No heartbeat from core client for 30 sec - exiting 17:26:52 (1372): No heartbeat from core client for 30 sec - exiting 17:26:53 (1372): No heartbeat from core client for 30 sec - exiting 17:26:54 (1372): No heartbeat from core client for 30 sec - exiting 17:26:55 (1372): No heartbeat from core client for 30 sec - exiting 17:26:56 (1372): No heartbeat from core client for 30 sec - exiting 17:27:27 (1372): No heartbeat from core client for 30 sec - exiting 17:27:28 (1372): No heartbeat from core client for 30 sec - exiting 17:27:29 (1372): No heartbeat from core client for 30 sec - exiting 17:27:30 (1372): No heartbeat from core client for 30 sec - exiting 17:27:31 (1372): No heartbeat from core client for 30 sec - exiting 17:27:32 (1372): No heartbeat from core client for 30 sec - exiting 17:27:33 (1372): No heartbeat from core client for 30 sec - exiting 17:27:34 (1372): No heartbeat from core client for 30 sec - exiting 17:27:35 (1372): No heartbeat from core client for 30 sec - exiting 17:27:36 (1372): No heartbeat from core client for 30 sec - exiting 17:27:37 (1372): No heartbeat from core client for 30 sec - exiting 17:27:38 (1372): No heartbeat from core client for 30 sec - exiting 17:27:39 (1372): No heartbeat from core client for 30 sec - exiting 17:28:10 (1372): No heartbeat from core client for 30 sec - exiting 17:28:11 (1372): No heartbeat from core client for 30 sec - exiting 17:28:12 (1372): No heartbeat from core client for 30 sec - exiting 17:28:13 (1372): No heartbeat from core client for 30 sec - exiting 17:28:14 (1372): No heartbeat from core client for 30 sec - exiting 17:28:15 (1372): No heartbeat from core client for 30 sec - exiting 17:28:16 (1372): No heartbeat from core client for 30 sec - exiting 17:28:17 (1372): No heartbeat from core client for 30 sec - exiting 17:28:18 (1372): No heartbeat from core client for 30 sec - exiting 17:28:19 (1372): No heartbeat from core client for 30 sec - exiting 17:28:20 (1372): No heartbeat from core client for 30 sec - exiting 17:28:21 (1372): No heartbeat from core client for 30 sec - exiting 17:28:22 (1372): No heartbeat from core client for 30 sec - exiting 17:28:23 (1372): No heartbeat from core client for 30 sec - exiting 17:28:24 (1372): No heartbeat from core client for 30 sec - exiting 17:28:25 (1372): No heartbeat from core client for 30 sec - exiting 17:28:26 (1372): No heartbeat from core client for 30 sec - exiting 17:28:27 (1372): No heartbeat from core client for 30 sec - exiting 17:28:28 (1372): No heartbeat from core client for 30 sec - exiting 17:28:29 (1372): No heartbeat from core client for 30 sec - exiting 17:28:30 (1372): No heartbeat from core client for 30 sec - exiting 17:28:31 (1372): No heartbeat from core client for 30 sec - exiting 17:28:32 (1372): No heartbeat from core client for 30 sec - exiting 17:28:33 (1372): No heartbeat from core client for 30 sec - exiting 17:28:34 (1372): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 0, checkPID=0, selfPID=0, iMonCtr=0 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 11:25:50 (6568): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:33:17 (2808): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:34:28 (6848): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 11:37:26 (6232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:50:16 (5992): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:50:57 (5368): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 12:50:58 (5368): No heartbeat from core client for 30 sec - exiting 12:50:59 (6088): Can't acquire lockfile (32) - waiting 35s 19:26:47 (6088): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 02:52:47 (4744): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:05 (6512): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 05:55:06 (6512): No heartbeat from core client for 30 sec - exiting 05:55:07 (6512): No heartbeat from core client for 30 sec - exiting 05:55:08 (6512): No heartbeat from core client for 30 sec - exiting 05:55:09 (6512): No heartbeat from core client for 30 sec - exiting 05:55:10 (6512): No heartbeat from core client for 30 sec - exiting 05:55:11 (6512): No heartbeat from core client for 30 sec - exiting 13:01:05 (6440): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 13:01:06 (6440): No heartbeat from core client for 30 sec - exiting 13:01:07 (6440): No heartbeat from core client for 30 sec - exiting 13:01:08 (6440): No heartbeat from core client for 30 sec - exiting 13:01:09 (6440): No heartbeat from core client for 30 sec - exiting 13:01:10 (6440): No heartbeat from core client for 30 sec - exiting 13:01:11 (6440): No heartbeat from core client for 30 sec - exiting 13:01:12 (6440): No heartbeat from core client for 30 sec - exiting 13:01:13 (6440): No heartbeat from core client for 30 sec - exiting 13:01:14 (6440): No heartbeat from core client for 30 sec - exiting 13:01:15 (6440): No heartbeat from core client for 30 sec - exiting 13:04:17 (4284): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... forrtl: severe (47): write to READONLY file, unit 6, file D:\BOINC\Data\projects\climateprediction.net\famous_uh5x_1599_200_006655088\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown Stack trace terminated abnormally. Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=5220, iMonCtr=1 Model crash detected, will try to restart... Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... 20:14:55 (2000): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 20:14:56 (2000): No heartbeat from core client for 30 sec - exiting 20:14:57 (2000): No heartbeat from core client for 30 sec - exiting 20:15:01 (2000): No heartbeat from core client for 30 sec - exiting 20:15:02 (2000): No heartbeat from core client for 30 sec - exiting 20:15:03 (2000): No heartbeat from core client for 30 sec - exiting 20:15:04 (2000): No heartbeat from core client for 30 sec - exiting 20:18:35 (5204): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 02:30:45 (3028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:41:59 (2620): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:42:56 (5588): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:43:32 (656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:44:46 (5196): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:46:57 (5176): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 23:20:55 (3060): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 19:13:21 (5028): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... 19:17:16 (3780): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:18:33 (3868): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:16 (2304): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:20:17 (2304): No heartbeat from core client for 30 sec - exiting 19:20:18 (2304): No heartbeat from core client for 30 sec - exiting 19:20:19 (2304): No heartbeat from core client for 30 sec - exiting 19:20:20 (2304): No heartbeat from core client for 30 sec - exiting 19:20:21 (2304): No heartbeat from core client for 30 sec - exiting 19:20:22 (2304): No heartbeat from core client for 30 sec - exiting 19:20:23 (2304): No heartbeat from core client for 30 sec - exiting 19:20:24 (2304): No heartbeat from core client for 30 sec - exiting 19:20:25 (2304): No heartbeat from core client for 30 sec - exiting 19:20:59 (5128): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:21:00 (5128): No heartbeat from core client for 30 sec - exiting 19:21:01 (5128): No heartbeat from core client for 30 sec - exiting 19:21:54 (3164): No heartbeat from core client for 30 sec - exiting 19:21:55 (3164): No heartbeat from core client for 30 sec - exiting 19:21:56 (3164): No heartbeat from core client for 30 sec - exiting 19:21:57 (3164): No heartbeat from core client for 30 sec - exiting 19:21:58 (3164): No heartbeat from core client for 30 sec - exiting 19:21:59 (3164): No heartbeat from core client for 30 sec - exiting 19:22:00 (3164): No heartbeat from core client for 30 sec - exiting 19:22:01 (3164): No heartbeat from core client for 30 sec - exiting 19:22:02 (3164): No heartbeat from core client for 30 sec - exiting 19:22:03 (3164): No heartbeat from core client for 30 sec - exiting 19:22:04 (3164): No heartbeat from core client for 30 sec - exiting 19:22:05 (3164): No heartbeat from core client for 30 sec - exiting 19:22:06 (3164): No heartbeat from core client for 30 sec - exiting 19:22:09 (3164): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:23:00 (3812): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:24:11 (5504): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:25:43 (804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 19:25:50 (804): No heartbeat from core client for 30 sec - exiting 19:25:51 (804): No heartbeat from core client for 30 sec - exiting 19:25:52 (804): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:18:32 (6272): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_uh5x_1599_200_006655088/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_uh5x_1599_200_006655088/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_uh5x_1599_200_006655088/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_uh5x_1599_200_006655088/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_uh5x_1599_200_006655088/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file D:\BOINC\Data/projects/climateprediction.net/famous_uh5x_1599_200_006655088/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy Sorry, too many model crashes! :-( 17:21:43 (5856): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
06 Sep 2010 14:42:13	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,422,746	426,838	0.3000
06 Sep 2010 13:58:01	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,413,386	424,012	0.3000
06 Sep 2010 13:08:45	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,404,026	421,180	0.3000
06 Sep 2010 12:21:41	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,394,666	418,318	0.2999
06 Sep 2010 11:35:35	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,385,306	415,453	0.2999
06 Sep 2010 10:45:41	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,375,946	412,633	0.2999
06 Sep 2010 09:58:59	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,366,586	409,781	0.2999
06 Sep 2010 09:06:34	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,357,226	406,910	0.2998
06 Sep 2010 02:58:55	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,347,866	404,032	0.2998
06 Sep 2010 02:10:38	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,338,506	401,171	0.2997
06 Sep 2010 01:22:24	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,329,146	398,321	0.2997
06 Sep 2010 00:34:38	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,319,786	395,467	0.2996
06 Sep 2010 00:01:23	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,310,426	392,628	0.2996
06 Sep 2010 00:01:20	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,301,066	389,780	0.2996
05 Sep 2010 22:04:57	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,291,706	386,927	0.2995
05 Sep 2010 21:17:29	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,282,346	384,061	0.2995
05 Sep 2010 20:29:35	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,272,986	381,179	0.2994
05 Sep 2010 19:54:14	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,263,626	378,304	0.2994
05 Sep 2010 19:54:14	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,254,266	375,418	0.2993
05 Sep 2010 17:54:18	1057648	11526285	famous_uh5x_1599_200_006655088_4	1,244,906	372,512	0.2992