Task 12552338

Name	famous_vlbd_1799_200_006713391_5
Workunit	6916644
Created	2 Feb 2011, 0:16:51 UTC
Sent	2 Feb 2011, 0:16:58 UTC
Report deadline	4 May 2011, 7:44:09 UTC
Received	8 Mar 2011, 14:14:26 UTC
Server state	Over
Outcome	Computation error
Client state	Compute error
Exit status	22 (0x00000016) Unknown error code
Computer ID	1131252
Run time	16 days 13 hours 39 min 47 sec
CPU time	16 days 3 hours 40 min 43 sec
Validate state	Invalid
Credit	5,126.44
Device peak FLOPS	1.35 GFLOPS
Application version	UK Met Office FAMOUS v6.11 windows_intelx86
Stderr	<core_client_version>6.10.58</core_client_version> <![CDATA[ <message> The device does not recognize the command. (0x16) - exit code 22 (0x16) </message> <stderr_txt> CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:20:13 (1716): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... forrtl: The requested operation cannot be performed on a file with a user-mapped section open. forrtl: severe (38): error during write, unit 6, file L:\Bonic Data\projects\climateprediction.net\famous_vlbd_1799_200_006713391\dataout\stdout_um.txt Image PC Routine Line Source famous_um_6.11_wi 008846D2 Unknown Unknown Unknown famous_um_6.11_wi 0088196C Unknown Unknown Unknown famous_um_6.11_wi 0080BD3E Unknown Unknown Unknown famous_um_6.11_wi 0080B95B Unknown Unknown Unknown famous_um_6.11_wi 007F0945 Unknown Unknown Unknown famous_um_6.11_wi 007F00F5 Unknown Unknown Unknown famous_um_6.11_wi 007BA86B Unknown Unknown Unknown famous_um_6.11_wi 007BA0FF Unknown Unknown Unknown famous_um_6.11_wi 007BA0A0 Unknown Unknown Unknown famous_um_6.11_wi 00776C15 Unknown Unknown Unknown famous_um_6.11_wi 007CA0F6 Unknown Unknown Unknown kernel32.dll 7C817067 Unknown Unknown Unknown Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2628, iMonCtr=1 Model crash detected, will try to restart... 16:47:31 (348): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:11:15 (1860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 11:40:28 (188): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:18:30 (2108): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:42:57 (2232): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:45:04 (688): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2188, selfPID=2188, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:27:34 (2544): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2464, selfPID=2464, iMonCtr=1 No Process Handle Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2548, selfPID=2548, iMonCtr=1 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 15:43:25 (1860): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 16:34:46 (1492): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2092, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2092, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2092, iMonCtr=1 Model crash detected, will try to restart... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=2092, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Suspended CPDN Monitor - Suspend request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 Worker:: CPDN process is not running, exiting, bRetVal = 1, checkPID=2432, selfPID=2432, iMonCtr=1 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Controller:: CPDN process is not running, exiting, bRetVal = 1, checkPID=0, selfPID=7964, iMonCtr=1 Model crash detected, will try to restart... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 16:35:31 (1656): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... 10:35:03 (2220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 13:28:40 (1896): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 60 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 61 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 68 - Return code = 16 BUFFIN: Read Failed: No such file or directory BUFFIN: C I/O Error feof - Unit 69 - Return code = 16 CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 17:14:24 (3020): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 17:23:42 (9220): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... Suspended CPDN Monitor - Suspend request from BOINC... 12:21:27 (6696): No heartbeat from core client for 30 sec - exiting Suspended CPDN Monitor - No 'heartbeat' from BOINC... 12:21:28 (6696): No heartbeat from core client for 30 sec - exiting CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:50:45 (9092): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 12:20:09 (7616): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... CPDN Monitor - Quit request from BOINC... CPDN Monitor - Quit request from BOINC... 13:53:09 (3216): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... 14:44:40 (1432): No heartbeat from core client for 30 sec - exiting CPDN Monitor - No 'heartbeat' from BOINC... cpdnmonitor: cannot open input file L:\Bonic Data/projects/climateprediction.net/famous_vlbd_1799_200_006713391/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file L:\Bonic Data/projects/climateprediction.net/famous_vlbd_1799_200_006713391/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file L:\Bonic Data/projects/climateprediction.net/famous_vlbd_1799_200_006713391/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file L:\Bonic Data/projects/climateprediction.net/famous_vlbd_1799_200_006713391/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file L:\Bonic Data/projects/climateprediction.net/famous_vlbd_1799_200_006713391/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy cpdnmonitor: cannot open input file L:\Bonic Data/projects/climateprediction.net/famous_vlbd_1799_200_006713391/dataout/ocean_restart.day after 11 attempts Model crashed: READ_FLH: I/O error tmp/pipe_dummy Sorry, too many model crashes! :-( 09:16:29 (3028): called boinc_finish </stderr_txt> ]]>

Latest Trickles Received
Time Sent (UTC)	Host ID	Result ID	Result Name	Timestep	CPU Time (sec)	Average (sec/TS)
26 Feb 2011 19:51:34	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,553,786	1,395,100	0.8979
26 Feb 2011 19:06:59	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,544,426	1,392,045	0.9013
26 Feb 2011 19:06:06	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,535,066	1,388,722	0.9047
26 Feb 2011 16:52:59	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,525,706	1,385,401	0.9080
26 Feb 2011 15:57:16	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,516,346	1,382,110	0.9115
26 Feb 2011 15:06:33	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,506,986	1,378,820	0.9150
26 Feb 2011 14:10:49	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,497,626	1,375,531	0.9185
26 Feb 2011 13:15:01	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,488,266	1,372,240	0.9220
26 Feb 2011 12:18:54	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,478,906	1,368,950	0.9257
26 Feb 2011 11:22:52	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,469,546	1,365,662	0.9293
26 Feb 2011 10:31:54	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,460,186	1,362,373	0.9330
26 Feb 2011 09:36:05	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,450,826	1,359,088	0.9368
26 Feb 2011 08:40:03	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,441,466	1,355,752	0.9405
26 Feb 2011 07:43:44	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,432,106	1,352,261	0.9442
26 Feb 2011 06:42:41	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,422,746	1,348,775	0.9480
26 Feb 2011 05:46:25	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,413,386	1,345,285	0.9518
26 Feb 2011 04:44:16	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,404,026	1,341,798	0.9557
26 Feb 2011 03:48:28	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,394,666	1,338,319	0.9596
26 Feb 2011 02:49:27	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,385,306	1,334,832	0.9636
26 Feb 2011 01:53:34	1131252	12552338	famous_vlbd_1799_200_006713391_5	1,375,946	1,331,344	0.9676