climateprediction.net home page

Upload Failure


Advanced search

Message boards : Number crunching : Upload Failure

AuthorMessage
Lockleys
Send message
Joined: Jan 13 07
Posts: 126
Credit: 4,285,176
RAC: 2,020
Message 43140 - Posted 5 Oct 2011 9:39:58 UTC

    Servers all appear OK, but I am getting a repetitive upload failure, namely
    05/10/2011 10:27:54 | climateprediction.net | Temporarily failed upload of hadam3p_eu_6dhw_2001_1_007479505_2_9.zip: connect() failed
    Other models on the same PC are so far uploading just fine.
    Any clues?

    Waldmeister
    Send message
    Joined: Jun 13 11
    Posts: 32
    Credit: 145,646
    RAC: 660
    Message 43141 - Posted 5 Oct 2011 9:56:09 UTC - in response to Message 43140.

      I am having about the same problem for one and a half days.

      But look at the last announcement in the "news and announcement" thread :

      http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=5447

      Maybe that's the explanation, but I do not know for sure.

      Eirik Redd
      Send message
      Joined: Aug 31 04
      Posts: 257
      Credit: 27,941,560
      RAC: 13,465
      Message 43142 - Posted 5 Oct 2011 10:19:44 UTC - in response to Message 43141.

        Yeah , there's an upload server with a problem, like you said refer to the "News and announcement" thread

        What may be somewhat confusing is that uploads go to any of several servers -- so if your particular work unit is set to upload to working servers -- no problem.

        If your work unit was set to use the currently broken server -- just wait -- expect they will have the hardware replaced soon. When that happens upload will proceed normally -- no data lost.

        Even individual upload files within a work unit may be scheduled for different upload servers -- If you are really curious look in your BOINC/client_state.xml file -- all the details are there -- more details than most of us can grok.

        "Noli perspirare"

        keep crunching and thanks for posting.


        ____________

        Lockleys
        Send message
        Joined: Jan 13 07
        Posts: 126
        Credit: 4,285,176
        RAC: 2,020
        Message 43143 - Posted 5 Oct 2011 12:11:06 UTC

          Missed that one. Got it now. Thanks, guys.

          Waldmeister
          Send message
          Joined: Jun 13 11
          Posts: 32
          Credit: 145,646
          RAC: 660
          Message 43149 - Posted 7 Oct 2011 8:03:35 UTC

            What's up folks? Problem with the disk resolved or what?

            I am having two files that need to be uploaded to complete my task. Task itself is done since early tuesday and I am waiting to finish the job.

            I am not going to start another task until I am sure uploads work.

            Les Bayliss
            Forum moderator
            Send message
            Joined: Sep 5 04
            Posts: 5411
            Credit: 9,006,998
            RAC: 3,097
            Message 43150 - Posted 7 Oct 2011 8:12:08 UTC - in response to Message 43149.

              There hasn't been any news yet.


              ____________
              Backups: Here

              [AF>France>Aquitaine>Cote-Adour-et-Gaves]Bernard 64250
              Send message
              Joined: Sep 25 09
              Posts: 5
              Credit: 238,226
              RAC: 0
              Message 43153 - Posted 7 Oct 2011 12:38:17 UTC

                Last modified: 7 Oct 2011 12:39:37 UTC

                For one of my WU (hadam3p_eu_2t5i_1963_1_007428406_2), it seems that all trickles have been received, but I still have 3 files waiting for upload ( hadam3p_eu_2t5i_1963_1_007428406_2_9.zip, hadam3p_eu_2t5i_1963_1_007428406_2_11.zip, hadam3p_eu_2t5i_1963_1_007428406_2_12.zip ) so the task cannot complete.
                It's strange that hadam3p_eu_2t5i_1963_1_007428406_2_10.zip is not in the list.
                ____________

                Les Bayliss
                Forum moderator
                Send message
                Joined: Sep 5 04
                Posts: 5411
                Credit: 9,006,998
                RAC: 3,097
                Message 43154 - Posted 7 Oct 2011 15:36:27 UTC - in response to Message 43153.

                  It's strange that hadam3p_eu_2t5i_1963_1_007428406_2_10.zip is not in the list.

                  It's not at all strange. Different zip numbers go to different upload servers to spread the load, just as Eirik Redd said a few posts previous to this.


                  ____________
                  Backups: Here

                  Floyd
                  Send message
                  Joined: Sep 21 11
                  Posts: 1
                  Credit: 33,348
                  RAC: 0
                  Message 43271 - Posted 25 Oct 2011 18:49:45 UTC

                    Last modified: 25 Oct 2011 19:21:55 UTC

                    (EDIT) It went thru now , thx anyway !

                    I know that the project has been down for a few days , BUT , I still can't upload... I am getting :

                    10/25/2011 1:42:09 PM | climateprediction.net | update requested by user
                    10/25/2011 1:42:12 PM | climateprediction.net | Sending scheduler request: Requested by user.
                    10/25/2011 1:42:12 PM | climateprediction.net | Reporting 2 completed tasks, not requesting new tasks, sending trickle-up message
                    10/25/2011 1:43:23 PM | climateprediction.net | Scheduler request failed: Server returned nothing (no headers, no data)
                    10/25/2011 1:43:39 PM | | Project communication failed: attempting access to reference site
                    10/25/2011 1:43:42 PM | | Internet access OK - project servers may be temporarily down.
                    10/25/2011 1:44:52 PM | climateprediction.net | Fetching scheduler list
                    10/25/2011 1:44:55 PM | climateprediction.net | Master file download succeeded
                    10/25/2011 1:45:01 PM | climateprediction.net | Sending scheduler request: To send trickle-up message.
                    10/25/2011 1:45:01 PM | climateprediction.net | Reporting 2 completed tasks, not requesting new tasks
                    10/25/2011 1:46:11 PM | climateprediction.net | Scheduler request failed: Server returned nothing (no headers, no data)
                    10/25/2011 1:46:26 PM | | Project communication failed: attempting access to reference site
                    10/25/2011 1:46:29 PM | | Internet access OK - project servers may be temporarily down.

                    I have been getting pretty much this same thing for several days , I have 2 WU's that are done , and a trickle up that needs to go thru ...
                    ____________

                    Dean M
                    Send message
                    Joined: Oct 26 04
                    Posts: 20
                    Credit: 11,198,441
                    RAC: 2,831
                    Message 43297 - Posted 26 Oct 2011 19:58:14 UTC

                      I have been sending trickles during the outage. Now that the project is up and running again I am sending trickles but they are not being posted to the workunit page. I hope all this crunching time isn't being wasted. Please advise if we keep going or abort and start a fresh unit.


                      ____________

                      Les Bayliss
                      Forum moderator
                      Send message
                      Joined: Sep 5 04
                      Posts: 5411
                      Credit: 9,006,998
                      RAC: 3,097
                      Message 43298 - Posted 26 Oct 2011 20:03:13 UTC - in response to Message 43297.

                        The bulk of the repair work has been finished. But there's still tidying up to do.
                        And two of the things still to fix, are the trickles and the graph display from the zip data.

                        As for starting a new model, they're all gone. :(


                        ____________
                        Backups: Here

                        Dean M
                        Send message
                        Joined: Oct 26 04
                        Posts: 20
                        Credit: 11,198,441
                        RAC: 2,831
                        Message 43299 - Posted 26 Oct 2011 20:09:00 UTC - in response to Message 43298.

                          Today is my 7 year anniversary and you say all the work is gone!! Gosh, I hope you can make more.
                          ____________

                          Les Bayliss
                          Forum moderator
                          Send message
                          Joined: Sep 5 04
                          Posts: 5411
                          Credit: 9,006,998
                          RAC: 3,097
                          Message 43301 - Posted 26 Oct 2011 22:19:43 UTC - in response to Message 43299.

                            Please see my post of the 28th of August here in the News and Announcements thread at the top of the Number crunching section regarding more work.

                            This thread can be subscribed to, to keep yourself informed about the project.


                            ____________
                            Backups: Here

                            dm1111
                            Send message
                            Joined: Sep 3 04
                            Posts: 1
                            Credit: 155,919
                            RAC: 0
                            Message 43928 - Posted 8 Mar 2012 15:25:56 UTC

                              I have problems reporting finished work.
                              Every time when trying to report i get a "Error in request message: fgets() failed".
                              The work unit was uploaded without any problems.
                              The work-unit-ID is: wuid=7578887
                              What can i do?
                              ____________

                              Les Bayliss
                              Forum moderator
                              Send message
                              Joined: Sep 5 04
                              Posts: 5411
                              Credit: 9,006,998
                              RAC: 3,097
                              Message 43929 - Posted 8 Mar 2012 21:46:53 UTC - in response to Message 43928.

                                I think that's a server side problem, possibly because the server load was too high at the time. In which case it'll fix itself after a while.


                                ____________
                                Backups: Here

                                Brickhead
                                Avatar
                                Send message
                                Joined: Aug 31 04
                                Posts: 4
                                Credit: 8,942,192
                                RAC: 0
                                Message 43939 - Posted 13 Mar 2012 16:20:37 UTC

                                  2012-03-12 16:06:50 Started upload of hadam3p_eu_ajcw_1976_1_007808357_1_9.zip
                                  2012-03-12 16:06:53 [error] Error reported by file upload server: Server is out of disk space

                                  ____________

                                  Les Bayliss
                                  Forum moderator
                                  Send message
                                  Joined: Sep 5 04
                                  Posts: 5411
                                  Credit: 9,006,998
                                  RAC: 3,097
                                  Message 43941 - Posted 13 Mar 2012 21:21:23 UTC - in response to Message 43939.

                                    This has been happening a bit lately. Also various other error messages, as the server tries to cope with the huge inflow of data.

                                    The project people know about it, and are keeping an eye on things.

                                    Just let BOINC keep trying. :)

                                    Profile JIM
                                    Send message
                                    Joined: Dec 31 07
                                    Posts: 679
                                    Credit: 4,128,527
                                    RAC: 2,418
                                    Message 43964 - Posted 2 Apr 2012 3:34:43 UTC

                                      I see that uploader1.atm is down again. Does anyone know what's wrong. Has it just filled up or is it or is there a real problem. Any idea when it will be up again. I already have 2 zip files stuck in my transfer tab.

                                      ____________

                                      Profile Dave Jackson
                                      Send message
                                      Joined: May 15 09
                                      Posts: 849
                                      Credit: 645,754
                                      RAC: 154
                                      Message 43965 - Posted 2 Apr 2012 7:46:59 UTC - in response to Message 43964.

                                        Last modified: 2 Apr 2012 7:52:50 UTC

                                        Doubtless Jonathan and team will be on to it shortly. Still pre-0900 on a Monday morning uk time.

                                        Profile Dave Jackson
                                        Send message
                                        Joined: May 15 09
                                        Posts: 849
                                        Credit: 645,754
                                        RAC: 154
                                        Message 43966 - Posted 2 Apr 2012 10:18:02 UTC - in response to Message 43965.

                                          Looks like it is up and running again.

                                          Profile Ananas
                                          Forum moderator
                                          Send message
                                          Joined: Oct 31 04
                                          Posts: 336
                                          Credit: 3,316,482
                                          RAC: 0
                                          Message 43995 - Posted 11 Apr 2012 18:46:45 UTC - in response to Message 43929.

                                            Last modified: 11 Apr 2012 19:44:00 UTC

                                            I think that's a server side problem, possibly because the server load was too high at the time. In which case it'll fix itself after a while.


                                            It is a server side problem, the scheduler (parser) tries to read 256 bytes from sched_request.xml and doesn't get those.

                                            There is no syntax or sanity check yet, just reading stuff into the buffer fails.

                                            It already happened on 3 boxes for me, two of which got work in the meantime, one still struggling.

                                            The file handle is most likely not null because it does check that (a bunch of statements before trying fgets() though).

                                            Unfortunately they don't report errno so it's not so easy to tell the exact reason.

                                            p.s.: the upload error and the scheduler error are not necessarily related (2 different programs) but the chance is high that the same thing causes them. The fgets() problem has been reported in a bunch of other projects like lhc, simap and seti

                                            Lockleys
                                            Send message
                                            Joined: Jan 13 07
                                            Posts: 126
                                            Credit: 4,285,176
                                            RAC: 2,020
                                            Message 44046 - Posted 19 Apr 2012 12:51:16 UTC

                                              I'm getting upload failures on zip file 13 from AM3P EU models today.

                                              Richard Haselgrove
                                              Send message
                                              Joined: Jan 1 07
                                              Posts: 299
                                              Credit: 6,878,929
                                              RAC: 13
                                              Message 44047 - Posted 19 Apr 2012 13:44:14 UTC - in response to Message 44046.

                                                I'm getting upload failures on zip file 13 from AM3P EU models today.

                                                Me too. Specifically,

                                                19/04/2012 14:39:35 | climateprediction.net | [error] Error reported by file upload server: can't open file

                                                Silverthorne
                                                Send message
                                                Joined: Aug 7 04
                                                Posts: 1
                                                Credit: 316,125
                                                RAC: 0
                                                Message 44050 - Posted 19 Apr 2012 21:42:52 UTC - in response to Message 44046.

                                                  I'm getting upload failures on zip file 13 from AM3P EU models today.




                                                  Same here.

                                                  4/19/2012 1:40:54 PM | climateprediction.net | Temporarily failed upload of hadam3p_eu_ag6v_1990_1_007841613_1_13.zip: transient upload error

                                                  ____________

                                                  marpes
                                                  Send message
                                                  Joined: Nov 11 04
                                                  Posts: 8
                                                  Credit: 15,188,494
                                                  RAC: 1,298
                                                  Message 44055 - Posted 20 Apr 2012 10:35:03 UTC

                                                    My one of:
                                                    20. 4. 2012 9:33:18 | climateprediction.net | Started upload of hadam3p_saf_0shh_1964_1_006860493_1_13.zip
                                                    20. 4. 2012 9:33:20 | climateprediction.net | Temporarily failed upload of hadam3p_saf_0shh_1964_1_006860493_1_13.zip: transient HTTP error
                                                    20. 4. 2012 9:33:20 | climateprediction.net | Backing off 4 hr 18 min 58 sec on upload of hadam3p_saf_0shh_1964_1_006860493_1_13.zip

                                                    error state is more than 24 hours
                                                    ____________

                                                    Lockleys
                                                    Send message
                                                    Joined: Jan 13 07
                                                    Posts: 126
                                                    Credit: 4,285,176
                                                    RAC: 2,020
                                                    Message 44056 - Posted 20 Apr 2012 11:10:37 UTC - in response to Message 44046.

                                                      I'm getting upload failures on zip file 13 from AM3P EU models today.


                                                      My zip files 13 are now uploading.

                                                      Eirik Redd
                                                      Send message
                                                      Joined: Aug 31 04
                                                      Posts: 257
                                                      Credit: 27,941,560
                                                      RAC: 13,465
                                                      Message 44058 - Posted 21 Apr 2012 11:58:45 UTC - in response to Message 44056.

                                                        Last modified: 21 Apr 2012 12:10:04 UTC

                                                        Now it's upload problem hadam3p_saf_28zu_1975_1_007240600_1_3.zip
                                                        since about 10:19 Zulu
                                                        Reports "transient upload problem"
                                                        Only on SAF final .13

                                                        At _present/>
                                                        <url>http://cpdn-upload2.oerc.ox.ac.uk/cgi-bin/file_upload_handler</url/>

                                                        Sorry -- usually wait a day or two before reporting upload problems -- should have waited until Monday in any case -- the staff always fix these things -- the saf not in my script -- apologies -- this can wait a few days.

                                                        EDIT -- "not a biggie problem at all"
                                                        ____________

                                                        Klaus Trost
                                                        Send message
                                                        Joined: Dec 6 05
                                                        Posts: 1
                                                        Credit: 250,722
                                                        RAC: 0
                                                        Message 44059 - Posted 21 Apr 2012 16:11:44 UTC

                                                          Last modified: 21 Apr 2012 16:12:22 UTC

                                                          i have the same Problem:

                                                          Sa 21 Apr 18:06:34 2012 | climateprediction.net | Started upload of hadam3p_eu_94qe_1966_1_007726190_0_13.zip
                                                          Sa 21 Apr 18:06:35 2012 | climateprediction.net | [error] Error reported by file upload server: can't open file
                                                          Sa 21 Apr 18:06:35 2012 | climateprediction.net | Temporarily failed upload of hadam3p_eu_94qe_1966_1_007726190_0_13.zip: transient upload error
                                                          Sa 21 Apr 18:06:35 2012 | climateprediction.net | Backing off 3 hr 17 min 32 sec on upload of hadam3p_eu_94qe_1966_1_007726190_0_13.zip


                                                          error state is more than 24 hours

                                                          Profile Dave Jackson
                                                          Send message
                                                          Joined: May 15 09
                                                          Posts: 849
                                                          Credit: 645,754
                                                          RAC: 154
                                                          Message 44060 - Posted 21 Apr 2012 16:43:56 UTC - in response to Message 44059.

                                                            Most likely the server is full up. Whatever the problem it is unlikely that anyone will get a chance to look at the server till Monday after 0900 UK time.

                                                            Dave

                                                            Les Bayliss
                                                            Forum moderator
                                                            Send message
                                                            Joined: Sep 5 04
                                                            Posts: 5411
                                                            Credit: 9,006,998
                                                            RAC: 3,097
                                                            Message 44061 - Posted 21 Apr 2012 19:49:21 UTC

                                                              There is a problem with the server which can't be fixed remotely, so the project people need to get physical access to it.
                                                              This won't happen until Monday morning UK time.
                                                              And then it may take a while for them to find out what's wrong, and even more time to fix it. Especially if replacement parts are needed.


                                                              ____________
                                                              Backups: Here

                                                              Les Bayliss
                                                              Forum moderator
                                                              Send message
                                                              Joined: Sep 5 04
                                                              Posts: 5411
                                                              Credit: 9,006,998
                                                              RAC: 3,097
                                                              Message 44062 - Posted 22 Apr 2012 19:54:21 UTC - in response to Message 44061.

                                                                Andy went in on Sunday and gave the server a good talking to.
                                                                It's now working again.

                                                                ____________
                                                                Backups: Here

                                                                Profile Dave Jackson
                                                                Send message
                                                                Joined: May 15 09
                                                                Posts: 849
                                                                Credit: 645,754
                                                                RAC: 154
                                                                Message 44063 - Posted 22 Apr 2012 20:28:52 UTC - in response to Message 44062.

                                                                  Never thought of trying that when my machine is misbehaving, must try it next time I have a problem.

                                                                  Profile Dave Jackson
                                                                  Send message
                                                                  Joined: May 15 09
                                                                  Posts: 849
                                                                  Credit: 645,754
                                                                  RAC: 154
                                                                  Message 44071 - Posted 23 Apr 2012 10:01:35 UTC - in response to Message 44063.

                                                                    Looks like some more talking is needed.

                                                                    Mon 23 Apr 2012 10:55:31 BST | climateprediction.net | Started upload of hadam3p_eu_98iv_1963_1_007852746_0_11.zip
                                                                    Mon 23 Apr 2012 10:55:32 BST | climateprediction.net | [error] Error reported by file upload server: can't open log file '../log_uploader1/file_upload_handler.log' (errno: 9)
                                                                    Mon 23 Apr 2012 10:55:32 BST | climateprediction.net | Temporarily failed upload of hadam3p_eu_98iv_1963_1_007852746_0_11.zip: transient upload error
                                                                    Mon 23 Apr 2012 10:55:32 BST | climateprediction.net | Backing off 1 hr 50 min 55 sec on upload of hadam3p_eu_98iv_1963_1_007852746_0_11.zip


                                                                    Dave

                                                                    Profile Mike*
                                                                    Send message
                                                                    Joined: Nov 2 07
                                                                    Posts: 1
                                                                    Credit: 332,900
                                                                    RAC: 0
                                                                    Message 44073 - Posted 23 Apr 2012 10:30:48 UTC

                                                                      I found simpler way.
                                                                      Abort and disconnect.

                                                                      Profile Dave Jackson
                                                                      Send message
                                                                      Joined: May 15 09
                                                                      Posts: 849
                                                                      Credit: 645,754
                                                                      RAC: 154
                                                                      Message 44074 - Posted 23 Apr 2012 10:42:00 UTC - in response to Message 44073.

                                                                        Once the relevant server is working again it will accept the zip files. Why bother running the project only to waste the computation time by aborting?

                                                                        Dave

                                                                        Profile Jonathan Miller
                                                                        Forum moderator
                                                                        Project administrator
                                                                        Project developer
                                                                        Volunteer developer
                                                                        Send message
                                                                        Joined: Mar 28 11
                                                                        Posts: 35
                                                                        Credit: 82,588
                                                                        RAC: 0
                                                                        Message 44075 - Posted 23 Apr 2012 15:54:17 UTC

                                                                          Hi everyone,

                                                                          We are currently suffering two server failures - both serious hard disk issues, so I am configuring another to take over their roles before I get around to sorting out those problems.

                                                                          I will let you know how things proceed, but it will be at least 24 hours before we can consider ourselves back online.

                                                                          Please accept my apologies.

                                                                          Jonathan

                                                                          CPDN Sys-Admin

                                                                          Profile Dave Jackson
                                                                          Send message
                                                                          Joined: May 15 09
                                                                          Posts: 849
                                                                          Credit: 645,754
                                                                          RAC: 154
                                                                          Message 44077 - Posted 23 Apr 2012 19:57:38 UTC - in response to Message 44075.

                                                                            OK network activity suspended till tomorrow at least. That way it won't keep trying when everyone else is also trying.

                                                                            Dave

                                                                            Eirik Redd
                                                                            Send message
                                                                            Joined: Aug 31 04
                                                                            Posts: 257
                                                                            Credit: 27,941,560
                                                                            RAC: 13,465
                                                                            Message 44081 - Posted 24 Apr 2012 8:31:08 UTC - in response to Message 44077.

                                                                              Yeah -- just wait a while -- some files uploading now, some not.
                                                                              Patience, patience. It won't be long -- don't waste any wu.
                                                                              Don 't kill any process -- this happens sometimes and the support team at Oxford will fix it so nothing gets wasted.
                                                                              They've done it before and will do again -- nothing gets wasted.
                                                                              Donating another few terabytes might be welcome, but who can afford another 200 TB? or EB? or whatever the Bigabytes are now?
                                                                              Patience.
                                                                              They do get it right.
                                                                              Wait a day or two and all will be well.

                                                                              Really.
                                                                              ____________

                                                                              MarkJ
                                                                              Avatar
                                                                              Send message
                                                                              Joined: Mar 28 09
                                                                              Posts: 104
                                                                              Credit: 5,098,940
                                                                              RAC: 691
                                                                              Message 44086 - Posted 25 Apr 2012 8:01:35 UTC

                                                                                Last modified: 25 Apr 2012 8:20:24 UTC

                                                                                Looking at the server status page just now and 3 of the 7 upload servers are off-line. Must be some fairly major failures going on.

                                                                                I know Jonathan wrote that 2 of them had hard disk problems, so looks like they may need to replace a lot of the drives with new ones. Maybe we need a fund-raising drive to help things along? The donations page can be found HERE

                                                                                I don't know what the cost of a 2Tb server grade HDD is in the UK, I would guess around 70 pounds. Anyway i've made a donation to get things started.
                                                                                ____________
                                                                                BOINC blog

                                                                                Profile Dave Jackson
                                                                                Send message
                                                                                Joined: May 15 09
                                                                                Posts: 849
                                                                                Credit: 645,754
                                                                                RAC: 154
                                                                                Message 44087 - Posted 25 Apr 2012 14:47:32 UTC - in response to Message 44086.

                                                                                  Back down to two out again now. At least there are still plenty of work units going. I only have two cores so the transfer backlog isn't taking up too much disk space.

                                                                                  Dave

                                                                                  Les Bayliss
                                                                                  Forum moderator
                                                                                  Send message
                                                                                  Joined: Sep 5 04
                                                                                  Posts: 5411
                                                                                  Credit: 9,006,998
                                                                                  RAC: 3,097
                                                                                  Message 44088 - Posted 25 Apr 2012 22:17:04 UTC

                                                                                    All of the accumulated zips on my machines have now uploaded OK to the Oregon server, so it looks like they're keeping an eye on things there.


                                                                                    ____________
                                                                                    Backups: Here

                                                                                    Profile Dave Jackson
                                                                                    Send message
                                                                                    Joined: May 15 09
                                                                                    Posts: 849
                                                                                    Credit: 645,754
                                                                                    RAC: 154
                                                                                    Message 44094 - Posted 26 Apr 2012 10:38:07 UTC - in response to Message 44088.

                                                                                      zips 11,12 & 13 all failed due to transient http error on eu task. 1 on a new task went through ok. Maybe some will have to wait for the new hard disks to be installed.

                                                                                      Richard Haselgrove
                                                                                      Send message
                                                                                      Joined: Jan 1 07
                                                                                      Posts: 299
                                                                                      Credit: 6,878,929
                                                                                      RAC: 13
                                                                                      Message 44096 - Posted 26 Apr 2012 16:12:51 UTC

                                                                                        At the moment, I'm seeing zip 13 failing on all three regional models - they all go to the same Oxford server, I think.

                                                                                        Also, I'm getting failures on zips 1 to 12 for EU models only - that's the other Oxford server at fault.

                                                                                        But zips 1 to 12 for both the SAF and - provided Oregon can keep their server empty - PNW models are getting through. If you are running one of the later BOINC 6 versions, you may want to try a few transfer retries if you have SAF/PNW uploads (except 13) stuck on a machine which is also running one or more EU models.

                                                                                        Cartoonman
                                                                                        Send message
                                                                                        Joined: Oct 8 08
                                                                                        Posts: 2
                                                                                        Credit: 138,181
                                                                                        RAC: 756
                                                                                        Message 44100 - Posted 26 Apr 2012 20:44:01 UTC

                                                                                          I honestly have no issue keeping a few tasks in the transfer dock. Just dont abort the transfer. Wait till the upload server is online, thats all. Given that these WU's take a very long time to complete anyhow, it shouldn't affect your WU flow. Most likely, by the time the WU's in the queue are done, the upload server will be online. If not, just temporarily switch to another project. That doesn't require any effort.

                                                                                          Bockovic
                                                                                          Send message
                                                                                          Joined: Apr 7 10
                                                                                          Posts: 1
                                                                                          Credit: 21,838
                                                                                          RAC: 0
                                                                                          Message 44104 - Posted 27 Apr 2012 12:44:15 UTC

                                                                                            Last modified: 27 Apr 2012 12:46:08 UTC

                                                                                            One of my uploads got stock. Event log says:

                                                                                            27.4.2012 14:22:52 | | [error] No URL for file transfer of hadam3p_eu_9qs2_1972_1_007855565_1_2.zip
                                                                                            27.4.2012 14:22:52 | climateprediction.net | [error] Can't initialize file transfer for hadam3p_eu_9qs2_1972_1_007855565_1_2.zip
                                                                                            27.4.2012 14:22:52 | | Version change (6.12.34 -> 7.0.25)

                                                                                            and little later upload failed:

                                                                                            27.4.2012 14:32:13 | climateprediction.net | Started upload of hadam3p_eu_9qs2_1972_1_007855565_1_2.zip
                                                                                            27.4.2012 14:32:14 | climateprediction.net | Temporarily failed upload of hadam3p_eu_9qs2_1972_1_007855565_1_2.zip: transient HTTP error
                                                                                            27.4.2012 14:32:14 | climateprediction.net | Backing off 2 min 13 sec on upload of hadam3p_eu_9qs2_1972_1_007855565_1_2.zip



                                                                                            I just noticed it was sitting in the outbox after upgrading to new version. I thought it was sent out many days ago.

                                                                                            In any case is this error server related or something went wrong on my comp?

                                                                                            Les Bayliss
                                                                                            Forum moderator
                                                                                            Send message
                                                                                            Joined: Sep 5 04
                                                                                            Posts: 5411
                                                                                            Credit: 9,006,998
                                                                                            RAC: 3,097
                                                                                            Message 44105 - Posted 27 Apr 2012 13:12:39 UTC - in response to Message 44104.

                                                                                              It's the server.
                                                                                              Look at the Server Status in blue menu to the left, 5 from the bottom.


                                                                                              ____________
                                                                                              Backups: Here

                                                                                              Profile JIM
                                                                                              Send message
                                                                                              Joined: Dec 31 07
                                                                                              Posts: 679
                                                                                              Credit: 4,128,527
                                                                                              RAC: 2,418
                                                                                              Message 44107 - Posted 27 Apr 2012 18:38:59 UTC

                                                                                                I see that one of the broken servers is now fixed and back up. Good work guys. Is there any word on when uploader1.atm might be fixed? I now have 14 zip files stored in my transfer tab and 2 finished WU's that can't report.

                                                                                                ____________

                                                                                                ojum-le
                                                                                                Send message
                                                                                                Joined: May 5 07
                                                                                                Posts: 26
                                                                                                Credit: 6,315,194
                                                                                                RAC: 0
                                                                                                Message 44108 - Posted 27 Apr 2012 18:43:42 UTC - in response to Message 44107.

                                                                                                  I also hope for solving the problem:

                                                                                                  in que: 48 files aprx. 850 MBytes! 5 EU-Models finished

                                                                                                  Tomas
                                                                                                  Send message
                                                                                                  Joined: Sep 7 05
                                                                                                  Posts: 12
                                                                                                  Credit: 601,646
                                                                                                  RAC: 0
                                                                                                  Message 44109 - Posted 28 Apr 2012 19:27:15 UTC - in response to Message 44075.

                                                                                                    Hi everyone,

                                                                                                    We are currently suffering two server failures - both serious hard disk issues, so I am configuring another to take over their roles before I get around to sorting out those problems.

                                                                                                    I will let you know how things proceed, but it will be at least 24 hours before we can consider ourselves back online.

                                                                                                    Please accept my apologies.

                                                                                                    Jonathan

                                                                                                    CPDN Sys-Admin


                                                                                                    Just wondering. Do you have a backup \ mirror of the uploaded data or is the data lost?
                                                                                                    ____________

                                                                                                    Les Bayliss
                                                                                                    Forum moderator
                                                                                                    Send message
                                                                                                    Joined: Sep 5 04
                                                                                                    Posts: 5411
                                                                                                    Credit: 9,006,998
                                                                                                    RAC: 3,097
                                                                                                    Message 44110 - Posted 28 Apr 2012 20:41:17 UTC - in response to Message 44109.

                                                                                                      The server in question is only for the restart data, zip 13.
                                                                                                      It's a raid system, so it's unlikely that the data is lost.

                                                                                                      As to why it's taking so long to fix, there's no information from the project people, but that server was having intermittent problems for several months recently, and perhaps the entire server needs replacing. Some of the many being used are quite old, and inadequate, resource wise, for the tasks required of them.

                                                                                                      The main server, which also hosts this board, IS mirrored.


                                                                                                      ____________
                                                                                                      Backups: Here

                                                                                                      Eirik Redd
                                                                                                      Send message
                                                                                                      Joined: Aug 31 04
                                                                                                      Posts: 257
                                                                                                      Credit: 27,941,560
                                                                                                      RAC: 13,465
                                                                                                      Message 44119 - Posted 30 Apr 2012 11:36:11 UTC

                                                                                                        Just now noticed that one hadam3p***13.zip is uploading.
                                                                                                        Way to go!
                                                                                                        Thanks to the whole crew!
                                                                                                        ____________

                                                                                                        Eirik Redd
                                                                                                        Send message
                                                                                                        Joined: Aug 31 04
                                                                                                        Posts: 257
                                                                                                        Credit: 27,941,560
                                                                                                        RAC: 13,465
                                                                                                        Message 44120 - Posted 30 Apr 2012 11:49:46 UTC

                                                                                                          But please, everybody, don't hit the "retry now" button.
                                                                                                          The server status pages still show one server down,
                                                                                                          And we don't want to pile on to the new or substituted server.
                                                                                                          Patience is a virtue.
                                                                                                          Take care.
                                                                                                          ____________

                                                                                                          Tomas
                                                                                                          Send message
                                                                                                          Joined: Sep 7 05
                                                                                                          Posts: 12
                                                                                                          Credit: 601,646
                                                                                                          RAC: 0
                                                                                                          Message 44121 - Posted 30 Apr 2012 12:15:09 UTC - in response to Message 44119.

                                                                                                            Just now noticed that one hadam3p***13.zip is uploading.
                                                                                                            Way to go!
                                                                                                            Thanks to the whole crew!


                                                                                                            Hmm.. the server status report "Upload server uploader1.atm Not Running"
                                                                                                            ____________

                                                                                                            Les Bayliss
                                                                                                            Forum moderator
                                                                                                            Send message
                                                                                                            Joined: Sep 5 04
                                                                                                            Posts: 5411
                                                                                                            Credit: 9,006,998
                                                                                                            RAC: 3,097
                                                                                                            Message 44122 - Posted 30 Apr 2012 12:39:21 UTC

                                                                                                              I think that the Server Status page will continue to show that of the "real" server, not the aliased machine.

                                                                                                              The reason that it's taken so long, is that there were problems in setting up a substitute server, and then there was a wait while the Uni IT people applied the alias.


                                                                                                              ____________
                                                                                                              Backups: Here

                                                                                                              ed2353
                                                                                                              Send message
                                                                                                              Joined: Feb 15 06
                                                                                                              Posts: 44
                                                                                                              Credit: 15,365,078
                                                                                                              RAC: 6,325
                                                                                                              Message 44134 - Posted 1 May 2012 23:47:24 UTC

                                                                                                                Although the Uploader is green status, I have about 40 files that are not uploading at all, not even one has started to upload. Is the uploader REALLY working?

                                                                                                                Les Bayliss
                                                                                                                Forum moderator
                                                                                                                Send message
                                                                                                                Joined: Sep 5 04
                                                                                                                Posts: 5411
                                                                                                                Credit: 9,006,998
                                                                                                                RAC: 3,097
                                                                                                                Message 44135 - Posted 2 May 2012 3:34:02 UTC - in response to Message 44134.

                                                                                                                  The uploads are working for me, and have been for a day or so now.

                                                                                                                  Try clicking on one, and then click Retry Now.


                                                                                                                  ____________
                                                                                                                  Backups: Here

                                                                                                                  ed2353
                                                                                                                  Send message
                                                                                                                  Joined: Feb 15 06
                                                                                                                  Posts: 44
                                                                                                                  Credit: 15,365,078
                                                                                                                  RAC: 6,325
                                                                                                                  Message 44138 - Posted 2 May 2012 8:07:58 UTC - in response to Message 44135.

                                                                                                                    I've tried that a couple of times. All I get is the "retry in (hours) Project backoff (minutes)" messages.

                                                                                                                    I miscounted, its actually 80+ hadam3p zip files waiting, mostly pnw, with one eu and 2 saf.
                                                                                                                    They are from 18 completed tasks that are still sitting in my task list, saying uploading.

                                                                                                                    ed2353
                                                                                                                    Send message
                                                                                                                    Joined: Feb 15 06
                                                                                                                    Posts: 44
                                                                                                                    Credit: 15,365,078
                                                                                                                    RAC: 6,325
                                                                                                                    Message 44139 - Posted 2 May 2012 8:14:20 UTC - in response to Message 44138.

                                                                                                                      I forgot to add that in Messages I get "Internet access OK - project servers may be down", hence my original enquiry as to whether they are really working.

                                                                                                                      Profile Dave Jackson
                                                                                                                      Send message
                                                                                                                      Joined: May 15 09
                                                                                                                      Posts: 849
                                                                                                                      Credit: 645,754
                                                                                                                      RAC: 154
                                                                                                                      Message 44140 - Posted 2 May 2012 8:47:11 UTC - in response to Message 44139.

                                                                                                                        Just to check, is there a way to see from the task's page if zips have been transferred, like there is with the trickles which go to a different server and have been going through?

                                                                                                                        ed2353
                                                                                                                        Send message
                                                                                                                        Joined: Feb 15 06
                                                                                                                        Posts: 44
                                                                                                                        Credit: 15,365,078
                                                                                                                        RAC: 6,325
                                                                                                                        Message 44141 - Posted 2 May 2012 9:10:28 UTC - in response to Message 44140.

                                                                                                                          Last modified: 2 May 2012 9:11:05 UTC

                                                                                                                          Uploading suddenly started for me at 0956 BST.
                                                                                                                          I'm relieved that all that work has not been wasted!

                                                                                                                          marpes
                                                                                                                          Send message
                                                                                                                          Joined: Nov 11 04
                                                                                                                          Posts: 8
                                                                                                                          Credit: 15,188,494
                                                                                                                          RAC: 1,298
                                                                                                                          Message 44146 - Posted 4 May 2012 18:16:16 UTC

                                                                                                                            From yesterday I can see:
                                                                                                                            3. 5. 2012 20:31:16 | climateprediction.net | [error] Error reported by file upload server: can't open log file '../log_uploader1/file_upload_handler.log' (errno: 9)
                                                                                                                            3. 5. 2012 20:31:16 | climateprediction.net | Temporarily failed upload of hadam3p_eu_a3mc_1997_1_007861165_2_11.zip: transient upload error
                                                                                                                            .
                                                                                                                            4. 5. 2012 19:11:00 | climateprediction.net | [error] Error reported by file upload server: can't open log file '../log_uploader1/file_upload_handler.log' (errno: 9)
                                                                                                                            4. 5. 2012 19:11:00 | climateprediction.net | Temporarily failed upload of hadam3p_eu_a3mc_1997_1_007861165_2_12.zip: transient upload error
                                                                                                                            .
                                                                                                                            4. 5. 2012 19:53:29 | climateprediction.net | [error] Error reported by file upload server: can't open log file '../log_uploader1/file_upload_handler.log' (errno: 9)
                                                                                                                            4. 5. 2012 19:53:29 | climateprediction.net | Temporarily failed upload of hadam3p_eu_a3mc_1997_1_007861165_2_11.zip: transient upload error

                                                                                                                            On page Server Status is no problem.
                                                                                                                            Where problem is?
                                                                                                                            ____________

                                                                                                                            Les Bayliss
                                                                                                                            Forum moderator
                                                                                                                            Send message
                                                                                                                            Joined: Sep 5 04
                                                                                                                            Posts: 5411
                                                                                                                            Credit: 9,006,998
                                                                                                                            RAC: 3,097
                                                                                                                            Message 44147 - Posted 4 May 2012 20:27:21 UTC - in response to Message 44146.

                                                                                                                              It should just be temporary, (transient), at a time when the server was overloaded with computers wanting to upload files.

                                                                                                                              I've been having no problems.


                                                                                                                              ____________
                                                                                                                              Backups: Here

                                                                                                                              MarkJ
                                                                                                                              Avatar
                                                                                                                              Send message
                                                                                                                              Joined: Mar 28 09
                                                                                                                              Posts: 104
                                                                                                                              Credit: 5,098,940
                                                                                                                              RAC: 691
                                                                                                                              Message 44148 - Posted 5 May 2012 4:01:11 UTC - in response to Message 44147.

                                                                                                                                It should just be temporary, (transient), at a time when the server was overloaded with computers wanting to upload files.

                                                                                                                                I've been having no problems.


                                                                                                                                Nope. I am getting them too on some eu work units for zip files 11 and 12. I think the admins need to move the log file or something. It's probably run out of disk space (with the backlog of uploads that wouldn't be surprising). Hopefully Jonathan or one of the other guys will notice and fix it soon.

                                                                                                                                ____________
                                                                                                                                BOINC blog

                                                                                                                                Les Bayliss
                                                                                                                                Forum moderator
                                                                                                                                Send message
                                                                                                                                Joined: Sep 5 04
                                                                                                                                Posts: 5411
                                                                                                                                Credit: 9,006,998
                                                                                                                                RAC: 3,097
                                                                                                                                Message 44149 - Posted 5 May 2012 4:10:04 UTC - in response to Message 44148.

                                                                                                                                  I've just sent them a note, but it's 5am on Saturday morning.


                                                                                                                                  ____________
                                                                                                                                  Backups: Here

                                                                                                                                  Richard Haselgrove
                                                                                                                                  Send message
                                                                                                                                  Joined: Jan 1 07
                                                                                                                                  Posts: 299
                                                                                                                                  Credit: 6,878,929
                                                                                                                                  RAC: 13
                                                                                                                                  Message 44152 - Posted 5 May 2012 8:42:33 UTC - in response to Message 44148.

                                                                                                                                    It should just be temporary, (transient), at a time when the server was overloaded with computers wanting to upload files.

                                                                                                                                    I've been having no problems.

                                                                                                                                    Nope. I am getting them too on some eu work units for zip files 11 and 12. I think the admins need to move the log file or something. It's probably run out of disk space (with the backlog of uploads that wouldn't be surprising). Hopefully Jonathan or one of the other guys will notice and fix it soon.

                                                                                                                                    If it's either a log file problem or a lack of disk space, the reason for the upload failure would appear in your local BOINC client message/event log (which the admins can't see directly).

                                                                                                                                    Why not help them find the cause of the problem by quoting the error message which you can see?

                                                                                                                                    Nigel Garvey
                                                                                                                                    Send message
                                                                                                                                    Joined: May 5 10
                                                                                                                                    Posts: 34
                                                                                                                                    Credit: 584,860
                                                                                                                                    RAC: 90
                                                                                                                                    Message 44153 - Posted 5 May 2012 9:08:50 UTC - in response to Message 44152.

                                                                                                                                      Why not help them find the cause of the problem by quoting the error message which you can see?


                                                                                                                                      Hi.

                                                                                                                                      I'm getting the same thing with files 10 to 13 of a HADAM3P EU. The error's: "[error] Error reported by upload server: can't open log file '../log_uploader1/file_upload_handler.log' (errno: 9)".


                                                                                                                                      NG

                                                                                                                                      Alex Miller
                                                                                                                                      Send message
                                                                                                                                      Joined: Feb 18 09
                                                                                                                                      Posts: 4
                                                                                                                                      Credit: 97,447
                                                                                                                                      RAC: 0
                                                                                                                                      Message 44162 - Posted 6 May 2012 9:12:51 UTC - in response to Message 44153.

                                                                                                                                        Last modified: 6 May 2012 9:13:31 UTC

                                                                                                                                        I'm getting the same thing with files 10 to 13 of a HADAM3P EU. The error's: "[error] Error reported by upload server: can't open log file '../log_uploader1/file_upload_handler.log' (errno: 9)".



                                                                                                                                        Same problem for me too.

                                                                                                                                        Profile Dave Jackson
                                                                                                                                        Send message
                                                                                                                                        Joined: May 15 09
                                                                                                                                        Posts: 849
                                                                                                                                        Credit: 645,754
                                                                                                                                        RAC: 154
                                                                                                                                        Message 44171 - Posted 7 May 2012 8:31:39 UTC - in response to Message 44162.

                                                                                                                                          A server always seems to go down on bank holiday weekend. Is rain getting into the system?

                                                                                                                                          Profile [AF>Le_Pommier] Jerome_C2005
                                                                                                                                          Send message
                                                                                                                                          Joined: Oct 21 10
                                                                                                                                          Posts: 28
                                                                                                                                          Credit: 819,733
                                                                                                                                          RAC: 657
                                                                                                                                          Message 44172 - Posted 7 May 2012 17:51:01 UTC

                                                                                                                                            I have a http error while trying to upload a zip file + one of the upload servers is down (http://climateapps2.oerc.ox.ac.uk/cpdnboinc/server_status.html) but there are others online...

                                                                                                                                            Last time I tried to upload was 1 or 2 weeks ago (I crunch on an offline machine and move WU with USB key on a windows VM into my home Mac when they are terminated, and request new work from the VM), and it was down too... I have now 4 WU waiting for upload... the good thing is that the deadline is faaaaar away ;)

                                                                                                                                            Eirik Redd
                                                                                                                                            Send message
                                                                                                                                            Joined: Aug 31 04
                                                                                                                                            Posts: 257
                                                                                                                                            Credit: 27,941,560
                                                                                                                                            RAC: 13,465
                                                                                                                                            Message 44173 - Posted 7 May 2012 20:05:25 UTC

                                                                                                                                              Last modified: 7 May 2012 20:14:20 UTC

                                                                                                                                              Seems that the problem is only with the eu uploads -- pnw uploads work here, eu don't even the _01 uploads fail.
                                                                                                                                              The one server is reported as down.

                                                                                                                                              @Dave -- yeah Murphy's law says that all system failures will happen at the worst time.

                                                                                                                                              And on a re-assuring note -- in the last 8 years crunching for climateprediction -- the staff has always fixed problems without losing data. Usually in a day or 2, once it took almost 2 weeks when they had a serious hardware failure. They got mirrors they got logs they got backups.
                                                                                                                                              I've learned to trust their backups. Wish my own home backup system was as good.

                                                                                                                                              Please keep posting any upload problems here.

                                                                                                                                              And keep on crunching.

                                                                                                                                              Eric
                                                                                                                                              ____________

                                                                                                                                              Profile [AF>Le_Pommier] Jerome_C2005
                                                                                                                                              Send message
                                                                                                                                              Joined: Oct 21 10
                                                                                                                                              Posts: 28
                                                                                                                                              Credit: 819,733
                                                                                                                                              RAC: 657
                                                                                                                                              Message 44174 - Posted 7 May 2012 20:31:06 UTC

                                                                                                                                                As I said, since there is such a long deadline it's not a real problem, only a pain :D

                                                                                                                                                Duzzie
                                                                                                                                                Send message
                                                                                                                                                Joined: Aug 31 04
                                                                                                                                                Posts: 1
                                                                                                                                                Credit: 1,062,940
                                                                                                                                                RAC: 345
                                                                                                                                                Message 44176 - Posted 14 May 2012 9:15:06 UTC

                                                                                                                                                  Is there any update on this. I have a few EU uploads waiting and I have WUs still working but I don't want to waste time crunching the WUs if the issue isn't going to be resolved and the data lost.
                                                                                                                                                  ____________

                                                                                                                                                  Profile Dave Jackson
                                                                                                                                                  Send message
                                                                                                                                                  Joined: May 15 09
                                                                                                                                                  Posts: 849
                                                                                                                                                  Credit: 645,754
                                                                                                                                                  RAC: 154
                                                                                                                                                  Message 44177 - Posted 14 May 2012 10:22:11 UTC - in response to Message 44176.

                                                                                                                                                    Jonathan and the team will get this sorted, I know it has been a bit longer than usual on this occasion - maybe he is on holiday or something. In the years I have been with the project these problems have always been resolved without data loss - all the data is stored on a mirrored raid array and regularly backed up externally as well. Hopefully one day the project will get the money it deserves which would allow newer better hardware and more staff to look after it.

                                                                                                                                                    The only advice I can offer is a phrase I heard regularly when in the forces, "Hurry up and wait!"

                                                                                                                                                    Dave

                                                                                                                                                    MarkJ
                                                                                                                                                    Avatar
                                                                                                                                                    Send message
                                                                                                                                                    Joined: Mar 28 09
                                                                                                                                                    Posts: 104
                                                                                                                                                    Credit: 5,098,940
                                                                                                                                                    RAC: 691
                                                                                                                                                    Message 44178 - Posted 15 May 2012 11:56:15 UTC

                                                                                                                                                      uploader1.atm seems to back on-line and one of my eu work units thats been trying to upload for a week has finally gone through. A big thank you to the guys for fixing it up.
                                                                                                                                                      ____________
                                                                                                                                                      BOINC blog

                                                                                                                                                      Profile Dave Jackson
                                                                                                                                                      Send message
                                                                                                                                                      Joined: May 15 09
                                                                                                                                                      Posts: 849
                                                                                                                                                      Credit: 645,754
                                                                                                                                                      RAC: 154
                                                                                                                                                      Message 44179 - Posted 15 May 2012 13:58:20 UTC - in response to Message 44178.

                                                                                                                                                        Good to see the server back on line, have I been unobservant or is srv1.cpdn.psu.edu new in the past week or so? I must get back in the habit of backing up my work units - just lost a couple due to power outage. "The chances of a power of disk failure is proportional to at least the square of the time since the last backup."

                                                                                                                                                        Profile Thyme Lawn
                                                                                                                                                        Forum moderator
                                                                                                                                                        Send message
                                                                                                                                                        Joined: Aug 5 04
                                                                                                                                                        Posts: 1232
                                                                                                                                                        Credit: 10,409,271
                                                                                                                                                        RAC: 1,138
                                                                                                                                                        Message 44184 - Posted 17 May 2012 8:31:59 UTC - in response to Message 44179.

                                                                                                                                                          The project team believe they have resolved all of the upload server issues. If anyone is still having problems uploading please let us know.

                                                                                                                                                          have I been unobservant or is srv1.cpdn.psu.edu new in the past week or so?

                                                                                                                                                          That's an upload server which hasn't been used for some time Dave. I don't pay much attention to the green on the server status page, but it may well have been returned to active service.
                                                                                                                                                          ____________
                                                                                                                                                          "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

                                                                                                                                                          Profile [AF>Le_Pommier] Jerome_C2005
                                                                                                                                                          Send message
                                                                                                                                                          Joined: Oct 21 10
                                                                                                                                                          Posts: 28
                                                                                                                                                          Credit: 819,733
                                                                                                                                                          RAC: 657
                                                                                                                                                          Message 44202 - Posted 18 May 2012 20:12:45 UTC

                                                                                                                                                            Wow, I can upload !

                                                                                                                                                            Good news, it's been so long I wasn't able to do it that I have almost 800 MB to upload now, it's going to take a while with my DSL 110 KBPS upload...

                                                                                                                                                            Thanks for fixing this !

                                                                                                                                                            dCw2trX3xi
                                                                                                                                                            Send message
                                                                                                                                                            Joined: Sep 14 10
                                                                                                                                                            Posts: 11
                                                                                                                                                            Credit: 1,812,972
                                                                                                                                                            RAC: 0
                                                                                                                                                            Message 44205 - Posted 20 May 2012 1:44:32 UTC - in response to Message 44202.

                                                                                                                                                              Last modified: 20 May 2012 1:47:38 UTC

                                                                                                                                                              It's still broken here. I get logs full of:

                                                                                                                                                              [file_xfer_debug] URL: http://cpdn-restarts.oerc.ox.ac.uk/cgi-bin/file_upload_handler
                                                                                                                                                              [file_xfer_debug] FILE_XFER_SET::poll(): http op done, retval -107
                                                                                                                                                              [file_xfer_debug] file transfer status -107

                                                                                                                                                              when attempting to upload _13 hada* results.

                                                                                                                                                              I can ping http://cpdn-restarts.oerc.ox.ac.uk successfully from the machines running cpdn.

                                                                                                                                                              Les Bayliss
                                                                                                                                                              Forum moderator
                                                                                                                                                              Send message
                                                                                                                                                              Joined: Sep 5 04
                                                                                                                                                              Posts: 5411
                                                                                                                                                              Credit: 9,006,998
                                                                                                                                                              RAC: 3,097
                                                                                                                                                              Message 44206 - Posted 20 May 2012 2:32:12 UTC - in response to Message 44205.

                                                                                                                                                                Last modified: 20 May 2012 2:33:08 UTC

                                                                                                                                                                Welcome to the Twilight Zone.
                                                                                                                                                                Don't adjust your horizontal, your vertical, or your mind, just some spellings on your computer.

                                                                                                                                                                There's a problem with some flavours of Linux that cause a character string to become corrupted when it's stored on the computer in question.

                                                                                                                                                                This has been discussed in several threads, (probably under several Topics), on our php board. This is one thread.

                                                                                                                                                                The cure is to **carefully** edit client_state.xml, and correct the corruption. One spelling is hnndler, but there are others.

                                                                                                                                                                Do a search on the 4 character model name, then check each line until you find the upload section for zip 13, then look at the spelling.
                                                                                                                                                                ____________
                                                                                                                                                                Backups: Here

                                                                                                                                                                Eirik Redd
                                                                                                                                                                Send message
                                                                                                                                                                Joined: Aug 31 04
                                                                                                                                                                Posts: 257
                                                                                                                                                                Credit: 27,941,560
                                                                                                                                                                RAC: 13,465
                                                                                                                                                                Message 44207 - Posted 20 May 2012 6:35:23 UTC

                                                                                                                                                                  Oh dear == thought this problem had gone away -
                                                                                                                                                                  follow the advice on the thread Les referred to
                                                                                                                                                                  And please follow this advice about stopping BOINC and the CPDN models and doing a backup before editing the client-state.xml
                                                                                                                                                                  If you follow the procedure to correct the misspellings -uploads will start working.

                                                                                                                                                                  Keep on keeping on.


                                                                                                                                                                  ____________

                                                                                                                                                                  dCw2trX3xi
                                                                                                                                                                  Send message
                                                                                                                                                                  Joined: Sep 14 10
                                                                                                                                                                  Posts: 11
                                                                                                                                                                  Credit: 1,812,972
                                                                                                                                                                  RAC: 0
                                                                                                                                                                  Message 44208 - Posted 21 May 2012 17:43:57 UTC - in response to Message 44207.

                                                                                                                                                                    The upload handler string does not appear to be mis-spelled on the problem systems. It reads "http://cpdn-restarts.oerc.ox.ac.uk/cgi-bin/file_upload_handler". The problem is that the upload shouldn't go to "cpdn-restarts.oerc..." at all. It should go to "uploader.oerc...". I read this at http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=5447&nowrap=true#44076 , fixed the client-state.xml files, and everything's uploading fine now.

                                                                                                                                                                    Please update the server-status page to indicate which files go where, so that users can fix this problem more easily.

                                                                                                                                                                    Profile Dave Jackson
                                                                                                                                                                    Send message
                                                                                                                                                                    Joined: May 15 09
                                                                                                                                                                    Posts: 849
                                                                                                                                                                    Credit: 645,754
                                                                                                                                                                    RAC: 154
                                                                                                                                                                    Message 44209 - Posted 21 May 2012 19:53:01 UTC - in response to Message 44208.

                                                                                                                                                                      Please update the server-status page to indicate which files go where, so that users can fix this problem more easily.

                                                                                                                                                                      What you say makes perfect sense to me. However I am sure that there are more than a few who would make a total hash of this if the information was that easily available. Making people ask questions here probably weeds out some of those who would produce the proverbial dogs dinner.

                                                                                                                                                                      I also wonder why it was going to the wrong server. Is this to do with the fixing of the server problems recently, in particular the interim fix before it was properly fixed? If so, problems due to that source have in the past been fixed at the CPDN end and the uploads have eventually gone through.

                                                                                                                                                                      Dave

                                                                                                                                                                      Profile Dave Jackson
                                                                                                                                                                      Send message
                                                                                                                                                                      Joined: May 15 09
                                                                                                                                                                      Posts: 849
                                                                                                                                                                      Credit: 645,754
                                                                                                                                                                      RAC: 154
                                                                                                                                                                      Message 44210 - Posted 21 May 2012 19:57:24 UTC - in response to Message 44209.

                                                                                                                                                                        And despite what I just said, it would be nice to see what was meant to go where, then if a server was down we would be able to see whether or not that was the cause of our problems as opposed to just assuming.

                                                                                                                                                                        Dave

                                                                                                                                                                        Les Bayliss
                                                                                                                                                                        Forum moderator
                                                                                                                                                                        Send message
                                                                                                                                                                        Joined: Sep 5 04
                                                                                                                                                                        Posts: 5411
                                                                                                                                                                        Credit: 9,006,998
                                                                                                                                                                        RAC: 3,097
                                                                                                                                                                        Message 44211 - Posted 21 May 2012 22:23:40 UTC

                                                                                                                                                                          The mods do have a list, but it's a bit dynamic.

                                                                                                                                                                          The real problem, is that during the recent server failure, re-directs were put in place for all servers, so that in the event of future problems, the data could be quickly sent to another server somewhere else in Oxford.
                                                                                                                                                                          It appears that this needs to be checked again, against the current working server list.

                                                                                                                                                                          The upload server being used for every upload is listed in client_state.xml, where it's used by BOINC to address it's mail.

                                                                                                                                                                          The labels on the servers is (mostly) the physical location of the server, which makes it easier to know where to go where the server needs a physical re-boot, as against a ssh wakeup.

                                                                                                                                                                          I've just informed the project, so it'll get looked at in half a day. But there may be more data sets that are affected, in which case, it shouldn't be a problem for too long.


                                                                                                                                                                          ____________
                                                                                                                                                                          Backups: Here

                                                                                                                                                                          dCw2trX3xi
                                                                                                                                                                          Send message
                                                                                                                                                                          Joined: Sep 14 10
                                                                                                                                                                          Posts: 11
                                                                                                                                                                          Credit: 1,812,972
                                                                                                                                                                          RAC: 0
                                                                                                                                                                          Message 44212 - Posted 21 May 2012 23:01:22 UTC - in response to Message 44211.

                                                                                                                                                                            I've just informed the project, so it'll get looked at in half a day. But there may be more data sets that are affected, in which case, it shouldn't be a problem for too long.

                                                                                                                                                                            Thanks. It does seem very likely that others are also experiencing this problem, perhaps without noticing it.

                                                                                                                                                                            Snow Crash
                                                                                                                                                                            Send message
                                                                                                                                                                            Joined: Dec 19 09
                                                                                                                                                                            Posts: 4
                                                                                                                                                                            Credit: 256,301
                                                                                                                                                                            RAC: 0
                                                                                                                                                                            Message 44237 - Posted 28 May 2012 8:49:47 UTC

                                                                                                                                                                              5/28/2012 4:47:56 AM | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_azhc_1967_1_007885059_1_13.zip: No such file or directory

                                                                                                                                                                              Les Bayliss
                                                                                                                                                                              Forum moderator
                                                                                                                                                                              Send message
                                                                                                                                                                              Joined: Sep 5 04
                                                                                                                                                                              Posts: 5411
                                                                                                                                                                              Credit: 9,006,998
                                                                                                                                                                              RAC: 3,097
                                                                                                                                                                              Message 44238 - Posted 28 May 2012 8:52:23 UTC - in response to Message 44237.

                                                                                                                                                                                Last modified: 28 May 2012 8:58:27 UTC

                                                                                                                                                                                It's a power failure in one of the buildings.

                                                                                                                                                                                edit
                                                                                                                                                                                Which now seems to be fixed. 2 of my zip 13s are currently uploading.
                                                                                                                                                                                ____________
                                                                                                                                                                                Backups: Here

                                                                                                                                                                                Snow Crash
                                                                                                                                                                                Send message
                                                                                                                                                                                Joined: Dec 19 09
                                                                                                                                                                                Posts: 4
                                                                                                                                                                                Credit: 256,301
                                                                                                                                                                                RAC: 0
                                                                                                                                                                                Message 44246 - Posted 28 May 2012 13:30:20 UTC - in response to Message 44238.

                                                                                                                                                                                  sure they upload ... the client says so but once it gets to 100% I still get the same error.

                                                                                                                                                                                  Profile Dave Jackson
                                                                                                                                                                                  Send message
                                                                                                                                                                                  Joined: May 15 09
                                                                                                                                                                                  Posts: 849
                                                                                                                                                                                  Credit: 645,754
                                                                                                                                                                                  RAC: 154
                                                                                                                                                                                  Message 44247 - Posted 28 May 2012 13:38:43 UTC - in response to Message 44246.

                                                                                                                                                                                    Servers keep going up and down. My guess is they are rebuilding the raid array if it went off during a write but that is no more than a guess.

                                                                                                                                                                                    Dave

                                                                                                                                                                                    Profile Dave Jackson
                                                                                                                                                                                    Send message
                                                                                                                                                                                    Joined: May 15 09
                                                                                                                                                                                    Posts: 849
                                                                                                                                                                                    Credit: 645,754
                                                                                                                                                                                    RAC: 154
                                                                                                                                                                                    Message 44248 - Posted 28 May 2012 13:49:09 UTC - in response to Message 44247.

                                                                                                                                                                                      Zip1 just gone through without an error message afterwards, waiting to see about the 13s

                                                                                                                                                                                      Profile ex_brit
                                                                                                                                                                                      Avatar
                                                                                                                                                                                      Send message
                                                                                                                                                                                      Joined: Aug 26 04
                                                                                                                                                                                      Posts: 84
                                                                                                                                                                                      Credit: 344,628
                                                                                                                                                                                      RAC: 78
                                                                                                                                                                                      Message 44249 - Posted 28 May 2012 18:07:37 UTC

                                                                                                                                                                                        Last modified: 28 May 2012 18:07:58 UTC

                                                                                                                                                                                        Well that cpdnupload2.oerc server keeps going up and down like a yo-yo so my WU's never complete the upload.

                                                                                                                                                                                        ChinookFoehn
                                                                                                                                                                                        Send message
                                                                                                                                                                                        Joined: Aug 7 04
                                                                                                                                                                                        Posts: 83
                                                                                                                                                                                        Credit: 410,895
                                                                                                                                                                                        RAC: 0
                                                                                                                                                                                        Message 44250 - Posted 28 May 2012 19:00:35 UTC

                                                                                                                                                                                          All upload servers are shown to be operating yet I still can not complete the 13.zip upload of the PNW unit. It keeps reaching 100% and then fails.

                                                                                                                                                                                          -ChinookFöhn

                                                                                                                                                                                          Profile Dave Jackson
                                                                                                                                                                                          Send message
                                                                                                                                                                                          Joined: May 15 09
                                                                                                                                                                                          Posts: 849
                                                                                                                                                                                          Credit: 645,754
                                                                                                                                                                                          RAC: 154
                                                                                                                                                                                          Message 44251 - Posted 28 May 2012 19:09:53 UTC - in response to Message 44250.

                                                                                                                                                                                            As Les Helpfully reminded me in this thread http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=7378

                                                                                                                                                                                            zip13s go to the restart server to generate the next data set. Following the power outage in the server room, the server has been going on and off a lot today and I would guess that is something to do with the work they are doing to restore it to full functionality after the power outage. In the last 10 minutes it has gone from down to up again on the server status page but I think I will keep network activity suspended till tomorrow and try again then.

                                                                                                                                                                                            Dave

                                                                                                                                                                                            Snow Crash
                                                                                                                                                                                            Send message
                                                                                                                                                                                            Joined: Dec 19 09
                                                                                                                                                                                            Posts: 4
                                                                                                                                                                                            Credit: 256,301
                                                                                                                                                                                            RAC: 0
                                                                                                                                                                                            Message 44254 - Posted 29 May 2012 8:54:49 UTC

                                                                                                                                                                                              Every time I check the server status page it shows all servers "running" and I do not see a server named "restart" and my final upload is not being accepted.

                                                                                                                                                                                              Would be nice if they turned the uploads off entirely so I don't have to continue reuploading the file only to have it fail in the end.

                                                                                                                                                                                              Does the project staff typically post when these types of problems occur / get correted?

                                                                                                                                                                                              Les Bayliss
                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                              Send message
                                                                                                                                                                                              Joined: Sep 5 04
                                                                                                                                                                                              Posts: 5411
                                                                                                                                                                                              Credit: 9,006,998
                                                                                                                                                                                              RAC: 3,097
                                                                                                                                                                                              Message 44256 - Posted 29 May 2012 9:27:27 UTC - in response to Message 44254.

                                                                                                                                                                                                This problem was being discussed in the adjacent thread, had3pam_eu models not uploading, before you started to post.
                                                                                                                                                                                                Look there for news.

                                                                                                                                                                                                Sometimes one of the two project people post, but that's in the News and Announcements thread.
                                                                                                                                                                                                Most of the time they leave it to the moderators to pass on news. In both directions.

                                                                                                                                                                                                You'll only see the cpdn_restarts server name mentioned in the client_state.xml file, because all of the servers have aliases to make it simpler and faster to redirect data to an alternative server when one fails for any reason.


                                                                                                                                                                                                ____________
                                                                                                                                                                                                Backups: Here

                                                                                                                                                                                                Profile Dave Jackson
                                                                                                                                                                                                Send message
                                                                                                                                                                                                Joined: May 15 09
                                                                                                                                                                                                Posts: 849
                                                                                                                                                                                                Credit: 645,754
                                                                                                                                                                                                RAC: 154
                                                                                                                                                                                                Message 44257 - Posted 29 May 2012 9:37:18 UTC - in response to Message 44254.

                                                                                                                                                                                                  Last modified: 29 May 2012 9:39:37 UTC

                                                                                                                                                                                                  My guess is that turning the uploads off completely might interfere with something else or it may just be that under resourced as the project is they have more important things to do such as working on resolving the problem. If you look in your clientstate.xml file you will see entries such as these which give hints as to which server is being used.

                                                                                                                                                                                                  <file>
                                                                                                                                                                                                  <name>hadam3p_pnw_c3rx_1988_1_007937852_0_13.zip</name>
                                                                                                                                                                                                  <nbytes>0.000000</nbytes>
                                                                                                                                                                                                  <max_nbytes>150000000.000000</max_nbytes>
                                                                                                                                                                                                  <status>0</status>
                                                                                                                                                                                                  <upload_url>http://cpdn-restarts.oerc.ox.ac.uk/cgi-bin/file_upload_handler</upload_url>
                                                                                                                                                                                                  </file>
                                                                                                                                                                                                  Which I think refres uploader.oerc but it could be upload2.oerc.

                                                                                                                                                                                                  It was suggested in another thread that it may be a drive mounting problem or a daemon problem causing the uploads to fail @ 100%. Best thing is to turn off network access for BOINC till things are resolved unless you are running multiple projects in which case there is a way to do it but as I only run CPDN I have never learned it.

                                                                                                                                                                                                  Beat me to it Les, feel free to contradict anything I may have got wrong on the technical side.

                                                                                                                                                                                                  Snow Crash
                                                                                                                                                                                                  Send message
                                                                                                                                                                                                  Joined: Dec 19 09
                                                                                                                                                                                                  Posts: 4
                                                                                                                                                                                                  Credit: 256,301
                                                                                                                                                                                                  RAC: 0
                                                                                                                                                                                                  Message 44262 - Posted 29 May 2012 17:07:21 UTC - in response to Message 44256.

                                                                                                                                                                                                    This problem was being discussed in the adjacent thread, had3pam_eu models not uploading, before you started to post.
                                                                                                                                                                                                    Look there for news.

                                                                                                                                                                                                    Sometimes one of the two project people post, but that's in the News and Announcements thread.
                                                                                                                                                                                                    Most of the time they leave it to the moderators to pass on news. In both directions.

                                                                                                                                                                                                    You'll only see the cpdn_restarts server name mentioned in the client_state.xml file, because all of the servers have aliases to make it simpler and faster to redirect data to an alternative server when one fails for any reason.


                                                                                                                                                                                                    I get that you are busy and covering lots of ground but your *tone kind of sounds like I'm at fault for not reading a thread about EU models when I'm taling about finals for PNW. Perhaps redirecting me to that thread first would have worked better for all. Guess it really doesn't matter much as there are still reports saying it's still not working.

                                                                                                                                                                                                    While I understand short budgets and time constraints as an IT professional myself, this has turned into just one more typical alientating BOINC forum experience ... love the science, frustrated with the communication.

                                                                                                                                                                                                    Profile Iain Inglis
                                                                                                                                                                                                    Forum moderator
                                                                                                                                                                                                    Send message
                                                                                                                                                                                                    Joined: Jan 16 10
                                                                                                                                                                                                    Posts: 511
                                                                                                                                                                                                    Credit: 9,532
                                                                                                                                                                                                    RAC: 0
                                                                                                                                                                                                    Message 44263 - Posted 29 May 2012 17:49:12 UTC - in response to Message 44262.

                                                                                                                                                                                                      ... While I understand short budgets and time constraints as an IT professional myself ...

                                                                                                                                                                                                      The moderators are volunteers and have no budgets; their time is as constrained as their real life and motivation allow.

                                                                                                                                                                                                      On the other (PHP) board there was recently an exchange in which a user posted a question and got what seemed to me to be a very complete answer from another user, which provided not only the direct information required but also the context in which to interpret the answer. The first user then posted back to complain about "long-winded waffle". Provide a direct answer, as Les did here, and that will offend someone else for being abrupt. Essentially, you can't win.

                                                                                                                                                                                                      If your complaint is about the lack of communication from project staff, then I'm with you ++ - but please give other volunteers a break.

                                                                                                                                                                                                      Les Bayliss
                                                                                                                                                                                                      Forum moderator
                                                                                                                                                                                                      Send message
                                                                                                                                                                                                      Joined: Sep 5 04
                                                                                                                                                                                                      Posts: 5411
                                                                                                                                                                                                      Credit: 9,006,998
                                                                                                                                                                                                      RAC: 3,097
                                                                                                                                                                                                      Message 44267 - Posted 29 May 2012 22:20:30 UTC - in response to Message 44262.

                                                                                                                                                                                                        Both time and health constraints. It was getting cold, and past time to go to bed where it's warmer.

                                                                                                                                                                                                        People discuss matters in lots of threads whose title doesn't always reflect a current problem. It's always a good idea to look at the most recent thread or two when their time/date stamp shows that they're recent.

                                                                                                                                                                                                        As you don't seem familiar with this board, perhaps I should say:

                                                                                                                                                                                                        Warning

                                                                                                                                                                                                        Essential maintenance on our storage infrastructure will require our main uploads servers to go out of service at the following time:

                                                                                                                                                                                                        Wed 30 May 2012; 8:30-10:30 GMT

                                                                                                                                                                                                        The following servers will be affected

                                                                                                                                                                                                        cpdn-uploads2.oerc.ox.ac.uk
                                                                                                                                                                                                        uploader.oerc.ox.ac.uk
                                                                                                                                                                                                        cpdn-restarts.oerc.ox.ac.uk
                                                                                                                                                                                                        climateapps1.oerc.ox.ac.uk

                                                                                                                                                                                                        I anticipate that the service will be resumed within the allocated time period, and that downloads deferred during this period will catch up over the next few hours.

                                                                                                                                                                                                        If you experience any problems after this scheduled downtime, please let us know by posting on the boards.

                                                                                                                                                                                                        Jonathan Miller


                                                                                                                                                                                                        This message was posted to the following places:
                                                                                                                                                                                                        News and Announcement] (at the top of the Number crunching section)
                                                                                                                                                                                                        News and Announcements (near the top of our alternative PHP board)
                                                                                                                                                                                                        News (linked from the front page of this project.)

                                                                                                                                                                                                        The 1st of these can be subscribed to, and you'll get an email every time something is posted there.
                                                                                                                                                                                                        BUT you MUST have email messages activated.

                                                                                                                                                                                                        The 2nd location, on the PHP bard, requires a separate login, as it's the original board from back before BOINC started, and has nothing to do with this board. Any of the threads there can also be subscribed to.

                                                                                                                                                                                                        The 3rd location has an RSS feed.


                                                                                                                                                                                                        ____________
                                                                                                                                                                                                        Backups: Here

                                                                                                                                                                                                        Profile Dave Jackson
                                                                                                                                                                                                        Send message
                                                                                                                                                                                                        Joined: May 15 09
                                                                                                                                                                                                        Posts: 849
                                                                                                                                                                                                        Credit: 645,754
                                                                                                                                                                                                        RAC: 154
                                                                                                                                                                                                        Message 44268 - Posted 30 May 2012 5:59:06 UTC - in response to Message 44267.

                                                                                                                                                                                                          Good to see that my zip13s have now gone through.

                                                                                                                                                                                                          Profile tullio
                                                                                                                                                                                                          Send message
                                                                                                                                                                                                          Joined: Aug 6 04
                                                                                                                                                                                                          Posts: 183
                                                                                                                                                                                                          Credit: 306,165
                                                                                                                                                                                                          RAC: 38
                                                                                                                                                                                                          Message 44270 - Posted 30 May 2012 12:29:03 UTC

                                                                                                                                                                                                            My uploads are OK but on my task page I see only this:
                                                                                                                                                                                                            Task 14708840

                                                                                                                                                                                                            The page is blank
                                                                                                                                                                                                            Tullio

                                                                                                                                                                                                            ____________

                                                                                                                                                                                                            Les Bayliss
                                                                                                                                                                                                            Forum moderator
                                                                                                                                                                                                            Send message
                                                                                                                                                                                                            Joined: Sep 5 04
                                                                                                                                                                                                            Posts: 5411
                                                                                                                                                                                                            Credit: 9,006,998
                                                                                                                                                                                                            RAC: 3,097
                                                                                                                                                                                                            Message 44272 - Posted 30 May 2012 15:02:27 UTC - in response to Message 44270.

                                                                                                                                                                                                              The page isn't blank. It's just the server taking forever to download the page info.
                                                                                                                                                                                                              This is caused by a problem with the graphics image which is supposed to be on the page, but isn't, and a program is waiting for this to be returned from 'somewhere'.
                                                                                                                                                                                                              Known problem.

                                                                                                                                                                                                              I guess the server is also still getting hammered by the after effects of the recent power problem.


                                                                                                                                                                                                              ____________
                                                                                                                                                                                                              Backups: Here

                                                                                                                                                                                                              Profile Thyme Lawn
                                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                                              Send message
                                                                                                                                                                                                              Joined: Aug 5 04
                                                                                                                                                                                                              Posts: 1232
                                                                                                                                                                                                              Credit: 10,409,271
                                                                                                                                                                                                              RAC: 1,138
                                                                                                                                                                                                              Message 44273 - Posted 30 May 2012 15:29:37 UTC - in response to Message 44270.

                                                                                                                                                                                                                That's a slightly different case of the HadAM3P regional model page load problem Tullio. Normally the load stalls after displaying the perturbed parameters but your task is doing it after the header. The page does eventually finish loading (it took nearly 20 minutes for me). I'm going to raise this with the project team again.

                                                                                                                                                                                                                The task's trickle page opens without any problems.
                                                                                                                                                                                                                ____________
                                                                                                                                                                                                                "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

                                                                                                                                                                                                                Profile tullio
                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                Joined: Aug 6 04
                                                                                                                                                                                                                Posts: 183
                                                                                                                                                                                                                Credit: 306,165
                                                                                                                                                                                                                RAC: 38
                                                                                                                                                                                                                Message 44274 - Posted 30 May 2012 18:33:54 UTC

                                                                                                                                                                                                                  Thanks, Les and Thyme. Anyway, the task is running OK.
                                                                                                                                                                                                                  Tullio
                                                                                                                                                                                                                  ____________

                                                                                                                                                                                                                  Eirik Redd
                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                  Joined: Aug 31 04
                                                                                                                                                                                                                  Posts: 257
                                                                                                                                                                                                                  Credit: 27,941,560
                                                                                                                                                                                                                  RAC: 13,465
                                                                                                                                                                                                                  Message 44283 - Posted 2 Jun 2012 8:11:17 UTC

                                                                                                                                                                                                                    On server-status page uploader1.atm appears to be down.
                                                                                                                                                                                                                    No upload problems reported by clients here yet.

                                                                                                                                                                                                                    E
                                                                                                                                                                                                                    ____________

                                                                                                                                                                                                                    BarryAZ
                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                    Joined: Jul 13 05
                                                                                                                                                                                                                    Posts: 121
                                                                                                                                                                                                                    Credit: 11,299,434
                                                                                                                                                                                                                    RAC: 6
                                                                                                                                                                                                                    Message 44284 - Posted 2 Jun 2012 13:26:24 UTC - in response to Message 44283.

                                                                                                                                                                                                                      Upload problems here -- so you can now note that upload problems by clients have been reported -- since late Friday night here (west coast US time)

                                                                                                                                                                                                                      Profile Dave Jackson
                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                      Joined: May 15 09
                                                                                                                                                                                                                      Posts: 849
                                                                                                                                                                                                                      Credit: 645,754
                                                                                                                                                                                                                      RAC: 154
                                                                                                                                                                                                                      Message 44285 - Posted 2 Jun 2012 15:13:52 UTC - in response to Message 44284.

                                                                                                                                                                                                                        Is that with the full resolution ocean models? Both pnw and eu regional models seem to be uploading zips fine here but I don't have any of the HADAMCM3N models running at the moment. Presumably they go to a different server.


                                                                                                                                                                                                                        Dave

                                                                                                                                                                                                                        BarryAZ
                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                        Joined: Jul 13 05
                                                                                                                                                                                                                        Posts: 121
                                                                                                                                                                                                                        Credit: 11,299,434
                                                                                                                                                                                                                        RAC: 6
                                                                                                                                                                                                                        Message 44290 - Posted 4 Jun 2012 4:09:43 UTC - in response to Message 44285.

                                                                                                                                                                                                                          Yup - it is the hadam3p == seems, over the years that the server used to support the hadam application uploads/downloads is the most fragile of the servers. Other uploads have not had problems this week for me.

                                                                                                                                                                                                                          Perhaps we should redirect by avoiding that set (and server) down the road to reduce the load on that server...

                                                                                                                                                                                                                          Les Bayliss
                                                                                                                                                                                                                          Forum moderator
                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                          Joined: Sep 5 04
                                                                                                                                                                                                                          Posts: 5411
                                                                                                                                                                                                                          Credit: 9,006,998
                                                                                                                                                                                                                          RAC: 3,097
                                                                                                                                                                                                                          Message 44291 - Posted 4 Jun 2012 5:49:10 UTC - in response to Message 44290.

                                                                                                                                                                                                                            The servers have changed over the years, and aliases were put in place a couple of months ago to make it easier to redirect data in the event of a server problem.
                                                                                                                                                                                                                            I suspect that the current problem is that one of the servers has had problems since the recent planed outage a week ago.

                                                                                                                                                                                                                            And it's currently very early on the 3rd of a 4 day weekend in the UK.

                                                                                                                                                                                                                            But all PNW's are uploading OK. I haven't yet started on the current batch of EU's, so I don't know about them. I'll get one and see what happens in ~12 hours.


                                                                                                                                                                                                                            ____________
                                                                                                                                                                                                                            Backups: Here

                                                                                                                                                                                                                            Profile astroWX
                                                                                                                                                                                                                            Forum moderator
                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                            Joined: Aug 5 04
                                                                                                                                                                                                                            Posts: 1302
                                                                                                                                                                                                                            Credit: 38,269,014
                                                                                                                                                                                                                            RAC: 10,868
                                                                                                                                                                                                                            Message 44292 - Posted 4 Jun 2012 20:10:33 UTC

                                                                                                                                                                                                                              There was an upload issue with the U.Oregon (USA) server, which receives the first twelve PNW .zip files, a few hours ago but is okay now. (Penn State U. [also in USA] is in the server mix but I forget for which project[s]. We've always dealt with scattered servers --> scattered across U.Oxford and across the globe. Complicated.)
                                                                                                                                                                                                                              ____________
                                                                                                                                                                                                                              "We have met the enemy and he is us." -- Pogo
                                                                                                                                                                                                                              Greetings from coastal Washington state, the scenic US Pacific Northwest.

                                                                                                                                                                                                                              Les Bayliss
                                                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                              Joined: Sep 5 04
                                                                                                                                                                                                                              Posts: 5411
                                                                                                                                                                                                                              Credit: 9,006,998
                                                                                                                                                                                                                              RAC: 3,097
                                                                                                                                                                                                                              Message 44293 - Posted 4 Jun 2012 20:38:41 UTC

                                                                                                                                                                                                                                The first EU zip was at about 8 hours, and this has now uploaded OK.
                                                                                                                                                                                                                                All of the first 12 zips go to the same server, so there shouldn't be any problem there.
                                                                                                                                                                                                                                PNW zips are also uploading OK, as are the restart zips.

                                                                                                                                                                                                                                Any problems with the regional models would therefore mostly likely be due to network problems, and/or heavy loading on the servers, which should be transitory.


                                                                                                                                                                                                                                ____________
                                                                                                                                                                                                                                Backups: Here

                                                                                                                                                                                                                                Profile Greg van Paassen
                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                Joined: Nov 17 07
                                                                                                                                                                                                                                Posts: 142
                                                                                                                                                                                                                                Credit: 4,271,370
                                                                                                                                                                                                                                RAC: 1
                                                                                                                                                                                                                                Message 44294 - Posted 4 Jun 2012 21:41:33 UTC

                                                                                                                                                                                                                                  Not exactly "transitory" in the usual meaning of the word... I have had the same two EU zip files failing to upload since early Saturday my time - that would be Friday afternoon in UTC (Friday evening in British Daylight Time). Currently there are another 104 zips in line behind these two. Oh, well.

                                                                                                                                                                                                                                  Les Bayliss
                                                                                                                                                                                                                                  Forum moderator
                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                  Joined: Sep 5 04
                                                                                                                                                                                                                                  Posts: 5411
                                                                                                                                                                                                                                  Credit: 9,006,998
                                                                                                                                                                                                                                  RAC: 3,097
                                                                                                                                                                                                                                  Message 44295 - Posted 4 Jun 2012 22:46:57 UTC - in response to Message 44294.

                                                                                                                                                                                                                                    In that case you're suffering from the "too-many-zips" problem, in which case it's up to you to fix it.

                                                                                                                                                                                                                                    Thyme Lawn has posted several cures for upload problems, of which there are a few varieties.
                                                                                                                                                                                                                                    To start with, try this one.


                                                                                                                                                                                                                                    ____________
                                                                                                                                                                                                                                    Backups: Here

                                                                                                                                                                                                                                    Profile Greg van Paassen
                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                    Joined: Nov 17 07
                                                                                                                                                                                                                                    Posts: 142
                                                                                                                                                                                                                                    Credit: 4,271,370
                                                                                                                                                                                                                                    RAC: 1
                                                                                                                                                                                                                                    Message 44298 - Posted 5 Jun 2012 11:13:33 UTC - in response to Message 44295.

                                                                                                                                                                                                                                      Thanks, Les, I appreciate the help.

                                                                                                                                                                                                                                      I added the time-out option to cc_config.xml, and set Boinc to try three transfers at a time as well. It's not helping, though: when I turned network activity back on, the first three files changed status to 'Uploading', and then within two seconds changed back to 'Retry in 09:30:00'. (With the time being different for each, of course.) Since the upload fails straight away it can't be a time-out problem.

                                                                                                                                                                                                                                      It's not the "hnndler" problem, either. The problem is with uploader1. Its http server appears to be running OK, but attempting to access the file upload handler returns HTTP 500, internal server error.

                                                                                                                                                                                                                                      I've set the http_transfer_debug option and I will see what new insight that brings when the first time-out expires.

                                                                                                                                                                                                                                      Profile Gordon Hartman
                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                      Joined: Aug 31 04
                                                                                                                                                                                                                                      Posts: 5
                                                                                                                                                                                                                                      Credit: 337,210
                                                                                                                                                                                                                                      RAC: 0
                                                                                                                                                                                                                                      Message 44300 - Posted 5 Jun 2012 15:12:38 UTC - in response to Message 44298.

                                                                                                                                                                                                                                        I can't upload either!

                                                                                                                                                                                                                                        6/5/2012 11:11:50 AM | climateprediction.net | Started upload of hadam3p_eu_8ck3_2005_1_007870650_0_12.zip
                                                                                                                                                                                                                                        6/5/2012 11:11:52 AM | climateprediction.net | Temporarily failed upload of hadam3p_eu_8ck3_2005_1_007870650_0_12.zip: transient HTTP error
                                                                                                                                                                                                                                        6/5/2012 11:11:52 AM | climateprediction.net | Backing off 5 hr 36 min 54 sec on upload of hadam3p_eu_8ck3_2005_1_007870650_0_12.zip





                                                                                                                                                                                                                                        ____________

                                                                                                                                                                                                                                        Profile Dave Jackson
                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                        Joined: May 15 09
                                                                                                                                                                                                                                        Posts: 849
                                                                                                                                                                                                                                        Credit: 645,754
                                                                                                                                                                                                                                        RAC: 154
                                                                                                                                                                                                                                        Message 44302 - Posted 5 Jun 2012 18:17:06 UTC

                                                                                                                                                                                                                                          Both EU and PNW zips uploading fine from here and have been all of the 4 day weekend.

                                                                                                                                                                                                                                          Dave

                                                                                                                                                                                                                                          Les Bayliss
                                                                                                                                                                                                                                          Forum moderator
                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                          Joined: Sep 5 04
                                                                                                                                                                                                                                          Posts: 5411
                                                                                                                                                                                                                                          Credit: 9,006,998
                                                                                                                                                                                                                                          RAC: 3,097
                                                                                                                                                                                                                                          Message 44303 - Posted 5 Jun 2012 18:23:22 UTC - in response to Message 44298.

                                                                                                                                                                                                                                            Greg

                                                                                                                                                                                                                                            When I had a lot stacked up during the recent pileup, I waited until the two that wanted to start first had timed out, clicked on one of the others way down the queue, and then on Retry now. That one started and then another one that hadn't had a turn yet. Both of then completed their upload, and then the others followed as fast as a bunch of sheep trying to get out of a barn.
                                                                                                                                                                                                                                            I think that the original two still hung around for a while, but eventually they uploaded too.


                                                                                                                                                                                                                                            ____________
                                                                                                                                                                                                                                            Backups: Here

                                                                                                                                                                                                                                            Les Bayliss
                                                                                                                                                                                                                                            Forum moderator
                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                            Joined: Sep 5 04
                                                                                                                                                                                                                                            Posts: 5411
                                                                                                                                                                                                                                            Credit: 9,006,998
                                                                                                                                                                                                                                            RAC: 3,097
                                                                                                                                                                                                                                            Message 44304 - Posted 5 Jun 2012 20:23:39 UTC

                                                                                                                                                                                                                                              I've just finished uploading a number of files to Oxford. These included PNW, EU, and a restart zip. On one computer I simultaneously downloaded the numerous files needed to start a new EU model.
                                                                                                                                                                                                                                              All of these went through with no problems.

                                                                                                                                                                                                                                              So I'm fairly sure that any transfer problems people are having are occurring either on their computer, or at one of the nodes between them and Oxford.

                                                                                                                                                                                                                                              More information will be needed to help than just "I can't upload files". :)
                                                                                                                                                                                                                                              If and when I get time/remember, I'll try and locate all of the help posts that have been made over the past few years, and make a sticky linking to them.


                                                                                                                                                                                                                                              ____________
                                                                                                                                                                                                                                              Backups: Here

                                                                                                                                                                                                                                              Profile Greg van Paassen
                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                              Joined: Nov 17 07
                                                                                                                                                                                                                                              Posts: 142
                                                                                                                                                                                                                                              Credit: 4,271,370
                                                                                                                                                                                                                                              RAC: 1
                                                                                                                                                                                                                                              Message 44305 - Posted 6 Jun 2012 5:58:34 UTC

                                                                                                                                                                                                                                                Last modified: 6 Jun 2012 6:04:48 UTC

                                                                                                                                                                                                                                                Thanks again, Les.

                                                                                                                                                                                                                                                I suspended all work and re-started the boinc client (i.e., the background process, not the boinc manager). That seems to have done the trick --- seven failures, but the eighth file uploaded OK and now the ninth file is uploading.

                                                                                                                                                                                                                                                EDIT: The successful uploads are going to cpdn-upload2.oerc.ox.ac.uk. The problems are with uploader1.atm.ox.ac.uk.

                                                                                                                                                                                                                                                Harri Liljeroos
                                                                                                                                                                                                                                                Avatar
                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                Joined: Dec 9 05
                                                                                                                                                                                                                                                Posts: 6
                                                                                                                                                                                                                                                Credit: 4,654,367
                                                                                                                                                                                                                                                RAC: 2,703
                                                                                                                                                                                                                                                Message 44306 - Posted 6 Jun 2012 5:58:38 UTC

                                                                                                                                                                                                                                                  I'm experiencing also problems with uploads. I just watched zips from three different wu's try to transfer. Zip files from two wus went just fine, from on wu they stop after 2 seconds (zip files 4-12). This failing wu was finished on 3rd of June, so Boinc has been trying to send those zips for several days, the zip 13 has already uploaded successfully.

                                                                                                                                                                                                                                                  Here is some message log (I removed some lines for other projects):


                                                                                                                                                                                                                                                  cimw7718.cimcorp5.com

                                                                                                                                                                                                                                                  812 climateprediction.net 06.06.12 08:40:08 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_4.zip
                                                                                                                                                                                                                                                  813 climateprediction.net 06.06.12 08:40:08 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_5.zip
                                                                                                                                                                                                                                                  814 climateprediction.net 06.06.12 08:40:10 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_4.zip: HTTP error
                                                                                                                                                                                                                                                  815 climateprediction.net 06.06.12 08:40:10 Backing off 10 hr 17 min 25 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_4.zip
                                                                                                                                                                                                                                                  816 climateprediction.net 06.06.12 08:40:10 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_5.zip: HTTP error
                                                                                                                                                                                                                                                  817 climateprediction.net 06.06.12 08:40:10 Backing off 6 hr 58 min 2 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_5.zip
                                                                                                                                                                                                                                                  818 climateprediction.net 06.06.12 08:40:10 Started upload of hadam3p_eu_cic0_2009_1_007981905_0_2.zip
                                                                                                                                                                                                                                                  819 climateprediction.net 06.06.12 08:40:10 Started upload of hadam3p_eu_d2mw_2006_1_007969626_0_1.zip
                                                                                                                                                                                                                                                  820 climateprediction.net 06.06.12 08:42:11 Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_2.zip
                                                                                                                                                                                                                                                  821 climateprediction.net 06.06.12 08:42:11 Started upload of hadam3p_eu_cic0_2009_1_007981905_0_3.zip
                                                                                                                                                                                                                                                  822 climateprediction.net 06.06.12 08:42:14 Finished upload of hadam3p_eu_d2mw_2006_1_007969626_0_1.zip
                                                                                                                                                                                                                                                  823 climateprediction.net 06.06.12 08:42:14 Started upload of hadam3p_eu_cic0_2009_1_007981905_0_4.zip
                                                                                                                                                                                                                                                  824 climateprediction.net 06.06.12 08:44:22 Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_3.zip
                                                                                                                                                                                                                                                  825 climateprediction.net 06.06.12 08:44:22 Started upload of hadam3p_eu_d2mw_2006_1_007969626_0_2.zip
                                                                                                                                                                                                                                                  826 climateprediction.net 06.06.12 08:44:28 Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_4.zip
                                                                                                                                                                                                                                                  827 climateprediction.net 06.06.12 08:44:28 Started upload of hadam3p_eu_cic0_2009_1_007981905_0_5.zip
                                                                                                                                                                                                                                                  833 climateprediction.net 06.06.12 08:45:44 Finished upload of hadam3p_eu_d2mw_2006_1_007969626_0_2.zip
                                                                                                                                                                                                                                                  834 climateprediction.net 06.06.12 08:45:44 Started upload of hadam3p_eu_d2mw_2006_1_007969626_0_3.zip
                                                                                                                                                                                                                                                  835 climateprediction.net 06.06.12 08:45:51 Finished upload of hadam3p_eu_cic0_2009_1_007981905_0_5.zip
                                                                                                                                                                                                                                                  836 climateprediction.net 06.06.12 08:45:53 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_6.zip
                                                                                                                                                                                                                                                  837 climateprediction.net 06.06.12 08:45:54 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_6.zip: HTTP error
                                                                                                                                                                                                                                                  838 climateprediction.net 06.06.12 08:45:54 Backing off 6 hr 29 min 1 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_6.zip
                                                                                                                                                                                                                                                  839 climateprediction.net 06.06.12 08:45:54 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_7.zip
                                                                                                                                                                                                                                                  840 climateprediction.net 06.06.12 08:45:55 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_7.zip: HTTP error
                                                                                                                                                                                                                                                  841 climateprediction.net 06.06.12 08:45:55 Backing off 6 hr 15 min 20 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_7.zip
                                                                                                                                                                                                                                                  842 climateprediction.net 06.06.12 08:45:55 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_8.zip
                                                                                                                                                                                                                                                  843 climateprediction.net 06.06.12 08:45:56 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_8.zip: HTTP error
                                                                                                                                                                                                                                                  844 climateprediction.net 06.06.12 08:45:56 Backing off 10 hr 15 min 43 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_8.zip
                                                                                                                                                                                                                                                  845 climateprediction.net 06.06.12 08:47:03 Finished upload of hadam3p_eu_d2mw_2006_1_007969626_0_3.zip
                                                                                                                                                                                                                                                  846 climateprediction.net 06.06.12 08:47:05 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_9.zip
                                                                                                                                                                                                                                                  847 climateprediction.net 06.06.12 08:47:05 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_10.zip
                                                                                                                                                                                                                                                  848 climateprediction.net 06.06.12 08:47:07 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_9.zip: HTTP error
                                                                                                                                                                                                                                                  849 climateprediction.net 06.06.12 08:47:07 Backing off 3 hr 10 min 20 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_9.zip
                                                                                                                                                                                                                                                  850 climateprediction.net 06.06.12 08:47:07 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_10.zip: HTTP error
                                                                                                                                                                                                                                                  851 climateprediction.net 06.06.12 08:47:07 Backing off 1 hr 0 min 0 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_10.zip
                                                                                                                                                                                                                                                  852 climateprediction.net 06.06.12 08:47:08 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_11.zip
                                                                                                                                                                                                                                                  853 climateprediction.net 06.06.12 08:47:08 Started upload of hadam3p_eu_82s2_2002_1_007750214_1_12.zip
                                                                                                                                                                                                                                                  854 climateprediction.net 06.06.12 08:47:09 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_11.zip: HTTP error
                                                                                                                                                                                                                                                  855 climateprediction.net 06.06.12 08:47:09 Backing off 1 hr 5 min 3 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_11.zip
                                                                                                                                                                                                                                                  856 climateprediction.net 06.06.12 08:47:09 Temporarily failed upload of hadam3p_eu_82s2_2002_1_007750214_1_12.zip: HTTP error
                                                                                                                                                                                                                                                  857 climateprediction.net 06.06.12 08:47:09 Backing off 2 hr 15 min 53 sec on upload of hadam3p_eu_82s2_2002_1_007750214_1_12.zip
                                                                                                                                                                                                                                                  858 climateprediction.net 06.06.12 08:54:40 Resuming task hadam3p_eu_cic0_2009_1_007981905_0 using hadam3p_eu version 609


                                                                                                                                                                                                                                                  ____________

                                                                                                                                                                                                                                                  Profile Dave Jackson
                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                  Joined: May 15 09
                                                                                                                                                                                                                                                  Posts: 849
                                                                                                                                                                                                                                                  Credit: 645,754
                                                                                                                                                                                                                                                  RAC: 154
                                                                                                                                                                                                                                                  Message 44307 - Posted 6 Jun 2012 7:16:52 UTC - in response to Message 44306.

                                                                                                                                                                                                                                                    Check the two previous messages - they should help.

                                                                                                                                                                                                                                                    Dave

                                                                                                                                                                                                                                                    Harri Liljeroos
                                                                                                                                                                                                                                                    Avatar
                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                    Joined: Dec 9 05
                                                                                                                                                                                                                                                    Posts: 6
                                                                                                                                                                                                                                                    Credit: 4,654,367
                                                                                                                                                                                                                                                    RAC: 2,703
                                                                                                                                                                                                                                                    Message 44308 - Posted 6 Jun 2012 7:36:16 UTC - in response to Message 44307.

                                                                                                                                                                                                                                                      Restarting Boinc or rebooting the computer does not help. HTTP error in 2 seconds when retrying to transfer. Using win 7 64bit and Boinc 6.12.34.
                                                                                                                                                                                                                                                      ____________

                                                                                                                                                                                                                                                      Profile Greg van Paassen
                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                      Joined: Nov 17 07
                                                                                                                                                                                                                                                      Posts: 142
                                                                                                                                                                                                                                                      Credit: 4,271,370
                                                                                                                                                                                                                                                      RAC: 1
                                                                                                                                                                                                                                                      Message 44309 - Posted 6 Jun 2012 8:02:17 UTC - in response to Message 44308.

                                                                                                                                                                                                                                                        Harri - try this:

                                                                                                                                                                                                                                                        Open another tab in your browser and go to this address: http://cpdn-upload2.oerc.ox.ac.uk.

                                                                                                                                                                                                                                                        The response should be a page that says

                                                                                                                                                                                                                                                        Climate Prediction.net Upload Server

                                                                                                                                                                                                                                                        This server is part of the Climate Prediction.net project. Please visit climateprediction.net to participate.
                                                                                                                                                                                                                                                        If you don't get that, there is a connection problem - perhaps a blacklisting. To check, open a command prompt (Win+R cmd.exe) and run
                                                                                                                                                                                                                                                        tracert cpdn-upload2.oerc.ox.ac.uk

                                                                                                                                                                                                                                                        The trace should complete in less than a minute, listing fewer than 30 numbered lines and with not too many '*'s in the output.

                                                                                                                                                                                                                                                        If that is OK, you'll need to set the file_xfer_debug and http_xfer_debug options in cc_config.xml and get boinc to re-read it. Let me know if you want help with that.

                                                                                                                                                                                                                                                        Les Bayliss
                                                                                                                                                                                                                                                        Forum moderator
                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                        Joined: Sep 5 04
                                                                                                                                                                                                                                                        Posts: 5411
                                                                                                                                                                                                                                                        Credit: 9,006,998
                                                                                                                                                                                                                                                        RAC: 3,097
                                                                                                                                                                                                                                                        Message 44310 - Posted 6 Jun 2012 8:16:11 UTC

                                                                                                                                                                                                                                                          Last modified: 6 Jun 2012 13:36:43 UTC

                                                                                                                                                                                                                                                          In case it helps, a couple of hours ago I put in a request for someone to give the server that's down a good kick in the reset button.
                                                                                                                                                                                                                                                          Oxford should be starting to stop dreaming and get back to work about now, so we'll see what happens.
                                                                                                                                                                                                                                                          ____________
                                                                                                                                                                                                                                                          Backups: Here

                                                                                                                                                                                                                                                          Profile Dave Jackson
                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                          Joined: May 15 09
                                                                                                                                                                                                                                                          Posts: 849
                                                                                                                                                                                                                                                          Credit: 645,754
                                                                                                                                                                                                                                                          RAC: 154
                                                                                                                                                                                                                                                          Message 44311 - Posted 6 Jun 2012 9:39:39 UTC - in response to Message 44309.

                                                                                                                                                                                                                                                            Last modified: 6 Jun 2012 9:42:11 UTC

                                                                                                                                                                                                                                                            "If you don't get that, there is a connection problem - perhaps a blacklisting. To check, open a command prompt (Win+R cmd.exe) and run
                                                                                                                                                                                                                                                            tracert cpdn-upload2.oerc.ox.ac.uk

                                                                                                                                                                                                                                                            The trace should complete in less than a minute, listing fewer than 30 numbered lines and with not too many '*'s in the output."

                                                                                                                                                                                                                                                            Anyone trying to do the same from a linux box should use the traceroute command.

                                                                                                                                                                                                                                                            Harri Liljeroos
                                                                                                                                                                                                                                                            Avatar
                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                            Joined: Dec 9 05
                                                                                                                                                                                                                                                            Posts: 6
                                                                                                                                                                                                                                                            Credit: 4,654,367
                                                                                                                                                                                                                                                            RAC: 2,703
                                                                                                                                                                                                                                                            Message 44312 - Posted 6 Jun 2012 9:44:44 UTC - in response to Message 44310.

                                                                                                                                                                                                                                                              @ Les

                                                                                                                                                                                                                                                              Maybe it was the size of the boot that they used to kick the server, anyway now the stuck transfers are flowing smoothly. Thanks for the help.

                                                                                                                                                                                                                                                              @ Greg

                                                                                                                                                                                                                                                              The response was as you described, so probably no blacklisting was involved.
                                                                                                                                                                                                                                                              ____________

                                                                                                                                                                                                                                                              Profile Greg van Paassen
                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                              Joined: Nov 17 07
                                                                                                                                                                                                                                                              Posts: 142
                                                                                                                                                                                                                                                              Credit: 4,271,370
                                                                                                                                                                                                                                                              RAC: 1
                                                                                                                                                                                                                                                              Message 44318 - Posted 6 Jun 2012 22:56:17 UTC

                                                                                                                                                                                                                                                                All flowing smoothly here, too. :)

                                                                                                                                                                                                                                                                Starfire
                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                Joined: Feb 5 05
                                                                                                                                                                                                                                                                Posts: 17
                                                                                                                                                                                                                                                                Credit: 1,581,796
                                                                                                                                                                                                                                                                RAC: 11
                                                                                                                                                                                                                                                                Message 44325 - Posted 8 Jun 2012 8:03:45 UTC

                                                                                                                                                                                                                                                                  I'm currently unable to upload ZIP 12 from a HADAM3P EU task. The upload goes to 100% and then errors out:


                                                                                                                                                                                                                                                                  08.06.2012 10:00:41 | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/incoming/uploader/hadam3p_eu_d7tn_2008_1_007976349_2_12.zip: Permission denied
                                                                                                                                                                                                                                                                  08.06.2012 10:00:41 | climateprediction.net | Temporarily failed upload of hadam3p_eu_d7tn_2008_1_007976349_2_12.zip: transient upload error


                                                                                                                                                                                                                                                                  The final ZIP 13 uploaded without any problems.

                                                                                                                                                                                                                                                                  Les Bayliss
                                                                                                                                                                                                                                                                  Forum moderator
                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                  Joined: Sep 5 04
                                                                                                                                                                                                                                                                  Posts: 5411
                                                                                                                                                                                                                                                                  Credit: 9,006,998
                                                                                                                                                                                                                                                                  RAC: 3,097
                                                                                                                                                                                                                                                                  Message 44326 - Posted 8 Jun 2012 8:16:13 UTC

                                                                                                                                                                                                                                                                    The server's down again, possible filled up with all the data.

                                                                                                                                                                                                                                                                    And now it's back up.


                                                                                                                                                                                                                                                                    ____________
                                                                                                                                                                                                                                                                    Backups: Here

                                                                                                                                                                                                                                                                    Starfire
                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                    Joined: Feb 5 05
                                                                                                                                                                                                                                                                    Posts: 17
                                                                                                                                                                                                                                                                    Credit: 1,581,796
                                                                                                                                                                                                                                                                    RAC: 11
                                                                                                                                                                                                                                                                    Message 44327 - Posted 8 Jun 2012 8:23:33 UTC

                                                                                                                                                                                                                                                                      It seems to come and go ;)

                                                                                                                                                                                                                                                                      When it's "gone" I get "connect() failed" as the error. At least I have some of these in the logs too. Normally I ignore these because usually it just means the server isn't up ;)

                                                                                                                                                                                                                                                                      Profile Dave Jackson
                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                      Joined: May 15 09
                                                                                                                                                                                                                                                                      Posts: 849
                                                                                                                                                                                                                                                                      Credit: 645,754
                                                                                                                                                                                                                                                                      RAC: 154
                                                                                                                                                                                                                                                                      Message 44329 - Posted 8 Jun 2012 10:33:17 UTC - in response to Message 44327.

                                                                                                                                                                                                                                                                        And down again. :-(

                                                                                                                                                                                                                                                                        Eirik Redd
                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                        Joined: Aug 31 04
                                                                                                                                                                                                                                                                        Posts: 257
                                                                                                                                                                                                                                                                        Credit: 27,941,560
                                                                                                                                                                                                                                                                        RAC: 13,465
                                                                                                                                                                                                                                                                        Message 44336 - Posted 9 Jun 2012 5:36:26 UTC - in response to Message 44329.

                                                                                                                                                                                                                                                                          Yup -- eu intermediate uploads not going - was clear less than a day ago - now several uploads failing - no servers showing down on server status page -
                                                                                                                                                                                                                                                                          And of course it is the weekend.
                                                                                                                                                                                                                                                                          Same what I always say -- the crew will figure it out -- and lose no data --
                                                                                                                                                                                                                                                                          Patience is a virtue -- reporting problems is a bigger virtue --
                                                                                                                                                                                                                                                                          losing no data is the big big big virtue.
                                                                                                                                                                                                                                                                          But sometimes it takes a while.

                                                                                                                                                                                                                                                                          Keep on crunching.
                                                                                                                                                                                                                                                                          Keep on reporting any problems.

                                                                                                                                                                                                                                                                          Thanks to all.
                                                                                                                                                                                                                                                                          E
                                                                                                                                                                                                                                                                          ____________

                                                                                                                                                                                                                                                                          Profile Greg van Paassen
                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                          Joined: Nov 17 07
                                                                                                                                                                                                                                                                          Posts: 142
                                                                                                                                                                                                                                                                          Credit: 4,271,370
                                                                                                                                                                                                                                                                          RAC: 1
                                                                                                                                                                                                                                                                          Message 44337 - Posted 9 Jun 2012 5:50:13 UTC

                                                                                                                                                                                                                                                                            Last modified: 9 Jun 2012 5:56:22 UTC

                                                                                                                                                                                                                                                                            To save bandwidth/data caps being wasted one could set 'Network Activity Suspended in the Boinc Manager's Activity menu.

                                                                                                                                                                                                                                                                            Of course that affects all projects. And of course there is no option in the 'Projects' tab to hold uploads for an individual project, and never will be...

                                                                                                                                                                                                                                                                            Profile Thyme Lawn
                                                                                                                                                                                                                                                                            Forum moderator
                                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                                            Joined: Aug 5 04
                                                                                                                                                                                                                                                                            Posts: 1232
                                                                                                                                                                                                                                                                            Credit: 10,409,271
                                                                                                                                                                                                                                                                            RAC: 1,138
                                                                                                                                                                                                                                                                            Message 44338 - Posted 9 Jun 2012 9:11:58 UTC - in response to Message 44337.

                                                                                                                                                                                                                                                                              Of course that affects all projects. And of course there is no option in the 'Projects' tab to hold uploads for an individual project, and never will be...

                                                                                                                                                                                                                                                                              I submitted a BOINC client patch to implement that almost 3 years ago (BOINC Trac ticket #139) but it was never applied. I've just updated that ticket with an alternative suggestion (scheduler replies include a status indication for upload and download servers, clients don't attempt transfers to/from unavailable servers and issue periodic scheduler requests to determine when transfer is possible).
                                                                                                                                                                                                                                                                              ____________
                                                                                                                                                                                                                                                                              "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

                                                                                                                                                                                                                                                                              Reed
                                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                                              Joined: May 27 12
                                                                                                                                                                                                                                                                              Posts: 3
                                                                                                                                                                                                                                                                              Credit: 32,115
                                                                                                                                                                                                                                                                              RAC: 31
                                                                                                                                                                                                                                                                              Message 44345 - Posted 9 Jun 2012 23:36:48 UTC

                                                                                                                                                                                                                                                                                I've been having the same problem continuously for several days now, I believe since 6 June. One zip failing to upload, but only after getting to 100%. It's uploaded and failed at 100% at least eight times now, probably several more when I wasn't looking. Restarts, etc. have no effect.

                                                                                                                                                                                                                                                                                Profile Greg van Paassen
                                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                                Joined: Nov 17 07
                                                                                                                                                                                                                                                                                Posts: 142
                                                                                                                                                                                                                                                                                Credit: 4,271,370
                                                                                                                                                                                                                                                                                RAC: 1
                                                                                                                                                                                                                                                                                Message 44346 - Posted 10 Jun 2012 3:38:22 UTC - in response to Message 44338.

                                                                                                                                                                                                                                                                                  Of course that affects all projects. And of course there is no option in the 'Projects' tab to hold uploads for an individual project, and never will be...

                                                                                                                                                                                                                                                                                  I submitted a BOINC client patch to implement that almost 3 years ago ...
                                                                                                                                                                                                                                                                                  Precisely, T. L. That was ... three, nearly four BOINC versions ago?

                                                                                                                                                                                                                                                                                  I guess we should be grateful that BOINC still mostly does the basics OK. (Except on Macs, where the BOINC project randomly changes the required ownership and permissions on the client's data folder from version to version.)

                                                                                                                                                                                                                                                                                  :-/

                                                                                                                                                                                                                                                                                  Profile Greg van Paassen
                                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                                  Joined: Nov 17 07
                                                                                                                                                                                                                                                                                  Posts: 142
                                                                                                                                                                                                                                                                                  Credit: 4,271,370
                                                                                                                                                                                                                                                                                  RAC: 1
                                                                                                                                                                                                                                                                                  Message 44347 - Posted 10 Jun 2012 3:56:47 UTC - in response to Message 44345.

                                                                                                                                                                                                                                                                                    Reed, welcome to climateprediction.net, a.k.a. CPDN.

                                                                                                                                                                                                                                                                                    You've already discovered that this is not a 'set and forget' project, nor even 'plug and play'. :-/

                                                                                                                                                                                                                                                                                    If you have a data cap on your ISP account, I recommend doing one of two things pro tem., to prevent your data cap being used up with fruitless upload attempts by BOINC. For either of these you'll need to change to the BOINC Manager's "advanced view".

                                                                                                                                                                                                                                                                                    Either: in the activity menu, select "Network Activity Suspended". This will isolate your BOINC work from the internet entirely.

                                                                                                                                                                                                                                                                                    Or, in the "Tools" menu, select "Computing Preferences". On the dialogue box that pops up, select the "network usage" tab. Just below half-way, there is a section "Network Usage Allowed". Set this to only allow network usage for one or two hours per day. That will limit the damage but still upload trickles and allow BOINC to download work ... once the project has some more.

                                                                                                                                                                                                                                                                                    Reed
                                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                                    Joined: May 27 12
                                                                                                                                                                                                                                                                                    Posts: 3
                                                                                                                                                                                                                                                                                    Credit: 32,115
                                                                                                                                                                                                                                                                                    RAC: 31
                                                                                                                                                                                                                                                                                    Message 44352 - Posted 10 Jun 2012 19:13:33 UTC - in response to Message 44347.

                                                                                                                                                                                                                                                                                      Thanks. I don't need to worry about total bandwidth usage. But I don't much care for its slowing my connection over and over to no purpose.

                                                                                                                                                                                                                                                                                      Mostly what I want to know is, is there any way to know when we may expect this to begin working again, so I can shut off its network usage till there's a chance of it going through. Is there any way to get an indicator of this? Given that there have been no notifications of anything wrong, I'm guessing not... (It would, as everyone has said, be better if we could shut off just one project's network usage.)

                                                                                                                                                                                                                                                                                      Of course, now no other BOINC projects seem to be downloading new work, either, so maybe something bigger is going on. I sure wish BOINC had been built as intelligently as the original SETI@home.

                                                                                                                                                                                                                                                                                      /complaint

                                                                                                                                                                                                                                                                                      Profile Dave Jackson
                                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                                      Joined: May 15 09
                                                                                                                                                                                                                                                                                      Posts: 849
                                                                                                                                                                                                                                                                                      Credit: 645,754
                                                                                                                                                                                                                                                                                      RAC: 154
                                                                                                                                                                                                                                                                                      Message 44353 - Posted 10 Jun 2012 19:25:10 UTC - in response to Message 44352.

                                                                                                                                                                                                                                                                                        My guess is things are likely to work again some time Monday after the university staff get to work. I know they have a meeting which will be looking at the issues with the servers as well as some issues to do with credits. Hopefully the servers issues will take priority and we will get some feedback at some stage about what they are doing re the server issues.

                                                                                                                                                                                                                                                                                        Les Bayliss
                                                                                                                                                                                                                                                                                        Forum moderator
                                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                                        Joined: Sep 5 04
                                                                                                                                                                                                                                                                                        Posts: 5411
                                                                                                                                                                                                                                                                                        Credit: 9,006,998
                                                                                                                                                                                                                                                                                        RAC: 3,097
                                                                                                                                                                                                                                                                                        Message 44354 - Posted 10 Jun 2012 21:40:15 UTC - in response to Message 44352.

                                                                                                                                                                                                                                                                                          Reed

                                                                                                                                                                                                                                                                                          Have you Subscribed to the News and Announcement thread at the top of the Number Crunching section? If you do you'll get an email when ever a new post is made there.
                                                                                                                                                                                                                                                                                          And that's where a notice will be made that problems "should" be fixed, not in threads scattered all over this board.

                                                                                                                                                                                                                                                                                          There's a similar News thread near the top of our alternative PHP board, here.


                                                                                                                                                                                                                                                                                          ____________
                                                                                                                                                                                                                                                                                          Backups: Here

                                                                                                                                                                                                                                                                                          Profile Dave Jackson
                                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                                          Joined: May 15 09
                                                                                                                                                                                                                                                                                          Posts: 849
                                                                                                                                                                                                                                                                                          Credit: 645,754
                                                                                                                                                                                                                                                                                          RAC: 154
                                                                                                                                                                                                                                                                                          Message 44355 - Posted 10 Jun 2012 21:48:55 UTC - in response to Message 44354.

                                                                                                                                                                                                                                                                                            However the news thread has been rather silent on this issue. I know that is to do with the project people not feeding the information through to the moderators and suspect that if they fed something through every time there is a problem it would slow down the problem being fixed. Also they are not there to put anything onto the news thread over the weekend.

                                                                                                                                                                                                                                                                                            Les Bayliss
                                                                                                                                                                                                                                                                                            Forum moderator
                                                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                                                            Joined: Sep 5 04
                                                                                                                                                                                                                                                                                            Posts: 5411
                                                                                                                                                                                                                                                                                            Credit: 9,006,998
                                                                                                                                                                                                                                                                                            RAC: 3,097
                                                                                                                                                                                                                                                                                            Message 44357 - Posted 10 Jun 2012 22:38:06 UTC - in response to Message 44355.

                                                                                                                                                                                                                                                                                              Well, if you think a post there will help, I'll make one.

                                                                                                                                                                                                                                                                                              Digby
                                                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                                                              Joined: Feb 17 06
                                                                                                                                                                                                                                                                                              Posts: 64
                                                                                                                                                                                                                                                                                              Credit: 3,717,892
                                                                                                                                                                                                                                                                                              RAC: 12
                                                                                                                                                                                                                                                                                              Message 44359 - Posted 11 Jun 2012 7:58:36 UTC

                                                                                                                                                                                                                                                                                                Yes I am also having problems. I have over 50 files to be uploaded and I am unable to receive any new files... Lets hope the server is sorted today because I have no projects to work on at the moment. They are all finished but the server will not take them...

                                                                                                                                                                                                                                                                                                Cheers :o

                                                                                                                                                                                                                                                                                                P.S. Is it possible to upload a screen shot...?

                                                                                                                                                                                                                                                                                                Profile Dave Jackson
                                                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                                                Joined: May 15 09
                                                                                                                                                                                                                                                                                                Posts: 849
                                                                                                                                                                                                                                                                                                Credit: 645,754
                                                                                                                                                                                                                                                                                                RAC: 154
                                                                                                                                                                                                                                                                                                Message 44360 - Posted 11 Jun 2012 8:07:22 UTC - in response to Message 44357.

                                                                                                                                                                                                                                                                                                  "Well, if you think a post there will help, I'll make one."

                                                                                                                                                                                                                                                                                                  I think it might reduce the amount of discussion that doesn't really achieve much in the other threads.[/quote]

                                                                                                                                                                                                                                                                                                  P John
                                                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                                                  Joined: Dec 18 11
                                                                                                                                                                                                                                                                                                  Posts: 4
                                                                                                                                                                                                                                                                                                  Credit: 64,484
                                                                                                                                                                                                                                                                                                  RAC: 0
                                                                                                                                                                                                                                                                                                  Message 44361 - Posted 11 Jun 2012 8:32:30 UTC - in response to Message 43140.

                                                                                                                                                                                                                                                                                                    Servers all appear OK, but I am getting a repetitive upload failure, namely
                                                                                                                                                                                                                                                                                                    05/10/2011 10:27:54 | climateprediction.net | Temporarily failed upload of hadam3p_eu_6dhw_2001_1_007479505_2_9.zip: connect() failed
                                                                                                                                                                                                                                                                                                    Other models on the same PC are so far uploading just fine.
                                                                                                                                                                                                                                                                                                    Any clues?

                                                                                                                                                                                                                                                                                                    I'm getting exactly the same problem, though when I checked the servers uploader1.atm and cpdnuploader3.oerc were both down. I've currently got 6 tasks waiting to be uploaded. They say they upload 100% then reset to zero on me. The log says "11/06/2012 09:10:05 | climateprediction.net | Temporarily failed upload of hadam3p_eu_czzs_2003_1_007966202_1_7.zip: transient upload error". This has been happening for a couple of days now. Anyone have any ideas what's going on? Or when it'll get fixed? Thanks all.

                                                                                                                                                                                                                                                                                                    nairb
                                                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                                                    Joined: Sep 3 04
                                                                                                                                                                                                                                                                                                    Posts: 23
                                                                                                                                                                                                                                                                                                    Credit: 587,927
                                                                                                                                                                                                                                                                                                    RAC: 119
                                                                                                                                                                                                                                                                                                    Message 44363 - Posted 11 Jun 2012 17:08:41 UTC

                                                                                                                                                                                                                                                                                                      All servers are green but no uploads here either.

                                                                                                                                                                                                                                                                                                      Nairb
                                                                                                                                                                                                                                                                                                      ____________
                                                                                                                                                                                                                                                                                                      <img border="0" src="http://boinc.mundayweb.com/one/stats.php?userID=343" />

                                                                                                                                                                                                                                                                                                      Ryle*
                                                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                                                      Joined: Aug 17 05
                                                                                                                                                                                                                                                                                                      Posts: 11
                                                                                                                                                                                                                                                                                                      Credit: 1,414,760
                                                                                                                                                                                                                                                                                                      RAC: 465
                                                                                                                                                                                                                                                                                                      Message 44364 - Posted 11 Jun 2012 18:39:58 UTC - in response to Message 44363.

                                                                                                                                                                                                                                                                                                        Last modified: 11 Jun 2012 18:40:56 UTC

                                                                                                                                                                                                                                                                                                        The servers are probably being hammered from all people uploading. That's what I'm guessing. We must be patient, they will surely get things on right track again.

                                                                                                                                                                                                                                                                                                        Profile Dave Jackson
                                                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                                                        Joined: May 15 09
                                                                                                                                                                                                                                                                                                        Posts: 849
                                                                                                                                                                                                                                                                                                        Credit: 645,754
                                                                                                                                                                                                                                                                                                        RAC: 154
                                                                                                                                                                                                                                                                                                        Message 44368 - Posted 12 Jun 2012 7:11:51 UTC - in response to Message 44364.

                                                                                                                                                                                                                                                                                                          I suspect it is not just the servers being hammered as the problem still persists now. Also from here at any rate, the pnw models go through fine but not the eu ones. Cant speak for the others as I have no data.

                                                                                                                                                                                                                                                                                                          Digby
                                                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                                                          Joined: Feb 17 06
                                                                                                                                                                                                                                                                                                          Posts: 64
                                                                                                                                                                                                                                                                                                          Credit: 3,717,892
                                                                                                                                                                                                                                                                                                          RAC: 12
                                                                                                                                                                                                                                                                                                          Message 44369 - Posted 12 Jun 2012 7:52:18 UTC

                                                                                                                                                                                                                                                                                                            Last modified: 12 Jun 2012 7:54:37 UTC

                                                                                                                                                                                                                                                                                                            Yes all my models are hadam3p_eu3...they are all complete and awaiting to be uploaded...progress is 0%...however last night it appears that a single file called hadam3p_eu_d0ef_2002_1_007966729_0_13.zip managed to get uploaded.

                                                                                                                                                                                                                                                                                                            Plainly something is not working. I have four files on Retry (project backoff) etc and 55+ files on Upload Pending (project backoff).

                                                                                                                                                                                                                                                                                                            I have stopped processing any models and cannot upload anything...

                                                                                                                                                                                                                                                                                                            The servers all appear green...any ideas?

                                                                                                                                                                                                                                                                                                            Profile Dave Jackson
                                                                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                                                                            Joined: May 15 09
                                                                                                                                                                                                                                                                                                            Posts: 849
                                                                                                                                                                                                                                                                                                            Credit: 645,754
                                                                                                                                                                                                                                                                                                            RAC: 154
                                                                                                                                                                                                                                                                                                            Message 44370 - Posted 12 Jun 2012 8:10:32 UTC - in response to Message 44369.

                                                                                                                                                                                                                                                                                                              Last modified: 12 Jun 2012 8:19:27 UTC

                                                                                                                                                                                                                                                                                                              Other than patience, no. I have suspended my eu units as I have others that can run for the moment. There is now a post on the news thread [url]http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=5447 which suggests it might just be congestion with so many people trying to upload at the same time and one having gone through suggests that might be correct. That particular server was down for quite a long time. Otherwise, I am sure the project people are on to it even if they don't have a quick fix.http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=5447

                                                                                                                                                                                                                                                                                                              Les Bayliss
                                                                                                                                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                                                                              Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                              Posts: 5411
                                                                                                                                                                                                                                                                                                              Credit: 9,006,998
                                                                                                                                                                                                                                                                                                              RAC: 3,097
                                                                                                                                                                                                                                                                                                              Message 44371 - Posted 12 Jun 2012 9:03:14 UTC

                                                                                                                                                                                                                                                                                                                What has to be kept in mind is: Horses for courses.
                                                                                                                                                                                                                                                                                                                Or, in this case, that ALL zip 13s, (the big restart data files), go to their own server, and zips 1-12 of the PNW models go directly to the researchers for that area, onto a server at Oregon University.

                                                                                                                                                                                                                                                                                                                Which leaves the Eu 1-12, the SAF 1-12, and the Coupled Ocean models going to various servers at Oxford. Which are having a hard time of it at the moment.

                                                                                                                                                                                                                                                                                                                For what it's worth, I'm also on a long term timeout while they wait their turn.
                                                                                                                                                                                                                                                                                                                In fact I've shut down BOINC on both computers to wait it out.


                                                                                                                                                                                                                                                                                                                ____________
                                                                                                                                                                                                                                                                                                                Backups: Here

                                                                                                                                                                                                                                                                                                                Profile Dave Jackson
                                                                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                                                                Joined: May 15 09
                                                                                                                                                                                                                                                                                                                Posts: 849
                                                                                                                                                                                                                                                                                                                Credit: 645,754
                                                                                                                                                                                                                                                                                                                RAC: 154
                                                                                                                                                                                                                                                                                                                Message 44376 - Posted 12 Jun 2012 13:10:57 UTC

                                                                                                                                                                                                                                                                                                                  One of my eu zips has just gone through. The other one has gone to back off again though. Might not be completely rigorous but in my mind it proves the congestion theory.

                                                                                                                                                                                                                                                                                                                  Profile Randi
                                                                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                                                                  Joined: Jun 28 07
                                                                                                                                                                                                                                                                                                                  Posts: 17
                                                                                                                                                                                                                                                                                                                  Credit: 1,587,118
                                                                                                                                                                                                                                                                                                                  RAC: 670
                                                                                                                                                                                                                                                                                                                  Message 44377 - Posted 12 Jun 2012 13:30:30 UTC

                                                                                                                                                                                                                                                                                                                    I came to see if there was any info on the problem. When I saw a post mentioning eu vs. pnw models, I went to verify that mine were eu.

                                                                                                                                                                                                                                                                                                                    6 of 7 have now uploaded!

                                                                                                                                                                                                                                                                                                                    Thanks to everyone who worked on solving the problem and responding to our questions!

                                                                                                                                                                                                                                                                                                                    Waldmeister
                                                                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                                                                    Joined: Jun 13 11
                                                                                                                                                                                                                                                                                                                    Posts: 32
                                                                                                                                                                                                                                                                                                                    Credit: 145,646
                                                                                                                                                                                                                                                                                                                    RAC: 660
                                                                                                                                                                                                                                                                                                                    Message 44378 - Posted 12 Jun 2012 13:50:22 UTC

                                                                                                                                                                                                                                                                                                                      Interesting.

                                                                                                                                                                                                                                                                                                                      I have one task with uploads of zips 5 through 12 previously pending.
                                                                                                                                                                                                                                                                                                                      Just a few minutes ago the zip12-file transferred up completely.
                                                                                                                                                                                                                                                                                                                      So, Zips 5-11 still pending.

                                                                                                                                                                                                                                                                                                                      Slowly, slowly, we come to terms... :D

                                                                                                                                                                                                                                                                                                                      Profile Dave Jackson
                                                                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                                                                      Joined: May 15 09
                                                                                                                                                                                                                                                                                                                      Posts: 849
                                                                                                                                                                                                                                                                                                                      Credit: 645,754
                                                                                                                                                                                                                                                                                                                      RAC: 154
                                                                                                                                                                                                                                                                                                                      Message 44379 - Posted 12 Jun 2012 14:26:55 UTC - in response to Message 44378.

                                                                                                                                                                                                                                                                                                                        My last pending zip now gone :-)

                                                                                                                                                                                                                                                                                                                        Waldmeister
                                                                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                                                                        Joined: Jun 13 11
                                                                                                                                                                                                                                                                                                                        Posts: 32
                                                                                                                                                                                                                                                                                                                        Credit: 145,646
                                                                                                                                                                                                                                                                                                                        RAC: 660
                                                                                                                                                                                                                                                                                                                        Message 44380 - Posted 12 Jun 2012 16:58:03 UTC - in response to Message 44379.

                                                                                                                                                                                                                                                                                                                          mine too! :D

                                                                                                                                                                                                                                                                                                                          nairb
                                                                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                                                                          Joined: Sep 3 04
                                                                                                                                                                                                                                                                                                                          Posts: 23
                                                                                                                                                                                                                                                                                                                          Credit: 587,927
                                                                                                                                                                                                                                                                                                                          RAC: 119
                                                                                                                                                                                                                                                                                                                          Message 44381 - Posted 12 Jun 2012 17:21:24 UTC

                                                                                                                                                                                                                                                                                                                            All gone thru. A+++++
                                                                                                                                                                                                                                                                                                                            ____________
                                                                                                                                                                                                                                                                                                                            <img border="0" src="http://boinc.mundayweb.com/one/stats.php?userID=343" />

                                                                                                                                                                                                                                                                                                                            glaesum
                                                                                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                                                                                            Joined: Feb 24 06
                                                                                                                                                                                                                                                                                                                            Posts: 47
                                                                                                                                                                                                                                                                                                                            Credit: 603,024
                                                                                                                                                                                                                                                                                                                            RAC: 50
                                                                                                                                                                                                                                                                                                                            Message 44382 - Posted 12 Jun 2012 17:39:02 UTC - in response to Message 44309.

                                                                                                                                                                                                                                                                                                                              Mine are still struggling. I have had two pairs of EU _4.zip and _5.zips waiting for some days. I have a couple of PNW models reporting fine on a laptop on the same network/router. Meanwhile I've suspended the cpdn EU wus to stop adding to the queue and running some SIMAP resends for June until things clear up. (I'm keeping one of the laptop PNW models going to confirm it still sends it's zips though they'll be infrequent.)

                                                                                                                                                                                                                                                                                                                              Using Greg's test [msg: June 6th 8:02utc]
                                                                                                                                                                                                                                                                                                                              http://climateapps2.oerc.ox.ac.uk/cpdnboinc/forum_thread.php?id=7300&nowrap=true#44309

                                                                                                                                                                                                                                                                                                                              My browser fails the first test and won't connect to the upload server page but passes the second 'tracert' test with no '*'s and only 16 to 18 cmd lines.
                                                                                                                                                                                                                                                                                                                              Not sure what that all means... ...not quite tecchie enough to understand. :)

                                                                                                                                                                                                                                                                                                                              /pg

                                                                                                                                                                                                                                                                                                                              from Greg: "Open another tab in your browser and go to this address: http://cpdn-upload2.oerc.ox.ac.uk.

                                                                                                                                                                                                                                                                                                                              The response should be a page that says
                                                                                                                                                                                                                                                                                                                              Climate Prediction.net Upload Server

                                                                                                                                                                                                                                                                                                                              This server is part of the Climate Prediction.net project. Please visit climateprediction.net to participate.
                                                                                                                                                                                                                                                                                                                              If you don't get that, there is a connection problem - perhaps a blacklisting. To check, open a command prompt (Win+R cmd.exe) and run
                                                                                                                                                                                                                                                                                                                              tracert cpdn-upload2.oerc.ox.ac.uk

                                                                                                                                                                                                                                                                                                                              The trace should complete in less than a minute, listing fewer than 30 numbered lines and with not too many '*'s in the output."

                                                                                                                                                                                                                                                                                                                              Les Bayliss
                                                                                                                                                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                                                                                              Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                              Posts: 5411
                                                                                                                                                                                                                                                                                                                              Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                              RAC: 3,097
                                                                                                                                                                                                                                                                                                                              Message 44385 - Posted 12 Jun 2012 20:58:20 UTC

                                                                                                                                                                                                                                                                                                                                One of the programs failed, but has been restarted.
                                                                                                                                                                                                                                                                                                                                But now one of the servers has 'fallen over'. Not surprising, with the amount of data trying to get to it.
                                                                                                                                                                                                                                                                                                                                It may just have filled up, but what ever the problem is, everyone just move back and give it some air. :)
                                                                                                                                                                                                                                                                                                                                The project people will be back in 10-12 hours.


                                                                                                                                                                                                                                                                                                                                ____________
                                                                                                                                                                                                                                                                                                                                Backups: Here

                                                                                                                                                                                                                                                                                                                                Profile Greg van Paassen
                                                                                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                                                                                Joined: Nov 17 07
                                                                                                                                                                                                                                                                                                                                Posts: 142
                                                                                                                                                                                                                                                                                                                                Credit: 4,271,370
                                                                                                                                                                                                                                                                                                                                RAC: 1
                                                                                                                                                                                                                                                                                                                                Message 44388 - Posted 13 Jun 2012 3:19:20 UTC - in response to Message 44382.

                                                                                                                                                                                                                                                                                                                                  Hi glaesum,

                                                                                                                                                                                                                                                                                                                                  I'm in NZ, so it often takes me a while to catch up with the day's posts.

                                                                                                                                                                                                                                                                                                                                  How are things now? I gather from other posts that most people's files are getting through now. Certainly mine have been, since about 6 hours ago when I turned networking back on.

                                                                                                                                                                                                                                                                                                                                  The two tests I gave just confirm that there is a path through the internet from your machine to the the cpdnupload2 server. But there are other servers, and maybe the problem is with one of them. If you are still having problems, try re-starting the boinc service - rebooting your machine is the easiest way of doing that. It worked for me, once! :) If that doesn't work, the next step is to set up "http transfer debugging" in cc_config.xml, so we can get more information about what boinc is trying to do, and where it's trying to do it.

                                                                                                                                                                                                                                                                                                                                  Les Bayliss
                                                                                                                                                                                                                                                                                                                                  Forum moderator
                                                                                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                                                                                  Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                                  Posts: 5411
                                                                                                                                                                                                                                                                                                                                  Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                                  RAC: 3,097
                                                                                                                                                                                                                                                                                                                                  Message 44389 - Posted 13 Jun 2012 3:52:19 UTC

                                                                                                                                                                                                                                                                                                                                    My large collection have now all uploaded, so things have improved.


                                                                                                                                                                                                                                                                                                                                    ____________
                                                                                                                                                                                                                                                                                                                                    Backups: Here

                                                                                                                                                                                                                                                                                                                                    Digby
                                                                                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                                                                                    Joined: Feb 17 06
                                                                                                                                                                                                                                                                                                                                    Posts: 64
                                                                                                                                                                                                                                                                                                                                    Credit: 3,717,892
                                                                                                                                                                                                                                                                                                                                    RAC: 12
                                                                                                                                                                                                                                                                                                                                    Message 44390 - Posted 13 Jun 2012 7:52:37 UTC

                                                                                                                                                                                                                                                                                                                                      Last modified: 13 Jun 2012 8:10:32 UTC

                                                                                                                                                                                                                                                                                                                                      Yes all my files have now uploaded. Yesterday I actually got error messages from the server before the upload started...I had planned to send in those server error messages but I got sidetracked....

                                                                                                                                                                                                                                                                                                                                      So all files are uploaded but the Project has no tasks to work on...I hope that changes soon...

                                                                                                                                                                                                                                                                                                                                      marpes
                                                                                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                                                                                      Joined: Nov 11 04
                                                                                                                                                                                                                                                                                                                                      Posts: 8
                                                                                                                                                                                                                                                                                                                                      Credit: 15,188,494
                                                                                                                                                                                                                                                                                                                                      RAC: 1,298
                                                                                                                                                                                                                                                                                                                                      Message 44391 - Posted 13 Jun 2012 13:02:19 UTC

                                                                                                                                                                                                                                                                                                                                        Upload all of .zip files is OK.
                                                                                                                                                                                                                                                                                                                                        Some PCs have tasks for 3-4 days only. Hopefully, the new job soon.
                                                                                                                                                                                                                                                                                                                                        ____________

                                                                                                                                                                                                                                                                                                                                        glaesum
                                                                                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                                                                                        Joined: Feb 24 06
                                                                                                                                                                                                                                                                                                                                        Posts: 47
                                                                                                                                                                                                                                                                                                                                        Credit: 603,024
                                                                                                                                                                                                                                                                                                                                        RAC: 50
                                                                                                                                                                                                                                                                                                                                        Message 44392 - Posted 13 Jun 2012 14:11:59 UTC - in response to Message 44391.

                                                                                                                                                                                                                                                                                                                                          hi Greg, all's well. In the small hours I was just about to shut down network activity overnight when I found everything had finally uploaded ok. I think it was several people reporting success in the previous hours that got me slightly concerned while it looked like the servers all had green flags.

                                                                                                                                                                                                                                                                                                                                          let's hope the long period of recurrent server problems is eventually overcome.
                                                                                                                                                                                                                                                                                                                                          now I just need to clear the cache of SIMAPs before returning to my HADAM eu models! :)

                                                                                                                                                                                                                                                                                                                                          nairb
                                                                                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                                                                                          Joined: Sep 3 04
                                                                                                                                                                                                                                                                                                                                          Posts: 23
                                                                                                                                                                                                                                                                                                                                          Credit: 587,927
                                                                                                                                                                                                                                                                                                                                          RAC: 119
                                                                                                                                                                                                                                                                                                                                          Message 44410 - Posted 15 Jun 2012 14:37:34 UTC

                                                                                                                                                                                                                                                                                                                                            Its all Green...... but uploads not playing again with:-

                                                                                                                                                                                                                                                                                                                                            15/06/2012 15:33:17 | climateprediction.net | Temporarily failed upload of hadam3p_eu_d9gm_2008_1_007978472_0_6.zip: transient upload error

                                                                                                                                                                                                                                                                                                                                            Maybe it will become UNtransient in a while.

                                                                                                                                                                                                                                                                                                                                            Nairb
                                                                                                                                                                                                                                                                                                                                            ____________
                                                                                                                                                                                                                                                                                                                                            <img border="0" src="http://boinc.mundayweb.com/one/stats.php?userID=343" />

                                                                                                                                                                                                                                                                                                                                            Profile Jonathan Miller
                                                                                                                                                                                                                                                                                                                                            Forum moderator
                                                                                                                                                                                                                                                                                                                                            Project administrator
                                                                                                                                                                                                                                                                                                                                            Project developer
                                                                                                                                                                                                                                                                                                                                            Volunteer developer
                                                                                                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                                                                                                            Joined: Mar 28 11
                                                                                                                                                                                                                                                                                                                                            Posts: 35
                                                                                                                                                                                                                                                                                                                                            Credit: 82,588
                                                                                                                                                                                                                                                                                                                                            RAC: 0
                                                                                                                                                                                                                                                                                                                                            Message 44411 - Posted 15 Jun 2012 15:15:18 UTC

                                                                                                                                                                                                                                                                                                                                              Last modified: 18 Jun 2012 9:17:51 UTC

                                                                                                                                                                                                                                                                                                                                              EDIT: This issue has now been resolved, and problems encountered as of 10.00 am BST on 18 June 2012 are likely to be due to the servers struggling to catch up with a weekend of uploads.

                                                                                                                                                                                                                                                                                                                                              Sorry everyone, but the data centre in which we host our servers has had a mystery network problem since Thursday 14 June 15:45 GMT.

                                                                                                                                                                                                                                                                                                                                              The problem manifests as intermittent loss of network within the data centre, which means that the project servers cannot communicate with oneanother.

                                                                                                                                                                                                                                                                                                                                              The error below is because the upload server you are trying to use cannot see the network file storage that it want to write your result to.

                                                                                                                                                                                                                                                                                                                                              I have taken cpdn-upload2.oerc and cpdn-restarts.oerc offine until this is resolved.

                                                                                                                                                                                                                                                                                                                                              I am dependent upon other people to look at this issue, and it is now gone 4 pm on the last day of University term, so I don't hold out any hope at all of a fix until Monday.

                                                                                                                                                                                                                                                                                                                                              Sorry.

                                                                                                                                                                                                                                                                                                                                              Jonathan

                                                                                                                                                                                                                                                                                                                                              Profile Dave Jackson
                                                                                                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                                                                                                              Joined: May 15 09
                                                                                                                                                                                                                                                                                                                                              Posts: 849
                                                                                                                                                                                                                                                                                                                                              Credit: 645,754
                                                                                                                                                                                                                                                                                                                                              RAC: 154
                                                                                                                                                                                                                                                                                                                                              Message 44428 - Posted 18 Jun 2012 10:29:45 UTC

                                                                                                                                                                                                                                                                                                                                                Last of 6 uploads almost finished now, so looks like all is sorted.

                                                                                                                                                                                                                                                                                                                                                Dave

                                                                                                                                                                                                                                                                                                                                                nairb
                                                                                                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                                                                                                Joined: Sep 3 04
                                                                                                                                                                                                                                                                                                                                                Posts: 23
                                                                                                                                                                                                                                                                                                                                                Credit: 587,927
                                                                                                                                                                                                                                                                                                                                                RAC: 119
                                                                                                                                                                                                                                                                                                                                                Message 44433 - Posted 18 Jun 2012 22:43:52 UTC

                                                                                                                                                                                                                                                                                                                                                  Works for me too.
                                                                                                                                                                                                                                                                                                                                                  ____________
                                                                                                                                                                                                                                                                                                                                                  <img border="0" src="http://boinc.mundayweb.com/one/stats.php?userID=343" />

                                                                                                                                                                                                                                                                                                                                                  Steve Camilleri
                                                                                                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                                                                                                  Joined: Nov 27 05
                                                                                                                                                                                                                                                                                                                                                  Posts: 4
                                                                                                                                                                                                                                                                                                                                                  Credit: 68,875
                                                                                                                                                                                                                                                                                                                                                  RAC: 22
                                                                                                                                                                                                                                                                                                                                                  Message 44434 - Posted 19 Jun 2012 19:09:55 UTC

                                                                                                                                                                                                                                                                                                                                                    still seems to be down for me:
                                                                                                                                                                                                                                                                                                                                                    19/06/2012 21:08:19 | climateprediction.net | project resumed by user
                                                                                                                                                                                                                                                                                                                                                    19/06/2012 21:08:23 | climateprediction.net | Started upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
                                                                                                                                                                                                                                                                                                                                                    19/06/2012 21:08:25 | climateprediction.net | Temporarily failed upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: can't resolve hostname
                                                                                                                                                                                                                                                                                                                                                    19/06/2012 21:08:25 | climateprediction.net | Backing off 3 hr 4 min 35 sec on upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
                                                                                                                                                                                                                                                                                                                                                    19/06/2012 21:08:28 | | Project communication failed: attempting access to reference site
                                                                                                                                                                                                                                                                                                                                                    19/06/2012 21:08:31 | | Internet access OK - project servers may be temporarily down.

                                                                                                                                                                                                                                                                                                                                                    Les Bayliss
                                                                                                                                                                                                                                                                                                                                                    Forum moderator
                                                                                                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                                                                                                    Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                                                    Posts: 5411
                                                                                                                                                                                                                                                                                                                                                    Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                                                    RAC: 3,097
                                                                                                                                                                                                                                                                                                                                                    Message 44435 - Posted 19 Jun 2012 21:32:53 UTC - in response to Message 44434.

                                                                                                                                                                                                                                                                                                                                                      The servers have been working OK for some time now, so your problem must be due to a different cause.

                                                                                                                                                                                                                                                                                                                                                      How many files are stuck?
                                                                                                                                                                                                                                                                                                                                                      Have you shut down both parts of BOINC, and then restarted them?
                                                                                                                                                                                                                                                                                                                                                      Have you tried the above, and restarted the computer while BOINC was stopped?


                                                                                                                                                                                                                                                                                                                                                      ____________
                                                                                                                                                                                                                                                                                                                                                      Backups: Here

                                                                                                                                                                                                                                                                                                                                                      Profile Thyme Lawn
                                                                                                                                                                                                                                                                                                                                                      Forum moderator
                                                                                                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                                                                                                      Joined: Aug 5 04
                                                                                                                                                                                                                                                                                                                                                      Posts: 1232
                                                                                                                                                                                                                                                                                                                                                      Credit: 10,409,271
                                                                                                                                                                                                                                                                                                                                                      RAC: 1,138
                                                                                                                                                                                                                                                                                                                                                      Message 44437 - Posted 20 Jun 2012 7:06:35 UTC - in response to Message 44434.

                                                                                                                                                                                                                                                                                                                                                        19/06/2012 21:08:25 | climateprediction.net | Temporarily failed upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: can't resolve hostname

                                                                                                                                                                                                                                                                                                                                                        That task (the second one listed for WU hadam3p_eu_4j8v_1999_1_007309876) was issued on 27th June 2011. I wonder if the upload file is being sent to a system on the oucs.ox.ac.uk network rather than the oerc.ox.ac.uk network. A redirect was in place to handle that, but if that's not working it would explain the hostname resolution failure.

                                                                                                                                                                                                                                                                                                                                                        To find out exactly what's happening create the file cc_config.xml (using a plain text editor, e.g. notepad rather than wordpad) in your BOINC data directory containing the following:

                                                                                                                                                                                                                                                                                                                                                        <cc_config>
                                                                                                                                                                                                                                                                                                                                                        <log_flags>
                                                                                                                                                                                                                                                                                                                                                        <http_debug>1</http_debug>
                                                                                                                                                                                                                                                                                                                                                        </log_flags>
                                                                                                                                                                                                                                                                                                                                                        </cc_config>

                                                                                                                                                                                                                                                                                                                                                        In BOINC Manager's advanced view click Advanced - Read config file, force an upload retry on the Transfers tab, and post the resulting HTTP debug messages. Once you've got the messages you shpuld change the <http_debug> value in the file from 1 to 0 and reload the config file to disable the debug messages; once you've done that you can delete the cc_config.xml.
                                                                                                                                                                                                                                                                                                                                                        ____________
                                                                                                                                                                                                                                                                                                                                                        "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

                                                                                                                                                                                                                                                                                                                                                        Steve Camilleri
                                                                                                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                                                                                                        Joined: Nov 27 05
                                                                                                                                                                                                                                                                                                                                                        Posts: 4
                                                                                                                                                                                                                                                                                                                                                        Credit: 68,875
                                                                                                                                                                                                                                                                                                                                                        RAC: 22
                                                                                                                                                                                                                                                                                                                                                        Message 44460 - Posted 25 Jun 2012 23:54:04 UTC - in response to Message 44437.

                                                                                                                                                                                                                                                                                                                                                          Thanks Thyme Lawn seems like you nailed it. Event Log with debug on below.
                                                                                                                                                                                                                                                                                                                                                          Question is, do I dump the WU or can it be sent somewhere useful?

                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:06 | | Re-reading cc_config.xml
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:06 | | log flags: file_xfer, sched_ops, task, http_debug
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:15 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle 'C:\Program Files\BOINC\ca-bundle.crt'
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:15 | climateprediction.net | [http] HTTP_OP::libcurl_exec(): ca-bundle set
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:15 | climateprediction.net | Started upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Connection #0 seems to be dead!
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Closing connection #0
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Connection #1 seems to be dead!
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Closing connection #1
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Could not resolve host: climateapps1.oucs.ox.ac.uk; Host not found
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:16 | climateprediction.net | [http] [ID#6] Info: Closing connection #0
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:16 | climateprediction.net | [http] HTTP error: Couldn't resolve host name
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:17 | climateprediction.net | Temporarily failed upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: can't resolve hostname
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:17 | climateprediction.net | Backing off 5 hr 37 min 15 sec on upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:32 | | Project communication failed: attempting access to reference site
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:32 | | [http] HTTP_OP::init_get(): http://www.google.com/
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:32 | | [http] HTTP_OP::libcurl_exec(): ca-bundle set
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:33 | | [http] [ID#0] Info: About to connect() to www.google.com port 80 (#0)
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:33 | | [http] [ID#0] Info: Trying 173.194.35.49...
                                                                                                                                                                                                                                                                                                                                                          26/06/2012 01:47:33 | | [http] [ID#0] Info: Connected to www.google.com (173.194.35.49) port 80 (#0)

                                                                                                                                                                                                                                                                                                                                                          Les Bayliss
                                                                                                                                                                                                                                                                                                                                                          Forum moderator
                                                                                                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                                                                                                          Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                                                          Posts: 5411
                                                                                                                                                                                                                                                                                                                                                          Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                                                          RAC: 3,097
                                                                                                                                                                                                                                                                                                                                                          Message 44461 - Posted 26 Jun 2012 0:31:22 UTC - in response to Message 44460.

                                                                                                                                                                                                                                                                                                                                                            I'll pass on a note to the project, but it's still night time there at the moment.


                                                                                                                                                                                                                                                                                                                                                            ____________
                                                                                                                                                                                                                                                                                                                                                            Backups: Here

                                                                                                                                                                                                                                                                                                                                                            Profile Jonathan Miller
                                                                                                                                                                                                                                                                                                                                                            Forum moderator
                                                                                                                                                                                                                                                                                                                                                            Project administrator
                                                                                                                                                                                                                                                                                                                                                            Project developer
                                                                                                                                                                                                                                                                                                                                                            Volunteer developer
                                                                                                                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                                                                                                                            Joined: Mar 28 11
                                                                                                                                                                                                                                                                                                                                                            Posts: 35
                                                                                                                                                                                                                                                                                                                                                            Credit: 82,588
                                                                                                                                                                                                                                                                                                                                                            RAC: 0
                                                                                                                                                                                                                                                                                                                                                            Message 44465 - Posted 26 Jun 2012 14:18:27 UTC - in response to Message 44460.

                                                                                                                                                                                                                                                                                                                                                              Last modified: 26 Jun 2012 14:21:16 UTC

                                                                                                                                                                                                                                                                                                                                                              We are no longer allowed to use the DNS names containing .OUCS
                                                                                                                                                                                                                                                                                                                                                              We have moved over to using .OERC.

                                                                                                                                                                                                                                                                                                                                                              We were graciously allowed to use the domain name for 9 months during the transition period. My requests for a transition period of a year were vetoed by managers in my department, for their own inexplicable reasons.

                                                                                                                                                                                                                                                                                                                                                              If you are trying to upload to climateapps1.oucs.ox.ac.uk then you now need

                                                                                                                                                                                                                                                                                                                                                              cpdn-restarts.oerc.ox.ac.uk

                                                                                                                                                                                                                                                                                                                                                              129.67.195.121

                                                                                                                                                                                                                                                                                                                                                              If you could convince your machine (perhaps through /etc/hosts on a linux machine) to use this address instead, then you can upload it.

                                                                                                                                                                                                                                                                                                                                                              Otherwise, I am afraid you will have to trash it.

                                                                                                                                                                                                                                                                                                                                                              Jonathan

                                                                                                                                                                                                                                                                                                                                                              Profile Thyme Lawn
                                                                                                                                                                                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                                                                                                                              Joined: Aug 5 04
                                                                                                                                                                                                                                                                                                                                                              Posts: 1232
                                                                                                                                                                                                                                                                                                                                                              Credit: 10,409,271
                                                                                                                                                                                                                                                                                                                                                              RAC: 1,138
                                                                                                                                                                                                                                                                                                                                                              Message 44466 - Posted 26 Jun 2012 20:28:42 UTC - in response to Message 44465.

                                                                                                                                                                                                                                                                                                                                                                Alternatively you could stop BOINC, edit the client_state.xml file to change every occurrence of climateapps1.oucs.ox.ac.uk to cpdn-restarts.oerc.ox.ac.uk and restart BOINC.

                                                                                                                                                                                                                                                                                                                                                                The project has disabled upload signatures (forced by changes in BOINC 7.*) so you shouldn't have to worry about leaving anything inside <signed_xml></signed_xml> blocks unchanged.
                                                                                                                                                                                                                                                                                                                                                                ____________
                                                                                                                                                                                                                                                                                                                                                                "The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

                                                                                                                                                                                                                                                                                                                                                                Digby
                                                                                                                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                                                                                                                Joined: Feb 17 06
                                                                                                                                                                                                                                                                                                                                                                Posts: 64
                                                                                                                                                                                                                                                                                                                                                                Credit: 3,717,892
                                                                                                                                                                                                                                                                                                                                                                RAC: 12
                                                                                                                                                                                                                                                                                                                                                                Message 44467 - Posted 27 Jun 2012 10:15:02 UTC

                                                                                                                                                                                                                                                                                                                                                                  Last modified: 27 Jun 2012 10:16:34 UTC

                                                                                                                                                                                                                                                                                                                                                                  Ok I am a bit confused...

                                                                                                                                                                                                                                                                                                                                                                  For a long time I had 8 models crunching. I was happy to do the 400hr models but for separate reasons on three occasions in the past 12 months I had a Windoze 7 blue screen of death and this unfortunately trashed some of those models. I was also happy to do the 100hr models because there was a reduced chance of corrupting models.

                                                                                                                                                                                                                                                                                                                                                                  Recently there have been problems uploading results and lengthy delays to get this resolved. Once this upload was achieved there was a paucity of models to download but eventually after a week I got two models.

                                                                                                                                                                                                                                                                                                                                                                  I have kept BOINC internet access open all the time to try and get new models to maximise my processor, but to no avail. This morning the 2 models I have tried to upload three times (I have to stop my internet access while this happens) but they errored:

                                                                                                                                                                                                                                                                                                                                                                  <code>27/06/2012 09:39:43 climateprediction.net Started upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:39:52 climateprediction.net Started upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:49:09 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_99fs_1959_1_007732288_2_1.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:49:09 climateprediction.net Temporarily failed upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:49:09 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:49:24 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_ct11_1999_1_007995766_1_1.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:49:24 climateprediction.net Temporarily failed upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:49:24 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:50:10 climateprediction.net Started upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:50:24 climateprediction.net Started upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:59:30 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_99fs_1959_1_007732288_2_1.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:59:30 climateprediction.net Temporarily failed upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:59:30 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:59:50 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_ct11_1999_1_007995766_1_1.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:59:50 climateprediction.net Temporarily failed upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 09:59:50 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:00:50 climateprediction.net Started upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:00:50 climateprediction.net Started upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:10:29 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_ct11_1999_1_007995766_1_1.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:10:29 climateprediction.net Temporarily failed upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:10:29 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_ct11_1999_1_007995766_1_1.zip
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:10:31 climateprediction.net [error] Error reported by file upload server: can't write file /storage/incoming/uploader/hadam3p_eu_99fs_1959_1_007732288_2_1.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:10:31 climateprediction.net Temporarily failed upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 10:10:31 climateprediction.net Backing off 1 min 0 sec on upload of hadam3p_eu_99fs_1959_1_007732288_2_1.zip </code>

                                                                                                                                                                                                                                                                                                                                                                  Ques:
                                                                                                                                                                                                                                                                                                                                                                  1) Is this temporary or do I have to make changes within the BOINC setup?
                                                                                                                                                                                                                                                                                                                                                                  2) Is it my imagination or is the climate change project a bit 'rocky' at the moment?

                                                                                                                                                                                                                                                                                                                                                                  If the answer to ques 2 is yes then that would be a shame. Is it short of money? Is there not a grant to do this research?

                                                                                                                                                                                                                                                                                                                                                                  Climate Change is a real threat and we need to have analytical ammunition to fight the corporates and politicians who want to carry on with 'business as usual' and trash the planet.

                                                                                                                                                                                                                                                                                                                                                                  Cheers

                                                                                                                                                                                                                                                                                                                                                                  BTW apologies for this broad post.

                                                                                                                                                                                                                                                                                                                                                                  MarkJ
                                                                                                                                                                                                                                                                                                                                                                  Avatar
                                                                                                                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                                                                                                                  Joined: Mar 28 09
                                                                                                                                                                                                                                                                                                                                                                  Posts: 104
                                                                                                                                                                                                                                                                                                                                                                  Credit: 5,098,940
                                                                                                                                                                                                                                                                                                                                                                  RAC: 691
                                                                                                                                                                                                                                                                                                                                                                  Message 44468 - Posted 27 Jun 2012 10:20:25 UTC

                                                                                                                                                                                                                                                                                                                                                                    Last modified: 27 Jun 2012 10:22:30 UTC

                                                                                                                                                                                                                                                                                                                                                                    I am getting the same error, except I am doing pnw models. I manages to get to 100% and then fails.

                                                                                                                                                                                                                                                                                                                                                                    27/06/2012 6:37:29 PM | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                    27/06/2012 6:37:29 PM | climateprediction.net | Temporarily failed upload of hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: transient upload error


                                                                                                                                                                                                                                                                                                                                                                    Maybe its run out of room again?
                                                                                                                                                                                                                                                                                                                                                                    ____________
                                                                                                                                                                                                                                                                                                                                                                    BOINC blog

                                                                                                                                                                                                                                                                                                                                                                    MarkJ
                                                                                                                                                                                                                                                                                                                                                                    Avatar
                                                                                                                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                                                                                                                    Joined: Mar 28 09
                                                                                                                                                                                                                                                                                                                                                                    Posts: 104
                                                                                                                                                                                                                                                                                                                                                                    Credit: 5,098,940
                                                                                                                                                                                                                                                                                                                                                                    RAC: 691
                                                                                                                                                                                                                                                                                                                                                                    Message 44469 - Posted 27 Jun 2012 10:51:59 UTC


                                                                                                                                                                                                                                                                                                                                                                      27/06/2012 8:47:03 PM | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_bcry_1998_1_008005674_0_13.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                      27/06/2012 8:47:03 PM | climateprediction.net | Temporarily failed upload of hadam3p_pnw_bcry_1998_1_008005674_0_13.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                      27/06/2012 8:47:03 PM | climateprediction.net | Backing off 3 min 47 sec on upload of hadam3p_pnw_bcry_1998_1_008005674_0_13.zip
                                                                                                                                                                                                                                                                                                                                                                      27/06/2012 8:47:11 PM | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                      27/06/2012 8:47:11 PM | climateprediction.net | Temporarily failed upload of hadam3p_pnw_b9c1_1977_1_008004972_0_13.zip: transient upload error


                                                                                                                                                                                                                                                                                                                                                                      Mine are uploading for 10-15 minutes and then failing at the 100% mark, then going into retry where they waste another 15 minutes trying to upload again and so on and so on. A total waste of bandwidth.
                                                                                                                                                                                                                                                                                                                                                                      ____________
                                                                                                                                                                                                                                                                                                                                                                      BOINC blog

                                                                                                                                                                                                                                                                                                                                                                      glaesum
                                                                                                                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                                                                                                                      Joined: Feb 24 06
                                                                                                                                                                                                                                                                                                                                                                      Posts: 47
                                                                                                                                                                                                                                                                                                                                                                      Credit: 603,024
                                                                                                                                                                                                                                                                                                                                                                      RAC: 50
                                                                                                                                                                                                                                                                                                                                                                      Message 44470 - Posted 27 Jun 2012 13:15:55 UTC - in response to Message 44469.

                                                                                                                                                                                                                                                                                                                                                                        same here on my old machine with a European model. (just noticed boinc not running on the other machine so no news from that machine until it runs for a bit).

                                                                                                                                                                                                                                                                                                                                                                        Les Bayliss
                                                                                                                                                                                                                                                                                                                                                                        Forum moderator
                                                                                                                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                                                                                                                        Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                                                                        Posts: 5411
                                                                                                                                                                                                                                                                                                                                                                        Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                                                                        RAC: 3,097
                                                                                                                                                                                                                                                                                                                                                                        Message 44471 - Posted 27 Jun 2012 15:46:28 UTC

                                                                                                                                                                                                                                                                                                                                                                          New problem people. Patience.


                                                                                                                                                                                                                                                                                                                                                                          ____________
                                                                                                                                                                                                                                                                                                                                                                          Backups: Here

                                                                                                                                                                                                                                                                                                                                                                          glaesum
                                                                                                                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                                                                                                                          Joined: Feb 24 06
                                                                                                                                                                                                                                                                                                                                                                          Posts: 47
                                                                                                                                                                                                                                                                                                                                                                          Credit: 603,024
                                                                                                                                                                                                                                                                                                                                                                          RAC: 50
                                                                                                                                                                                                                                                                                                                                                                          Message 44472 - Posted 27 Jun 2012 17:03:30 UTC - in response to Message 44471.

                                                                                                                                                                                                                                                                                                                                                                            ok :(
                                                                                                                                                                                                                                                                                                                                                                            I've fetched work for other active projects for a day or two and shut network activity while waiting for news. /pg

                                                                                                                                                                                                                                                                                                                                                                            Bob
                                                                                                                                                                                                                                                                                                                                                                            Send message
                                                                                                                                                                                                                                                                                                                                                                            Joined: Dec 20 04
                                                                                                                                                                                                                                                                                                                                                                            Posts: 6
                                                                                                                                                                                                                                                                                                                                                                            Credit: 3,744,854
                                                                                                                                                                                                                                                                                                                                                                            RAC: 203
                                                                                                                                                                                                                                                                                                                                                                            Message 44473 - Posted 27 Jun 2012 18:40:37 UTC

                                                                                                                                                                                                                                                                                                                                                                              With All due respect to the Moderator; New Problem, this thread goes back to Oct 2011, it might be new for a few but to some of us it appears to be old.

                                                                                                                                                                                                                                                                                                                                                                              Maybe it is just that the system is too big, too complicated, and that the support staff are overwhelmed by it all. To me at least it starting to appear like a giant game of whack a mole, patch this problem and the tooth paste finds another weak spot in the system.

                                                                                                                                                                                                                                                                                                                                                                              Some of us have limited bandwidth and small pipes that we have to share with others, I do not know about you, but my last mile on a really good day is 700K down and 100K up, on a typical day it is on the order of 450K down and 60K up, but that is my problem, but when there is only one service in your area you are just wood chips.

                                                                                                                                                                                                                                                                                                                                                                              It would be better for us out on the edge for you to shut down the receiving services, Post a notice to Boinc, and message to your message board that the service is shut down until the issue is fixed. That way our local Boinc client will see that service fails to connect and it will keep expand the delay time between attempts to upload as it attempts to perform the upload, I believe that is a feature was installed in the clients to keep them from swamping a server when they were restored to service.

                                                                                                                                                                                                                                                                                                                                                                              As it is now I have to manually suspend/enable the network, in order to keep this zip file upload from sapping all of what little bandwidth I do have.


                                                                                                                                                                                                                                                                                                                                                                              ____________

                                                                                                                                                                                                                                                                                                                                                                              Les Bayliss
                                                                                                                                                                                                                                                                                                                                                                              Forum moderator
                                                                                                                                                                                                                                                                                                                                                                              Send message
                                                                                                                                                                                                                                                                                                                                                                              Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                                                                              Posts: 5411
                                                                                                                                                                                                                                                                                                                                                                              Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                                                                              RAC: 3,097
                                                                                                                                                                                                                                                                                                                                                                              Message 44474 - Posted 27 Jun 2012 20:27:59 UTC

                                                                                                                                                                                                                                                                                                                                                                                By 'new', I meant since the university network failure of a couple of weeks ago.
                                                                                                                                                                                                                                                                                                                                                                                Since this thread started, there have been many occasions of upload problems, which have varied in nature.

                                                                                                                                                                                                                                                                                                                                                                                The current problem seems to be a repeat of a one from a few weeks ago; a 'disk mount' failure in the storage server that's fed by the upload server.
                                                                                                                                                                                                                                                                                                                                                                                The upload server won't know that there's no storage server until it tries to transfer the data at the end of the upload.
                                                                                                                                                                                                                                                                                                                                                                                This is being discussed, and hopefully a fix will emerge to give the uploader some feedback before it OKs the client computer to start an upload.

                                                                                                                                                                                                                                                                                                                                                                                Also, Oxford Uni has just started it's 'Long Vacation', so there's probably only a skeleton staff looking after network problems. And no night shift.

                                                                                                                                                                                                                                                                                                                                                                                Other changes have also been discussed in recent weeks. These may take a while, and be transparent to users anyway.


                                                                                                                                                                                                                                                                                                                                                                                ____________
                                                                                                                                                                                                                                                                                                                                                                                Backups: Here

                                                                                                                                                                                                                                                                                                                                                                                Steve Camilleri
                                                                                                                                                                                                                                                                                                                                                                                Send message
                                                                                                                                                                                                                                                                                                                                                                                Joined: Nov 27 05
                                                                                                                                                                                                                                                                                                                                                                                Posts: 4
                                                                                                                                                                                                                                                                                                                                                                                Credit: 68,875
                                                                                                                                                                                                                                                                                                                                                                                RAC: 22
                                                                                                                                                                                                                                                                                                                                                                                Message 44475 - Posted 27 Jun 2012 20:35:59 UTC - in response to Message 44466.

                                                                                                                                                                                                                                                                                                                                                                                  Thanks Thyme & Jonathan. I'm running a WinXP box, so decided editing the xml file might be a quicker/easier solution. This started uploading, but seems like I have similar result with failure at 100% (34Mb file takes about 15mins to up/l).
                                                                                                                                                                                                                                                                                                                                                                                  event log as follows:
                                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 22:23:20 | climateprediction.net | Started upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip
                                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 22:32:55 | climateprediction.net | [error] Error reported by file upload server: can't write file /storage/cpdn-restarts/incoming/uploader/hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: Input/output error
                                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 22:32:55 | climateprediction.net | Temporarily failed upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip: transient upload error
                                                                                                                                                                                                                                                                                                                                                                                  27/06/2012 22:32:55 | climateprediction.net | Backing off 4 hr 0 min 13 sec on upload of hadam3p_eu_4j8v_1999_1_007309876_1_13.zip

                                                                                                                                                                                                                                                                                                                                                                                  Will hang about to see if this clears in the next few days.
                                                                                                                                                                                                                                                                                                                                                                                  Thanks for your guidance.

                                                                                                                                                                                                                                                                                                                                                                                  Les Bayliss
                                                                                                                                                                                                                                                                                                                                                                                  Forum moderator
                                                                                                                                                                                                                                                                                                                                                                                  Send message
                                                                                                                                                                                                                                                                                                                                                                                  Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                                                                                  Posts: 5411
                                                                                                                                                                                                                                                                                                                                                                                  Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                                                                                  RAC: 3,097
                                                                                                                                                                                                                                                                                                                                                                                  Message 44476 - Posted 27 Jun 2012 20:41:02 UTC - in response to Message 44475.

                                                                                                                                                                                                                                                                                                                                                                                    Steve

                                                                                                                                                                                                                                                                                                                                                                                    I'm afraid that you've come up against another problem that started a few hours ago.
                                                                                                                                                                                                                                                                                                                                                                                    If you can, Suspend the BOINC network connection to save your bandwidth.


                                                                                                                                                                                                                                                                                                                                                                                    ____________
                                                                                                                                                                                                                                                                                                                                                                                    Backups: Here

                                                                                                                                                                                                                                                                                                                                                                                    Steve Camilleri
                                                                                                                                                                                                                                                                                                                                                                                    Send message
                                                                                                                                                                                                                                                                                                                                                                                    Joined: Nov 27 05
                                                                                                                                                                                                                                                                                                                                                                                    Posts: 4
                                                                                                                                                                                                                                                                                                                                                                                    Credit: 68,875
                                                                                                                                                                                                                                                                                                                                                                                    RAC: 22
                                                                                                                                                                                                                                                                                                                                                                                    Message 44477 - Posted 27 Jun 2012 20:52:58 UTC - in response to Message 44476.

                                                                                                                                                                                                                                                                                                                                                                                      yes I figured. Meanwhile I went off and had a look at BAM and it said it wasn't communicating right with CPDN, so *bright idea* I disconnected the project in my manager...and seems like that reset the project...oops. So end of thread here me thinks. Is the upload file still stored and that I can send manually or have i really blooped?
                                                                                                                                                                                                                                                                                                                                                                                      Also is there a discussion somewhere on the BAM/CPDN in these forums that I can contribute to?

                                                                                                                                                                                                                                                                                                                                                                                      Les Bayliss
                                                                                                                                                                                                                                                                                                                                                                                      Forum moderator
                                                                                                                                                                                                                                                                                                                                                                                      Send message
                                                                                                                                                                                                                                                                                                                                                                                      Joined: Sep 5 04
                                                                                                                                                                                                                                                                                                                                                                                      Posts: 5411
                                                                                                                                                                                                                                                                                                                                                                                      Credit: 9,006,998
                                                                                                                                                                                                                                                                                                                                                                                      RAC: 3,097
                                                                                                                                                                                                                                                                                                                                                                                      Message 44478 - Posted 27 Jun 2012 21:26:27 UTC - in response to Message 44477.

                                                                                                                                                                                                                                                                                                                                                                                        Yes, Reset does just that - deletes everything. This is mostly for when a project program has become corrupt, and you need to get a new copy. But all of your data files go too. :(
                                                                                                                                                                                                                                                                                                                                                                                        When I said Suspend, I meant in the BOINC manager's menu.

                                                                                                                                                                                                                                                                                                                                                                                        Re: BAM/cpdn. Not really. There have been posts in the past about problems with this combination, because for some reason, some of the BAM functions don't work with updating cpdn stuff. Changes wanted, e.g. to pref settings, need to be done manually here, and not through BAM.


                                                                                                                                                                                                                                                                                                                                                                                        ____________
                                                                                                                                                                                                                                                                                                                                                                                        Backups: Here

                                                                                                                                                                                                                                                                                                                                                                                        Profile Dave Jackson
                                                                                                                                                                                                                                                                                                                                                                                        Send message
                                                                                                                                                                                                                                                                                                                                                                                        Joined: May 15 09
                                                                                                                                                                                                                                                                                                                                                                                        Posts: 849
                                                                                                                                                                                                                                                                                                                                                                                        Credit: 645,754
                                                                                                                                                                                                                                                                                                                                                                                        RAC: 154
                                                                                                                                                                                                                                                                                                                                                                                        Message 44482 - Posted 28 Jun 2012 6:08:41 UTC - in response to Message 44478.

                                                                                                                                                                                                                                                                                                                                                                                          Upload from full ocean model went through fine - presumably going to a different server and pnws except for the 13 going to Oregon State mean I should be OK for a few days.

                                                                                                                                                                                                                                                                                                                                                                                          Richard Haselgrove
                                                                                                                                                                                                                                                                                                                                                                                          Send message
                                                                                                                                                                                                                                                                                                                                                                                          Joined: Jan 1 07
                                                                                                                                                                                                                                                                                                                                                                                          Posts: 299
                                                                                                                                                                                                                                                                                                                                                                                          Credit: 6,878,929
                                                                                                                                                                                                                                                                                                                                                                                          RAC: 13
                                                                                                                                                                                                                                                                                                                                                                                          Message 44483 - Posted 28 Jun 2012 14:30:23 UTC

                                                                                                                                                                                                                                                                                                                                                                                            _13.zip files from the regional models are uploading OK now.

                                                                                                                                                                                                                                                                                                                                                                                            Post to thread

                                                                                                                                                                                                                                                                                                                                                                                            Message boards : Number crunching : Upload Failure




                                                                                                                                                                                                                                                                                                                                                                                            Copyright © 2002-2014 climateprediction.net