Okay, makes sense! Thanks again :)

Well, uploads are failing again :(
But this time the reason seems to be clear ;)

28.02.2012 12:11:07 | climateprediction.net | Started upload of hadam3p_eu_apqs_1987_1_007816633_0_1.zip
28.02.2012 12:11:08 | climateprediction.net | [error] Error reported by file upload server: Server is out of disk space
28.02.2012 12:11:08 | climateprediction.net | Temporarily failed upload of hadam3p_eu_apqs_1987_1_007816633_0_1.zip: transient upload error

And upload2.oerc is down according to server status page.

Just for the record, I had the same message a couple of hours before it was reported here, and notified the project people.
Apparently the server had filled up without triggering the alarms. It's now been shut down from getting more data, and what's on it is being transferred to a storage server. But this is done over the university network, which is a slow process.

Keep an eye on the server status page.

---

Backups: Here

Network activity suspended - As it is going to be several days till I complete a task, I will wait till the backlog of those who don't visit the boards is finished before resuming.

Left it overnight after server status page showed all servers back on line. Still getting this when hadam3peu tries to upload zip files.
Thu 01 Mar 2012 10:07:52 GMT | climateprediction.net | Started upload of hadam3p_eu_ap6s_1992_1_007815913_0_8.zip
Thu 01 Mar 2012 10:07:52 GMT | climateprediction.net | Started upload of hadam3p_eu_ap6s_1992_1_007815913_0_9.zip
Thu 01 Mar 2012 10:07:53 GMT | climateprediction.net | [error] Error reported by file upload server: can't open log file '../log_aforgomon/file_upload_handler.log' (errno: 9)
Thu 01 Mar 2012 10:07:53 GMT | climateprediction.net | [error] Error reported by file upload server: can't open log file '../log_aforgomon/file_upload_handler.log' (errno: 9)
Thu 01 Mar 2012 10:07:53 GMT | climateprediction.net | Temporarily failed upload of hadam3p_eu_ap6s_1992_1_007815913_0_8.zip: transient upload error
Thu 01 Mar 2012 10:07:53 GMT | climateprediction.net | Backing off 3 hr 46 min 36 sec on upload of hadam3p_eu_ap6s_1992_1_007815913_0_8.zip
Thu 01 Mar 2012 10:07:53 GMT | climateprediction.net | Temporarily failed upload of hadam3p_eu_ap6s_1992_1_007815913_0_9.zip: transient upload error
Thu 01 Mar 2012 10:07:53 GMT | climateprediction.net | Backing off 37 min 16 sec on upload of hadam3p_eu_ap6s_1992_1_007815913_0_9.zip
Thu 01 Mar 2012 10:08:01 GMT | | Suspending network activity - user request
Currently I have 4 in the queue which means I suspect some of those with fast boxes with more cores have a lot more!

Dave

The project team are investigating the problem.

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

Thanks

Dave

Uploads to cpdnupload2.oerc should be okay now, but multiple attempts might be required until the server catches up with the backlog (my first upload after the fix took 4 attempts).

---

"The ultimate test of a moral society is the kind of world that it leaves to its children." - Dietrich Bonhoeffer

Thanks all,
last 2 going through now. Even this morning, it took more than one attempt to get them all through.

Dave

Hi everyone,

We have had a number of problems with our upload servers over the last few weeks.
I believe I have addressed all of the operational issues that prevented file uploads, and the machines are now working to clear the considerable backlog.

I apologise for the inconvenience that this has caused.


Jonathan

CPDN Sys-Admin

Cannot upload a *.zip file:

27-May-2012 02:00:31 [climateprediction.net] Started upload of hadam3p_eu_cppn_2002_1_007991468_0_2.zip
27-May-2012 02:04:58 [---] Project communication failed: attempting access to reference site
27-May-2012 02:04:59 [climateprediction.net] Temporarily failed upload of hadam3p_eu_cppn_2002_1_007991468_0_2.zip: HTTP error
27-May-2012 02:04:59 [climateprediction.net] Backing off 3 hr 18 min 33 sec on upload of hadam3p_eu_cppn_2002_1_007991468_0_2.zip
27-May-2012 02:05:00 [---] Internet access OK - project servers may be temporarily down.

Tullio

---

Hi, tullio,

Please note server status, linked near the bottom in the blue block on the left. (Must be the weekend!)

---

"We have met the enemy and he is us." -- Pogo
Greetings from coastal Washington state, the scenic US Pacific Northwest.

Hi, tullio,

Please note server status, linked near the bottom in the blue block on the left. (Must be the weekend!)

But uploader1.atm is running! That should apply to the hadam3p_eu models.Cheers.
Tullio

---

Hi Tulio,
If memory serves me correctly not all the zip files from the EU models go to the same server. Unfortunately, I can never remember which ones go where but I think 11,12, and 13 go somewhere else from the rest.

Dave.

zip 13 for all 3 types contains the data needed to create the next data set in the series, and goes to the restart server.

But Dave's comment has triggered an old memory about the other zips being divided into odd and even numbers for 2 different servers to spread the load.
It's possible that this is still the way it works.

---

Backups: Here

None eu files can upload from here -- the pnw's go when they get a chance after the backlog of eu's .
Will be at least 5 days if read postings right.
Patience again.
But a bit of a grump -- how little budget for hardware is there?
Maybe contribute a lousy 20-50K USD?
Meanwhile I'll read Ulysses and compare to the free audio version .
"Universally that person's acumen is esteemed very little perceptive -- "
Patience.

---

Apparently it's a power problem. More later.

---

Backups: Here

Looks like it's now been fixed. 2 of my zip 13s are currently uploading.

---

Backups: Here

"Looks like it's now been fixed. 2 of my zip 13s are currently uploading."

Sorry Les, power might be back but there is still some fixing to do.

Dave

I am getting the following error on uploading - zip13 files show as going through on transfers tab and get to 100% but then I get the project backoff for 1 hour message. I also get an upload failed message in the event log.
Mon 28 May 2012 11:28:55 BST | climateprediction.net | Started upload of hadam3p_pnw_byb0_1979_1_007930763_1_13.zip
Mon 28 May 2012 11:36:14 BST | climateprediction.net | Started upload of hadam3p_pnw_bbny_1997_1_007901421_1_13.zip
Mon 28 May 2012 11:49:03 BST | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_byb0_1979_1_007930763_1_13.zip: No such file or directory
Mon 28 May 2012 11:49:03 BST | climateprediction.net | Temporarily failed upload of hadam3p_pnw_byb0_1979_1_007930763_1_13.zip: transient upload error
Mon 28 May 2012 11:49:03 BST | climateprediction.net | Backing off 56 min 2 sec on upload of hadam3p_pnw_byb0_1979_1_007930763_1_13.zip
Mon 28 May 2012 11:56:07 BST | climateprediction.net | [error] Error reported by file upload server: can't open file /storage/cpdn-restarts/incoming/uploader/hadam3p_pnw_bbny_1997_1_007901421_1_13.zip: No such file or directory
Mon 28 May 2012 11:56:07 BST | climateprediction.net | Temporarily failed upload of hadam3p_pnw_bbny_1997_1_007901421_1_13.zip: transient upload error