Each model has a lot of files open that need to be saved.
If a model is in the process of check pointing at the time of a crash, then what's on the disk will be part old save, and part new.
When it tries to start up again, the files don't match, so the model can't start.

The more models that get crammed into the newer computers with their large number of processors, the more likely it will be that there will be constant check pointing.

Each model has a lot of files open that need to be saved.
If a model is in the process of check pointing at the time of a crash, then what's on the disk will be part old save, and part new.
When it tries to start up again, the files don't match, so the model can't start.

The more models that get crammed into the newer computers with their large number of processors, the more likely it will be that there will be constant check pointing.

I can see how that would fail, but that's because it's a very bad design. The next checkpoint should be saved to a 2nd file, then the old one deleted, then the new one renamed if necessary. Almost every program does this, including say a word processor. When you click save, you can see a temporary file appearing for a fraction of a second. If the power is cut off while that is happening, the original file is not destroyed.

Same happened with two on a working machine, which I rebooted cleanly. Should Boinc not gracefully shut down running CPDN tasks itself?

You are right, BOINC should restart the task from the last checkpoint reached. In the past, my memory is of this being a bigger problem with Linux tasks but I haven't had a problem with it recently, even when I have updated the Linux kernel which requires a reboot. My experience a few years ago was that a kernel change combined with a reboot greatly increased the chances of tasks crashing.

To minimise the chances of tasks crashing, I suspend tasks individually, exit BOINC manager and client before rebooting. On restarting, I resume tasks one at a time, allowing a couple of minutes between resuming individual tasks. I don't know if on the most recent task types this makes any difference but it used to. I don't know what happens with other projects. For a fair comparison you might need to look at something like LHC@home which like CPDN has a large number of files open at once, all of which need closing down by BOINC when exiting.

If you reboot without exiting BOINC first, again in theory tasks should resume from previous checkpoint but experience tells me that doing so dramatically increases the chances of failure though last time I had a power failure, all tasks survived.

I am not really sure if this is a BOINC issue or a CPDN one which makes sorting it out difficult.

Surely Windows waits for Boinc to close all files first? In the case of LHC, I get an impatient warning from Windows saying Virtualbox has "active connections". I can click shut down anyway, or cancel. I click cancel, then watch in the task manager until I can see no processing, disk activity, or network activity is happening, then shut down again. Or sometimes I remember to close Boinc first, wait until zero activity, then shut down Windows. Windows still claims Virtualbox has "active connections" which I just ignore. I'm not going to go into Virtualbox itself and mess around to stop stuff. LHC tasks seem to be quite robust though, I've only ever seen them go wrong if the system crashes. And once when a hard disk was failing, although not producing errors, it was going very slowly, so the LHC tasks were giving up waiting for disk access and saying "computation error" in Boinc. It took someone in LHC to tell me they'd seen a certain error in the log file so I knew the disk was too old and tired!

I can't have power problems on my main machine as it's protected by a UPS, but I haven't gone to the expense of a larger one for the 6 Boinc-only machines. If the UPS goes onto battery mode, Boinc immediately suspends to make the battery last longer, and the monitors are turned off. If the battery is almost empty, the PC hibernates, so any tasks should resume where they left off. Although I rarely get proper powercuts, I do get the odd fraction of a second powercut. And the voltage varies from 241 to 256V (it should be 230V), so the transformer in the UPS levels that off. More to protect the house lighting actually, since I had a lot of LED lights fail with bad voltages. And because once a 0.5 second powercut caused a bad corruption of the system disk and I lost a few documents. The cheapest thing to do is buy a second hand UPS with a busted battery, and replace it by connecting it to two (if it needs 24V) or more large car batteries. Much cheaper than the sealed ones that come with it, and you get a lot of run time!

Whatever the fine details are, it works.
I tested this a few days ago with a new test model, and it worked perfectly.

So it's your computers and the way that you use them.
And any one else silently having the same problem.

Just something that will have to be lived with I guess.

Whatever the fine details are, it works.
I tested this a few days ago with a new test model, and it worked perfectly.

So it's your computers and the way that you use them.
And any one else silently having the same problem.

Just something that will have to be lived with I guess.

I can't think of anything unusual that would cause this. It's either a computer crash or power failure while CPDN is running, or me simply clicking restart in Windows without closing Boinc first. I would imagine most folk experience these things. It's a small chance it will happen, and usually only 1 or 2 of the running tasks break, but larger if it was a hard crash/power off instead of just rebooting.

If you deliberately power off a Windows machine to simulate a powercut for example, I'm sure after a few times, some of the tasks would cause a computation error.

If you deliberately power off a Windows machine to simulate a powercut for example, I'm sure after a few times, some of the tasks would cause a computation error.

I have seen hard resets i.e. power outage or by hitting the power switch crash tasks on more projects than just CPDN. However, while not having coded for decades and never anything as complex as the climate models or BOINC itself, I still say that it should be possible to write the code so that even then it can resume after a checkpoint. I accept that that might mean more disk usage during computation. The problem is that most of the code used in the executeable files for CPDN is over a million lines of fortran (before compiling) that comes from the met office and is used under a license that does not allow the sort of playing with the code that might be needed to resolve the problem. It will be interesting to see what happens with OpenIFS eventually as that code is open source.

---

Please do not private message myself or other moderators for help. This limits the number of people who are able to help and deprives others who may benefit from the answer.

If you deliberately power off a Windows machine to simulate a powercut for example, I'm sure after a few times, some of the tasks would cause a computation error.

I have seen hard resets i.e. power outage or by hitting the power switch crash tasks on more projects than just CPDN. However, while not having coded for decades and never anything as complex as the climate models or BOINC itself, I still say that it should be possible to write the code so that even then it can resume after a checkpoint. I accept that that might mean more disk usage during computation. The problem is that most of the code used in the executeable files for CPDN is over a million lines of fortran (before compiling) that comes from the met office and is used under a license that does not allow the sort of playing with the code that might be needed to resolve the problem. It will be interesting to see what happens with OpenIFS eventually as that code is open source.

A million! Ok, we should be happy it works at all!

A million! Ok, we should be happy it works at all!

Yes, when thinking about what we would like the code to be like, I am reminded of the story of someone in a remote rural village asking for directions. The local thinks for a minute before replying,

"Well if I wanted to go there, I wouldn't want to be starting from here."

---

Please do not private message myself or other moderators for help. This limits the number of people who are able to help and deprives others who may benefit from the answer.

A million! Ok, we should be happy it works at all!

Yes, when thinking about what we would like the code to be like, I am reminded of the story of someone in a remote rural village asking for directions. The local thinks for a minute before replying,

"Well if I wanted to go there, I wouldn't want to be starting from here."

So in that story, I'm the innocent person asking for directions, and the programmers are the ones giving deliberately obtuse answers to evade the problem? :-P

And is that from a Monty Python sketch? I've heard it before somewhere.

The programmers are in the UK Met Office, who aren't involved in this project.
So you're just making things up.

And no, the quote is Not Monty Python. It's much older than that.

And no, the quote is Not Monty Python. It's much older than that.

Earliest reference I found to it being in print with a cursory web search was 1924 but it may be quite a bit older even than that.

---

Please do not private message myself or other moderators for help. This limits the number of people who are able to help and deprives others who may benefit from the answer.

Patience, I suppose.

The programmers are in the UK Met Office, who aren't involved in this project.
So you're just making things up.

And no, the quote is Not Monty Python. It's much older than that.

I'm not making anything up, I didn't say who I was having a go at. But I have no idea why you used the quote.

And no, the quote is Not Monty Python. It's much older than that.

Earliest reference I found to it being in print with a cursory web search was 1924 but it may be quite a bit older even than that.

I probably heard someone else repeating it, a stand up comedian or something.

Thank you guys who overclock or have failed tasks. I am getting tasks with 2 or 3 at the end and i run them okay.

---

Thank you guys who overclock or have failed tasks. I am getting tasks with 2 or 3 at the end and i run them okay.

Some might be mine. In which case you owe me a pint.

For Linux users, there may be another new model type on its way. HadSM4. These are similar to HadAM4 but using a slab model of the ocean rather than surface temperatures. First six month runs of these seem to not have problems in testing. (Five have completed. My five have about four hours to go.) I would guess there will be an official notice about these closer to them appearing on the main site. Time scale for these is at present anyone's guess but as Les said in another thread,

"Don't forget to keep breathing."

---

Please do not private message myself or other moderators for help. This limits the number of people who are able to help and deprives others who may benefit from the answer.

There are more UK Met Office HadCM3 shorts available.
I will leave them for the Windows users.

There are more UK Met Office HadCM3 shorts available.
I will leave them for the Windows users.

Well that didn't last, I just told 6 machines to grab some, and none are left.

There are more UK Met Office HadCM3 shorts available.
I will leave them for the Windows users.

Well that didn't last, I just told 6 machines to grab some, and none are left.

... I'm not sure there were any, as there haven't been any additions to the work unit list.