Linux server not restarting properly after an image restore

[peter-h]

This relates
https://www.eevblog.com/forum/programming/mystery-process-adding-options-rotate-to-resolv-conf-file/

This is standard virtual hosting. The hosting company offers an image backup, and a restore. They don't actually say they shut the server down for this, and clearly they do not otherwise the website (or whatever) would be dead for the backup time (a few mins, or more).

I wonder if I should have shut the server down explicitly?

The backup I restored on that occassion was one of the 3am automatic ones, so the server was running.

It looks like if doing a backup which might seriously be restored, don't use an automatic one but do it manually and with the server shut down.

AIUI, Windows backups (e.g. Trueimage) do a "shadow copy" whatever that is. They would suffer from the same issues, surely?

[Nominal Animal]

As I said here, for the filesystem image to be consistent, you want the system to be quiescent. If the host is notified somehow, before a snapshot is taken, you can add a script to quesce the server; exactly how, depends on the distro and workload.  And if the host is not notified, you can't do much.  Scheduling a script to do "sync ; echo 3 > /proc/sys/vm/drop_caches ; sync" a few minutes before the expected snapshot based on wall clock time, might be useful anyhow: it just makes sure that the kernel writes all changes to filesystems to their respective devices, in a non-destructive manner.

If the image is for the whole VM, including processor state, quiescing is not needed.  All that happens on image restore is that the kernel sees a jump forward in time (which is okay), and running processes connected to network sockets (TCP, mainly) see those time out.

"Shadow copy", VM snapshots, and LVM2 snapshots, are all "copy on write": the state is recorded, but in a way that allows further modifications to be made to the state without interrupting the copy, because those modifications are stored separately.

[golden_labels]

We can at best speculate. It’s best to ask the hosting provider. There are companies, which shut down servers for this kind of maintenance.

Unless, as indicated by Nominal Animal, they backup the entire VM state (which I doubt): for a perfect backup the system has to be either shut down, suspended to disk, or all persistent filesystems need to be mounted read-only and flushed. The last option is cumbersome in modern deployments, and suspending isn’t often seen on servers. This leaves the first option: shutting down.

The reason for that is the need to complete pending output operations⁽¹⁾ and to flush the caches to storage. This is separate from the filesystem inconsistency from different storage blocks being backed up at a different time. The latter can be trivially avoided by using copy-on-write (COW), provided either by the virtual machine itself or the volume manager beneath.

If the backup may be imperfect, the answer changes. COW permits instant snapshots from a running system, with no worries about inconsistency from writes being made during backup. The effect is the same as if you cut off power. The most recently written data may be lost or inconsistent, but a reliable system should recover from that. If that is acceptable, just follow what Nominal Animal said: drop caches, sync soon before the backup to minimize damage.

⁽¹⁾ In program logic sense. A good example: logs. Log files don’t need to be closed. Only writing an entry has to be completed.

[peter-h]

I emailed the hosting company (a very good outfit I've used for many years) but their replies were not detailed enough. They basically say that if you restore over a running system you should reboot afterwards. Well, obviously!

It looks like they just make a snapshot of the HD and that's it. If you want a good backup you need to do a manual one, shut down the server, do the backup, and restart it.

So fairly obviously all the regular (automatic) backups are "probably 99% good"

[peter-h]

The most likely explanation is that some code in the hosting setup (intended to set up the IP, DHCP, etc) messes with that conf file.