Veeam on Linux

[PlainName]

I do not backup system files, because the configuration files includes the list of packages currently installed.

Presumably that applies only to things installed via the repo's package manager? What about Windows apps installed into Wine? Or user-installed stuff (say, esp-idf)? The package manager wouldn't know about those, so they would have to be reinstalled manually, wouldn't they?

Because I backup user files, I rarely restore a system exactly to how it was.  Instead, I use the opportunity to upgrade to the latest stable versions, verify key applications work,

Getting back to this, because it's a biggie and affects my entire strategy.

Recently my Nextcloud server ran out of disk space, despite user files only taking up 10% of what should be available. I had this problem some time back and tracked it down to the Nextcloud cron not being run - it needs to be hooked into the system crontab or set to run whenever anyone uses the web interface (no-one does). I fixed that with a bit of customisation, and it was fine. Until it wasn't. Turns out that some update to the OS (not Nextcloud - I am too clever to let that update without notice) vaped or changed something so that the cron update no longer worked. Indeed, the thing no longer exists.

Next, happily programming away when one day I found subversion couldn't check in to repos on the network. Took me a long time to figure out it was a silent update to Kaspersky AV that caused the issue (the primary reason for switching to Bitdefender - Kaspersky is unfixable for this). The thing to note is that in tracking it down I was able to restore before and after backups very quickly and thus be able to zero in on what the differences were. A simple config or data backup would never have allowed this since the before backup would be just as broken as the after one. Also, until I needed to restore whatever it was that fixed things, I didn't know that I needed it.

These sort of things happen and, as you say, periodically one has to figure stuff out and take the opportunity to update (or block from updating!), etc. But midway through a disaster recovery is absolutely not the time to do that. No, I am not going to fart around trying to work out why this apparently compatible install doesn't work, or breaks something. Typically, a disaster is really a disaster, and there is no need to make it a bigger one.

So that's where I am coming from. I want to know I can recover as quickly and perfectly as possible from an actual disaster, and then as a lower priority I can think of icing like how to undelete a fat finger moment, or compare this version to last weeks, etc.

[rsjsouza]

Interesting discussion; I have been involved with this and my only solution at the time was to simply partition the various development environments (multiple SDKs hosted in multiple versions of Ubuntu) in separate virtual machines.

The concept of re-creating an environment is very interesting when you are actually interested in updating the tools around it; in my case, however, I needed to keep the environments as similar as possible. I also had to distribute them across teams, so the VM was the easiest (although huge in size).

In Windows-based environments, I used to create a system image of the main hard drive with all updates and bare minimum utilities that allowed to move forward with reinstalling everything - however, these images have a drawback of not working if the underlying machine is different enough (drivers would start to fail).

Nowadays in my home system (Windows-based) I pretty much do a data backup to a NAS and also keep the original installers of the utilities I use - after I got burned, I don't trust the internet anymore to have the exact versions of what I need. I still need to trust and provision ($$$) for a cloud service.

[Nominal Animal]

I do not backup system files, because the configuration files includes the list of packages currently installed.

Presumably that applies only to things installed via the repo's package manager?

Yes.

What about Windows apps installed into Wine?

Those are under user home directory, $HOME/.wine/.

Or user-installed stuff (say, esp-idf)? The package manager wouldn't know about those, so they would have to be reinstalled manually, wouldn't they?

Yes.  I do not install such things (say, Arduino and Teensyduino, me being a Teensy user) in the system, but keep them under my own user home directory.  Their binaries are launched by symlinks or scripts in $HOME/bin/, which is the first directory in my PATH.  Same for stuff I pull from git.  In that sense, they are "user files", just like stuff installed under Wine.

I fixed that with a bit of customisation, and it was fine. Until it wasn't. Turns out that some update to the OS (not Nextcloud - I am too clever to let that update without notice) vaped or changed something so that the cron update no longer worked. Indeed, the thing no longer exists.

Hell, that should not happen.  System updates should not overwrite customized configuration files without asking, and package managers should complain loudly if trying to remove a package whose configuration directory has user-customized files.

The likeliest explanation is accidentally uninstalling anacron (and losing /etc/anacrontab), but otherwise it sounds like a serious bug.

On servers with nginx or Apache, I do use a completely custom configuration directory for the service, instead of the /etc/apache2/ or /etc/nginx/ default in Debian derivatives.  This is to ensure that system updates do not affect the service configuration at all.  It is an exception, though.

These sort of things happen and, as you say, periodically one has to figure stuff out and take the opportunity to update (or block from updating!), etc.

In my experience, very rarely; but out workflows and the tools we use are completely different, which I think explains the difference.

(Me using "disposable" virtual machines to experiment, is one way I keep my main working environment "clean", and reduces the likelihood of such events significantly.)

So that's where I am coming from. I want to know I can recover as quickly and perfectly as possible from an actual disaster, and then as a lower priority I can think of icing like how to undelete a fat finger moment, or compare this version to last weeks, etc.

It sounds like LVM snapshots would work best for you, with both / (root) and /home (home) on the same partition.

An LVM snapshot is an instant snapshot of an LVM volume.  It can be mounted as a separate device, without affecting the volume it is a snapshot of, as long as the LVM Volume Group has sufficient unused disk space for it.  (It is copy-on-write.)  You can have as many of them as you like, you can mount them and explore and compare them, and so on.

You can revert the volume to any previous snapshot with just a single command.

Using LVM2 snapshots can be mounted read-write can be made read-write, so you can make a snapshot, then restore the contents of that snapshot from offline storage, and then revert to that.

Unfortunately, I don't know of any GUI backup tools that do that.
It also only works if one tells the installer to use LVM at system installation time, and gives the LVM volume groups much more disk space than the actual partitions (so that there is always disk space for the snapshots available).

However, based on your description, I suspect Timeshift and Back in Time might work for you.  If you have a NAS running Linux you store the backups to, you might wish to try out Bacula.
(I do not use them myself, and have not installed them.  I've only seen them mentioned and read their descriptions.)

[Jr460]

Unfortunately, I don't know of any GUI backup tools that do that.
It also only works if one tells the installer to use LVM at system installation time, and gives the LVM volume groups much more disk space than the actual partitions (so that there is always disk space for the snapshots available).

Current workplace has "standards" that say you will use LVM, otherwise your system fails "checks".  Had almost the same thing at other places many years before, you will use LVM, or you will rebuild the server from scratch.

Let's see 20+ years ago, HP's Omniback, which changed names to "Data Protector", and I have no idea what it is called now, or if HPE sold it off to someone else.

Had big Oracle database server, other than boot disk, data was all on a fiber channel attached SAN.  Backup master server had connections to SAN and had a tape library.  With no scripts, it would do a split mirror backup.  When backup started, it would run an optional script of your writing on the database server, then tell the SAN device to split the mirror it had.   It would then run a post-split script on the database server.  then it would activate the VGs in read-only mode, and mount the file systems and back them up.   Once all done rip things done on the backup server and tell the SAN to update the mirror.   The SAN kept an update list, so it was never a full re-mirror.   Now we did have a pre and post split script and in the first one it flushed redo logs, checkpoints the DB, etc. and then suspends IO.  The post script let the IO fly again.   A backup in the middle of the night and the database would juts seem pause for a few seconds.

I know this not exactly what you said.   Also Veritas Netbackup will snapshot backups with LV, or SAN mirror, or filesystem snaps, whatever you want in the GUI.

[bd139]

Just a point on LVM snapshots. They’re not 100% consistent. They’re file system consistent but they are not application level consistent. Not everything fsyncs properly.

[PlainName]

Hell, that should not happen.  System updates should not overwrite customized configuration files without asking

Unexpected, yes, since OS updates are usually 'safe'. On Windows too pre-10 . May have been something I did (or, more likely, didn't do), but water under the bridge now.

(Me using "disposable" virtual machines to experiment, is one way I keep my main working environment "clean", and reduces the likelihood of such events significantly.)

That's generally what I do, and if it's a fairly safe app I'll run it under Sandboxie (a kind of chroot thing). But sometimes one slips through, and there are some normally OK apps that just won't work in a VM - FilterForge, I think, is one such (and they lost a sale because of that).

It sounds like LVM snapshots would work best for you, with both / (root) and /home (home) on the same partition.

Might do! It's not quite the same as VSS though - the system has to be effectively stopped to take the snapshot, and you need at least as much snapshot space as the volume you're snapshotting. It has potential, though.

Unfortunately, I don't know of any GUI backup tools that do that.

That's a rather big fly in the ointment. Still, it might be both doable and worth it

I suspect Timeshift and Back in Time might work for you.

Worth taking a closer browse after a quick look, thanks. I'd briefly looked at bacula before, but it is again pushing the problem further up the road. And the interesting bits (snapshotting) are only in the enterprise version, for which they are too embarrassed to say upfront how much it costs.

[PlainName]

Just a point on LVM snapshots. They’re not 100% consistent. They’re file system consistent but they are not application level consistent. Not everything fsyncs properly.

Indeed, and when it comes to stuff like databases they might not even be current. Applies to any kind of snapshotting (so not just LVM but also my Windows stuff).

[bitwelder]

For bare metal recovery and various levels of backup on my Linux machines generally I use a combination of:
- Clonezilla (to take once maybe a year a cold image of the disks, and e.g. NFS it to my NAS). I use it also with encrypted disks, Windows images, etc.
- Timeshift (to save daily snapshots of the live filesystems)
- plus additional rsync, syncthing and similar (for frequently changing work directories)

The tools are all free

[Benta]

On my (several) Lubuntu laptops, I use two types of backup. My machines all have a main user (admin/sudo) of course, but I often also add guest accounts for friends needing a laptop temporarily.

For my main account, I use:
1: Timeshift for system backup. This backs up all system files, plus the hidden files/directories in /home. I use this very often when I'm experimenting with new installations, or mucking around with system configurations. It does NOT back up /home generally.
2: BackInTime for /home backups. This backs up all user files except the hidden files/directories (I've added a rule to exclude those).

This split works well for me. The guest accounts can also use BackInTime freely for their own stuff. Timeshift is exclusively for me.

Both are incremental and take up little space except for the initial backup.

[PlainName]

Thanks, both. The thing that gets me about these is the single backup - you have your original data, and your backup, and that's it. So for me I would need to duplicate that backup destination somehow, which means another job.

Thinking it through, I couldn't use an external drive for the destination (which is what I do now) because I could only use one drive. Yes, I could then copy that, but that's an extra manual step which means it's not going to happen often. Or I could have a network drive as destination, and an automated copying system for that. But now we've added at least one more box, plus the automated whatever now needs to be backed up, etc.

One thing I forgot to ask: have you used your processes in anger? If so, often or just a one-off?

[Foxxz]

I evaluated Veeam in a corporate environment close to 10 years ago. The security of the product was abysmal. It was sending VMWare creds in plaintext to the Veeam agents to backup VMs. And it seemed like the agents that ran on individual systems at the time had no security at all allowing full RW access to the filesystem. There were CVEs for the latter that I remember. Take a look here to see what current security issues it has
https://www.cvedetails.com/vulnerability-list/vendor_id-15994/Veeam.html

[PlainName]

They do seem to have dropped the ball a couple of times.