Making backups using rsync

[Karel]

For creating backups on external harddrives I use a shell script, something like:

rsync -vrth --progress --delete /data/src_dir /var/run/media/harddrive/dest_dir

The only disadvantage is that, if I accidently erase a file and make a backup, the file is gone forever.
Because of that I'm thinking about to make incremental backups using tar --create --listed-incremental
and put it together with rsync in a shell script.

Any thoughts, ideas?

[Twoflower]

If you want to keep the deleted files, the --backup and --backup-dir options are your friend.

[mrflibble]

Use rsnapshot. That uses rsync + LVM snapshots for its backups, and yes, it is incremental. So when you remove file and then run rsnapshot once, the file is still avilable in an older part of incremental backup.

[sleemanj]

Use borg backup, way nicer, incremental, de-duped, compressed and mountable backups with optional encryption.  The executable is statically built stand-alone so you can just drop that on your backup device to have it available when the shit hits the fan - https://www.borgbackup.org/

Alternative you can use rsnapshot, which uses rsync and creates archives based around hardlinks - https://rsnapshot.org/

[ataradov]

+1 for borg. Works perfectly. And if you want off-site backups, rsync.net has special pricing for borg hosting accounts. It is very cheap.

[RoGeorge]

Use borg backup, way nicer, incremental, de-duped, compressed and mountable backups with optional encryption.  The executable is statically built stand-alone so you can just drop that on your backup device to have it available when the shit hits the fan - https://www.borgbackup.org/

There is a dd example but It's not clear from the demo videos, can Borg backup an entire OS disk (Linux) while keeping the enumerated advantages?

[ataradov]

No, I don't think Borg is for backing up the whole image. It will save files, of course, so to that extent it will work. But it will not give you an image that you can simply write into a partition and have a bootable computer.

It is really more for user data backup. At least that's the way I understand it. May be there is some way to make a full OS backups. I have not investigated it much.

[bd139]

Borg has some problems I found on large datasets. At around the 1TiB mark on lots of small files it shit a gigantic brick one day irrecoverably.

I switched to rdiff-backup which is an extrapolation of rsync that does extremely fast incremental backups so you can get point in time recovery. Also the latest increment is always a copy of the source filesystem so you can restore with "cp" if the tools ever fall over worst case. https://github.com/rdiff-backup/rdiff-backup

Also don't back the entire machine up. Total waste of space and time. Only things that are valuable are in /etc /var/ /home in theory. I tend to just dump the machine package list out and /home and that's it. If it's a server box it is deployed from ansible therefore you can recreate the "desired state" with that

[sleemanj]

I switched to rdiff-backup

I used to use rdiff-backup, it's also good.

For my (somewhat peculiar with lots of duplicate files on my system for reasons) usage pattern though the de-dup of borg is a god-send...

Code: [Select]

                       Original size      Compressed size    Deduplicated size
This archive:              428.52 GB            240.07 GB            483.39 MB
All archives:                5.73 TB              3.19 TB            201.89 GB

Yeah that's 5.73 TB of backups in 201 GB of space total, and the most recent compressed and de-duped to a 10x reduction.

I use a separate borg repo for my vmware image/disks backups and it's similarly impressive on the de-dup...

Code: [Select]

                       Original size      Compressed size    Deduplicated size                                                                                                                             
This archive:              572.59 GB            281.86 GB            638.64 MB                                                                                                                             
All archives:               11.36 TB              5.59 TB            233.19 GB                                                                                                                             


[ve7xen]

I used to use rdiff-backup.

I have now fully switched to using a BTRFS filesystem and btrbk. Incremental, copy-on-write, any version immediately accessible without any 'recovery' process, transfers are extremely fast & small since there is no comparison process, the journal just needs to be run forward. It has both online and offline dedupe if you wanted it, but using COW reflinks or old school hard links probably makes more sense than block-level copies in the first place. Dedupe is more for VM storage, for example, where you can't alter the semantics but many machines will have identical blocks.

[brucehoult]

+1 for borg. Works perfectly. And if you want off-site backups, rsync.net has special pricing for borg hosting accounts. It is very cheap.

OK, borg looks interesting.

It seems remote backups are supported using:
1) ssh to a remote system with borg installed. (fastest)
2) sshfs or similar
3) I guess anything else that looks locally mounted e.g. DropBox, Google Drive, iCloud Drive.

Question: can you simply copy whole repos from one place to another? For example, make a repo locally, mass copy it to Google Drive, and then do incremental backups to the Google Drive. Is the speed reasonable doing so?

[ataradov]

I have not tried it, but it should be possible. It just creates a bunch of files and there is nothing that ties them to the target system.

[nctnico]

For creating backups on external harddrives I use a shell script, something like:

rsync -vrth --progress --delete /data/src_dir /var/run/media/harddrive/dest_dir

The only disadvantage is that, if I accidently erase a file and make a backup, the file is gone forever.
Because of that I'm thinking about to make incremental backups using tar --create --listed-incremental
and put it together with rsync in a shell script.

Any thoughts, ideas?

Remove the --delete from the line above and files won't be deleted. The downside is that you have to prune the backup every once in a while.

[Karel]

For creating backups on external harddrives I use a shell script, something like:

rsync -vrth --progress --delete /data/src_dir /var/run/media/harddrive/dest_dir

The only disadvantage is that, if I accidently erase a file and make a backup, the file is gone forever.
Because of that I'm thinking about to make incremental backups using tar --create --listed-incremental
and put it together with rsync in a shell script.

Any thoughts, ideas?

Remove the --delete from the line above and files won't be deleted. The downside is that you have to prune the backup every once in a while.

Agreed but I will still loose data in case the content of a file has changed, no?

[nctnico]

True.

[kripton2035]

this script mimics the mac time machine, which does incremential backups
https://github.com/samdoran/rsync-time-machine

[brucehoult]

this script mimics the mac time machine, which does incremential backups
https://github.com/samdoran/rsync-time-machine

Time Machine has a big advantage, because Apple modified their file system to support hard linking directories.

With other file systems you either have to recreate the entire directory tree in each incremental backup and hard link only the unchanged files, or else symlink the unchanged directories, which complicates selective pruning of backups later.

Or store the directory tree in some other way such as using a normal file to store each directory as git does. Or a database.

[bitman]

For creating backups on external harddrives I use a shell script, something like:

rsync -vrth --progress --delete /data/src_dir /var/run/media/harddrive/dest_dir

The only disadvantage is that, if I accidently erase a file and make a backup, the file is gone forever.
Because of that I'm thinking about to make incremental backups using tar --create --listed-incremental
and put it together with rsync in a shell script.

Any thoughts, ideas?

Remove the --delete from the line above and files won't be deleted. The downside is that you have to prune the backup every once in a while.

If you want to restore and have the files you had when you backed up, --delete is absolutely required.  A file in the wrong location can easily change behavior of software, so if the intention with the backup is to ever do a full restore, --delete is a must.  There are other options if you want to put those deleted files in a safe place.

[bd139]

Yes like rdiff-backup 

[bitman]

For creating backups on external harddrives I use a shell script, something like:

rsync -vrth --progress --delete /data/src_dir /var/run/media/harddrive/dest_dir

The only disadvantage is that, if I accidently erase a file and make a backup, the file is gone forever.
Because of that I'm thinking about to make incremental backups using tar --create --listed-incremental
and put it together with rsync in a shell script.

Any thoughts, ideas?

If you want full backup and features, look at deja-dup - it too can use rsync but a lot of other protocols too. What's great is that it keeps indexes of files, over multiple backups and if you're using Nautilus in Gnome it's connected directly with the backup metadata allowing you to restore files/directories with a single click in the gui.  A nice feature is that it will use cloud services as well as a local server like you do with scp/rsync.

rsync is great - very flexible, but it's solely focused on the latest backup. So when you run it, you overwrite what the backup before did. That may be exactly what you want, but do realize that a crash during backup will make your destination invalid (old/new files in a big mess). Then again, when the backup fails typically you can just re-run it once you're resolved the issue that made it crash.

Another note - for stuff I work on - writing code, documentation etc. I use 'git' as my backup. It's much better at keeping track of things and even allows easy sharing/contributions to happen.  But I do have deja-dup running daily to deal with everything else.

[Karel]

What about making incremental backups using Tar? Anybody tried that?

https://www.gnu.org/software/tar/manual/html_node/Incremental-Dumps.html

https://www.unixmen.com/performing-incremental-backups-using-tar/

https://www.theurbanpenguin.com/incremental-backups-with-tar/

[sleemanj]

I incorporate an incremental tar backup function into the websites I develop (which may be deployed on places I don't control and may or may not have good server backups).  Now, if anybody actually uses it...

One thing that did surprise me when making it is... tar is not nearly as standardised or universal as you think.  I found that the incremental archives especially, some tar implementations were ok, others not so much, some were ok with one format and not another or whatever.

Rather than trying to find a good middle-ground I just specified in the backup restoration notes intended for future me or developers, "use gnu tar".

For that reason I would use incremental tar only and exactly for this... to allow easy downloads of incremental archives to "admins", it's old and crusty for anything else.

[bd139]

tar is the reason so many alternatives exist. Literally the only stable functions are the POSIX defined ones and most of them are poorly implemented.

[Karel]

Considering that I'm using Linux & GNU tar, I could give it a try.
I'm thinking to put it into a shell script together with rsync.

[Zucca]

I use 7z, it has tons of options (incremental or not) and it save space.
At every workday end I fire a .bat which locks the pc, makes the 7z backup and then turn off the PC.

7z a %_backup_path_final_name%.7z -p%_psw% -w%temp% -mhe -ssw -bb -spf2 @%_cfiles_path% >%_log_file% 2>&1

It's surely not the best solution for everybody, but the best for me.

PS: My trusty 7z backup file saved my ass a couple of times already.

PSS: Oh shit I have to sign up ASAP to some cloud backup service, probably going to blackblaze for future FreeNAS integration.