Are your backups up to date?

[amspire]

I'm using BackBlaze now for HD backups

Looks like Backblaze have really good prices. I do worry about how long a company can last when it has data storage costs 1/4 of its competitors. At that price, they would probably have to have disks last at least two years before they even have a chance of making money. Don't know how they do it especially if they need to have redundancy. As long as they keep up the prices though, they are a fabulous deal.

Just watch out for a sudden price jump in the future. It can be traumatic when you have Terabytes of cloud storage and suddenly the price skyrockets. It has happened before and often there is no option but to pay. Microsoft used to offer unlimited cloud storage with office and some people did have Terabytes stored. Then they suddenly introduce the 1T free cap for Office 365 and so the people had the choice of paying for the extra storage or dumping the cloud files.

For my backup, I use a number of 2 disk RAID1 NAS boxes. With 10T NAS drives available, the 2 disk boxes will be all I need for a long time. I sync the PC's to the NAS boxes with rsync, and have a scheduled daily script file to rsync the different of-site NAS boxes. With the exception of video files and Altera/Xilinx installation programs, I seem to be able to comfortably sync remote NAS boxes via ADSL2+ connections overnight. In a couple of years, I hopefully will have enough bandwidth to sync everything. The NAS boxes have BTRFS daily snapshots, so I can access the backup from previous days at any site. The BTRFS snapshots provide excellent protection from crytpoviruses. I do a weekly scrub on the NAS boxes to eliminate bitrot in the mirrored drives.

The basic principle is for a reliable backup, you want at least two copies locally (the mirrored disk) and at least one off-site backup. You also have to remember that a backup is only a backup as long as the original files are still on your PC. If you delete the files from your PC, then you should make an extra archival copy. Otherwise a "backup" ends up becoming the primary storage for the files, and it then it needs a backup.

[bd139]

I've got about 200Gb of data and I'm using the following:

1. time machine offline on 512 gig SSD for lazy local backups
2. rdiff-backup hourly to an HP microserver with FreeBSD + ZFS mirror on two 1 TiB disks.
3. rsync that to an EBS volume on an AWS instance nightly.
4. Everything major (pictures/documents/irrecoverable stuff) is synced to iCloud as well and is on my phone handset.

I've just set the whole stack above up this week. All automated.

AWS + EBS is quite expensive but cheaper than the other options. You don't get screwed hard until you actually have to do a restore - costs money to pull 200Gb out of AWS.

[Mr. Scram]

I've got about 200Gb of data and I'm using the following:

1. time machine offline on 512 gig SSD for lazy local backups
2. rdiff-backup hourly to an HP microserver with FreeBSD + ZFS mirror on two 1 TiB disks.
3. rsync that to an EBS volume on an AWS instance nightly.
4. Everything major (pictures/documents/irrecoverable stuff) is synced to iCloud as well and is on my phone handset.

I've just set the whole stack above up this week. All automated.

AWS + EBS is quite expensive but cheaper than the other options. You don't get screwed hard until you actually have to do a restore - costs money to pull 200Gb out of AWS.

Is your system able to deal with creeping corruption? All backups seem fairly immediate. That can be an issue when something's been encrypting your files on the down low, or corrupt memory causes random and intermittent corruption of files.

[bd139]

rdiff-backup is “permanently incremental” I.e you can roll back to any point in time. Checksums everything. If anything is corrupted or you get cryptolocker then it will diff the corruption as a file increment so you can go backwards in time until the file is fixed. Same with time machine.

Corruption is handled via multiple target media as well.

We use it for production backups for 15 years now. ZFS for about 5 years. It’s amazingly solid. Trick is buy good disks and use good operating systems.

[Mr. Scram]

rdiff-backup is “permanently incremental” I.e you can roll back to any point in time. Checksums everything. If anything is corrupted or you get cryptolocker then it will diff the corruption as a file increment so you can go backwards in time until the file is fixed. Same with time machine.

Corruption is handled via multiple target media as well.

We use it for production backups for 15 years now. ZFS for about 5 years. It’s amazingly solid. Trick is buy good disks and use good operating systems.

I assume you have a way of preventing malware from eating up all your incrementals and spreading throughout your network? Do you have something that's disconnected from the rest of the network somehow?

[bd139]

Yes time machine backups are offline and two drives are cycled.

[ChrisLX200]

I have about 70TB diskspace spread through my network, NAS and sundry external drives (built using decent drives and ICYBOX enclosures). With astro-imaging I can generate ~10GB data per night, about half of that is all-sky camera recordings which is not critical, the rest is CCD raw images and processed images which I want to keep safe. The QNAP NAS runs Linux and the external drive connected to that is fairly well isolated from common virus attack - but it is still scanned of course.

[amspire]

I assume you have a way of preventing malware from eating up all your incrementals and spreading throughout your network? Do you have something that's disconnected from the rest of the network somehow?

The good thing about using a ZFS or BTRFS NAS is that snapshots are instant to create and become read only. The snapshots are part of the filesystem - not files on a drive. I have had a cryptovirus start to infect a NAS box at a company I sometimes help, and in just an hour, it had managed to encrypt about 20G of files on the NAS. The snapshots from the previous day were all fine, but since then, I have disabled all SMB Windows network file sharing to the NAS boxes. Cryptoviruses do know how to find and attack network shares. They also target and attached USB drives, so a backup USB drive permanently plugged into a PC is pretty useless. It is much more valuable to attack and encrypt files on a server then a workstation.

Also when I sync the local NAS box to a offsite one, I always initiate the connection from the offsite box - there is no information on the local NAS box that a virus could use to initiate a connection to the remote one. Any connection is always on terms dictated by the offsite NAS.

[Red Squirrel]

Raid is not a backup, but I use Linux md raid on my file server for all mass data storage so don't have to worry about drive failures or at very least not a single one.  Beauty is not having to lose any productivity, no need to copy data anywhere or restore anything.  Just pop another drive in and I'm good to go. (or wait, does not matter, but I like to do it asap)

For actual backups I have various rsync jobs that copy data to one of the raid arrays that is dedicated for backups, I also have a drive dock that I do a backup to when I remember... and now that I think of it, it's been a while.  I have several drives I rotate around with different jobs assigned to them.   Typically I bring the large one to work which covers my most critical stuff.  That is like my total last resort backup if the house burns down type deal.   I also have some offsite rsync backup jobs that run weekly to my web server.  I need to also look at cloud, but I'd want something that supports rsync so I don't need to come up with a different way to automate it.   Actually I need to look at rdiff backup too.   Overall I do need a massage storage upgrade so I can have better backups. Most of my backups are simply a mirror, with some oddball stuff dated monthly/weekly/daily such as databases.


Oh and to those curious this is my file server setup:



The middle one is where all the data is, the top one is a VM server (only has an OS drive) and the bottom stuff I don't really use but it's some SAN enclosures that are hooked up to the file server via fibre channel.  Was awesome at the time, but for the power it uses and fact that I can't put my own drives in it, I find it's practically useless, so I will take them out if I need the rack space.

Right now the file server has 3 raid arrays:

Code: [Select]

Filesystem            Size  Used Avail Use% Mounted on
/dev/md0              5.4T  3.5T  1.7T  69% /volumes/raid1
/dev/md1              6.3T  5.4T  571G  91% /volumes/raid2
/dev/md3              7.2T  1.8T  5.1T  26% /volumes/raid3

Not that much disk space by today's standards but I can expand as I need.  md1 is a bunch of 1TB drives, I am due to upgrade that array to much larger drives.  In fact all my arrays use rather small drives as that's what was economical to get at the time.   md0 and md3 are raid 10, md1 is raid 5.  That is my oldest array, it's dated 2008.  It's been transplanted between like 3 different machines lol.  That's the beauty of software raid though, you're not depending on specific hardware.

[BradC]

I assume you have a way of preventing malware from eating up all your incrementals and spreading throughout your network? Do you have something that's disconnected from the rest of the network somehow?

The good thing about using a ZFS or BTRFS NAS is that snapshots are instant to create and become read only. The snapshots are part of the filesystem - not files on a drive.

I lost a 16TB RAID6 to creeping corruption. In my instance I had a dicey SATA controller that gave read errors under heavy load on 2 ports, but never enough to even register. Where it did the damage was stripe read-modify-write cycles on parity where the chunk size was 1M (so 8M stripes). Over probably 7 months it slowly corrupted 2 drives on the array until it caused noticeable damage. By then I'd lost a considerable amount of (mostly replaceable) data and corrupted a whole archive of years of digital photos that were not backed up.

ZFS would probably have mitigated that, but it was strictly Solaris only at the time. BTRFS hadn't even been thought of.

Along with effective and tested backups, I now run periodic md5 checks over the array and keep an eye on the array scrub mismatch_cnt (which is how I noticed the issue in the first place).

The day ZFS hits the mainline kernel I might switch.

[Red Squirrel]

ZFS is something I want to look into at some point myself.  MD raid has been pretty solid though, but ZFS has lot of interesting features as well.

[amspire]

I assume you have a way of preventing malware from eating up all your incrementals and spreading throughout your network? Do you have something that's disconnected from the rest of the network somehow?

The good thing about using a ZFS or BTRFS NAS is that snapshots are instant to create and become read only. The snapshots are part of the filesystem - not files on a drive.

I lost a 16TB RAID6 to creeping corruption. In my instance I had a dicey SATA controller that gave read errors under heavy load on 2 ports, but never enough to even register. Where it did the damage was stripe read-modify-write cycles on parity where the chunk size was 1M (so 8M stripes). Over probably 7 months it slowly corrupted 2 drives on the array until it caused noticeable damage. By then I'd lost a considerable amount of (mostly replaceable) data and corrupted a whole archive of years of digital photos that were not backed up.

ZFS would probably have mitigated that, but it was strictly Solaris only at the time. BTRFS hadn't even been thought of.

Along with effective and tested backups, I now run periodic md5 checks over the array and keep an eye on the array scrub mismatch_cnt (which is how I noticed the issue in the first place).

The day ZFS hits the mainline kernel I might switch.

This is the problem with raid - particularly as the disks get bigger. Raid 1 (Mirroring)  is the only raid I have ever chosen to use. I could be talked into using RAID 10.  To me, if you want to use RAID5/6, you may as well be looking multiple RAID systems set up to run like a SAN array to give hardware redundancy. Otherwise RAID5/6 needs a good backup, and you have to be able to accept the down time while you are rebuilding the repaired or replaced RAID drive from the backup.

BTRFS is probably not as RAM hungry as ZFS and is pretty solid. ZFS has been part of FreeBSD since 2007, so if you want a really solid ZFS, that is probably the way to go. Netgear have been using BTRFS in their NAS boxes for many years now. RAID has been retarded historically as imperfect in BTRFS. I have been using it on a server myself in a RAID 1 configuration for 3 years without any issue. Netgear has been using mdadm for its raid and is then running a single drive BTRFS on top of the raid. Netgear for ages was using BTRFS at about version 0.2 and it was very reliable then.

Both these file systems offer Scrub to help protect against bitrot, but you do have to schedule the Scrub - it is not automatic. It also take a really long time for huge filesystems.

[kripton2035]

I assume you have a way of preventing malware from eating up all your incrementals and spreading throughout your network? Do you have something that's disconnected from the rest of the network somehow?

The good thing about using a ZFS or BTRFS NAS is that snapshots are instant to create and become read only. The snapshots are part of the filesystem - not files on a drive.

I lost a 16TB RAID6 to creeping corruption. In my instance I had a dicey SATA controller that gave read errors under heavy load on 2 ports, but never enough to even register. Where it did the damage was stripe read-modify-write cycles on parity where the chunk size was 1M (so 8M stripes). Over probably 7 months it slowly corrupted 2 drives on the array until it caused noticeable damage. By then I'd lost a considerable amount of (mostly replaceable) data and corrupted a whole archive of years of digital photos that were not backed up.

ZFS would probably have mitigated that, but it was strictly Solaris only at the time. BTRFS hadn't even been thought of.

Along with effective and tested backups, I now run periodic md5 checks over the array and keep an eye on the array scrub mismatch_cnt (which is how I noticed the issue in the first place).

The day ZFS hits the mainline kernel I might switch.

it's a good practice to backup the NAS ... on a simple drive but backup.
also good to make a backup that does not stay in the same building.

[brucehoult]

I'm using BackBlaze now for HD backups

Looks like Backblaze have really good prices. I do worry about how long a company can last when it has data storage costs 1/4 of its competitors. At that price, they would probably have to have disks last at least two years before they even have a chance of making money. Don't know how they do it especially if they need to have redundancy. As long as they keep up the prices though, they are a fabulous deal.

I would suspect they are counting on everyone backing up the same OS files, apps, porn, commercial music and movies etc and only having to store one copy of them.

[BradC]

This is the problem with raid - particularly as the disks get bigger. Raid 1 (Mirroring)  is the only raid I have ever chosen to use.

No, that was the problem with dicey hardware, an inadequate backup / integrity checking mechanism and my complete ignorance of the risk and management factors. Nothing to do with RAID.

BTRFS is probably not as RAM hungry as ZFS and is pretty solid.

I'm just not comfortable with BTRFS. A couple of us (myself and some mates) tried it on some larger filesystems earlier in the year (>30TB) using what were theoretically stable parity features, but after it trashed a number of filesystems we walked away (as did Redhat apparently). Most of those crashes were not parity related and were apparently due to the filesystem filling up. Nice, an "oh, I've run out of space" error becomes "Oh, I've lost an entire filesystem, I wonder what happened". Restoring 30TB from backup takes a while. I'm sure if we'd worked with the BTRFS list and developers we might have salvaged some stuff, but man "fool me once...". Why would you trust it?

Aside from bad hardware, I've never lost an ext filesystem. In fact even *with* bad hardware I didn't lose the filesystem, I just ended up with some stripes across my jpegs and other miscellaneous file damage. Metadata was completely recoverable with a simple fsck. A BTRFS crash was a total irrecoverable loss.

Make no mistake, my failure would have been caught sooner with regular scrubs (even on my RAID6), so it was a combination of bad hardware and ignorance of the mechanics on my part. Heck, I wasn't doing regular scrubs or even SMART tests. I didn't even know what I didn't know.

RAID1 / RAID10 have plenty of failure modes, but when you start getting the requirement for larger data pools, they're just not as cost effective. You need to learn to mitigate the risks associated with other storage topologies.

[BradC]

I'm using BackBlaze now for HD backups

Looks like Backblaze have really good prices. I do worry about how long a company can last when it has data storage costs 1/4 of its competitors. At that price, they would probably have to have disks last at least two years before they even have a chance of making money. Don't know how they do it especially if they need to have redundancy. As long as they keep up the prices though, they are a fabulous deal.

I would suspect they are counting on everyone backing up the same OS files, apps, porn, commercial music and movies etc and only having to store one copy of them.

That won't work. Your data is encrypted at your end with your key, so there's no way they can de-dupe.

Backblaze are actually really open about a lot of what they do and how they work, it's really worth reading up on how they do what they do. I don't see them going anywhere anytime soon. They've put together a storage methodology that works for them from a cost perspective, and they're constantly getting bigger (not smaller). They've been around long enough now that if it wasn't a sustainable business for them, they'd have gone bang.

[bd139]

Just as a heads up, tarsnap are pretty good as well. They front end S3 with something less crazy and offer source code etc.

[kulla]

I ran mine backups on 4 drive NAS backed up to BackBlaze.

But backups as backups means nothing if they are not tested before disaster happens, both scheduled and unscheduled.

[woody]

Yes, they are.

Every 30 minutes copy of new/changed, volatile data from pc -> nas1, to a $weekday directory
Once per day all new/changed data to nas1, to the same $weekday directory
Every night all new/changed data from nas1 to nas2.
Nas1 in mirror RAID, nas2 in mirror RAID, 200 km away from nas1.
I use basic file copy, so no smart compressing, incremental, journalling, state-of-the-art tricks that need even more fancy tricks to get my files out of the backup again. Also no cloud, no recurring costs and, knock wood, so far no problems.
Apart from this once in a blue moon I make an image backup. Why I do that is not clear to me; for the few times in my life I ran into disk problems I always started with a freshly installed OS.

And I think TS's portable 2TB drive might not be two drives in RAID, but one 'thick' one (12mm).

[VK5RC]

NAS 2x2TB raid 0 configured, backed up weekly.

Whhhhyyyyyy.

Apologies ,  I always get the RAID 0 and 1 numbers the wrong way around, it is 1.

[CJay]

Yes, testing your backups is as important as making them. That's not theoretical either, I've seen an example of a narrow escape just last month.

Could not agree more, backing up and not testing that
A. they can be restored
B. What is restored isn't digital gibberish

is a complete waste of time.

I've seen it a few times where corporates haven't had good backups for, in one case, years.

[Monkeh]

NAS 2x2TB raid 0 configured, backed up weekly.

Whhhhyyyyyy.

Apologies ,  I always get the RAID 0 and 1 numbers the wrong way around, it is 1.

It's easy enough to remember - AID 0 is not RAID, and describes exactly how much help you'll get when (not if) it fails.

[bd139]

NAS 2x2TB raid 0 configured, backed up weekly.

Whhhhyyyyyy.

Apologies ,  I always get the RAID 0 and 1 numbers the wrong way around, it is 1.

It's easy enough to remember - AID 0 is not RAID, and describes exactly how much help you'll get when (not if) it fails.

Perhaps I'm just a sociopath but I can't help laughing slightly when something major goes wrong where there is a big misunderstanding in the tech like that. It's always like the punch line for a German joke.

Yes, testing your backups is as important as making them. That's not theoretical either, I've seen an example of a narrow escape just last month.

Could not agree more, backing up and not testing that
A. they can be restored
B. What is restored isn't digital gibberish

is a complete waste of time.

I've seen it a few times where corporates haven't had good backups for, in one case, years.

I worked for a company once. Well I say worked for, but accidentally landed chief technical monkey position because I needed to eat. Turned out they had been blindly cycling tapes for about 2 years. When I reviewed the steaming turd I was landed with I noticed that the external VS160 DLT drive wasn't even connected to the server. The SCSI cable was down the back of the rack. I think someone had moved stuff around and left it like that. Still the tapes ejected and got inserted so they had the illusion of a backup

[CJay]

I worked for a company once. Well I say worked for, but accidentally landed chief technical monkey position because I needed to eat. Turned out they had been blindly cycling tapes for about 2 years. When I reviewed the steaming turd I was landed with I noticed that the external VS160 DLT drive wasn't even connected to the server. The SCSI cable was down the back of the rack. I think someone had moved stuff around and left it like that. Still the tapes ejected and got inserted so they had the illusion of a backup

It'd have been fast too and none of those pesky backup errors

[bd139]

They actually didn’t notice they weren’t running and didn’t check for errors. Someone just went into the cupboard in the office; took the tape out and put another one in once a day. What happened between two consecutive events wasn’t their problem.