Backup strategies

[AndyC_772]

Hi all,

I'm interested to compare notes about how we all keep our data safe from equipment failure, viruses, operator errors and the like. It seems to be a subject where it's all too easy to learn the hard way that a backup strategy wasn't adequate, but it's harder to find good information on 'best practice' in advance of a disaster happening.

My important data lives on a NAS box with three discs in a RAID array, powered by a UPS. Overnight, it runs an incremental backup to a separate, encrypted USB drive, and I have 3 of these drives which I rotate once a month, so there's always one which is physically disconnected and less than a month old, and another which is more than a month old (just in case something gets accidentally deleted and I don't notice for a while). (Note to self, the spare USB discs should probably live in a fire safe).

I've also recently added a second NAS box with two discs in a mirror configuration, and that receives an incremental backup overnight too. It lives in a different room and has only a wireless connection to the main NAS. This box exists in case a fault in the main NAS (say, a catastrophic power supply failure) takes out all the drives in it as well, including the current USB device. I don't keep any drive letters mapped to this box, and my PC user account only has read access to it.

What I like about my backup strategy is that it's fully automated; now I've set it up, I don't have to do anything besides rotate the USB discs once a month. That's important to me, as I'd inevitably forget or not bother to back things up otherwise.

I don't like the idea of using a "cloud" backup service. I need to know, very clearly and specifically, who has access to my data, and in what jurisdiction it may be kept. These awkward questions seem all too easily glossed over or ignored, and that's no good.

If anyone thinks I'm unprotected against a likely failure, then do please explain, I'm keen to make worthwhile improvements.

Ta
Andy.

[German_EE]

I've just upgraded all of my storage and the new system is as follows.

Everything is kept on a 6 TB NAS drive, movies, personal stuff, the lot, but if you look at the directory tree it naturally separates into three distinct areas which are labelled red, green or blue in the file manager. These three drive areas are backed up to color coded 2 TB USB drives. Backups are tested.

I don't use Seagate drives.

I've had to deal with a drive failure at one point and it's NOT a nice feeling.

[bookaboo]

I use a reputable cloud backup service and once every few month I do a manual backup of vital files  (accounts, programs & drawings) to my home PC via the cloud (upload them at work, download them at home).
Not as secure as those already stated but the chances of losing all my data via dual disasters is next to none.

While the cloud may not be 100% secure it's a risk I'm willing to take given that my data is non sensitive.

[nctnico]

I ditched UPSs a decade ago. They caused more outages than they solved. I'm also not convinced RAID is a good method due to the manual intervention required to fix it. I have 3 backups which are updated incrementally using Rsync (only add, not delete) every day. One on a slow hard drive inside my PC/server for easy access, one in a NAS which sits in a fire protected place and one USB drive in the attic above the flooding level (I'm about 4 meters below sea level so flooding is a risk). I take the USB drive with me when I leave for longer periods.

For one particular customer I also backup to Dropbox but that goes through a different user account on my machine in 2 steps so the Dopbox stuff is isolated from the rest.

[eas]

The obvious holes:

1) you don't mention how you verify that any of this is working as expected and that you can restore as expected.
2) You don't mention your offsite backup, other than that you don't trust cloud-based solutions.

Cloud-based services aren't the only option for off-site backup.

More importantly, is your risk-assesment well thought through? Are the risks you attribute to cloud-backup really larger than the risk of data-loss due to something like your house burning, or flooding?  Think long and hard about this, because, the way I see it, your current position is that your data is more important to a hypothetical someone else than it is to you. If that is the case, why are you keeping it and backing it up in the first place?

Once you've considered that, you should revisit your look at cloud-based backup services. The requirements you describe don't sound unique, and I'd be very surprised if there aren't options that can address your doubts, though probably not among basic consumer-oriented offerings (That doesn't mean they have to be esoteric or expensive though).

Personally, I don't bother with RAID for my home server any more. It can enhance availability at the cost of complexity and maintenance, but it is no substitute for backup (not that you seem to be relying on it for such). I'd rather use the extra disks for rsync snapshot backups (using rsnapsot makes it easy to create and maintain).

[Mechanical Menace]

Me and a friend keep daily incremental backups of each others file servers. We have a WiFi network that's meant primarily for that. That way we both have regular off site backups. Also do the typical monthly disk imaging etc but fingers crossed have never had to fall back on one yet. The latest are kept at my mothers, the oldest come back with me to become the latest.

I also happily back certain things up to various internet services. Even if it wasn't encrypted that's all just family pictures and home movies, my own crappy music from when I thought I was going to be the next Garry Moore etc. Nothing "important," nothing secret, but probably means more to me than everything else combined lol.

[eas]

I also happily back certain things up to various internet services. Even if it wasn't encrypted that's all just family pictures and home movies, my own crappy music from when I thought I was going to be the next Garry Moore etc. Nothing "important," nothing secret, but probably means more to me than everything else combined lol.

Personally, I think that even that stuff should be as secure and safe as you want it to be.

That said, I think in some ways, at least half the value of personal backup and data management tools (like Evernote) is simply that they make you think you'll be able to find your old stuff when you need it, which frees you to do and think about other, more immediate concerns. Even if, at some later date, they fail in that promise, they've probably still done you a huge service by letting you get on with things over the previous years.

[codeboy2k]

I'm going through a double disk RAID failure right now, as some of you know from my own thread on the matter.

What I've learned is that hard disks are shite, they WILL fail, and backups are a necessity.  RAID is not good enough. And as was said by one of the members, if you need to make a backup anyways, why use RAID? (or at least, why use a complex RAID like RAID5 -- just mirror it and back it up).

Cloud backup was never an option for me or I would have done it.  I have limited bandwidth and limited GB of data allocation before they start charging me overage per GB so using the cloud for GB's of storage was not an option.

Learn from me.  Having no backup strategy is never going to end well

[krish2487]

I have seen/read opinions of others and while they all have their preferences, I ll just write down what I do to ensure safety of data.


Office - Main workstation PC - RAID 1


Office - Local NAS (also RAID 1) - Daily scheduled complete backup of workstation PC. (All the User files, Drawings, code etc).


Home - Remote Backup (Also RAID 1) - Complete mirror of Local NAS running at office (synced using owncloud).


As an added advantage, using owncloud to sync remote and local allows me to access the data on the web .
It helps that I have dedicated bandwidth of 4 MBps and static IPs for both sites. Just as a number, I transfer roughly about 400 MB of data everyday. I find that bandwidth is a cheaper expenditure than disks. The redundancy / downtime is presently at 15 mins in case of a complete hardware failure (like a fire)  [size=78%] [/size]

[sleemanj]

I have a USB 3.0 drive and use "rsnapshot" to create nightly snapshots to it, this makes it pretty easy for me to recover an accidentally deleted or screwed up file as each night's snapshot is a complete filesystem tree (using hardlinks so each file is only stored physically once) and I can (could) easily pull unplug the drive and plug in another one every so often.

Only the important otherwise-non-recoverable stuff gets backed up, as for the OS itself (Ubuntu) I just store the package selections and contents of /etc and /usr/local so I could if push came to shove rebuild the system to roughly as it is now from those.

I really should get around to hanging another off a network connection in an outbuilding and do a weekly rsync of the snapshot drive across to it.

[AndyC_772]

The obvious holes:

1) you don't mention how you verify that any of this is working as expected and that you can restore as expected.
2) You don't mention your offsite backup, other than that you don't trust cloud-based solutions.

Every time I rotate the USB discs, I can at least see how much capacity is used. Other than actually performing a full restore - presumably to a fresh set of spare discs - what test would you recommend?

The most recently unplugged USB disc is usually kept off-site.

More importantly, is your risk-assesment well thought through? Are the risks you attribute to cloud-backup really larger than the risk of data-loss due to something like your house burning, or flooding?  Think long and hard about this, because, the way I see it, your current position is that your data is more important to a hypothetical someone else than it is to you. If that is the case, why are you keeping it and backing it up in the first place?

I have both important data that I don't want to lose, and data which I need to keep private.

A house fire is probably the worst case scenario, though even then, my customers generally have their own copies of completed projects, so it's just work-in-progress that would be truly lost. That, and all my personal data, music, photos and the like, of course.

Once you've considered that, you should revisit your look at cloud-based backup services.

Maybe that's worth a thread of its own. What specific guarantees or assurances should I be looking for and expecting a cloud provider to make, and how do the major players stack up in that respect?

[Halcyon]

Disregard people who suggest one particular brand of HDD is most unreliable. It may be true. But what of it?
There is no perfectly reliable HDD. That's why you need backups.

I don't think you should disregard experience from those in the field. There are disks that perform worse and are less reliable than others. My advice would be if you're running a proper (hardware) RAID set up, stick to proper enterprise drives. Those sub-$100 consumer drives aren't up to the task. Your backup should be a last-resort tactic. If you can sustain a stable RAID with good hardware, when a failure does occur, you'll see little or no downtime and you have other options still up your sleeve. It's the worst feeling when you're restoring from the only other source of your data, hoping that none of it is corrupt and all of the media is readable.

Unless you're backing up constantly, your backup in most cases will still be missing the latest data.

[lewis]

What specific guarantees or assurances should I be looking for and expecting a cloud provider to make, and how do the major players stack up in that respect?

I only ever trust cloud providers with data that I would be happy for anyone to view. In other words, whenever something goes on the cloud, any cloud, no matter the provider, I assume someone is looking at it.

Don't forget encryption too. Vastly simplifying, you have obligations under the Data Protection Act to keep customer's data confidential. There was a case recently where a small business was fined a significant sum by the ICO for DPA breaches when they had their laptop stolen. The laptop was password protected but crucially not encrypted.

My backup system involves daily backups to office NAS (5x WD red 4TB with 2 disks redundancy) and weekly backups via USB drive to home laptop. All encrypted. Lots of manual pissing about, and lots of managing encryption keys! To be honest, I'm looking for a decent backup solution too.

[EEVblog]

Depends if you need to back up video or not.
For my needs as a video content producer, the only cost-effective and suitable backup and archive medium is hard drives.
Cloud doesn't work for massive amounts of video files.
I keep two hard drives (different brands) with the same data.
Just for the raw video for episodes 500-715, that fill a 2TB drive, and that requirement always increases as I shoot at ever higher bitrates.
For all my photo, documents, design files and other stuff, my 1TB Dropbox is perfect, so it's backed up in the cloud and on several local machine that are synced.

[nctnico]

What I'm missing from this thread is how to recover the OS and installed software quickly. It takes me several days working 9 to 5 to get everything re-installed and configured. Nowadays I run Windows software from virtual machines and the VM images are included in my backups so the Windows part of my 'operation' is peanuts to revive. But I'm also doing a lot of development on the Linux host system as well. I will need to work out a backup strategy for that too...

[Red Squirrel]

I have a 24 bay file server that I built, there are currently 3 separate raid arrays on it, an old raid 5 that was ported from my main server, then two newer arrays which are raid 10's. Most of my environment is virtualized so this stores VMs and actual raw data.  The old raid5 is used mostly for backups so stuff on the other arrays or system files in the VMs etc are backed up nightly.  I have 19TB total of space on that server. Not that much by today's standards, but I tend to only add more drives as I need more space.  Eventually I'll want to retire that old raid 5 array and make a bigger raid 10. Might even look at ZFS at some point.

I have a drive dock which I use to run manual backups as well, though I need to buy more drives as a lot of them are failing.  Basically I treat them like tapes and have a small rotation going, just to act as cold backups as you don't want all your backups on spinning drives in case of odd ball situation like an earthquake or hackers or something along those lines.

For power backup (almost just as important as file backup) I have an inverter-charger with 4 100AH flooded cell batteries.  I'm good for about 4 hours for the servers.  Just have a basic SOHO UPS for workstation so that gets shut down mostly right away if power goes out. Load is about 50 amps on the DC side for server stuff.   I eventually want to upgrade this setup to a 48v dual conversion setup like telcos use, that way there is less chance of something going wrong during switch over, as it's always running off the inverter which is always connected to the batteries.  I'd increase the amount of batteries too which would allow me to add more stuff to it like my PCs and maybe even my TV setup. 

[AndyC_772]

What I'm missing from this thread is how to recover the OS and installed software quickly.

In my case, this really isn't a major issue. A day of hassle to reinstall Windows, email, PCB software and a couple of compilers.

Realistically, in the event of my PC dying, I'm much more concerned about not being able to replace it with a Windows 7 machine, and being forced onto 8.1.

[AndyC_772]

Depends if you need to back up video or not.
For my needs as a video content producer, the only cost-effective and suitable backup and archive medium is hard drives.

Do people still use tape these days? I'd have thought that archiving video would be a perfect fit for tape backup... large capacity, very occasional access, and all that data is held in a relatively small number of large files.

I only ever trust cloud providers with data that I would be happy for anyone to view. In other words, whenever something goes on the cloud, any cloud, no matter the provider, I assume someone is looking at it.

Me too, and that's the problem. My business data is confidential, and I don't want it crossing borders.

[Halcyon]

In some ways crossing borders has advantages, because it creates legal difficulties for any government entity trying to access your data. Not that it really matters if they get it.

Depends on the country/provider. Most large companies have a dedicated e-mail address or request process for such data and a warrant is not always required. Facebook and Apple are two I have directly dealt which personally which require little more than online (intranet) form filled out. Most companies (big and small) will almost bend over backwards for requests by the Government. I don't see it necessarily as a bad thing though. The outcome is almost always positive and in the interest of the general public. Mind you, the cops have no interest in your nudie photos etc...

[AndyC_772]

It's not that I'm that bothered about national security services accessing my data out of curiosity. What I don't want, is to find that I'm deemed to have inadvertently exported something from somewhere which required a licence, or which results in a breach of customer confidence, even though everything I "export" originally came from my own PC anyway.

Life is, frankly, too short to spend it conducting an audit of which countries allow what code / images / personal data to cross borders to where, and which cloud providers offer absolute cast-iron guarantees regarding where my data ends up.

[eas]

Disregard people who suggest one particular brand of HDD is most unreliable. It may be true. But what of it?
There is no perfectly reliable HDD. That's why you need backups.

I don't think you should disregard experience from those in the field. There are disks that perform worse and are less reliable than others. My advice would be if you're running a proper (hardware) RAID set up, stick to proper enterprise drives. Those sub-$100 consumer drives aren't up to the task. Your backup should be a last-resort tactic. If you can sustain a stable RAID with good hardware, when a failure does occur, you'll see little or no downtime and you have other options still up your sleeve. It's the worst feeling when you're restoring from the only other source of your data, hoping that none of it is corrupt and all of the media is readable.

Again, I disagree with the idea that RAID should be the foundation of a data preservation strategy. It is most useful for addressing availability, capacity, throughput and latency, and even then, there are other options that should be considered alongside RAID. In situations where RAID is appropriate, it needs to be accompanied by appropriate operational practices. First and foremost, a RAID increases the likelihood of a single failure in the storage subsystem, as well as increasing the risk from a single failure. Because of this, while RAID provides the possibility of continued operation in the face of a failure, a failure should receive immediate attention. It is hard for me to imagine that any of them are justified for secondary storage for one or two people, unless you want to make a hobby of being a storage admin.

The idea that hardware RAID is "proper," and software RAID isn't, is misguided and ill-informed. Hardware RAID brings a number of downsides. Firmwares can be buggy, they add additional complexity, and with it and more places for things to go wrong and mistakes to be made by part-time admins. In the event of failure of the RAID hardware, recovery generally requires acquiring the appropriate proprietary, possibly obsolete hardware. Recovery from some failure modes is complicated because data and metadata are spread across multiple disks using potentially undocumented datastructures.

Unless you're backing up constantly, your backup in most cases will still be missing the latest data.

True, but with OS-provided filesystem notifications, the backup window can be quite small. Moreover, it doesn't sound like the OP is necessarily using this for working data in the first place.

RAID also does nothing to protect against accidental file deletion and file corruption. It can reduce the chances of filesystem corruption, but once that happens, it does nothing to help with recovery.

The obvious holes:

1) you don't mention how you verify that any of this is working as expected and that you can restore as expected.
2) You don't mention your offsite backup, other than that you don't trust cloud-based solutions.

Every time I rotate the USB discs, I can at least see how much capacity is used. Other than actually performing a full restore - presumably to a fresh set of spare discs - what test would you recommend?

Well, disk-space is relatively cheap, so making sure there are no errors when copying back from a backup isn't necessarily that big an expense. Another approach would be to script restoration of, say, 10% of all the files. An orthogonal piece is to use something to compute signatures of all the files at the time of backup and then check all or a subset of them.

Checking the amount of capacity in use depends on the filesystem being in a consistent state, but an inconsistent filesystem is one of the things you want to guard against, and a consistent filesystem doesn't ensure that critical files are indeed backed-up and in good order. Making a regular filesystem check (not repair) of your backup disks is probably a good thing to add to the mix.

The most recently unplugged USB disc is usually kept off-site.

Ah, sorry if you mentioned that and I missed it. That's a good easy step to take.

Maybe that's worth a thread of its own. What specific guarantees or assurances should I be looking for and expecting a cloud provider to make, and how do the major players stack up in that respect?

I think the question of access to backed-up data, and the question of jurisdiction and data crossing borders are probably pretty common -- they are often raised, and often backed with well considered reasons. Any backup provider that doesn't provide crisp answers is either interested in a different segment of the market, or incompetent at business, and likely, technology.

I am not recommending them, but in the US, at least, I'm quite happy with Crashplan and impressed by the completeness of their offering. For paying customers, there is the option of using a custom encryption key to encrypt the backup. I am using encryption, but I haven't enabled this specific option yet, in part because I'm not ready to tackle the key management.

[eas]

What I'm missing from this thread is how to recover the OS and installed software quickly. It takes me several days working 9 to 5 to get everything re-installed and configured. Nowadays I run Windows software from virtual machines and the VM images are included in my backups so the Windows part of my 'operation' is peanuts to revive. But I'm also doing a lot of development on the Linux host system as well. I will need to work out a backup strategy for that too...

Well, one reasonable approach is to only use the host linux system for virtualization, and do your linux development in a linux vm.

Beyond that, linux is more amenable to simple filesystem copy backups/restores than Windows. In addition, for debian based systems, its pretty straightforward to capture the basic package manager configuration, including customer repositories, and to dump a list of installed packages/versions that can then be fed back. For stuff from tarballs, a dumbass shell-script is often enough. Don't worry too much about dealing with exceptions. In most cases, if it fails, it doesn't take much to fix the problem and rerun the script.  I think one challenge with Linux is that there are many many options and the reasons to choose among them aren't always obvious. If they were, I'd probably have a better answer

[ez24]

What I'm missing from this thread is how to recover the OS and installed software quickly.

I use a disk duplicator, if I remember it was about $60.  When I get a new computer, I install the software that I know I will not be messing with anytime soon.  Then I duplicate it with new drive and put the new drive into the computer and finish installing things like Firefox.  And store the original a safe place.

Once a new drive failed and I put back the original drive to make sure it worked and then duplicated it again.  Never had to reinstall OS since I got the duplicator.  The only thing to remember is you have to use the same size or larger, you cannot go down in size.

Another time when a computer failed and I got a newer model with no OS, a duplicated drive worked (from another computer) but I have to install new drivers.

Also I used the duplicator to back up a full 3 tb drive to another 3 tb drive.  Then I hooked both up to a computer and turned on the computer and both drives crashed.  I spent months and every software solution I could find without any luck.  I lost 3 tb of data.  The moral is do not hook up duplicates at the same time.  It would have been ok if I had powered one up and let the computer recognize it then power up the other.  Maybe someone could tell me the technical reason why this happened.

I have used this method on about 20 computers so far and I could not live without one.  The only downside is you need another drive for each backup.


Edit:  The drives were not destroyed, just the data.  I re-formatted them and I am still using them.  I am not sure what "brick" means, ie does it mean software or hardware destruction?

[TerraHertz]

Maybe someone could tell me the technical reason why this happened.

It's worth keeping in mind that hard drive command sets include the ability to reflash the hard drive firmware. Which means there's potential for hostile code on the PC to deliberately brick hard drives, by reflashing the drive firmware with garbage. There are obvious scenarios in which this could happen (hackers, intelligence agencies, etc), but there are also non-obvious ones.

I have a story that may be relevant. Long ago I swore I'd never give MS another cent, for anything, under any circumstances. So I have an interest in unofficial versions of Windows. A few years ago I was setting up a system using an 'optimized and minimized' version of WinXP. The system HD was all FAT32 partitions. I'd spent about a month on and off getting the system customized and all utils I wanted installed. Never even a hint of any problems. I reached the point where I was happy with it, and so should make a duplicate system image. A suitable spare drive I had was NTFS. Before reformatting it I thought I'd check what was on it. Powered down the system, plugged in the NTFS as 2nd HD, powered up. Spent a brief moment looking at the contents, decided it held nothing important and formatting it would be OK. I wasn't logged in as Administrator, and the quickest way to switch was via a reboot. So shut down the system and powered up again. Btw, both drives were PATA, and on separate cables. I didn't touch anything during this sequence.

On this power up, the system wouldn't boot. 'Missing hard disk'.
On investigation, the system boot HD I was about to mirror, was now a non-HD. Bricked, not responding to the BIOS at all.

So, that drive just happened to become a brick, in the couple of minutes when this 'black' Windows install saw an NTFS drive for the first time ever, after being fine for a month of use, many power cycles, many reboots, both from power downs and ctl-alt-del restarts.
It's not an impossible coincidence, but it sure makes me suspicious.
If I was a paranoid person (and I am) I'd suspect there's code buried in Microsoft NTFS drivers that runs some kind of 'am I a legitimate Windows install?' check, and if not, causes as much pain as possible.
Now if I was writing a 'hidden last resort protection' code bomb like that (and I have in the past) I'd make it non-deterministic in operation, ie it wouldn't always act, and it wouldn't always do the same thing when it did act. Repeatability is a weakness and allows for easier recognition, identification and removal. Also, since I'm convinced Microsoft management are the worst imaginable kind of bastards, I don't think it's beyond the realm of possibility that such drive-bricking behavior might also happen at rare, random times, solely to create a background of such failures in which more targeted failures don't stand out so much. Which would put a couple of other such sudden total drive-failures I've had in a different light.

One of those projects I'll probably never get around to, is to set up that same scenario, but with target HD having the _write line to the HD firmware flash physically cut, and a logic analyzer looking at the PATA interface and that write line. Also using a system with a CPU emulator installed. It would be cool to catch it happening, and document exactly what code did the drive bricking. Imagine the class action lawsuit, if it turned out MS Windows actually was provably deliberately bricking drives now and then.


Edit to add:

PowerCartel, a blog, and forum, focusing on scavenging and reusing lithium ion rechargeable batteries.
http://powercartel.com/
http://discourse.powercartel.com/  <-- not working?

Thanks for the links! You should start a thread on this. 18650 batteries are great, and I'm already finding lots of surplus ones. Very interested in discussions of salvage/restoration & charge management techniques.

Pic below - the green ones are all good, as they were found (in DECT phone battery packs) while still charged, mostly brand new. The yellow ones are an interesting experiment-to-be, in that they are from a portable power tool pack and are COMPLETELY flat. 0.0V. But the first one I've tried charging is showing signs it might recover somewhat. Still to be seen how it goes with multiple 'resuscitation cycles'. I'm very curious about how badly these batteries are degraded by mistreatment.

[Halcyon]

If I was a paranoid person (and I am) I'd suspect there's code buried in Microsoft NTFS drivers that runs some kind of 'am I a legitimate Windows install?' check, and if not, causes as much pain as possible.

That sort of code doesn't exist in Windows. Could you imagine how many drives would be bricked if that was the case?

You would have been better off just using a legitimate Windows XP install disc rather than stuffing around with someone else's work. You would have been up and running in a few hours rather than weeks.