OT: Backing up video files long term

[EEVblog]

Opinions invited:
What is the best way to physically backup all my raw (and edited) video files for the long term?
I can fit maybe 500 episodes  + other misc and family stuff onto a 2TB drive, and I currently have a NetGear ReadyNAS RAID system to do that, plus the odd drive here and there, but it's close to full.
Yes, I know that RAID is NOT proper "backup" so please spare me the lecture 

I'm thinking maybe 3 separate hard drives in 3 locations per every 500 episodes?
Or should I keep multiple RAID disk arrays, enclosures and all in different locations?

Online cloudy solutions will likely eat into my data allowance at the lab, need to look into that, but can't rely on it solely.

[free_electron]

get a few harddisk in usb enclosure.

for long term storage : two sets. one off-site ( at home ) one in lab. preventing fire disasters.
drives are to be in a good schock munted enclosure and powered off. get drives that physically retract heads off-platter.
drives are to be filled with data in 1 shot. then powered down until needed. let's say you got 2.3 terabyte of data : 'seal' 2 terabyte onto two 2 terabyte drives.
the remainder is kept alive on your nas system (until you have collected enough to fill another drive.

i would use 1 terabyte drives. that way you can fill the drive faster and 'seal it' ( the goal is to mimimize the amount of power cyles on the drive. )

we d this for project data. once a project is over an the design can be archived it is dumped to a set of brand-spanking new harddisks in one shot. they are written , verified and then powered off.
one drive is stored in the archival safe on site , one drive goes to an archival company. third and fourth sets are created at other sites around the world through our network. even if a drive is not full it is considered 'spent' on a project.  drives are cheap.

avoid nas boxes as the plague for this.. nas boxes often use a custom or weird  file system. if the nas hardware dies the drive is unreadable unless you have an identical nas. never mind the nas may use ext3 or anther linux filesystem. if you have it formatted in a linksys nas it will not be readable by a wd nas... even though they both use linux internally. there is metadata and journalling and other proprietary stuff and that may not be compatible between nas boxes from different brands !
a drive in usb enclosure does not have this problem. they are simple sata drives formatted in whatever filesystem used by the computer that wrote them. if the usb translator dies : plonk another one on there and it will work. worst case hook it to an esata or sata port and it will work.

[XOIIO]

I can't recall for sure, but don't tape drives have pretty decent storage if kept safe (eg away from magnetic interference sources)?

I'd say just some standard drives in some insanely good packaging would be the best option though.

You could also laser cut the binary into metal sheets to decode later 

[EEVblog]

drives are to be in a good schock munted enclosure and powered off. get drives that physically retract heads off-platter.

Don't they all do that these days?

I've got a 3.5" hot swap sata dock in my lab machine, so can use that to backup onto individual drives and then store them in different locations.
If I used 2 drives in two locations, plus maybe a cloud solution as well, then that's gotta cover all eventualities.
I'll keep the NAS for local use on the most recent stuff.
My desktop editing machine has two hard drives (in addition to the SSD boot drive), incrementally mirrored daily.

[KK]

I use LTO tapes. I've been using tape for decades quantum-DDS-DLT-LTO.

Copies on site, and in a bank vault.

LTO-4 drives use 800/1600GB tapes. Video will be somewhere between the two numbers depending on compressibility.

LTO-6 drives are out now with 2.5/5.0TB capacities.

Good long term archival storage. The LTO-4 drive does 550GB/hr using A SCSI interface.

LTO-6 is up to 1.4TB/hr

I have had situations where I needed a backup copy off 8 year old tapes. And it was not a problem for several TB's of restores.

I think optical storage has around 20 years or so shelf life because of eventual rot, and the magnetic tapes are around that lifespan.

[EEVblog]

Currently backing up 61 episodes worth of footage (284GB) to the networked NAS, and that's going to take 8 hours at 10MB/sec 

[SeanB]

Careful on tape. Very long life, but if the drives are no longer made they are essentially useless ornaments. I have old backups made on assorted tapes, that are no longer usable as the drive or the software no longer is usable on modern computers.

You might look into a BD recorder and store the video as data on them, making 2 copies as additional insurance. Store the discs in jewel cases inside a brown paper envelope, and keep in a box in a cabinet. A lot more space, but there you at least can read them with current drives, and the drives likely will be around for a while. No proprietary software needed to read them.

[XOIIO]

Good to have confirmation that tape drives do work well as I thought, I have a dell powervault tape storage system, it's pretty sweet, I just need the cable to hook it up to my server.

Currently backing up 61 episodes worth of footage (284GB) to the networked NAS, and that's going to take 8 hours at 10MB/sec 

Would it not be faster to copy it to an external drive then directly onto the NAS, rather than over the network?

You could also announce your channel is closing, when that happened with photonicinduction there was a torrent of every single video he uploaded, it's probably still alive, I should re-download it to watch some of his old stuff.

[Richard Crowley]

I don't trust ANY kind of field-burnable optical media.  Not LONG term archival storage IMHO, IME.

99.99% of this planets most valuable data is backed up and archived (two different functions) on digital magnetic tape.
While the concern for long-term viability of the format/equipment is a concern,  reviewing (and re-writing, if necessary) archive tapes every n years is only prudent if the data is worth it.
Hard drives are iffy, IMHO. I have too many of them that just died on the shelf in long-term archive service.
If I wanted to do that, I would make at least two copies on DIFFERENT BRAND hard drives, and store them sealed in plastic zipper bags in separate locations (one at the office, and the other at home, etc.)  The reason for different brands is critical because there have been several cases of bad batches of hard drives which created catastrophic problems for people "putting all their bits in one basket".
I would also at least plug in the hard drives every year or two just to confirm that still spin up and are readable, etc.

I use raw drives for archive storage and use a docking station like this to access them...



http://www.amazon.com/Plugable-Lay-Flat-Docking-Station-ASM1053E/dp/B00APP6694

And there are many storage options to protect raw drives when you aren't accessing them.  For example...



http://www.ebay.com/itm/like/300540328686?lpid=82

[XOIIO]

A really slick option would be to take the drive(s), find a good solid enclosure, cap off the top, fill the outside with spray foam all the way to the end, then have some other type of shock absorbing foam around that. Then maybe rubber bumpers and a metal case around that to close it off. You could also make a cover with spray foam if the metal case closed up.

[TheBorg]

I don't trust ANY kind of field-burnable optical media.  Not LONG term archival storage IMHO, IME.

99.99% of this planets most valuable data is backed up and archived (two different functions) on digital magnetic tape.
While the concern for long-term viability of the format/equipment is a concern,  reviewing (and re-writing, if necessary) archive tapes every n years is only prudent if the data is worth it.
Hard drives are iffy, IMHO. I have too many of them that just died on the shelf in long-term archive service.
If I wanted to do that, I would make at least two copies on DIFFERENT BRAND hard drives, and store them sealed in plastic zipper bags in separate locations (one at the office, and the other at home, etc.)  The reason for different brands is critical because there have been several cases of bad batches of hard drives which created catastrophic problems for people "putting all their bits in one basket".
I would also at least plug in the hard drives every year or two just to confirm that still spin up and are readable, etc.

I use raw drives for archive storage and use a docking station like this to access them...



http://www.amazon.com/Plugable-Lay-Flat-Docking-Station-ASM1053E/dp/B00APP6694

And there are many storage options to protect raw drives when you aren't accessing them.  For example...



http://www.ebay.com/itm/like/300540328686?lpid=82

Would ESD be an issue? Seems to me a plastic bag would have the potential to fry circuits on the control boards. Agree on the different brands though.
That being said, I store a couple of raw drives in a plastic drawer around here...

[EEVblog]

A really slick option would be to take the drive(s), find a good solid enclosure, cap off the top, fill the outside with spray foam all the way to the end, then have some other type of shock absorbing foam around that. Then maybe rubber bumpers and a metal case around that to close it off. You could also make a cover with spray foam if the metal case closed up.

Pelican case! 

[XOIIO]

A really slick option would be to take the drive(s), find a good solid enclosure, cap off the top, fill the outside with spray foam all the way to the end, then have some other type of shock absorbing foam around that. Then maybe rubber bumpers and a metal case around that to close it off. You could also make a cover with spray foam if the metal case closed up.

Pelican case! 

Pretty much, I think the spray insulation foam might be a good other layer to add, then put it in a pelican case that just pinches it between the foam cones.

Although I think we are getting into territory of dropping the hard drives, from considerable heights rather than just storage 

Guess it would be good to have videos survive a nuclear blast though 

[abaxas]

This doesn't sound like a backup issue at all. It's an archiving one.

The first question should be, do you actually need to keep the raw video? I know it's human nature to hoard everything but does it actually have any use?

[EEVblog]

This doesn't sound like a backup issue at all. It's an archiving one.

Yes, basically archiving.

The first question should be, do you actually need to keep the raw video? I know it's human nature to hoard everything but does it actually have any use?

I want to keep it for the reasons I can't think of right now!
One basic requirement is that the original source material is higher quality than what is uploaded to youtube.
Recent ones are pretty close in quality, but anything below say 400 or so was optimised for upload time and not video quality.
And there are potentially outtakes and other extra stuff.
Plus the editor project files, and I can go back in and re-render the original video if needed, or make new variations, extended versions etc.

It's silly to just throw it all away if I can at the very least store it all on a few hard drives that cost sub $100 a pop. So might as well do that.
Lets say it costs me a few hundred dollars every 500 videos in storage, that's pretty cheap I think.

[EEVblog]

Yes, I'm considering getting a quality Bluray drive and brand name discs as an extra backup.
Can probably fit 5 average episodes on each disc.

Seems you can buy x8 and x16 write speed BR drives, but only x4 write media?

[Legit-Design]

Overspeed writing? Like in normal CD-R times, could write a cd really fast. Or could write it really slow and be sure it would not crap out at some point, when some bit didn't settle there correctly.

http://www.soundonsound.com/sos/nov04/articles/qa1104-3.htm

Different burn speeds do indeed have a measurable effect on the quality of the signal burned into a CD-R, but it also depends on the quality of the CD burner and the CD-R itself.

So there might be some difference in terms of reliability. I'm sure some audiophiles can also hear the difference?

[XOIIO]

Never tried burning standard files with imgburn but it should allow you to verify those as well, it has that option for ISO's at least, it helps make sure there were no mistakes.

[Legit-Design]

Never tried burning standard files with imgburn but it should allow you to verify those as well, it has that option for ISO's at least, it helps make sure there were no mistakes.

md5sum
sha1sum
works every time.

[XOIIO]

Never tried burning standard files with imgburn but it should allow you to verify those as well, it has that option for ISO's at least, it helps make sure there were no mistakes.

md5sum
sha1sum
works every time.

What is the difference between the two? Never heard of sha1sum, a bit of googling just hints that it's a bigger hash? Also saw mention of SHA256

[Legit-Design]

What is the difference between the two? Never heard of sha1sum, a bit of googling just hints that it's a bigger hash? Also saw mention of SHA256

MD5 collisions. Security. Tinfoil hats.
Remember stuxnet? And it's close relative Flame? http://arstechnica.com/security/2012/06/flame-crypto-breakthrough/
Some really big entity took really smart people and state of the art supercomputers, then they made something that was not suppose to happen and made it happen.

[SirNick]

Disc chemistry will dictate the best burn rate.  Or at least it has for previous optical media formats.  Don't over-speed them, don't under-speed them.  Choose a speed that it seems to like best and stick to it.  I just let the drive choose unless I have some reason to override this.  I have CD-Rs I burnt back in circa 2000 that are still readable.  Good burner hardware and quality media is the key here.

So, what I would do in your case is this:  Buy yourself the largest established HDD at the time.  That is, don't buy a 4TB when they're new.  Stick to the 2TB models for a while.  This is how I populate my (home-built rack-mounted Linux MD RAID) NAS.  When the 1TB drives filled up, I took them out and replaced them with 2TB drives.  I then used the 1TB drives to back up the NAS.  Since I wasn't losing 1/4 of my space due to RAID-5 overhead, this was pretty much enough until I filled up the 2TB drives.  I'm about due to replace those with larger disks, and the cycle begins again.  This way, you're not leaving drives for decades, but constantly refreshing the hardware.  Just use a USB dock to copy from NAS to backup HDD, then store it somewhere safe.  You don't need to bomb-proof the things, just put them somewhere they won't be disturbed, in the little plastic clam-shell the new drives came in.

It would also be wise to back up individual episodes (or a couple, depending on how much space it takes) on BD-R.  Keep these in sleeves in a binder or storage box so you can go right to the material of interest if/when you decide to revisit old footage.  If the disc fails, go to your HDD backup.  This will keep the power-cycle count low on the spindle motor.  The optical disc won't care how much it's used.

MD5Sums of all the data couldn't hurt, so you can tell if there was bit-rot along the way.  But usually, the medium itself will have a CRC of its own at some level that will fail in obvious ways (read errors) so, meh..  Keep in mind, it's merely an alert that something has gone bad, not prevention, so consider the worth of that feature before going to any trouble.  I do have a script that updates a per-directory md5sum on my media share for this purpose, so I can eventually write another script that compares this to the actual file checksums and lets me know if a movie file has gone bad.

[nctnico]

The problem with storing hard drives, tapes, disks, etc is that you don't know whether it is good or not until you plug it in. Interfaces and media becoming obsolete is also a problem. Nowadays I keep backups on 3 seperate disks in 3 seperate systems which are synchronised using rsync every night and supplemented with the work I did during the day. Because the drives are active I know immediately when one of the drives has failed so I can take immediate action to replace it. If the disks are full I just add/replace the disks. It acts more or less a distributed RAID array which protects me from losing data by fire, flooding and theft.

[pickle9000]

I have used every kind of media over the years for backup but I find now I tend to use two hard drives, fill, unpowered, and store one off site. I use exactly the same drive and I prefer WD drives (personal preference).

Two years ago I transferred many of the files from old drives / media to a pair of new hard drives. I did actually find the odd file fault but was able to recover a good one from the other source. So for me the lesson is two backups are far superior than one regardless of the media.

The only other thing is that I keeps rough paper records with the drives. That has proven to be a very big plus over the years.

[SirNick]

Because the drives are active I know immediately when one of the drives has failed so I can take immediate action to replace it. If the disks are full I just add/replace the disks. It acts more or less a distributed RAID array which protects me from losing data by fire, flooding and theft.

Still.... RAID and rsync offer zero protection from faults above the physical layer, like file system corruption or accidental deletion.  All it takes is one virus or a rogue rm -rf and everything's gone.  As long as you catch this before it cascades, great.  You're working within a window constrained by single-point-of-failure on one end, and margin-for-error on the other.

Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)  I have had some develop media errors over time (so have two backups if the data is important), but the usual fate is throwing away those that were just too small to be of any practical use anymore.