Author Topic: OT: Backing up video files long term  (Read 34937 times)

0 Members and 1 Guest are viewing this topic.

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
OT: Backing up video files long term
« on: July 03, 2014, 03:21:33 am »
Opinions invited:
What is the best way to physically backup all my raw (and edited) video files for the long term?
I can fit maybe 500 episodes  + other misc and family stuff onto a 2TB drive, and I currently have a NetGear ReadyNAS RAID system to do that, plus the odd drive here and there, but it's close to full.
Yes, I know that RAID is NOT proper "backup" so please spare me the lecture  :P

I'm thinking maybe 3 separate hard drives in 3 locations per every 500 episodes?
Or should I keep multiple RAID disk arrays, enclosures and all in different locations?

Online cloudy solutions will likely eat into my data allowance at the lab, need to look into that, but can't rely on it solely.
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #1 on: July 03, 2014, 03:37:12 am »
get a few harddisk in usb enclosure.

for long term storage : two sets. one off-site ( at home ) one in lab. preventing fire disasters.
drives are to be in a good schock munted enclosure and powered off. get drives that physically retract heads off-platter.
drives are to be filled with data in 1 shot. then powered down until needed. let's say you got 2.3 terabyte of data : 'seal' 2 terabyte onto two 2 terabyte drives.
the remainder is kept alive on your nas system (until you have collected enough to fill another drive.

i would use 1 terabyte drives. that way you can fill the drive faster and 'seal it' ( the goal is to mimimize the amount of power cyles on the drive. )

we d this for project data. once a project is over an the design can be archived it is dumped to a set of brand-spanking new harddisks in one shot. they are written , verified and then powered off.
one drive is stored in the archival safe on site , one drive goes to an archival company. third and fourth sets are created at other sites around the world through our network. even if a drive is not full it is considered 'spent' on a project.  drives are cheap.

avoid nas boxes as the plague for this.. nas boxes often use a custom or weird  file system. if the nas hardware dies the drive is unreadable unless you have an identical nas. never mind the nas may use ext3 or anther linux filesystem. if you have it formatted in a linksys nas it will not be readable by a wd nas... even though they both use linux internally. there is metadata and journalling and other proprietary stuff and that may not be compatible between nas boxes from different brands !
a drive in usb enclosure does not have this problem. they are simple sata drives formatted in whatever filesystem used by the computer that wrote them. if the usb translator dies : plonk another one on there and it will work. worst case hook it to an esata or sata port and it will work.
« Last Edit: July 03, 2014, 03:46:11 am by free_electron »
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #2 on: July 03, 2014, 04:25:24 am »
I can't recall for sure, but don't tape drives have pretty decent storage if kept safe (eg away from magnetic interference sources)?

I'd say just some standard drives in some insanely good packaging would be the best option though.

You could also laser cut the binary into metal sheets to decode later  :-DD

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #3 on: July 03, 2014, 04:27:42 am »
drives are to be in a good schock munted enclosure and powered off. get drives that physically retract heads off-platter.

Don't they all do that these days?

I've got a 3.5" hot swap sata dock in my lab machine, so can use that to backup onto individual drives and then store them in different locations.
If I used 2 drives in two locations, plus maybe a cloud solution as well, then that's gotta cover all eventualities.
I'll keep the NAS for local use on the most recent stuff.
My desktop editing machine has two hard drives (in addition to the SSD boot drive), incrementally mirrored daily.
 

Offline KK

  • Regular Contributor
  • *
  • Posts: 100
  • Country: us
Re: OT: Backing up video files long term
« Reply #4 on: July 03, 2014, 04:28:59 am »
I use LTO tapes. I've been using tape for decades quantum-DDS-DLT-LTO.

Copies on site, and in a bank vault.

LTO-4 drives use 800/1600GB tapes. Video will be somewhere between the two numbers depending on compressibility.

LTO-6 drives are out now with 2.5/5.0TB capacities.

Good long term archival storage. The LTO-4 drive does 550GB/hr using A SCSI interface.

LTO-6 is up to 1.4TB/hr

I have had situations where I needed a backup copy off 8 year old tapes. And it was not a problem for several TB's of restores.

I think optical storage has around 20 years or so shelf life because of eventual rot, and the magnetic tapes are around that lifespan.
« Last Edit: July 03, 2014, 04:37:31 am by KK »
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #5 on: July 03, 2014, 04:31:41 am »
Currently backing up 61 episodes worth of footage (284GB) to the networked NAS, and that's going to take 8 hours at 10MB/sec  :=\
 

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 15379
  • Country: za
Re: OT: Backing up video files long term
« Reply #6 on: July 03, 2014, 04:42:10 am »
Careful on tape. Very long life, but if the drives are no longer made they are essentially useless ornaments. I have old backups made on assorted tapes, that are no longer usable as the drive or the software no longer is usable on modern computers.

You might look into a BD recorder and store the video as data on them, making 2 copies as additional insurance. Store the discs in jewel cases inside a brown paper envelope, and keep in a box in a cabinet. A lot more space, but there you at least can read them with current drives, and the drives likely will be around for a while. No proprietary software needed to read them.
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #7 on: July 03, 2014, 04:45:48 am »
Good to have confirmation that tape drives do work well as I thought, I have a dell powervault tape storage system, it's pretty sweet, I just need the cable to hook it up to my server.

Currently backing up 61 episodes worth of footage (284GB) to the networked NAS, and that's going to take 8 hours at 10MB/sec  :=\

Would it not be faster to copy it to an external drive then directly onto the NAS, rather than over the network?

You could also announce your channel is closing, when that happened with photonicinduction there was a torrent of every single video he uploaded, it's probably still alive, I should re-download it to watch some of his old stuff.

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4310
  • Country: us
  • KE7GKP
Re: OT: Backing up video files long term
« Reply #8 on: July 03, 2014, 05:02:08 am »
I don't trust ANY kind of field-burnable optical media.  Not LONG term archival storage IMHO, IME.

99.99% of this planets most valuable data is backed up and archived (two different functions) on digital magnetic tape.
While the concern for long-term viability of the format/equipment is a concern,  reviewing (and re-writing, if necessary) archive tapes every n years is only prudent if the data is worth it.
Hard drives are iffy, IMHO. I have too many of them that just died on the shelf in long-term archive service.
If I wanted to do that, I would make at least two copies on DIFFERENT BRAND hard drives, and store them sealed in plastic zipper bags in separate locations (one at the office, and the other at home, etc.)  The reason for different brands is critical because there have been several cases of bad batches of hard drives which created catastrophic problems for people "putting all their bits in one basket".
I would also at least plug in the hard drives every year or two just to confirm that still spin up and are readable, etc.

I use raw drives for archive storage and use a docking station like this to access them...



http://www.amazon.com/Plugable-Lay-Flat-Docking-Station-ASM1053E/dp/B00APP6694

And there are many storage options to protect raw drives when you aren't accessing them.  For example...



http://www.ebay.com/itm/like/300540328686?lpid=82
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #9 on: July 03, 2014, 05:13:09 am »
A really slick option would be to take the drive(s), find a good solid enclosure, cap off the top, fill the outside with spray foam all the way to the end, then have some other type of shock absorbing foam around that. Then maybe rubber bumpers and a metal case around that to close it off. You could also make a cover with spray foam if the metal case closed up.

Offline TheBorg

  • Frequent Contributor
  • **
  • Posts: 345
  • Country: us
  • Hoping to start an EE degree soon...
Re: OT: Backing up video files long term
« Reply #10 on: July 03, 2014, 05:18:45 am »
I don't trust ANY kind of field-burnable optical media.  Not LONG term archival storage IMHO, IME.

99.99% of this planets most valuable data is backed up and archived (two different functions) on digital magnetic tape.
While the concern for long-term viability of the format/equipment is a concern,  reviewing (and re-writing, if necessary) archive tapes every n years is only prudent if the data is worth it.
Hard drives are iffy, IMHO. I have too many of them that just died on the shelf in long-term archive service.
If I wanted to do that, I would make at least two copies on DIFFERENT BRAND hard drives, and store them sealed in plastic zipper bags in separate locations (one at the office, and the other at home, etc.)  The reason for different brands is critical because there have been several cases of bad batches of hard drives which created catastrophic problems for people "putting all their bits in one basket".
I would also at least plug in the hard drives every year or two just to confirm that still spin up and are readable, etc.

I use raw drives for archive storage and use a docking station like this to access them...



http://www.amazon.com/Plugable-Lay-Flat-Docking-Station-ASM1053E/dp/B00APP6694

And there are many storage options to protect raw drives when you aren't accessing them.  For example...



http://www.ebay.com/itm/like/300540328686?lpid=82

Would ESD be an issue? Seems to me a plastic bag would have the potential to fry circuits on the control boards. Agree on the different brands though.
That being said, I store a couple of raw drives in a plastic drawer around here...
Youtube Channel - Assimilated Circuits
We are Borg. You will be assimilated. Resistance is futile.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #11 on: July 03, 2014, 06:04:42 am »
A really slick option would be to take the drive(s), find a good solid enclosure, cap off the top, fill the outside with spray foam all the way to the end, then have some other type of shock absorbing foam around that. Then maybe rubber bumpers and a metal case around that to close it off. You could also make a cover with spray foam if the metal case closed up.

Pelican case!  :-+
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #12 on: July 03, 2014, 06:36:35 am »
A really slick option would be to take the drive(s), find a good solid enclosure, cap off the top, fill the outside with spray foam all the way to the end, then have some other type of shock absorbing foam around that. Then maybe rubber bumpers and a metal case around that to close it off. You could also make a cover with spray foam if the metal case closed up.

Pelican case!  :-+

Pretty much, I think the spray insulation foam might be a good other layer to add, then put it in a pelican case that just pinches it between the foam cones.

Although I think we are getting into territory of dropping the hard drives, from considerable heights rather than just storage  ^-^

Guess it would be good to have videos survive a nuclear blast though  ;D

Offline abaxas

  • Regular Contributor
  • *
  • Posts: 131
Re: OT: Backing up video files long term
« Reply #13 on: July 03, 2014, 09:00:11 am »
This doesn't sound like a backup issue at all. It's an archiving one.

The first question should be, do you actually need to keep the raw video? I know it's human nature to hoard everything but does it actually have any use?
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #14 on: July 03, 2014, 09:44:09 am »
This doesn't sound like a backup issue at all. It's an archiving one.

Yes, basically archiving.

Quote
The first question should be, do you actually need to keep the raw video? I know it's human nature to hoard everything but does it actually have any use?

I want to keep it for the reasons I can't think of right now!
One basic requirement is that the original source material is higher quality than what is uploaded to youtube.
Recent ones are pretty close in quality, but anything below say 400 or so was optimised for upload time and not video quality.
And there are potentially outtakes and other extra stuff.
Plus the editor project files, and I can go back in and re-render the original video if needed, or make new variations, extended versions etc.

It's silly to just throw it all away if I can at the very least store it all on a few hard drives that cost sub $100 a pop. So might as well do that.
Lets say it costs me a few hundred dollars every 500 videos in storage, that's pretty cheap I think.
« Last Edit: July 03, 2014, 09:46:34 am by EEVblog »
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #15 on: July 03, 2014, 10:29:06 am »
Yes, I'm considering getting a quality Bluray drive and brand name discs as an extra backup.
Can probably fit 5 average episodes on each disc.

Seems you can buy x8 and x16 write speed BR drives, but only x4 write media?
« Last Edit: July 03, 2014, 10:38:00 am by EEVblog »
 

Offline Legit-Design

  • Frequent Contributor
  • **
  • Posts: 562
Re: OT: Backing up video files long term
« Reply #16 on: July 03, 2014, 10:52:15 am »
Overspeed writing? Like in normal CD-R times, could write a cd really fast. Or could write it really slow and be sure it would not crap out at some point, when some bit didn't settle there correctly.

http://www.soundonsound.com/sos/nov04/articles/qa1104-3.htm
Quote
Different burn speeds do indeed have a measurable effect on the quality of the signal burned into a CD-R, but it also depends on the quality of the CD burner and the CD-R itself.
So there might be some difference in terms of reliability. I'm sure some audiophiles can also hear the difference?
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #17 on: July 03, 2014, 10:54:37 am »
Never tried burning standard files with imgburn but it should allow you to verify those as well, it has that option for ISO's at least, it helps make sure there were no mistakes.

Offline Legit-Design

  • Frequent Contributor
  • **
  • Posts: 562
Re: OT: Backing up video files long term
« Reply #18 on: July 03, 2014, 10:56:23 am »
Never tried burning standard files with imgburn but it should allow you to verify those as well, it has that option for ISO's at least, it helps make sure there were no mistakes.
md5sum
sha1sum
works every time.
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #19 on: July 03, 2014, 11:07:23 am »
Never tried burning standard files with imgburn but it should allow you to verify those as well, it has that option for ISO's at least, it helps make sure there were no mistakes.
md5sum
sha1sum
works every time.

What is the difference between the two? Never heard of sha1sum, a bit of googling just hints that it's a bigger hash? Also saw mention of SHA256
« Last Edit: July 03, 2014, 11:09:04 am by XOIIO »
 

Offline Legit-Design

  • Frequent Contributor
  • **
  • Posts: 562
Re: OT: Backing up video files long term
« Reply #20 on: July 03, 2014, 11:20:25 am »
What is the difference between the two? Never heard of sha1sum, a bit of googling just hints that it's a bigger hash? Also saw mention of SHA256
MD5 collisions. Security. Tinfoil hats.
Remember stuxnet? And it's close relative Flame? http://arstechnica.com/security/2012/06/flame-crypto-breakthrough/
Some really big entity took really smart people and state of the art supercomputers, then they made something that was not suppose to happen and made it happen.
 

Offline SirNick

  • Frequent Contributor
  • **
  • Posts: 589
Re: OT: Backing up video files long term
« Reply #21 on: July 03, 2014, 06:47:29 pm »
Disc chemistry will dictate the best burn rate.  Or at least it has for previous optical media formats.  Don't over-speed them, don't under-speed them.  Choose a speed that it seems to like best and stick to it.  I just let the drive choose unless I have some reason to override this.  I have CD-Rs I burnt back in circa 2000 that are still readable.  Good burner hardware and quality media is the key here.

So, what I would do in your case is this:  Buy yourself the largest established HDD at the time.  That is, don't buy a 4TB when they're new.  Stick to the 2TB models for a while.  This is how I populate my (home-built rack-mounted Linux MD RAID) NAS.  When the 1TB drives filled up, I took them out and replaced them with 2TB drives.  I then used the 1TB drives to back up the NAS.  Since I wasn't losing 1/4 of my space due to RAID-5 overhead, this was pretty much enough until I filled up the 2TB drives.  I'm about due to replace those with larger disks, and the cycle begins again.  This way, you're not leaving drives for decades, but constantly refreshing the hardware.  Just use a USB dock to copy from NAS to backup HDD, then store it somewhere safe.  You don't need to bomb-proof the things, just put them somewhere they won't be disturbed, in the little plastic clam-shell the new drives came in.

It would also be wise to back up individual episodes (or a couple, depending on how much space it takes) on BD-R.  Keep these in sleeves in a binder or storage box so you can go right to the material of interest if/when you decide to revisit old footage.  If the disc fails, go to your HDD backup.  This will keep the power-cycle count low on the spindle motor.  The optical disc won't care how much it's used.

MD5Sums of all the data couldn't hurt, so you can tell if there was bit-rot along the way.  But usually, the medium itself will have a CRC of its own at some level that will fail in obvious ways (read errors) so, meh..  Keep in mind, it's merely an alert that something has gone bad, not prevention, so consider the worth of that feature before going to any trouble.  I do have a script that updates a per-directory md5sum on my media share for this purpose, so I can eventually write another script that compares this to the actual file checksums and lets me know if a movie file has gone bad.
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 19369
  • Country: nl
    • NCT Developments
Re: OT: Backing up video files long term
« Reply #22 on: July 03, 2014, 07:21:14 pm »
The problem with storing hard drives, tapes, disks, etc is that you don't know whether it is good or not until you plug it in. Interfaces and media becoming obsolete is also a problem. Nowadays I keep backups on 3 seperate disks in 3 seperate systems which are synchronised using rsync every night and supplemented with the work I did during the day. Because the drives are active I know immediately when one of the drives has failed so I can take immediate action to replace it. If the disks are full I just add/replace the disks. It acts more or less a distributed RAID array which protects me from losing data by fire, flooding and theft.
« Last Edit: July 03, 2014, 07:22:51 pm by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #23 on: July 03, 2014, 07:36:02 pm »
I have used every kind of media over the years for backup but I find now I tend to use two hard drives, fill, unpowered, and store one off site. I use exactly the same drive and I prefer WD drives (personal preference).

Two years ago I transferred many of the files from old drives / media to a pair of new hard drives. I did actually find the odd file fault but was able to recover a good one from the other source. So for me the lesson is two backups are far superior than one regardless of the media.

The only other thing is that I keeps rough paper records with the drives. That has proven to be a very big plus over the years.
 

Offline SirNick

  • Frequent Contributor
  • **
  • Posts: 589
Re: OT: Backing up video files long term
« Reply #24 on: July 03, 2014, 10:59:52 pm »
Because the drives are active I know immediately when one of the drives has failed so I can take immediate action to replace it. If the disks are full I just add/replace the disks. It acts more or less a distributed RAID array which protects me from losing data by fire, flooding and theft.

Still.... RAID and rsync offer zero protection from faults above the physical layer, like file system corruption or accidental deletion.  All it takes is one virus or a rogue rm -rf and everything's gone.  As long as you catch this before it cascades, great.  You're working within a window constrained by single-point-of-failure on one end, and margin-for-error on the other.

Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)  I have had some develop media errors over time (so have two backups if the data is important), but the usual fate is throwing away those that were just too small to be of any practical use anymore.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #25 on: July 03, 2014, 11:07:01 pm »
It would also be wise to back up individual episodes (or a couple, depending on how much space it takes) on BD-R.  Keep these in sleeves in a binder or storage box so you can go right to the material of interest if/when you decide to revisit old footage.  If the disc fails, go to your HDD backup.  This will keep the power-cycle count low on the spindle motor.  The optical disc won't care how much it's used.

Yes, I think I'll do this in addition to the two hard drives.

 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #26 on: July 03, 2014, 11:08:30 pm »
I have used every kind of media over the years for backup but I find now I tend to use two hard drives, fill, unpowered, and store one off site. I use exactly the same drive and I prefer WD drives (personal preference).

As someone else noted, I wouldn't use the exact same drive for both backups, but two different brands.
I think I'll go buy a bunch of 1TB WD and 1TB Seagate drives.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #27 on: July 03, 2014, 11:10:18 pm »
Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)  I have had some develop media errors over time (so have two backups if the data is important), but the usual fate is throwing away those that were just too small to be of any practical use anymore.

Any archive or backup system, regardless of how complex, is a simple (or maybe not so simple) matter of probability theory.
 

Offline ovnr

  • Frequent Contributor
  • **
  • Posts: 658
  • Country: no
  • Lurker
Re: OT: Backing up video files long term
« Reply #28 on: July 03, 2014, 11:47:47 pm »
Well, if you want an online solution, Amazon's S3 Glacier storage is pretty nice. $10/month per terabyte stored, pretty reliable ("99.999999999% durability of objects in storage", survives the complete loss of two datacenters). Obviously more expensive than disks over time, especially for massive heaps of data - but I consider it manageable. I don't use it myself (yet) - I currently run two 6x2TB RAIDZ-2 arrays and generally swap disks when they get five years old; this costs me around $400/year compared to $2000/year, but it's offsite and as reliable as you can hope for.

I'm going to be replicating all of my irreplaceable data onto S3 eventually; it's just a couple TB or so. The rest can be rebuilt, but at considerable cost.
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #29 on: July 04, 2014, 12:23:26 am »
Moving to a more modern (backup) system allowed me to reduce the physical size of the backups, and throw out some very old (large and cluncky / slow) hardware. I'm not saying it was unreliable just old. It was also good to see that the data was still viable.

The main motivation was to reduce the size from that of a couch sized space to that which could fit in a shoebox. Remember not just the media and or drives but a couple old computer systems to go with it.   
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #30 on: July 04, 2014, 12:24:52 am »
Why not just buy the most reliable one?

Because the point is that if you use two identical drives then you have the added (albeit small) percentage that a design or manufacturing defect in that model of drive will render both of your backups useless in X years time.
Especially so if you buy two identical models from the same manufacturing batch. Likely if you go to the same store to buy them, as they will likely get them in boxes from the same batch.

"The most reliable" model is data that is essentially only available in hindsight.
 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 6298
  • Country: gb
Re: OT: Backing up video files long term
« Reply #31 on: July 04, 2014, 12:36:46 am »
Currently backing up 61 episodes worth of footage (284GB) to the networked NAS, and that's going to take 8 hours at 10MB/sec  :=\

And that's why they invented gigabit ethernet. 15 years ago.
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #32 on: July 04, 2014, 12:45:16 am »
Why not just buy the most reliable one?

Because the point is that if you use two identical drives then you have the added (albeit small) percentage that a design or manufacturing defect in that model of drive will render both of your backups useless in X years time.
Especially so if you buy two identical models from the same manufacturing batch. Likely if you go to the same store to buy them, as they will likely get them in boxes from the same batch.

"The most reliable" model is data that is essentially only available in hindsight.

I agree that using 2 different brands is a better system than what I have, no question. I do run the WD drives I use for a few hundred hours to help ensure they are up to snuff. That's not the same as using completely different hardware. My personal preference for WD relates to problems with the other brand Dave mentioned. Even so I will probably throw one into the mix next time I do a backup. 
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4310
  • Country: us
  • KE7GKP
Re: OT: Backing up video files long term
« Reply #33 on: July 04, 2014, 01:17:30 am »
Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)
I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.  This is why a single hard drive is NOT reliable backup or archival media IME. I have also seen raid arrays fail because they used several drives from a single defective batch.

Hard drives are OK for backup and archival storage, as long as you take appropriate precautions.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #34 on: July 04, 2014, 02:17:22 pm »
They are basically magneto-optical, which as we know is extremely robust and long lived. You still need to store them in a fairly cool, dark place so that they don't warp over time. Archival grade discs are being advertised as ideal for keeping your precious family videos for generations and testing does suggest that the claim is not exaggerated. Of course at 50GB/disc you will need a lot of discs, and ideally two copies for security. Unlike tape you shouldn't have any trouble reading the discs in 20 or 30 years time, considering that a modern BluRay player will still play an 80s CD.

BluRay is not even close to magneto-optical except maybe in reliability and I have doubts about that.  I still use my 3.5" magneto-optical drives which hold up to 640MB.

I would consider archival BluRay a good choice except for price.  Even multiply redundant hard drive storage is less expensive but I wonder what their unpowered lifetime is.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #35 on: July 04, 2014, 02:32:38 pm »
I have had good results backing up onto good single layer DVD+R disks but those will be too small for your use.  I generate and save PAR2 recovery files along with the data and write the disks out at the slowest burner speed with verification and both UDF and ISO file system data so in the event of a damaged disk, recovery is easier.  The only disks failures I have had in 10 years were either do to cheap media or actual physical damage and almost all of them were completely recoverable.  I suspect BluRay would be similarly reliable under these conditions.

For larger data sets I would go with two identical file servers running Linux or BSD using 4 disk RAID5 or perhaps one of the newer file systems which support data integrity like ZFS or Btrfs.
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 19369
  • Country: nl
    • NCT Developments
Re: OT: Backing up video files long term
« Reply #36 on: July 04, 2014, 03:46:07 pm »
Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)
I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.  This is why a single hard drive is NOT reliable backup or archival media IME.
I disagree. That particular failure mode is (often) caused by hard drives running too hot. If you keep hard drives cool you won't have this problem.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 3938
  • Country: ro
  • .
Re: OT: Backing up video files long term
« Reply #37 on: July 04, 2014, 04:19:08 pm »
I thought about this as well and came to the conclusion that just plain hard drives in a RAID is the most future proof in long term.

Bluray is only ok for temporary storage, a few months to a year. After that, just like with DVDs, you can't really trust the media.  It's cheap enough (2-3$ for 25 GB) to have a 3rd backup on blurays but not as a primary backup.

Tapes are cool, lots of storage relatively cheap and in small space but like others said, you have the problem of continuously having to upgrade your tapes to whatever comes up next year or to keep buying tape reader/writers (and have some as backups) just in case they're discontinued at some point and your main unit breaks down.

Even though I respect free_electron and his experience, I don't agree with his suggestion of turning on hard drive, storing the data and then turning off the drive. Maybe it's silly, but I don't trust a hard drive NOT failing at startup all of the sudden, I'd be more comfortable with having the hard drive running 24/7 and potentially catching some error in close to real time and have the hard drive replaced.
I would go with one of those units that allow you to connect 4 or more hard drives and set up a raid5 or better and check from time to time the state of the disks and take corrective measures when needed.
In your case, you could have an identical unit at home (but maybe with other drives or another series just in case there's a bad batch) and you could just configure the home unit to slowly copy throughout the day what you have stored at the office (doesn't have to fast, it can be just 1mbps upload for example, you have it running 24/7 anyway)

Maxxarcade (of youtube "fame") recently upgraded his raid storage from a server with lots of ATA drives to a Synology DS1813+ ( http://www.newegg.com/Product/Product.aspx?Item=N82E16822108138 ) that can hold 8 hard drives, has 4 network ports that can be linked together for higher throughput  etc etc. It's a bit expensive, but there's the cheaper DS1513+ also which can only hold 5 drives.
I think it's a good investment if you're serious about backup.



PS. For extra security, you could use a tool like PAR2 to create recovery volumes for the video files. With about 20% of the video size in recovery volumes, you can recover pretty much any video file if it's just a bunch of bits or bytes messed up randomly (it won't help if there's a continuous 10-20 MB of totally damaged data)
md5sum and sha1sum will just make a checksum and that tells you if the file got corrupted or not, but par also is capable of repairing the files.
« Last Edit: July 04, 2014, 04:23:22 pm by mariush »
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 1848
  • Country: sk
Re: OT: Backing up video files long term
« Reply #38 on: July 04, 2014, 04:30:01 pm »
ok i didn't read the whole thread, but my 2 cents :

1. blue-ray - do you guys have experience with 5+ years of storage of those ? i guess not. if blue-ray (it's cheap) then refresh every 2-3 years to fresh media.

2. tape drives - most expensive but it's a industry standard and very reliable. the best for archiving are the WORM LTO tapes  (Write Once Read Many) archived in controlled environment (humidity, temperature). but as mentioned it's freaking expensive solution to set up, with a good price per TB once  it's there, but still i wouldn't recommend this for archiving videos just because of the price tag (not just the hardware and tapes, but the archiving of the tapes in a controlled environment gets expensive). and let's not forget - you would need to refresh the tapes as well - it's usually 10+ years guaranteed for properly stored WORM media.

3. hard drives - cheap and reliable to some extent. but still i would recommend to keep 3 independent copies offline and regular check/refresh

i think at the end of the day it doesn't matter whether blue-ray or HDD - the cost is very similar for those methods and both would require regular refresh.
 

Online madires

  • Super Contributor
  • ***
  • Posts: 5110
  • Country: de
  • A qualified hobbyist ;)
Re: OT: Backing up video files long term
« Reply #39 on: July 04, 2014, 04:40:48 pm »
avoid nas boxes as the plague for this.. nas boxes often use a custom or weird  file system. if the nas hardware dies the drive is unreadable unless you have an identical nas. never mind the nas may use ext3 or anther linux filesystem. if you have it formatted in a linksys nas it will not be readable by a wd nas... even though they both use linux internally. there is metadata and journalling and other proprietary stuff and that may not be compatible between nas boxes from different brands !

The problem is based on the version of the linux kernel and the RAID tools used. I've managed to rescue NAS disks (RAID 1) by attaching them directly to a recent linux system. But I agree that USB disks are the best choice for backup at the moment, unless you have to backup a datacenter.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #40 on: July 04, 2014, 05:44:55 pm »
Forget hard drives, they are totally unsuitable for long term offline storage. Modern drives need to periodically re-write data to the platters. It slowly degrades over time, and the surface of the disc itself degrades due to dust and various natural processes that affect metals. The magnetic fields are extremely weak and even the earth's shifting magnetism slowly each bit down.

The coercivity of the media in hard drives is incredibly high to support their high data density in the face of thermally activated relaxation so external magnetic fields, even strong ones, have an insignificant effect.  Magneto-optical media is tough for the same reason.

Quote
They come with errors from the factory and rely on the controller to map them out on the fly - they certainly don't get a 100% test. Don't get me wrong, they are miracles of technology, but they do rely on being powered up and able to re-write data in the background to avoid slow degradation and correct errors before blocks become unreadable.

I agree about using them for long term offline storage although I suspect they will fair better than many alternatives.  For that reason, I would use them in a small self contained server and run them through a complete background scrub while updating the contents.

Quote
Amazon S3 Glacier is very interesting. Hopefully competition will continue to push prices down, but 1TB is enough for me to back up a couple of machines. Sure, I have more data than that, but I can always re-rip/re-download a lot of it. It really depends if there is any good backup software that supports it. I will look.

At least where I am in the US, internet based storage is totally useless because of both bandwidth and ISP transfer caps.
 

Offline IanJ

  • Supporter
  • ****
  • Posts: 1074
  • Country: scotland
  • Pro EE guy many years ago, now a hobby/home biz.
    • IanJohnston.com
Re: OT: Backing up video files long term
« Reply #41 on: July 04, 2014, 08:37:57 pm »
Hi,

My own backup solution for my Vegas Movie Studio video files is a 2nd drive on my PC where the files sit, and a QNAP NAS running RAID 5 (4 drives).
I also have a 2nd QNAP NAS running RAID 1 (2 drives) which is where I backup all my other data etc including my 1&1 dedicated server (web & email) backups.
All drives on both QNAPs are WD Enterprise edition RE3 drives.

Works for me, dead easy to administer. Have only ever had 1 drive failure (a 13 month old WD RE3) on the RAID 1 NAS.

Tips: Keep the server cupboard temperature stable, keep it dust free, and install a wee UPS.

Ian.
« Last Edit: July 04, 2014, 09:01:01 pm by IanJ »
Ian Johnston
www.ianjohnston.com
Manufacturer of the PDVS2 & PDVS2mini
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #42 on: July 04, 2014, 11:39:59 pm »
Hi,

My own backup solution for my Vegas Movie Studio video files is a 2nd drive on my PC where the files sit, and a QNAP NAS running RAID 5 (4 drives).
I also have a 2nd QNAP NAS running RAID 1 (2 drives) which is where I backup all my other data etc including my 1&1 dedicated server (web & email) backups.
All drives on both QNAPs are WD Enterprise edition RE3 drives.

Works for me, dead easy to administer. Have only ever had 1 drive failure (a 13 month old WD RE3) on the RAID 1 NAS.

Tips: Keep the server cupboard temperature stable, keep it dust free, and install a wee UPS.

Ian.
I'll guessing you use one that sends a shutdown signal when the batteries get below a certain percentage in case of long term power failure?

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: OT: Backing up video files long term
« Reply #43 on: July 05, 2014, 12:14:35 am »
The Library of Congress, when they designate something worthy of the time and effort required, save copies on hard disks and verify the data periodically, then when new hard disk tech comes out, they copy to new media while drives for old media still exist and are serviceable.

Last I heard they use a mix of LTO tape and hard drives.

LTO isn't going anywhere, and the new drives that come out are backwards compatible, IIRC.  LTO is pretty safe.  Hard disk is pretty safe.  I wouldn't bother with optical anything.
 

Offline SL4P

  • Super Contributor
  • ***
  • Posts: 2147
  • Country: au
  • There's more value if you figure it out yourself!
Re: OT: Backing up video files long term
« Reply #44 on: July 05, 2014, 04:52:28 am »
As mentioned previously, you are archiving.
You need to look a the access requirements vs the cost / time to retrieve a specific title etc.

Read about HSM strategy (Hierarchical Storage Management), and consider the various combinations of online, near-line and offline repositories.  The use of stub files can also provide near-instant online access, while loading the large part of for complete playout while the stub is playing.
« Last Edit: July 05, 2014, 04:54:18 am by SL4P »
Don't ask a question if you aren't willing to listen to the answer.
 

Offline IanJ

  • Supporter
  • ****
  • Posts: 1074
  • Country: scotland
  • Pro EE guy many years ago, now a hobby/home biz.
    • IanJohnston.com
Re: OT: Backing up video files long term
« Reply #45 on: July 05, 2014, 09:52:36 am »
I'll guessing you use one that sends a shutdown signal when the batteries get below a certain percentage in case of long term power failure?

That would be a bonus but actually my own UPS I fitted to help with a few 15 minute outages I have had in the past couple of years. The batteries have always lasted.
I also have the UPS keep my modem/routers alive.

One other issue with NAS boxes is I think it is important to power cycle every 6 months. I had a raid drive fail on one of my qnap NAS boxes, it was fine right up to the power up when the drive just failed to start. Hidden problems! Can't imagine maybe more than 1 drive being affected and then suddenly after a power cycle no data!

Seriously, for what Dave needs I would just stick with a good NAS box with quality drives.......and use it for other data also.

Ian.
Ian Johnston
www.ianjohnston.com
Manufacturer of the PDVS2 & PDVS2mini
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #46 on: July 05, 2014, 10:14:25 am »
Forget hard drives, they are totally unsuitable for long term offline storage. Modern drives need to periodically re-write data to the platters. It slowly degrades over time, and the surface of the disc itself degrades due to dust and various natural processes that affect metals. The magnetic fields are extremely weak and even the earth's shifting magnetism slowly each bit down. They come with errors from the factory and rely on the controller to map them out on the fly - they certainly don't get a 100% test. Don't get me wrong, they are miracles of technology, but they do rely on being powered up and able to re-write data in the background to avoid slow degradation and correct errors before blocks become unreadable.
I'm gonna be very blunt here : Stay off of whatever you have been smoking because you are talking out your ass. What you are saying is pure nonsense.

-The surface of the disk is not the recording layer ! That layer is buried. The surface layer is a lubricant with underneath a layer of cristalline carbon (basically grown diamond)
Then there are several other layers before you even hit the actual recording layer.

-earth magnetic field have no impact. You can put a permanent magnet on a harddisk case while it is running. No harm will happen to the data.

-all platters have bad zones. These are NOT mapped on the fly. The mapping happens during manufacturing and is stored in large lookup tables on the drive. If, during operation, the drive encounters problems with a sector it can add this to the existing table. The surface is 100% tested during manufacturing when the drive formatter writes the servo signals and sector marker wedges. This is information is doublechecked and any problems are marked in the lookup table. All harddisks go through a 24 to 48 hour burn-in cycle. It takes about 3 hours for the formatter to do its work ( do not confuse the formatting process in the factory with formatting a filesystem. It has nothing to do with each other. Once a drive is completely assembled and sealed you cannot alter the physical formatting as you need an additional head that is now no longer there. If you look at a harddisk you will find silver metallic small labels , either on the side , or between the cricuit board and case. They cover a hole and or slit. That is where the formatters head enters the drive to write the synchro track on the platters.

-as for the rewriting of data. That is absolute nonsense. We try to minimise the amount of writing.

I design those things for a living. None of what you are saying is true.
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #47 on: July 05, 2014, 10:19:06 am »
avoid nas boxes as the plague for this.. nas boxes often use a custom or weird  file system. if the nas hardware dies the drive is unreadable unless you have an identical nas. never mind the nas may use ext3 or anther linux filesystem. if you have it formatted in a linksys nas it will not be readable by a wd nas... even though they both use linux internally. there is metadata and journalling and other proprietary stuff and that may not be compatible between nas boxes from different brands !

The problem is based on the version of the linux kernel and the RAID tools used. I've managed to rescue NAS disks (RAID 1) by attaching them directly to a recent linux system. But I agree that USB disks are the best choice for backup at the moment, unless you have to backup a datacenter.
The problem is each nas seems to use its own thing. There are rarely updates to the firmware , some have bugs, some use reiser, so e use ext 2 , so e ext3 , some have journaling on, others off.
Nothing but trouble. Some even store the runtime image on the drive.

Usb box, format drive in ntfs and off you go. All computers can read ntfs these days. No need to screw around with obscure things .
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4310
  • Country: us
  • KE7GKP
Re: OT: Backing up video files long term
« Reply #48 on: July 05, 2014, 10:20:46 am »
-earth magnetic field have no impact. You can put a permanent magnet on a harddisk case while it is running. No harm will happen to the data.
You don't even have to put a magnet on your hard drive. If you disassemble one, you will end up with one of the strongest permanent magnets you have in the house. So strong, you could seriously hurt yourself with one if you got pinched between it and a piece of ferrous metal.

You can find all sorts of complete twaddle on the interweb. caveat lector
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4310
  • Country: us
  • KE7GKP
Re: OT: Backing up video files long term
« Reply #49 on: July 05, 2014, 10:26:27 am »
Forget hard drives, they are totally unsuitable for long term offline storage. Modern drives need to periodically re-write data to the platters. It slowly degrades over time, and the surface of the disc itself degrades due to dust and various natural processes that affect metals. The magnetic fields are extremely weak and even the earth's shifting magnetism slowly each bit down. They come with errors from the factory and rely on the controller to map them out on the fly - they certainly don't get a 100% test. Don't get me wrong, they are miracles of technology, but they do rely on being powered up and able to re-write data in the background to avoid slow degradation and correct errors before blocks become unreadable.
Some of what you say is true of "non-volatile" solid-state memory (i.e. "flash memory"). Which is why nobody I know who understands the physics considers NVram to be suitable for "archival" storage.

But NONE of what you wrote has anything to do with conventional rotating-disk hard drives. Complete nonsense.
 

Offline magetoo

  • Frequent Contributor
  • **
  • Posts: 284
  • Country: se
Re: OT: Backing up video files long term
« Reply #50 on: July 05, 2014, 10:45:26 am »
I design those things for a living. None of what you are saying is true.

Do you have a reference you could point to?

Asking because I've been hearing the exact same things that mojo-chan is saying: hard drives nowadays work not because they are error free, but because error correction is so good that the drive can cope with the inevitable faults; that those faults are a part of normal operation; that reading through a drive's sectors is necessary to spot "bits that are getting weak" (hand waving furiously) so that the controller can do its remapping magic, etc.

Would be nice to have something more to go on than "well, a guy on a forum said".
 

Offline magetoo

  • Frequent Contributor
  • **
  • Posts: 284
  • Country: se
Re: OT: Backing up video files long term
« Reply #51 on: July 05, 2014, 11:15:35 am »
For published episodes, maybe archive.org could be an option?  I know many people seem to be using them for video podcasts.

Since you have a ton of them, I would think that there might be some way to solve the problem of getting the bits there other than uploading (like mailing drives).  Their FAQ says to contact them, but doesn't go into details.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #52 on: July 05, 2014, 11:22:30 am »
For published episodes, maybe archive.org could be an option?

Never looked into it.
Although isn't that public only?
I consider my raw video files private, and not something I'd generally publicly release as a whole.
 

Offline magetoo

  • Frequent Contributor
  • **
  • Posts: 284
  • Country: se
Re: OT: Backing up video files long term
« Reply #53 on: July 05, 2014, 11:48:08 am »
Yes, it's not an option for everything.  But for things that are public anyway, the EEVblog episodes, it would be free backup.

(And we viewers get another place to get them from.)


Edit: But maybe I've misunderstood what you mean.  I guess the finished product is small compared to the raw video, so maybe that's not so relevant.
« Last Edit: July 05, 2014, 12:01:33 pm by magetoo »
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #54 on: July 05, 2014, 12:45:44 pm »
Edit: But maybe I've misunderstood what you mean.  I guess the finished product is small compared to the raw video, so maybe that's not so relevant.

Yes, finished and uploaded video is maybe 1/5 the size of the original raw files.
 

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 15379
  • Country: za
Re: OT: Backing up video files long term
« Reply #55 on: July 05, 2014, 02:27:53 pm »
I design those things for a living. None of what you are saying is true.

Do you have a reference you could point to?

Asking because I've been hearing the exact same things that mojo-chan is saying: hard drives nowadays work not because they are error free, but because error correction is so good that the drive can cope with the inevitable faults; that those faults are a part of normal operation; that reading through a drive's sectors is necessary to spot "bits that are getting weak" (hand waving furiously) so that the controller can do its remapping magic, etc.

Would be nice to have something more to go on than "well, a guy on a forum said".

Yes, it is good to refresh the drive info by doing a read every so often. SMART can do this if you ask for a full surface scan, and will take a few hours to run this, and will read all the data on the drive and see if it is all readable after error correction. If the error bursts get too big ( drive firmware dependant) for a sector the drive will then rewrite the corrected data and try again. If it decided the sector is no longer reliable it will use one of the spare sectors it has and add the bad sector to the bad block table and write the data elsewhere.

You can use SMART to do this or use other tools, the most advertised is Spinrite from GRC.com which can recover most data from bad blocks and try to get a last good read from failing bits by statistical analysis of bad bits in marginal cases. It also will do the same for failing MLC and SLC drives where the data slicer gives a bad reading.

If you ever have used Norton Disk Doctor to recover data off a flexible disk, or used TIP to recover a ZIP disk ( I have used it a lot and it works well on ZIP drives with poor sectors to recover the data) you have used Steve Gibson's software invention.
 

Offline SL4P

  • Super Contributor
  • ***
  • Posts: 2147
  • Country: au
  • There's more value if you figure it out yourself!
Re: OT: Backing up video files long term
« Reply #56 on: July 05, 2014, 02:41:05 pm »
Someone should mention that any dynamically writeable media (disk / tape / USB etc), that is visible as a logical volume - is still fair game for malware attacks.

The only 'archive' is not mounted, and the only 'safe archive' is rotated through offsite repositories durong the refresh/resync cycle.
Don't ask a question if you aren't willing to listen to the answer.
 

Online madires

  • Super Contributor
  • ***
  • Posts: 5110
  • Country: de
  • A qualified hobbyist ;)
Re: OT: Backing up video files long term
« Reply #57 on: July 05, 2014, 02:53:16 pm »
The problem is each nas seems to use its own thing. There are rarely updates to the firmware , some have bugs, some use reiser, so e use ext 2 , so e ext3 , some have journaling on, others off.
Nothing but trouble. Some even store the runtime image on the drive.

Actually, the various filesystems aren't a big problem. If the mount command can't detect the filesystem automatically you simply try all possible filesystems (read-only of course). And for the linux extended filesystems you got several choices to get your data. ext3 is ext2 plus journaling, i.e. you can mount an ext3 fs as ext2 read-only and read the files. It's also possible to recover a linux based RAID 5 if the disks are still ok. But that's not a task for the average user.

Quote
Usb box, format drive in ntfs and off you go. All computers can read ntfs these days. No need to screw around with obscure things .

Linux users prefer ext4 ;)
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 19369
  • Country: nl
    • NCT Developments
Re: OT: Backing up video files long term
« Reply #58 on: July 05, 2014, 03:23:26 pm »
Same with people advocating hard drives. Any kind of physical failure and you lose everything. Electrical failure might be recoverable but good luck finding an identical controller in 5 years time.
If that is your opinion you have not been reading too well. The consensus it to keep a backup on multiple hard drives with 24/7 health monitoring. If one fails you replace it and you gradually upgrade to new technology. A bluray disc isn't redundant and if a new technology becomes availabe you will have issues finding a backwards compatible reader.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #59 on: July 05, 2014, 04:22:10 pm »
As I already said, it's magneto optical rather than chemical so does not degrade in the same way as older DVD and CD recordable discs.

BluRay is not magneto-optical; it does not use any magnetic layer, does not use the Kerr effect for reading, and has no bias electromagnet.  It uses a spin-coated dye or pigment or a sputtered alloy or composite just like a recordable or rewritable DVD.
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #60 on: July 05, 2014, 05:06:18 pm »
-earth magnetic field have no impact. You can put a permanent magnet on a harddisk case while it is running. No harm will happen to the data.
You don't even have to put a magnet on your hard drive. If you disassemble one, you will end up with one of the strongest permanent magnets you have in the house. So strong, you could seriously hurt yourself with one if you got pinched between it and a piece of ferrous metal.

You can find all sorts of complete twaddle on the interweb. caveat lector
The field of that magnet is trapped in the horseshoe around the vcm coil.. But you are right. Those magnets are the strongest there are. In fact, they are so strong that , if you place two of them near each other , they will 'jump' towards each other and, on impact, will shatter. The speed and force of impact is such that the magnets crack and chips and slivers fly off. This is very dangerous. Always wear safety goggles when dealing with these suckers.
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 6298
  • Country: gb
Re: OT: Backing up video files long term
« Reply #61 on: July 05, 2014, 05:32:42 pm »
You can use SMART to do this or use other tools, the most advertised is Spinrite from GRC.com which can recover most data from bad blocks and try to get a last good read from failing bits by statistical analysis of bad bits in marginal cases. It also will do the same for failing MLC and SLC drives where the data slicer gives a bad reading.

SpinRite is an archaic joke and Gibson is a prolific liar.
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #62 on: July 05, 2014, 05:33:11 pm »
Yes, it is good to refresh the drive info by doing a read every so often. SMART can do this if you ask for a full surface scan, and will take a few hours to run this, and will read all the data on the drive and see if it is all readable after error correction. If the error bursts get too big ( drive firmware dependant) for a sector the drive will then rewrite the corrected data and try again. If it decided the sector is no longer reliable it will use one of the spare sectors it has and add the bad sector to the bad block table and write the data elsewhere.
WHAT ?
There is no such thing as a data refresh. There is no point in even reading periodically.

The safest data is data stored on a harddisk that has its heads retracted off platter and powered off.
-banging on it doesnt cause mechanical damage due to headstrikes as the heads are off disk , suspended and locked
-viruses cant corrupt data as it is powered down
-power surges can't fry the electronics as it is disconnected.
-the drive is a dust free sealed environment (not like a plastic wobbly disk , exposed to degradation due to light exposure, easily scratched and full of greasy fingerprints)

There is no need for surface scanning. The drives do this when reading as part of the reading process, but only for the sectors requested by the read. It is not true that they willy-nilly access passing sectors if they have no business there. A drive access only sectors requested by the host. If it encounters a read error or seek error it will retry a number of times and then mark the sector as bad. A bad sector will not prevent subsequent read attempts. It will prevent write attempts.

There is also a big difference between logical and physical mapping. You have no idea where your data really resides on the drive. Your data is also not stored as the chain of ones and zeroes you feed the disk. The data has redundancy added to it and the packet then goes through a scrambler to prevent having more than 5 consecutive ones or zeroes. One byte you feed is stored as 10 bit on the media.
Throw in the servo wedges and other stuff like lookup tables and you end up that a drive holding 1 terabyte of user data, actually stores about 1.3 terabyte. Your data is stored as 'symbols'. 5 bits are used to store 4. Depending on the order of symbols we can flick the symbol polarity.
Lets say we find out that two consecutive symbols end up being 01000 and 00010. This would give six consecutive seroes on the drive. We replace the second symbol with 11101 as this has the same user payload. A symbol can be stored as real or as inverted. It doesnt matter symbols are symmetrical. Additional symbols, containing redundant information are injected in the stream.
Additional techniques like viterbi encoding and maximum likelyhood algorithms can weed out even multi bit errors. The mathematics below it is very complex.

Optical disks use a similar mechanism. They actually store 11 bits for every byte you feed them... The 8 bit is expanded to 14 and then compressed to 11(overall compression reduces the 14 bit streams to an average of 11. Some packets are less compressable than others. This compression algorithm is tuned for music on cd's and video on DVD and bluray. The compressors also work on data of course but they are tuned for video or audio
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 1848
  • Country: sk
Re: OT: Backing up video files long term
« Reply #63 on: July 05, 2014, 05:46:48 pm »

WHAT ?
There is no such thing as a data refresh. There is no point in even reading periodically.


refresh to NEW media ! and yes it's a common practice.  so refresh to NEW harddrives, NEW blueray media, NEW tapes.

if you understood it as reading and writing back , then your understanding was wrong ;)
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #64 on: July 05, 2014, 05:49:57 pm »
You can use SMART to do this or use other tools, the most advertised is Spinrite from GRC.com which can recover most data from bad blocks and try to get a last good read from failing bits by statistical analysis of bad bits in marginal cases. It also will do the same for failing MLC and SLC drives where the data slicer gives a bad reading.

SpinRite is an archaic joke and Gibson is a prolific liar.
Spinrite works fine on very old RLL drives (the ones that needed dedicated controller cards with two flatcables). It also worked on early ide drives. His idea was fine and worked. But any drive above 300megabyte uses other techniques internally You cannot acces raw head data like you could in old drives.

Spi rite today is a good program that simp,y does multiple retries on bad sectors. Normally the drive will do a fixed number of tries , flag an error and give the datablock , errors included, to the host. And that's it. Spinrite is nothing more than a tool that, on a user level, keeps sending retry operations, grabbing the returned block , comparing them and filtering out what data is 'static' and what data is destroyed.
A block can be a few kilobyte. A damaged block is not completely corrupt. It may just be a few bits in the stream, there where the mechanical damage is(and the magnetic layer is removed). Only those bits will change from retry to retry. The rest stay static. So yes, you can recover partial data by collecting two or three retires and finding the static bits. If you are lucky and the filesystem also uses some form of error correction and redundancy you may retrieve the data. If there is a catastrophical failure and entire sectors, including the serve wedge, are gone there is no recovery. If the servo wedge and track data block (not the user data but the data used by the drive to find a sector) is destroyed you cannot find it. It can be retrieved on a spin stand but the drive itself does not have the capability to do that. It would cost too much to give the drive such mechanical precision. We simply employ electronic servo mechanisms to keep the head on track. If there is no indicator where the track is, the servo doesnt work. The drive does not have the mechanical precision to put it on a track and leave it there. A spinstand can do that, a drive can't.

I can go on for days explaining all the trickery and mechanisms involved. It is a very complex system, devised for maximum data safety.

I trust drives much more than optical (self recorded) media. Take a recorded dvd and let it lay in you office exposed to ambient light. Many degrade after a year or so. A lot of cheap ones use an organic material that degrades in light. So even if you keep em in the dark : when reading them the laser in the drive is exposing them....

There are long term storage grade disks havng a metal coating. But those can only be written at maximum 2 or 4 speed. I have very old recorded CD from traxdata. These are gold and you can clearly see the holes under a strong microscope. Recording those took 25 minutes for 650 megabyte. These disks are 16 or 17 years old and work fine. The last 'silver' one i wrote from emtec (basf) was corrupted after 3 months... Simply because it lay in the open.
Gimme a nice cast aluminum frame as enclosure any time.

But, keep three copies and store them in different places.
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #65 on: July 05, 2014, 05:55:55 pm »

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #66 on: July 05, 2014, 05:56:26 pm »
There is no need for surface scanning. The drives do this when reading as part of the reading process, but only for the sectors requested by the read. It is not true that they willy-nilly access passing sectors if they have no business there. A drive access only sectors requested by the host. If it encounters a read error or seek error it will retry a number of times and then mark the sector as bad. A bad sector will not prevent subsequent read attempts. It will prevent write attempts.

RAID controllers often include a scrubbing option which periodically reads an entire drive allowing the drive controller or the RAID controller to reallocate marginal or bad sectors.  Sectors may be reallocated on read even when the data is good or at least recovered.

Quote
There is also a big difference between logical and physical mapping.

There is a logical to physical mapping table and at least on older drives, spare sectors are distributed over the entire surface.  High end drives that used a dedicated servo track and supported low level formatting could optionally include a spare sector on every track lowing the number of reported bad sectors but also sacrificing capacity.
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #67 on: July 05, 2014, 05:58:01 pm »

WHAT ?
There is no such thing as a data refresh. There is no point in even reading periodically.


refresh to NEW media ! and yes it's a common practice.  so refresh to NEW harddrives, NEW blueray media, NEW tapes.

if you understood it as reading and writing back , then your understanding was wrong ;)

That's not what SeanB is talking about. Copying to new media is one kind of refresh. Fine with me.
He was talki g about reading the same drive periodically as to scan the surface for damage.

Data on a drive can o ly get damaged bqy mechanical impact (headstrike). So , by reasing you create an i stance where headstrike could happen. Are you sure that the data you just read is now still intact ? After all the head passes multiple times over the same track. The data marked ok in the first pass could now be destroyed during the second pass when accessing a different sector on that track. Oopsie.. Chicken and egg problem (and no, what you think is track 1 sector 1 and track 1 sector two are not necessarily adjacent to each other, in fact they may physicall lay on different tracks ! Physical and logical are two different things. You have no control over phyqsical, inly the drive knows. This is done te maximise throughput on the drive and shorten seek operations. I formation is staggered.
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Online nctnico

  • Super Contributor
  • ***
  • Posts: 19369
  • Country: nl
    • NCT Developments
Re: OT: Backing up video files long term
« Reply #68 on: July 05, 2014, 06:08:15 pm »
Yes, it is good to refresh the drive info by doing a read every so often. SMART can do this if you ask for a full surface scan, and will take a few hours to run this, and will read all the data on the drive and see if it is all readable after error correction. If the error bursts get too big ( drive firmware dependant) for a sector the drive will then rewrite the corrected data and try again. If it decided the sector is no longer reliable it will use one of the spare sectors it has and add the bad sector to the bad block table and write the data elsewhere.
WHAT ?
There is no such thing as a data refresh. There is no point in even reading periodically.

The safest data is data stored on a harddisk that has its heads retracted off platter and powered off.
-banging on it doesnt cause mechanical damage due to headstrikes as the heads are off disk , suspended and locked
-viruses cant corrupt data as it is powered down
-power surges can't fry the electronics as it is disconnected.
-the drive is a dust free sealed environment (not like a plastic wobbly disk , exposed to degradation due to light exposure, easily scratched and full of greasy fingerprints)
But you never know if it works until you power it up again. That is the big gamble besides other people grabbing the disk and use it for other purposes. And as others pointed out it may be impossible to replace the mechanical and electronical parts.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline miguelvp

  • Super Contributor
  • ***
  • Posts: 5549
  • Country: us
Re: OT: Backing up video files long term
« Reply #69 on: July 05, 2014, 06:11:07 pm »

WHAT ?
There is no such thing as a data refresh. There is no point in even reading periodically.


refresh to NEW media ! and yes it's a common practice.  so refresh to NEW harddrives, NEW blueray media, NEW tapes.

if you understood it as reading and writing back , then your understanding was wrong ;)

I agree, I've done many media refresh copies when we used tapes (the big ones like in the old movies) it was common practice to copy the media to new fresh tapes every so often (don't recall exactly how often now). Then send a copy off site in case of a fire or some other event that could destroy everything in site.

Hierarchical backup was the craze later in the mid/late 90's not sure where they are at now but they where VERY expensive. They would have hard drives, digital tapes and optical media as well. with redundant power supplies and hot swappable RAID drives, the tapes and optical had may bays and some robotics where involved to move tapes and discs around.
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #70 on: July 05, 2014, 06:20:18 pm »
There is no need for surface scanning. The drives do this when reading as part of the reading process, but only for the sectors requested by the read. It is not true that they willy-nilly access passing sectors if they have no business there. A drive access only sectors requested by the host. If it encounters a read error or seek error it will retry a number of times and then mark the sector as bad. A bad sector will not prevent subsequent read attempts. It will prevent write attempts.

RAID controllers often include a scrubbing option which periodically reads an entire drive allowing the drive controller or the RAID controller to reallocate marginal or bad sectors.  Sectors may be reallocated on read even when the data is good or at least recovered.

Quote
There is also a big difference between logical and physical mapping.

There is a logical to physical mapping table and at least on older drives, spare sectors are distributed over the entire surface.  High end drives that used a dedicated servo track and supported low level formatting could optionally include a spare sector on every track lowing the number of reported bad sectors but also sacrificing capacity.

Raid comtrollers can scrub : but the drive firmware needs to suppport it. You will need SAS drives. Your runof the mill ide or sata can't do that.

Dediated servo tracks no longer exist. That has gone away with the diskpacks. Damage to the servotrack was a catastrophy as the entire disk became unusable , true the head for servo was read only but a mechanical damage on that track was a disaster. In diskdrives as we know emfrom the PC world , servo data is interspersed with user data. That is the base operating principle of the drive.
There is a servotrack created by the formatter to lay down the servo wedges. Once the wedges have been written that servotrack is no longer accessible. It has never been accessible by the drive.

Here is how a drive is 'formatted' in the factory. You have a surface with no information on it. So how do you place markers ? The drive mechanics is not precise enough and only works if servo mechanisms can stabilise the head.

Well. Enter the formatter. An additional head is loaded on the outer rim of the drive. Speed control of the motor doesnt even work as we need to measure interval of servo wedges... Which have not been written yet.
So the formatter writes a single 'ping'. The motor controllr is told :disengage your servo , go in open loop and simply drive based on your crystal frequency.
The formatter now reads the ping and measures the time between two pings. This gives it accurate rotational speed (crystals drift enough so motor speed is off enough to have an impact on data retrieval). The formatter now tunes speed up or down by writing control register in the motor controller. Once the ping is seen at the required rate a square wave is now laid down. Every 'click' (o e to zero or zero to one) marks a potential sector change( not all tracks have the same amount of sectors. Track on the outside are longer than tracks on the inside )
We now have a precise physical marker coming from the disk that can tell the electronics : write now.

The formatter now grabs the headstack mechanically and moves it using its serve mechanism with anoptical encoder. So this is like a stepper motor with very fine precision. We walk the tracks and everytime there is a tick detected we write a servoburst and the track/sector datablock.
Ance this is done we can retract the head that was reading the ticks, close that opening, decouple the headstack and close that opening as well. The drive is now sealed and ready for surface test.
The servo mechanisms now all work : the servoburst controls both speed of the motor and timing of write and read operatins and the datablocks tells us where we are. Amplitude detectors keep us centered on the block. Now we write the remaining empty data with a pattern (not just all zeroes... We need changes there so we can do amplitude detection to find the center of the track)

Anyway. Like i said, i can go on and on about this. There are now event techniques where a formatter is no longer needed. The electronics that recovers data is so smart and adaptive that speed fluctuations are not a problem. It can adapt at will. Think of it as a serial port where the baidrate should be 9600 but can drift from 9000 to 10000 at random without data corruption. That is what the signal coming from the head looks like. The data samplers can adapt on the fly. You only need a track position but that can also be done because now we have velocity control on the voicecoil.
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Offline KK

  • Regular Contributor
  • *
  • Posts: 100
  • Country: us
Re: OT: Backing up video files long term
« Reply #71 on: July 05, 2014, 08:05:22 pm »
^^^ thanks for that insight. Very interesting.
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 3938
  • Country: ro
  • .
Re: OT: Backing up video files long term
« Reply #72 on: July 05, 2014, 08:19:31 pm »
(this is going a bit off topic but I'm genuinely curious)

free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a continuous segment...
I'm thinking this may allow slower rotation speeds which may improve reliability and reduce the heat generated.. or am I wrong about these?

Also, is there something physically or otherwise problematic with moving back to 5.25" format with todays technology, let's say in order to make a 10 TB hard drive using 4400-5400 rpm and several platters for companies  that want drives for storage? Is it just too hard to make such large platters or it's just considered old technology, they think it wouldn't sell? Or it would be too expensive to change factories to make it when ssd drives are gaining ground?

---

mojo_chan : hard drives fail in 3-5 years even without keeping them on 24/7.  I run my system 24/7 on and have four drives in the system, oldest is about 4 years old and works fine. I replaced one recently due to starting to develop bad sectors though.

Here's another article with statistics regarding hard drives and other hardware, from a large french retailer : http://linustechtips.com/main/topic/108284-huge-list-of-failure-rates-on-pc-components-french-but-i-translated-nearly-everything/  Point is these people don't all keep their systems on 24/7 and still have hdd failures (preemptive comment: yes, we may not know how those people that return hardware treated those drives, they could have punched the pc or dropped the drives on floor for all we know).
I agree with your on just the comment about lightning strikes or fires.. that's still a risk.
 
« Last Edit: July 05, 2014, 08:24:35 pm by mariush »
 

Online madires

  • Super Contributor
  • ***
  • Posts: 5110
  • Country: de
  • A qualified hobbyist ;)
Re: OT: Backing up video files long term
« Reply #73 on: July 05, 2014, 08:29:55 pm »
It makes more sense to outsource the task and use cloud storage, where the data is duplicated over multiple drives in different parts of the world and you don't have to worry about any of it. Just encrypt if you are worried about that sort of thing. Even on a slow internet connection bandwidth is infinite over time.

Have you ever tried to restore 1TB of data from a cloud based backup? I would prefer to restore from a local disk (in an USB box). But the larger problem is to transfer those 1TB to three different cloud storage services. Or do you got a STM64 or better for internet access?
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #74 on: July 05, 2014, 08:47:01 pm »
Raid comtrollers can scrub : but the drive firmware needs to suppport it. You will need SAS drives. Your runof the mill ide or sata can't do that.

SAS drives may have a specific command for it but SATA drives which will reallocate on read may be scrubbed just reading the drive.  Even my oldest SATA RAID controllers and drives support this.

Quote
Dediated servo tracks no longer exist. That has gone away with the diskpacks.

Dedicated servo designs have been gone for a long time.  I just meant to point out that back then, spare sectors were distributed across the disk surfaces to support reallocation and I am sure the same is still the case.

Quote
<stuff about embedded servo formatting>

The part I always found interesting is the economics.  Formatting and testing takes time so the only difference between two different drives with different capacities may be the time spent on the machine which writes and verifies the formatting.

Semiconductors are often the same way.  The difference between two different operational amplifiers may only be the time to test them.

 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 1848
  • Country: sk
Re: OT: Backing up video files long term
« Reply #75 on: July 05, 2014, 08:52:14 pm »
btw... don't believe to much in clouds ;) they're promissing a shitloads of 9s in availability, but that's just marketing... their technology is something like 99% or slightly better (like 99.5% upto 99.9%) and they're simply promising 99.99% or even 99.999%. that's marketing  not real availability or real protection of your data, they're simply taking the risk because usually you're achieving 100% during a month in case of no issues.  (it's ALL of the cloud services like that).
i'm not saying cloud is a lie, i'm saying don't overly stick to the marketing info :) if something wrong happens and you'll lose your data, they'll refund you the monthly fee (or part of it) but your data will be gone. so it's better to combine the cloud storage with a local mirrors if you really want to not lose your data.
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 1848
  • Country: sk
Re: OT: Backing up video files long term
« Reply #76 on: July 05, 2014, 09:11:03 pm »
btw... don't believe to much in clouds ;) they're promissing a shitloads of 9s in availability, but that's just marketing... their technology is something like 99% or slightly better (like 99.5% upto 99.9%) and they're simply promising 99.99% or even 99.999%. that's marketing  not real availability or real protection of your data, they're simply taking the risk because usually you're achieving 100% during a month in case of no issues.  (it's ALL of the cloud services like that).
i'm not saying cloud is a lie, i'm saying don't overly stick to the marketing info :) if something wrong happens and you'll lose your data, they'll refund you the monthly fee (or part of it) but your data will be gone. so it's better to combine the cloud storage with a local mirrors if you really want to not lose your data.

Do you really care for backup? Even a day or two a year down time wouldn't be much of an issue for backing up personal files.

if my data is lost then i do care ;) and it happens that storage arrays fail and some data is lost (regardless of the raid levels used). actually i witnessed a case when a storage replication between 2 big storage arrays (supposed to increase the data protection level) caused the actual data loss (bug in the firmware). so your data is not 100% safe in the clouds even if they do daily tape backups (you still might lose data for the last 24hours).
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #77 on: July 05, 2014, 09:13:02 pm »
free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a continuous segment...
I'm thinking this may allow slower rotation speeds which may improve reliability and reduce the heat generated.. or am I wrong about these?

Track pitch is too tight for them to remain in alignment on different surfaces.  This is also why tracking uses an embedded servo signal instead of a dedicated surface.  If the actuators were independent this would be possible but the gain is small because of access time and latency.  It is more economical to just use an additional drive.

Quote
Also, is there something physically or otherwise problematic with moving back to 5.25" format with todays technology, let's say in order to make a 10 TB hard drive using 4400-5400 rpm and several platters for companies  that want drives for storage? Is it just too hard to make such large platters or it's just considered old technology, they think it wouldn't sell? Or it would be too expensive to change factories to make it when ssd drives are gaining ground?

Access time and latency would suffer.  I suspect mechanical runout would limit density as well.  Quantum tried this for a while with their Bigfoot series of drives but found it uneconomical.  I remember a laptop drive manufacturer trying this with 3 inch drives which could home more than the common 2.5 inch drives yet still be used in a laptop form factor.
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #78 on: July 05, 2014, 09:13:03 pm »
Cool, 10TB free, probably not going to fill that up a long time, hopefully it counts for a few years XD

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #79 on: July 05, 2014, 09:19:02 pm »
free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a continuous segment...
I'm thinking this may allow slower rotation speeds which may improve reliability and reduce the heat generated.. or am I wrong about these?

Mainly cost, but also space in a given form factor. It probably wouldn't help that much anyway because modern drives implement Native Command Queuing, which reduces the time wasted waiting for the disc to rotate. The cost of the extra heads and electronics just isn't worth it, especially when high performance drives already have one head per platter and stripe data over each platter to increase performance.

Conner made a dual actuator drive called Chinook but it was not economical.

Parallel reading of the heads is not possible because of track misalignment between surfaces.  There is even a head setting time when switching heads between surfaces of the same platter.
 

Online madires

  • Super Contributor
  • ***
  • Posts: 5110
  • Country: de
  • A qualified hobbyist ;)
Re: OT: Backing up video files long term
« Reply #80 on: July 05, 2014, 09:45:53 pm »
On my current connection (120Mb) it would take about a day. You don't need to use three services, just one and also keep a local, offline backup.

Your cloud storage provider may go bankrupt any time, shut down services, be hacked or face a major disaster. Do you remember the major outage at Microsoft with a broken Hitachi SAN? Several 100,000 of mobile phone users lost their mobility services for several days. Some lost their entire data. Or take this forum for example. How often did it face an outage caused by power or network issues? Large datacenters had total outages, even the backup systems failed. And it will happen again, anytime, anywhere. It takes a lot to design and build a network service with an availability of 99.9% or more. But that doesn't imply that the service will offer that reliability over time. Maintenance procedures may change, failover testing may be never be done or not be part of a regular maintenance schedule. Another datacenter had a power outage and the backup power supply failed too,because it wasn't upgraded to support more servers. The initial deployment was designed for much less servers. "We have to cut down the budget, so let's build that for 30% load and extend it later on", the management said. Who needs capacity planning? When the complete power outage did happen, they knew what they had forgotten.
 

Online madires

  • Super Contributor
  • ***
  • Posts: 5110
  • Country: de
  • A qualified hobbyist ;)
Re: OT: Backing up video files long term
« Reply #81 on: July 05, 2014, 09:50:05 pm »
Conner made a dual actuator drive called Chinook but it was not economical.

Please see WD Black, e.g. WD2002FAEX.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #82 on: July 05, 2014, 10:23:59 pm »
Conner made a dual actuator drive called Chinook but it was not economical.

Please see WD Black, e.g. WD2002FAEX.
Dual actuators as Conner used them are completely different.  The Chinook included two completely separate sets of voice coil actuated head assemblies on opposite sides of the platters so a pair of reads or writes could be conducted simultaneously on two areas of the disk.  This also had the benefit of halving rotational latency.

http://en.wikipedia.org/wiki/Conner_Peripherals#Performance_issues_and_the_.22Chinook.22_dual-actuator_drive

The dual actuators that Western Digital uses are located on the same head assembly and raise the resonate frequency at the end of the head assembly allowing a higher bandwidth for better track following.  As far as I know, it does not allow reading or writing to multiple tracks at once and the maximum transfer rate of the drive is not increased beyond that of a single head.
 

Offline free_electron

  • Super Contributor
  • ***
  • Posts: 7373
  • Country: us
    • SiliconValleyGarage
Re: OT: Backing up video files long term
« Reply #83 on: July 06, 2014, 04:36:09 am »
(this is going a bit off topic but I'm genuinely curious)

free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a
mojo_chan : hard drives fail in 3-5 years even without keeping them on 24/7.  I run my system 24/7 on and have four drives in the system, oldest is about 4 years old and works fine. I replaced one recently due to starting to develop bad sectors though.

I agree with your on just the comment about lightning strikes or fires.. that's still a risk.
That is an old idea and has actually been tried , not with four but with two headstacks at opposite corners. You cant do four courners as the heads would bang i to each other the arc on which they move is made very large as to keep the head perpendicular to the track at all times. Now that we have dual actuators this could actually be attempted.

But there are other problems. You still have to wait for data to fly by underneath the head. So you could not use both heads simultaneously. So throughputwise you gain nothing. You could maybe seek a bit faster. While stack 1 is reading stack could be seeking the next block. But the amount of mechanics and the associated cost and failure modes make this unusable. Electronics cost nothing. 90% of the cost of a drive is mechanics. Of that cost 30% is the media (platters) 60% the headstack and its actuator and the rest the motor and case.

In short : you gain nothing and only drive up the cost. Other factor is the required physical room for a second headstack. This means smaller platters thus less capacity. Its a loose loose situation.

Dont confuse dual headstack drives with dual actuator drives. The dual actuator is a second 'motor' o. The headstack that can flex the head.

Here is the problem : the heads are at the end of an arm. This arm moves around a fixed point. So the heads essentially move in an arc. If you position the heads in the center of the usable space between hub and rim of the platter , the heads are perfectly positioned on a track. Move the head outwards or inwards from this center and the head now sits under an angle on the track. This the track will be wider !
So what we do is mount a piezo element at the tip of the arm where the head is. By flexing the piezo we can twist the head position so it becomes perpendicular again.

Imagine it like this.: stretch out your arm , hand flat , fingers pointed out in fro t of you. Draw parallel lines with your arm. Your fingers are perfectly aligned with the line in front of you. Move your arm left or right without bending wrist or elbow. Your fingers now no longer sit parallel with the lines, they sit under an angle. Flex your wrist to adjust your hand. That is what the second actuator does. It gives the head a 'wrist'.

This is done to make the tracks uniform in width thus upping the total amount fo data we can cram on a platter.
Professional Electron Wrangler.
Any comments, or points of view expressed, are my own and not endorsed , induced or compensated by my employer(s).
 

Online vk6zgo

  • Super Contributor
  • ***
  • Posts: 5332
  • Country: au
Re: OT: Backing up video files long term
« Reply #84 on: July 06, 2014, 04:59:21 am »

Thinking about tape storage:-

Analog video tape suffers badly from "print-through" from layer to layer after around 20 years.
I'm not sure how this would affect error rate of digital signals,though---the first generation of digital VTRs will be coming up on the magic 20 years about now,so that may give you some idea,although the format is probably different.

If you really want it to last,convert it to 35mm film stock,or even 70mm.
Film from 50 + years ago is still viewable,so there you are!
All providing, of course,that there is still anyone with the skills to make a movie projector in 50 years time!

 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 1848
  • Country: sk
Re: OT: Backing up video files long term
« Reply #85 on: July 06, 2014, 07:29:33 am »

Thinking about tape storage:-

Analog video tape suffers badly from "print-through" from layer to layer after around 20 years.
I'm not sure how this would affect error rate of digital signals,though---the first generation of digital VTRs will be coming up on the magic 20 years about now,so that may give you some idea,although the format is probably different.

If you really want it to last,convert it to 35mm film stock,or even 70mm.
Film from 50 + years ago is still viewable,so there you are!
All providing, of course,that there is still anyone with the skills to make a movie projector in 50 years time!

regarding the magnetic tapes - yes the format is different for LTO tapes... and manufacturers are promising  only approx 10 years for a reason (some of them more - that's why  i mentioned 10+ in my prev. post). and need to mention again - the 10years is valid only when stored in a controlled environment (temperature, humidity, dust-free). if you need to retain your data for more than a decade then a regular refresh to new tapes is the solution ;)
 

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: OT: Backing up video files long term
« Reply #86 on: July 06, 2014, 12:17:45 pm »
Hard drives are fascinating.

I've always wondered why we're married to quarter height drives in desktops these days.  Most PC cases I've used could easily mount a half or full-height drive easily, and it seems it would be an easy way to immediately double or quadruple capacity.

I saw a 10Mb drive once that was huge and had I think 8 platters and fixed heads everywhere.  Nothing moved except the platters.  I know that wouldn't be possible today, it was just neat.
 

Offline Shock

  • Super Contributor
  • ***
  • Posts: 3234
  • Country: au
Re: OT: Backing up video files long term
« Reply #87 on: July 07, 2014, 06:45:12 am »
This...

Buy some eSata connectors if you don't have them. Backup to disk by booting the PC with external disks connected. Use two different reliable brands. After backing up complete a SHA256 read verify against the original data and leave a copy of the checksum on the disk and the PC before shutting down and removing your external disks.

Keep one on premises and one off premises and every few months set a reminder to run a verify against the original SHA256 hashes on separate PC's

If you have problems with a disk do not attempt to verify another disk on that same system.  After the original backup keep both sets of hardware isolated to avoid accidental corruption by hardware and software etc.

Other things you can do to add reliability:
Buy them straight out of the box at the store (avoid mishandling)
Updating the firmware of the disks
Don't use an external enclosures
Store them an antistatic case or bag
Store them where they won't be handled or dropped
When the disks are new use a utility to read the SMART data from the disk then fill and zero them a few times then compare the SMART data again and evaluation their condition. You can also use utilities to perform more aggressive tests.
Soldering/Rework: Pace ADS200, Pace MBT350
Multimeters: Fluke 87V, 117, 27/FM               >>> WANTED STUFF <<<
Oszilloskopen: Rigol DS1054Z, Phillips PM3065
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #88 on: July 07, 2014, 06:51:47 am »
I've now got two new 2TB drives, one WD, one Seagate and currently archiving all my raw video onto those. Doing about 65-70MB/sec still takes 7-8 hours per drive.
Will then store in separate locations at lab and home.
I've also got a new Bluray burner coming and will also burn one copy onto those just to be sure. They will be handy for regular access if I need it. Will try and get some quality TDK or some other brand discs, but they don't seem to be cheap.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #89 on: July 07, 2014, 06:53:05 am »
If it's on YouTube what is there to worry about? You don't need to save every minute of your muttering

The raw files are not on youtube. Only a fool would spend 5 years collecting it all and then just throw it all away. You never know when you might need it for something you haven't even thought of yet...
Yes, if I lost it the world wouldn't end, but it's certainly worth some effort to preserve.
 

Offline XOIIO

  • Super Contributor
  • ***
  • Posts: 1592
  • Country: ca
Re: OT: Backing up video files long term
« Reply #90 on: July 07, 2014, 07:13:02 am »
You could spend the time you're using on these drives on your son instead. Wouldn't that be better?

I had an obsession with saving every photograph and video I took, until my 3TB WD crashed and the only photos I could get off my local cache were corrupted (pink stripes everywhere, colors inverted, etc).

Then I realized that the only valuable ones had been published and I could re-take the ones that I lost any time I needed to. Pretty humbling.

Oh boy, here we go, parenting genius here now.

FYI, you don't need to sit at the computer while you are copying the files.

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #91 on: July 07, 2014, 07:18:24 am »
You could spend the time you're using on these drives on your son instead. Wouldn't that be better?

He's at Pre-school as I do this, so no. And it's not a huge amount of time, I don't have to sit there watching the files copy.
You can say that about every thing I do in my life, including this forum.
You forget that this video is my career, and content I have worked hard for 5 years to produce. Are you saying it's not worth me spending a few hundred bucks and an hour or two of my time to archive it all?, that I should just throw it all away?
« Last Edit: July 07, 2014, 07:22:03 am by EEVblog »
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #92 on: July 07, 2014, 12:08:17 pm »
Considering Tencent will give you 10TB for free you would be kind of mad not to take advantage of it.

In my case I only have 500GB/month allowance at the lab, and 250GB/month at home, and both of them include uploads.
I just shut down my old ADSL at the lab that had unlimited (but slow) upload.
 

Offline dfmischler

  • Frequent Contributor
  • **
  • Posts: 548
  • Country: us
Re: OT: Backing up video files long term
« Reply #93 on: July 07, 2014, 12:48:45 pm »
Do you think it's best to leave an archival hard disk on the shelf untouched and untested until needed?  Or does it need to be exercised periodically to maintain bearings/grease/seals.  Or just periodically copy it to a new disk?  An interesting discussion about hard disks and archival media from another forum from a few years ago.  Does anybody know of any useful research to try to nail this down (I realize it's a moving target, but even old data would be better than none)?
 

Online madires

  • Super Contributor
  • ***
  • Posts: 5110
  • Country: de
  • A qualified hobbyist ;)
Re: OT: Backing up video files long term
« Reply #94 on: July 07, 2014, 12:55:56 pm »
I was hoping I didn't have to say it, but apparently I do. Relying on a single type of backup is dumb. You need more than one.

Exactly! We should say that load and clear, and more often! A lot of people store their data just on a single drive or believe the marketing departmant of the cloud storage provider.

Quote
Considering Tencent will give you 10TB for free you would be kind of mad not to take advantage of it.

Do you think a company would give you 10TB for free? Wait until you know what you'll have to pay for that free storage.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #95 on: July 07, 2014, 12:57:14 pm »
Do you think it's best to leave an archival hard disk on the shelf untouched and untested until needed?

As a matter of course I'd probably power them up once a year to check. And probably replace after 5 years or so?
The cost to replace them isn't much in the scheme of things.
 

Offline rob77

  • Super Contributor
  • ***
  • Posts: 1848
  • Country: sk
Re: OT: Backing up video files long term
« Reply #96 on: July 07, 2014, 01:29:24 pm »
Do you think it's best to leave an archival hard disk on the shelf untouched and untested until needed?

As a matter of course I'd probably power them up once a year to check. And probably replace after 5 years or so?
The cost to replace them isn't much in the scheme of things.

sounds good ;) but i would keep 2 sets of those drives (2 copies stored separately) . and actually i would probably go for less than 5 years - something like 3 years (but i'm a bit paranid :D )
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #97 on: July 07, 2014, 07:45:23 pm »
In my case I only have 500GB/month allowance at the lab, and 250GB/month at home, and both of them include uploads.
I just shut down my old ADSL at the lab that had unlimited (but slow) upload.

Ah yes, I forgot how had broadband is in Australia. I don't know how you cope, I pull >500GB in a day sometimes.

The US is rapidly catching up on two fronts; allowances are more often being enforced or going down while data demands are rising.  One good aspect though is that congestion at the local distribution level is also rising so bandwidth is limited during peak hours anyway.
 

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: OT: Backing up video files long term
« Reply #98 on: July 07, 2014, 07:53:05 pm »
The US is rapidly catching up on two fronts; allowances are more often being enforced or going down while data demands are rising.  One good aspect though is that congestion at the local distribution level is also rising so bandwidth is limited during peak hours anyway.

I have a 2000GB monthly quota (up & down) and I never even approach that limit, even when I go crazy on torrent.  In my area this is a common quota, and I'm not sure about the rest of the US.  My last connection had 150GB/month and I exceeded that every single month, even though the connection was about 15th the speed I have now.
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #99 on: July 07, 2014, 08:15:38 pm »
@Dave

Out of curiosity roughly how many gigs of files for say 30 minutes of YouTube video (raw and otherwise to be backed up)?

 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 3938
  • Country: ro
  • .
Re: OT: Backing up video files long term
« Reply #100 on: July 07, 2014, 08:35:00 pm »
Cameras record in full HD at about 17-28 mbps (varies from camera to camera, mine's best preset is 28mbps for 1080p 60fps).  At about 24mbps average and including sound, you're looking at around 3 MB/s or about 11 GB per hour of recorded video.

A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I recently did a 3 part series about temperature measuring with an AD595 and a microcontroller, about 2 hours of actual camera shot video got on youtube from about 3.5 hours of actual footage ... the folder with raw recordings 46 GB.
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4310
  • Country: us
  • KE7GKP
Re: OT: Backing up video files long term
« Reply #101 on: July 07, 2014, 09:23:28 pm »
Shooting ratios of 10:1 were not uncommon even back in the days of film. And with video, producers have become even more profligate. Shooting ratios for some types of productions (like documentaries) may approach 100:1.  Several multi-camera productions that I work on use ("iso recording") where we are recording each camera, in addition to recording the "live-switched" program.
 

Online madires

  • Super Contributor
  • ***
  • Posts: 5110
  • Country: de
  • A qualified hobbyist ;)
Re: OT: Backing up video files long term
« Reply #102 on: July 07, 2014, 09:26:21 pm »
Do you think a company would give you 10TB for free? Wait until you know what you'll have to pay for that free storage.

They are the same as any number of other companies that offer free storage. They hope you will use their other services, either ad-supported or paid. In Tencent's case their biggest product is the extremely popular QQ messenger. It's a bit like MSN or ICQ. I use it to chat to my girlfriend a lot - it's very popular in China. They also do email, web search, games and all the usual random crap. You don't have to use any of it though, you can just use the free drive space.

I just looked up the end user price for a WD40EFRX (4TB, 24x7, SOHO NAS/AV usage), it's about EUR 150. That would be one of the cheapest 24x7 disks on the market (but you wouldn't find such disks in a SAN). We assume that we need 4 disks for 10TB and some redundancy. So the disks are EUR 600. They will run about 3 years, i.e. EUR 200 per year. We also have to add the SAN, datacenter, maintenance, spares, internet traffic, new Ferrari for the boss and what have you. Let's say that would be EUR 2300 a year. That means the cloud storage provider pays EUR 2500 a year to offer you 10TB for free. That would be a nice DSO, and you would get another one next year. All for free, just by watching some ads?
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 3938
  • Country: ro
  • .
Re: OT: Backing up video files long term
« Reply #103 on: July 07, 2014, 09:36:18 pm »
They probably resort to the same mentality as shared hosting... they know few will use more than a few GB of backup and if someone uses more than a certain limit, they can start looking into the account and just say they close the account because of storing copyrighted content or something like that.

Other companies like backblaze use their own custom cases and use software that works similar to rsync, storing only differences between files.. see blog post about their storage pods: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #104 on: July 07, 2014, 10:04:08 pm »
Cameras record in full HD at about 17-28 mbps (varies from camera to camera, mine's best preset is 28mbps for 1080p 60fps).  At about 24mbps average and including sound, you're looking at around 3 MB/s or about 11 GB per hour of recorded video.

A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I recently did a 3 part series about temperature measuring with an AD595 and a microcontroller, about 2 hours of actual camera shot video got on youtube from about 3.5 hours of actual footage ... the folder with raw recordings 46 GB.

It's not surprising (the size), the amount of effort that goes into a 2 hour production is.

I'm an always amazed at people complaining about backups and the cost of hardware. My reply was always you invest years gathering that information, it is your business. How much would it cost to replace it if it was lost? Would that be the end of your business? I wasn't a computer guy, production testing and setup, but still it's a part of being in business. 
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #105 on: July 07, 2014, 10:13:39 pm »
Shooting ratios of 10:1 were not uncommon even back in the days of film. And with video, producers have become even more profligate. Shooting ratios for some types of productions (like documentaries) may approach 100:1.  Several multi-camera productions that I work on use ("iso recording") where we are recording each camera, in addition to recording the "live-switched" program.

I had a small amount of contact with local production cameraman 10 years ago. The shooting ratio is crazy. It is fun to watch productions being shot (not uncommon). Stargate often came out to a nearby lake and provincial park to film/shoot. It's very weird to the Stargate set up in a farmers field and a golf course just 100 feet away. 
 

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 15379
  • Country: za
Re: OT: Backing up video files long term
« Reply #106 on: July 07, 2014, 10:47:05 pm »
If you use pro photographers you might use around 5 photos from a shoot, but the photographer would have taken possibly 1000 photos and then developed the negatives only. That got rather expensive on 70mm film.
 

Offline Richard Crowley

  • Super Contributor
  • ***
  • Posts: 4310
  • Country: us
  • KE7GKP
Re: OT: Backing up video files long term
« Reply #107 on: July 07, 2014, 10:55:16 pm »
If you use pro photographers you might use around 5 photos from a shoot, but the photographer would have taken possibly 1000 photos and then developed the negatives only. That got rather expensive on 70mm film.

The difference between a professional photographer and an amateur is that the amateur shows you ALL their photos.
 

Offline SeanB

  • Super Contributor
  • ***
  • Posts: 15379
  • Country: za
Re: OT: Backing up video files long term
« Reply #108 on: July 07, 2014, 11:04:43 pm »
The Pro only shows you negatives, then you select those that have promise and make a positive transparency of them. Then the final ones are blown up and edited to remove the inevitable marks.
 

Offline Shock

  • Super Contributor
  • ***
  • Posts: 3234
  • Country: au
Re: OT: Backing up video files long term
« Reply #109 on: July 07, 2014, 11:18:52 pm »
I've now got two new 2TB drives, one WD, one Seagate and currently archiving all my raw video onto those. Doing about 65-70MB/sec still takes 7-8 hours per drive.
Will then store in separate locations at lab and home.
I've also got a new Bluray burner coming and will also burn one copy onto those just to be sure. They will be handy for regular access if I need it. Will try and get some quality TDK or some other brand discs, but they don't seem to be cheap.

Dave don't bother with Blu-ray.  It's as useful as tape drives which are totally a waste of time and expense.  Instead buy a 3rd disk in a usb or eSata caddy if you have to.

If you refer to my last post you also need to perform a proper verifies. Because backing up data is fine but you need to confirm it can be read byte exact to the original.  This needs to be repeatable over the life of the drive.

By using eSata you get the fastest backup and verify (read) speed you can.
Once you have sha256 hashes of the original data you no longer need to read it again.  You just read the backups.

You can use a utility like the PowerShell File Checksum Integrity Verifier (PsFCIV).
So no need for a backup program, all they do is add unnecessary complication and things to go wrong (unless you're backing up online databases).
Soldering/Rework: Pace ADS200, Pace MBT350
Multimeters: Fluke 87V, 117, 27/FM               >>> WANTED STUFF <<<
Oszilloskopen: Rigol DS1054Z, Phillips PM3065
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #110 on: July 07, 2014, 11:22:59 pm »
Cameras record in full HD at about 17-28 mbps (varies from camera to camera, mine's best preset is 28mbps for 1080p 60fps).

I record at 17Mbps as a compromise. Used to be 12Mbps.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #111 on: July 07, 2014, 11:25:47 pm »
A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I usually use about 80-90% of my raw material. A good lot of that is simply dead time at the start of the clip.
You record audio separately?
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 10930
  • Country: us
  • DavidH
Re: OT: Backing up video files long term
« Reply #112 on: July 07, 2014, 11:45:41 pm »
Other companies like backblaze use their own custom cases and use software that works similar to rsync, storing only differences between files.. see blog post about their storage pods: http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/
I remember studying what Backblaze published carefully when I looked into doing this for myself on a much smaller scale.  This article discussing various storage controllers is informative as well:

http://blog.zorinaq.com/?e=10
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #113 on: July 07, 2014, 11:51:23 pm »
The difference between a professional photographer and an amateur is that the amateur shows you ALL their photos.

I thought that was called "being a grandmother".
 

Offline apelly

  • Supporter
  • ****
  • Posts: 1039
  • Country: nz
Re: OT: Backing up video files long term
« Reply #114 on: July 08, 2014, 12:19:50 am »
Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)
I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.
Me too. Three anyway. One just yesterday.
I'd rather a Google clue, link, or some theory than "do this" (generally)
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #115 on: July 08, 2014, 12:42:17 am »
Now, an offline drive could indeed fail to start, but honestly, I can't ever recall having a drive that worked when it was powered down, stop working on power-up.  (I'm sure it's happened in the history of computing.)
I have seen SEVERAL (six or eight) hard drives in long-term archival service simply be DOA when you try to power them back up.
Me too. Three anyway. One just yesterday.

And thus the need for multiple backups. Hardware, media, fire, storage location, the list goes on.
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 3938
  • Country: ro
  • .
Re: OT: Backing up video files long term
« Reply #116 on: July 08, 2014, 01:30:13 am »
A 1 hour Youtube video can easily have up to 3-5 hours of content, takes, separate audio recording etc so it adds up.

I usually use about 80-90% of my raw material. A good lot of that is simply dead time at the start of the clip.
You record audio separately?

No. I either let the camera record with its own internal microphones (5.1 dolby something) or I plug a lavalier microphone in it and then it switches to stereo recording automatically.

For one video, i recorded some segments without lavalier mic so the audio was 5.1.  I wanted to have that same 5.1 sound and that same audio processing the camera does when showing the screen capture, so I put the camera on tripod by the monitor and recorded with the camera in sync with the screen capture software. Then I copied video from camera to pc and extracted the audio and threw away the video, so now my screen capture segments had the original stereo sound produced by the desktop capture software (using lavalier mic in the pc soundcard jack) and the 5.1 sound from the camera.

I've had segments messed up for lots of reasons .. ambulance driving with sirens behind my apartment building, phone ringing.. not a big deal but if I stop I sometimes decide I could say that "paragraph" in a better way and reshoot the last 5 minutes of an explanation or something like that. It adds up.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #117 on: July 08, 2014, 06:41:22 am »
Out of curiosity roughly how many gigs of files for say 30 minutes of YouTube video (raw and otherwise to be backed up)?

FYI, my latest 29min edited LVDC video is 4.86GB of raw video, with a total length of 39:46 over 53 clips.
There were a few clips in that were nothing and I didn't use. I don't bother deleting those in-camera at the time because it's just too much hassle to go into playback mode, select delete mode, switch back, and then reset my exposure setting (annoying bug in the Canon HF G10). Nor do I delete when nothing clips on the PC, it's just not worth the effort.
 

Offline pickle9000

  • Super Contributor
  • ***
  • Posts: 2136
  • Country: ca
Re: OT: Backing up video files long term
« Reply #118 on: July 08, 2014, 07:08:02 am »
Out of curiosity roughly how many gigs of files for say 30 minutes of YouTube video (raw and otherwise to be backed up)?

FYI, my latest 29min edited LVDC video is 4.86GB of raw video, with a total length of 39:46 over 53 clips.
There were a few clips in that were nothing and I didn't use. I don't bother deleting those in-camera at the time because it's just too much hassle to go into playback mode, select delete mode, switch back, and then reset my exposure setting (annoying bug in the Canon HF G10). Nor do I delete when nothing clips on the PC, it's just not worth the effort.

To be honest that seems ridiculously good, both in raw video and shooting ratio.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #119 on: July 08, 2014, 07:42:20 am »
To be honest that seems ridiculously good, both in raw video and shooting ratio.

Yeah, I think it's pretty good. Most of what comes out of my mouth ends up on camera.
If people are really keen to see that in action, I've posted some complete video of me shooting an entire mailbag video.
I don't fuss that much over the details, and I also rarely replay clips before I go edit. And also rarely go back to reshoot something after I watch it in the edit.
 

Offline EEVblog

  • Administrator
  • *****
  • Posts: 31232
  • Country: au
    • EEVblog
Re: OT: Backing up video files long term
« Reply #120 on: July 08, 2014, 08:10:14 am »
Dave, were you floating the notion of doing a video (or text blog) on your methods of videoblogging? The behind the scenes stuff about getting your edited video onto Youtube, monetising, dealing with comments and all the problems and traps a budding VBer might expect. Things that you have dealt with and hopefully solved.
Perhaps I imagined it because I would like to know. IIRC it was for your second channel.

Yes, I've mentioned it a few times it's something I want to do.
Just the methods of shooting/editing/tools etc would take several videos.
Not easy to shoot either because I'd have to setup multiple camera shots etc. e.g. to record how I record. Showing the effects of various settings etc. Not easy stuff to show that well.
The monetisation/comments thing would be another video again.
 

Offline SirNick

  • Frequent Contributor
  • **
  • Posts: 589
Re: OT: Backing up video files long term
« Reply #121 on: July 08, 2014, 08:37:58 pm »
Blu-ray is fine.  Will it last 20 years?  Probably not.  Some of the CD-Rs I burned on my first 4x writer are still readable, but things change.  Regardless, if you're not putting all your eggs in that basket, it's still a convenient way to store and catalog data for easy retrieval.  I envision something where each disc is labeled "raw footage from episode #327" or whatever.  If the disc fails, fine, go to the HDDs.  It's inexpensive to add an internal (or USB) reader to any computer, assuming it doesn't already have one, so accessibility is good.

As to HDD reliability, I have a whole stack of 40 and 80GB drives.  One developed bad sectors last time I tried to read it (about a year or two ago.)  Others still work and hold, for e.g., Hackintosh HDD images (because it's a laborious and painfully slow process to reinstall).  If I lose them, ... eh.

I have 4 drives (RAID-5) in a NAS.  When I replace those for bigger capacity drives, the old 4 become backup drives for the new ones.  NO RAID on the backups -- you don't want the hassle of trying to arrange for an appropriate read environment later on.  A USB-to-SATA cable or dock will be available for at least as long as the drive is viable.  This process of upgrade and trickle down keeps each generation of drives "fresh" within 2-5 years.  I plug them in every six months or so to backup new data.  (It really should be more often, but that's a risk I take.)  IMO, this is a decent compromise between leaving them at rest, and periodically checking for faults.

Since video archival won't be updated, doing a checksum validation of backed-up data once a year would suffice to catch drive faults or media read errors before both the BD-R and HDD have had a chance to fail.  Could a simultaneous failure still occur?  Yes, of course.  That's life.  Dave might be out one episode of raw video then.  I suspect he'll survive.  :)
 

Offline ve7xen

  • Frequent Contributor
  • **
  • Posts: 732
  • Country: ca
    • VE7XEN Blog
Re: OT: Backing up video files long term
« Reply #122 on: July 08, 2014, 11:38:20 pm »
If I were doing this:

- Replace commodity NAS with FreeNAS or NAS4Free running ZFS in a redundant mode as primary archive. Schedule regular scrubs. Comes with free replication to a remote server, if you decide that's feasible, plus block-level checksums and other useful features. Since this is fairly cheap, I would probably build a duplicate box at home and use USB media to carry the incremental snapshot home with you after you finish an episode. Don't remove your snapshots and you're safe from pretty much everything but simultaneous and catastrophic hardware or filesystem failure, and you have live access to all your data at any time.

- Secondary archive to BD with par2 or dvddisaster, keep in fire safe / safe deposit box / with data archival company

- Upgrade your network to GigE FFS!
« Last Edit: July 08, 2014, 11:43:03 pm by ve7xen »
73 de VE7XEN
 

Offline SirNick

  • Frequent Contributor
  • **
  • Posts: 589
Re: OT: Backing up video files long term
« Reply #123 on: July 09, 2014, 12:15:39 am »
Ethernet speed isn't necessarily the bottleneck.  I typically get ~30MB/s over CIFS from a Win 7 desktop to my NAS.  All Gig-E.  But, the NAS is RAID 5 on JFS, all in software on a Mobile Core 2 Duo mini ITX motherboard.

Some of those little toaster NASeses are running at a few hundred MHz, and may or may not have any hardware acceleration.  Gig-E is merely a bullet point for marketing for all the good it's going to do you.
 

Offline ve7xen

  • Frequent Contributor
  • **
  • Posts: 732
  • Country: ca
    • VE7XEN Blog
Re: OT: Backing up video files long term
« Reply #124 on: July 09, 2014, 12:45:23 am »
Ethernet speed isn't necessarily the bottleneck.  I typically get ~30MB/s over CIFS from a Win 7 desktop to my NAS.  All Gig-E.  But, the NAS is RAID 5 on JFS, all in software on a Mobile Core 2 Duo mini ITX motherboard.
You should be able to do much better. Any remotely modern hardware (which this is) should be able to saturate GigE if set up properly. I do software RAID6 and can get ~300MB/s read and 150MB/s write out of the array (which is poorly tuned for block size and stride). On NFS it saturates the GigE in either direction, but CIFS required a lot more messing around to work well and still doesn't match it for performance. Many times people complain about less than ideal performance it's due to using crappy Realtek NICs on one or both ends.

Quote
Some of those little toaster NASeses are running at a few hundred MHz, and may or may not have any hardware acceleration.  Gig-E is merely a bullet point for marketing for all the good it's going to do you.
Oh for sure, 10MB was just a "suspicious" number.
73 de VE7XEN
 

Offline SirNick

  • Frequent Contributor
  • **
  • Posts: 589
Re: OT: Backing up video files long term
« Reply #125 on: July 09, 2014, 01:15:45 am »
Definitely.  I have no doubt it can be improved dramatically.  I tried "dd if=/dev/zero of=/mnt/md" after I built it, and got significantly higher throughput than what I see from across the LAN.  I don't remember what it came out to... pretty sure it didn't even come close to a couple hundred MB/s, but it was obvious that performance over the network -- including and especially CIFS -- is a constraint.

That said, throughput is so rarely a concern that I haven't bothered to pursue any of the bottlenecks.  Can stream 1080p?  Check.  Decent chance of saturating a single-spindle source or destination?  Check.  OK, good enough then.

And of course there's an excellent chance that both ends use Realtek chipsets, as both the NAS mobo and the PC are using onboard LAN.  That's hardly the worst sin though.  It's also stacked up on top of a cabinet in the laundry-slash-boiler room, keeping those 1U PSU CapXon electrolytics cooled by 25-30C ambient temps. :box:  Wouldn't be caught dead implementing a system like that for a business, but at my own home?  Eh.  "Good enough for Australia."  :-+
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 3938
  • Country: ro
  • .
Re: OT: Backing up video files long term
« Reply #126 on: July 09, 2014, 01:23:25 am »
Quote
Ethernet speed isn't necessarily the bottleneck.  I typically get ~30MB/s over CIFS from a Win 7 desktop to my NAS.  All Gig-E.  But, the NAS is RAID 5 on JFS, all in software on a Mobile Core 2 Duo mini ITX motherboard.

It depends. For example, the $1000 NAS I linked to a couple of pages or so before ( Synology DS1813+ ) has 4 gigabit ports that can be linked together for a 4 gbps throughput.  If you have a smart switch capable to do port trunking, you're in business.
That NAS  uses 2.13 Ghz dual core Atom...

ps. You can spend 400$ on a hardware raid card with 8 sata ports like this one : http://www.newegg.com/Product/Product.aspx?Item=N82E16816151137

Add ~150-200$ on cpu, mb, case and ram and optionally get a 2-4 port gigabit network card from eBay and do port trunking.  The software's going to be a problem though, there is freenas and other ready made solutions but it's still not going to be as polished as that Synology comes from the factory.
« Last Edit: July 09, 2014, 01:29:51 am by mariush »
 

Offline Rigby

  • Super Contributor
  • ***
  • Posts: 1476
  • Country: us
  • Learning, very new at this. Righteous Asshole, too
Re: OT: Backing up video files long term
« Reply #127 on: July 09, 2014, 01:42:45 am »
Ethernet throughput matters a lot on what is copied.

Lots of small files will not saturate gig-e.  A few large files will saturate any link.  There is an overhead per file that impacts transfer speed in any per-file upload/download/copy.

To maximize throughput, zip up the files with a non-compressing archive format, and transfer them in that form.
 

Offline Legit-Design

  • Frequent Contributor
  • **
  • Posts: 562
 

Offline mariush

  • Super Contributor
  • ***
  • Posts: 3938
  • Country: ro
  • .
Re: OT: Backing up video files long term
« Reply #129 on: July 09, 2014, 02:34:08 am »
Way to go, posting an article last updated in 2007 and a graph showing cpu usage on a 300 Mhz Sun server.

Now a dual core i3 can probably run in circles and saturate 1gbps even without jumbo frames.
 

Offline SirNick

  • Frequent Contributor
  • **
  • Posts: 589
Re: OT: Backing up video files long term
« Reply #130 on: July 09, 2014, 03:10:20 am »
ps. You can spend 400$ on a hardware raid card with 8 sata ports like this one : http://www.newegg.com/Product/Product.aspx?Item=N82E16816151137

You can, and (IMHO) that would be a waste of $400.  ;)

I have a rack chassis with 4 bays.  I have a motherboard with four SATA ports (and a CF card slot).  Connectivity is covered.

"But what about hardware-optimized RAID then?"  Good point, Bob.  But where you see a purpose-built ASIC that increases IOPS, I see a data striping algorithm that requires the original hardware to recover data from my disks.  I once had a server with a proprietary controller card that made RAID really fast.  Then the card died, and it was NLA, and I no longer had access to the data on those disks.

"But what about backups?"  Yes, Bob, I had backups.  After a few hours of rebuilding the server to the point where the backup software client could be installed, it only took another hour or so to get the data back.  Or, you can pull a commodity PC off the shelf, plug in the NIC and power cables, stuff the hard drives in there, and have any Linux Live CD recognize the MD RAID volume and you're back online in 15 minutes.

On another note:  I once did something colosally stupid and offlined two of the four disks in my RAID-5 set.  An ordinary controller would have dumped the volume at that point.  But, MD RAID just complained furiously until I gave it the "dash F" option, which stands for "Shut the dash-F up and re-mount the volume regardless".  After replaying the journal (and running a background integrity check), it was 100% accessible.  No lost data.  I am a big, big fan of Linux MD RAID.
« Last Edit: July 09, 2014, 03:12:16 am by SirNick »
 

Offline ve7xen

  • Frequent Contributor
  • **
  • Posts: 732
  • Country: ca
    • VE7XEN Blog
Re: OT: Backing up video files long term
« Reply #131 on: July 09, 2014, 04:07:34 am »
ps. You can spend 400$ on a hardware raid card with 8 sata ports like this one : http://www.newegg.com/Product/Product.aspx?Item=N82E16816151137

Add ~150-200$ on cpu, mb, case and ram and optionally get a 2-4 port gigabit network card from eBay and do port trunking.  The software's going to be a problem though, there is freenas and other ready made solutions but it's still not going to be as polished as that Synology comes from the factory.
For the task at hand, I would say hardware RAID is a big no-no. It's expensive, you're not targeting high IOPS, and now your data depends on having a working RAID card of correct model and firmware. You can get an 8-port SAS HBA for under $100, if you even need one, since the 6 or 8 onboard SATA ports common these days are likely to be sufficient, at least to start. This guy did it recently with a pretty nice build for around $400 sans-drives. I would personally throw in an Intel PCIe NIC just because onboard NICs suck, but hardly required.

Also keep in mind that port-trunking/link aggregation chooses a link based on a hash of some portion of the header (often just the source/destination MAC addresses, though sometimes layer 3 or even layer 4 on higher-end gear). In other words, you're primarily concerned with throughput to a single client, this won't help at all, since all the traffic will end up on the same link.

Nas4Free is pretty polished these days, and gets you ZFS, which is hands down worth any slight increase in effort required over a commercial product. Automatic scrubbing and block checksums are exactly what you need for long-term archival. Snapshots, and ones which can be incrementally copied to a remote system by sneaker- or Inter-nets would also be pretty handy. Most of the commercial boxes are based on standard Linux technology, so you can plop the disks into any machine to recover your data if the NAS box goes tits up, but a nice benefit of building on Nas4Free/FreeNAS/Linux is that you know you can do this in a pinch and that the vendor isn't doing something weird (like Drobo and presumably others).

Edit: Ninja'd but I'll leave the post anyway :)
73 de VE7XEN
 

Offline Legit-Design

  • Frequent Contributor
  • **
  • Posts: 562
Re: OT: Backing up video files long term
« Reply #132 on: July 09, 2014, 05:17:23 am »
Way to go, posting an article last updated in 2007 and a graph showing cpu usage on a 300 Mhz Sun server.

Now a dual core i3 can probably run in circles and saturate 1gbps even without jumbo frames.
nice video with real life test.
That NAS box probably doesn't run processor faster than 300 Mhz something? It all depends on situation.
In my opinion it's a fair comparison of almost decade old server vs modern NAS box.
Just found this quote:
I have a small ioMega NAS box that actually gets almost double the throughput using jumbo frames, but that's because the large frames allow more data transfer with lower CPU utilization (it has a low-end ARM-compatible cpu)
« Last Edit: July 09, 2014, 05:18:56 am by Legit-Design »
 

Offline rdl

  • Super Contributor
  • ***
  • Posts: 2918
  • Country: us
Re: OT: Backing up video files long term
« Reply #133 on: July 10, 2014, 03:01:06 pm »
Any time the discussion of backing up data arises I am reminded of this video.





 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf