OT: Backing up video files long term

[rob77]

btw... don't believe to much in clouds they're promissing a shitloads of 9s in availability, but that's just marketing... their technology is something like 99% or slightly better (like 99.5% upto 99.9%) and they're simply promising 99.99% or even 99.999%. that's marketing  not real availability or real protection of your data, they're simply taking the risk because usually you're achieving 100% during a month in case of no issues.  (it's ALL of the cloud services like that).
i'm not saying cloud is a lie, i'm saying don't overly stick to the marketing info if something wrong happens and you'll lose your data, they'll refund you the monthly fee (or part of it) but your data will be gone. so it's better to combine the cloud storage with a local mirrors if you really want to not lose your data.

[rob77]

btw... don't believe to much in clouds they're promissing a shitloads of 9s in availability, but that's just marketing... their technology is something like 99% or slightly better (like 99.5% upto 99.9%) and they're simply promising 99.99% or even 99.999%. that's marketing  not real availability or real protection of your data, they're simply taking the risk because usually you're achieving 100% during a month in case of no issues.  (it's ALL of the cloud services like that).
i'm not saying cloud is a lie, i'm saying don't overly stick to the marketing info if something wrong happens and you'll lose your data, they'll refund you the monthly fee (or part of it) but your data will be gone. so it's better to combine the cloud storage with a local mirrors if you really want to not lose your data.

Do you really care for backup? Even a day or two a year down time wouldn't be much of an issue for backing up personal files.

if my data is lost then i do care and it happens that storage arrays fail and some data is lost (regardless of the raid levels used). actually i witnessed a case when a storage replication between 2 big storage arrays (supposed to increase the data protection level) caused the actual data loss (bug in the firmware). so your data is not 100% safe in the clouds even if they do daily tape backups (you still might lose data for the last 24hours).

[David Hess]

free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a continuous segment...
I'm thinking this may allow slower rotation speeds which may improve reliability and reduce the heat generated.. or am I wrong about these?

Track pitch is too tight for them to remain in alignment on different surfaces.  This is also why tracking uses an embedded servo signal instead of a dedicated surface.  If the actuators were independent this would be possible but the gain is small because of access time and latency.  It is more economical to just use an additional drive.

Also, is there something physically or otherwise problematic with moving back to 5.25" format with todays technology, let's say in order to make a 10 TB hard drive using 4400-5400 rpm and several platters for companies  that want drives for storage? Is it just too hard to make such large platters or it's just considered old technology, they think it wouldn't sell? Or it would be too expensive to change factories to make it when ssd drives are gaining ground?

Access time and latency would suffer.  I suspect mechanical runout would limit density as well.  Quantum tried this for a while with their Bigfoot series of drives but found it uneconomical.  I remember a laptop drive manufacturer trying this with 3 inch drives which could home more than the common 2.5 inch drives yet still be used in a laptop form factor.

[XOIIO]

Cool, 10TB free, probably not going to fill that up a long time, hopefully it counts for a few years XD

[David Hess]

free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a continuous segment...
I'm thinking this may allow slower rotation speeds which may improve reliability and reduce the heat generated.. or am I wrong about these?

Mainly cost, but also space in a given form factor. It probably wouldn't help that much anyway because modern drives implement Native Command Queuing, which reduces the time wasted waiting for the disc to rotate. The cost of the extra heads and electronics just isn't worth it, especially when high performance drives already have one head per platter and stripe data over each platter to increase performance.

Conner made a dual actuator drive called Chinook but it was not economical.

Parallel reading of the heads is not possible because of track misalignment between surfaces.  There is even a head setting time when switching heads between surfaces of the same platter.

[madires]

On my current connection (120Mb) it would take about a day. You don't need to use three services, just one and also keep a local, offline backup.

Your cloud storage provider may go bankrupt any time, shut down services, be hacked or face a major disaster. Do you remember the major outage at Microsoft with a broken Hitachi SAN? Several 100,000 of mobile phone users lost their mobility services for several days. Some lost their entire data. Or take this forum for example. How often did it face an outage caused by power or network issues? Large datacenters had total outages, even the backup systems failed. And it will happen again, anytime, anywhere. It takes a lot to design and build a network service with an availability of 99.9% or more. But that doesn't imply that the service will offer that reliability over time. Maintenance procedures may change, failover testing may be never be done or not be part of a regular maintenance schedule. Another datacenter had a power outage and the backup power supply failed too,because it wasn't upgraded to support more servers. The initial deployment was designed for much less servers. "We have to cut down the budget, so let's build that for 30% load and extend it later on", the management said. Who needs capacity planning? When the complete power outage did happen, they knew what they had forgotten.

[madires]

Conner made a dual actuator drive called Chinook but it was not economical.

Please see WD Black, e.g. WD2002FAEX.

[David Hess]

Conner made a dual actuator drive called Chinook but it was not economical.

Please see WD Black, e.g. WD2002FAEX.

Dual actuators as Conner used them are completely different.  The Chinook included two completely separate sets of voice coil actuated head assemblies on opposite sides of the platters so a pair of reads or writes could be conducted simultaneously on two areas of the disk.  This also had the benefit of halving rotational latency.

http://en.wikipedia.org/wiki/Conner_Peripherals#Performance_issues_and_the_.22Chinook.22_dual-actuator_drive

The dual actuators that Western Digital uses are located on the same head assembly and raise the resonate frequency at the end of the head assembly allowing a higher bandwidth for better track following.  As far as I know, it does not allow reading or writing to multiple tracks at once and the maximum transfer rate of the drive is not increased beyond that of a single head.

[free_electron]

(this is going a bit off topic but I'm genuinely curious)

free_electron, I've often wondered why hard drive manufacturers wouldn't design a hard drive that would have a head stack on each corner of the unit, so four in total. Just read either four tracks at the same time or a single track using all four heads and assemble the data into the cache of the drive in a
mojo_chan : hard drives fail in 3-5 years even without keeping them on 24/7.  I run my system 24/7 on and have four drives in the system, oldest is about 4 years old and works fine. I replaced one recently due to starting to develop bad sectors though.

I agree with your on just the comment about lightning strikes or fires.. that's still a risk.

That is an old idea and has actually been tried , not with four but with two headstacks at opposite corners. You cant do four courners as the heads would bang i to each other the arc on which they move is made very large as to keep the head perpendicular to the track at all times. Now that we have dual actuators this could actually be attempted.

But there are other problems. You still have to wait for data to fly by underneath the head. So you could not use both heads simultaneously. So throughputwise you gain nothing. You could maybe seek a bit faster. While stack 1 is reading stack could be seeking the next block. But the amount of mechanics and the associated cost and failure modes make this unusable. Electronics cost nothing. 90% of the cost of a drive is mechanics. Of that cost 30% is the media (platters) 60% the headstack and its actuator and the rest the motor and case.

In short : you gain nothing and only drive up the cost. Other factor is the required physical room for a second headstack. This means smaller platters thus less capacity. Its a loose loose situation.

Dont confuse dual headstack drives with dual actuator drives. The dual actuator is a second 'motor' o. The headstack that can flex the head.

Here is the problem : the heads are at the end of an arm. This arm moves around a fixed point. So the heads essentially move in an arc. If you position the heads in the center of the usable space between hub and rim of the platter , the heads are perfectly positioned on a track. Move the head outwards or inwards from this center and the head now sits under an angle on the track. This the track will be wider !
So what we do is mount a piezo element at the tip of the arm where the head is. By flexing the piezo we can twist the head position so it becomes perpendicular again.

Imagine it like this.: stretch out your arm , hand flat , fingers pointed out in fro t of you. Draw parallel lines with your arm. Your fingers are perfectly aligned with the line in front of you. Move your arm left or right without bending wrist or elbow. Your fingers now no longer sit parallel with the lines, they sit under an angle. Flex your wrist to adjust your hand. That is what the second actuator does. It gives the head a 'wrist'.

This is done to make the tracks uniform in width thus upping the total amount fo data we can cram on a platter.

[vk6zgo]

Thinking about tape storage:-

Analog video tape suffers badly from "print-through" from layer to layer after around 20 years.
I'm not sure how this would affect error rate of digital signals,though---the first generation of digital VTRs will be coming up on the magic 20 years about now,so that may give you some idea,although the format is probably different.

If you really want it to last,convert it to 35mm film stock,or even 70mm.
Film from 50 + years ago is still viewable,so there you are!
All providing, of course,that there is still anyone with the skills to make a movie projector in 50 years time!

[rob77]

Thinking about tape storage:-

Analog video tape suffers badly from "print-through" from layer to layer after around 20 years.
I'm not sure how this would affect error rate of digital signals,though---the first generation of digital VTRs will be coming up on the magic 20 years about now,so that may give you some idea,although the format is probably different.

If you really want it to last,convert it to 35mm film stock,or even 70mm.
Film from 50 + years ago is still viewable,so there you are!
All providing, of course,that there is still anyone with the skills to make a movie projector in 50 years time!

regarding the magnetic tapes - yes the format is different for LTO tapes... and manufacturers are promising  only approx 10 years for a reason (some of them more - that's why  i mentioned 10+ in my prev. post). and need to mention again - the 10years is valid only when stored in a controlled environment (temperature, humidity, dust-free). if you need to retain your data for more than a decade then a regular refresh to new tapes is the solution

[Rigby]

Hard drives are fascinating.

I've always wondered why we're married to quarter height drives in desktops these days.  Most PC cases I've used could easily mount a half or full-height drive easily, and it seems it would be an easy way to immediately double or quadruple capacity.

I saw a 10Mb drive once that was huge and had I think 8 platters and fixed heads everywhere.  Nothing moved except the platters.  I know that wouldn't be possible today, it was just neat.

[Shock]

This...

Buy some eSata connectors if you don't have them. Backup to disk by booting the PC with external disks connected. Use two different reliable brands. After backing up complete a SHA256 read verify against the original data and leave a copy of the checksum on the disk and the PC before shutting down and removing your external disks.

Keep one on premises and one off premises and every few months set a reminder to run a verify against the original SHA256 hashes on separate PC's

If you have problems with a disk do not attempt to verify another disk on that same system.  After the original backup keep both sets of hardware isolated to avoid accidental corruption by hardware and software etc.

Other things you can do to add reliability:
Buy them straight out of the box at the store (avoid mishandling)
Updating the firmware of the disks
Don't use an external enclosures
Store them an antistatic case or bag
Store them where they won't be handled or dropped
When the disks are new use a utility to read the SMART data from the disk then fill and zero them a few times then compare the SMART data again and evaluation their condition. You can also use utilities to perform more aggressive tests.

[EEVblog]

I've now got two new 2TB drives, one WD, one Seagate and currently archiving all my raw video onto those. Doing about 65-70MB/sec still takes 7-8 hours per drive.
Will then store in separate locations at lab and home.
I've also got a new Bluray burner coming and will also burn one copy onto those just to be sure. They will be handy for regular access if I need it. Will try and get some quality TDK or some other brand discs, but they don't seem to be cheap.

[EEVblog]

If it's on YouTube what is there to worry about? You don't need to save every minute of your muttering

The raw files are not on youtube. Only a fool would spend 5 years collecting it all and then just throw it all away. You never know when you might need it for something you haven't even thought of yet...
Yes, if I lost it the world wouldn't end, but it's certainly worth some effort to preserve.

[XOIIO]

You could spend the time you're using on these drives on your son instead. Wouldn't that be better?

I had an obsession with saving every photograph and video I took, until my 3TB WD crashed and the only photos I could get off my local cache were corrupted (pink stripes everywhere, colors inverted, etc).

Then I realized that the only valuable ones had been published and I could re-take the ones that I lost any time I needed to. Pretty humbling.

Oh boy, here we go, parenting genius here now.

FYI, you don't need to sit at the computer while you are copying the files.

[EEVblog]

You could spend the time you're using on these drives on your son instead. Wouldn't that be better?

He's at Pre-school as I do this, so no. And it's not a huge amount of time, I don't have to sit there watching the files copy.
You can say that about every thing I do in my life, including this forum.
You forget that this video is my career, and content I have worked hard for 5 years to produce. Are you saying it's not worth me spending a few hundred bucks and an hour or two of my time to archive it all?, that I should just throw it all away?

[EEVblog]

Considering Tencent will give you 10TB for free you would be kind of mad not to take advantage of it.

In my case I only have 500GB/month allowance at the lab, and 250GB/month at home, and both of them include uploads.
I just shut down my old ADSL at the lab that had unlimited (but slow) upload.

[dfmischler]

Do you think it's best to leave an archival hard disk on the shelf untouched and untested until needed?  Or does it need to be exercised periodically to maintain bearings/grease/seals.  Or just periodically copy it to a new disk?  An interesting discussion about hard disks and archival media from another forum from a few years ago.  Does anybody know of any useful research to try to nail this down (I realize it's a moving target, but even old data would be better than none)?

[madires]

I was hoping I didn't have to say it, but apparently I do. Relying on a single type of backup is dumb. You need more than one.

Exactly! We should say that load and clear, and more often! A lot of people store their data just on a single drive or believe the marketing departmant of the cloud storage provider.

Considering Tencent will give you 10TB for free you would be kind of mad not to take advantage of it.

Do you think a company would give you 10TB for free? Wait until you know what you'll have to pay for that free storage.

[EEVblog]

Do you think it's best to leave an archival hard disk on the shelf untouched and untested until needed?

As a matter of course I'd probably power them up once a year to check. And probably replace after 5 years or so?
The cost to replace them isn't much in the scheme of things.

[rob77]

Do you think it's best to leave an archival hard disk on the shelf untouched and untested until needed?

As a matter of course I'd probably power them up once a year to check. And probably replace after 5 years or so?
The cost to replace them isn't much in the scheme of things.

sounds good but i would keep 2 sets of those drives (2 copies stored separately) . and actually i would probably go for less than 5 years - something like 3 years (but i'm a bit paranid )

[David Hess]

In my case I only have 500GB/month allowance at the lab, and 250GB/month at home, and both of them include uploads.
I just shut down my old ADSL at the lab that had unlimited (but slow) upload.

Ah yes, I forgot how had broadband is in Australia. I don't know how you cope, I pull >500GB in a day sometimes.

The US is rapidly catching up on two fronts; allowances are more often being enforced or going down while data demands are rising.  One good aspect though is that congestion at the local distribution level is also rising so bandwidth is limited during peak hours anyway.

[Rigby]

The US is rapidly catching up on two fronts; allowances are more often being enforced or going down while data demands are rising.  One good aspect though is that congestion at the local distribution level is also rising so bandwidth is limited during peak hours anyway.

I have a 2000GB monthly quota (up & down) and I never even approach that limit, even when I go crazy on torrent.  In my area this is a common quota, and I'm not sure about the rest of the US.  My last connection had 150GB/month and I exceeded that every single month, even though the connection was about 15th the speed I have now.

[pickle9000]

@Dave

Out of curiosity roughly how many gigs of files for say 30 minutes of YouTube video (raw and otherwise to be backed up)?