DIY HDD platter swap?

[wraper]

It's not valuable which is why I'm willing to try to DIY it.   Just enough to get it to run for about 4 hours so I can image the drive, that's my goal.

May sound like your expectations are small. But actually is quiet a lot to expect from HDD which was opened and even more if something was replaced.

[AndyC_772]

I vote that a year from now, the cost of having used a professional data recovery service won't seem so bad, and it'll have been worth it.

Two discs failing so close together... damn, I feel for you. Who'd have thought any product could be made so consistently?

[wraper]

Who'd have thought any product could be made so consistently?

Seagate is very consistent... in failing HDDs 
https://www.backblaze.com/blog/3tb-hard-drive-failure/

Only 251 of the 4,190 Seagate 3TB hard drives deployed in 2012 are still in service as of March 31, 2015. Breaking it down:
blog_seagate_status

As a reminder, about 75% of the “Removed” drives failed one of the bench tests once they were removed from a Storage Pod.

Now look at the deployed graph  . Very similar to the failed graph. Seems they live exactly the same time and go kaput.

[codeboy2k]

re: the backblaze report: That 3TB drive is the same as my 2TB drives, the  7200.14  series.

and mine was deployed Nov 2012.  And also failed in Q3 2015.  Go figure.

Seagate is very consistent... in failing HDDs

I've had previous failures with seagate too.  And I've had good luck with Western Digital, but I've had those fail before, so it's not one vs. the other.
I think that todays higher density HDDs are just too tightly packed bits on the platters now, and that's very difficult to read back consistently,  The drives have to go through hoops to get a good read, and the heads are getting smaller and smaller too. It's bound to fail, and it's also evidenced by the fact that the common disk drive warranty is only 1 year now.

This time around I thought I would try Toshiba, which I believe to be Fujitsu.  Fujitsu has had a great hard drive division, I remember big Fujitsu removable packs back in the 80's, so they are one of the early pioneers of the technology.

I do think I need to buy 2 more drives and run them in RAID10 or RAIDZ2 this time around.  RAIDZ1 seems risky now.

[PlainName]

buy 2 more drives and run them in RAID10 or RAIDZ2

I think you should forget RAID. It's purpose is not as a backup (or to do away with backups) but just to remove the downtime associated with failed media. That is, RAID will warn you the data is at risk and allow you to replace bust media without losing access (temporarily!) to that data.

What it won't do is protect you from disasters. That power surge that takes out the entire rack, finger trouble, rats, etc. RAID does nothing for those, so you still need to have a coherent backup strategy AS WELL as running RAID. So why bother?

Since you must have the backup anyway, all you're really interested in is circumventing the several hours of downtime whilst you restore from backup (plus the time taken to acquire new media). For that, a mirror is pretty good. There is a low risk that bother drives will fail as you're dealing with a problem, but that will screw your bigger and better RAID too, as you found.

Instead of shoving two more drives in that RAID array, and hope no real disaster occurs, just use them as two backup sets, alternating them and/or preferentially keeping one off-site. Just not having one plugged into a live PC will make your data MUCH safer.

[Psi]

Hot swap raid mirror systems are useful for backups. Especially the all-in-one caddie versions that have a built in controller. They have one sata port so appear to the OS as a single drive.

They usually have 3 drives, 2 of which are kept in mirror and 1 is a hot spare.
At any time you can turn the key, pull out a drive and take it with you as a backup, the hot spare then kicks in.

I know a few companies that use this quite a lot. They unplug a drive every night as a take home backup and then plug it back in the next morning.

[george graves]

Spinrite does a think called dynostat recovery

Don't get me started on Spinrite - I'm convinced it doesn't work.  Steve Gibson is a really smart guy, and I bet back when he came up with the software (1980's) if worked 50% of the time when drives had the aerial density of a the hindenburg - but nope - I've tried it on about 25 drives I'd say - at least 20.  I've never recovered a drive with it - ever - not even a single file.

Anyways, he still pushed his software on Leo Laport's "TWIT" shows.  Kinda shamelessly for a product that doesn't work.  But he swears it does!   

[PlainName]

They usually have 3 drives, 2 of which are kept in mirror and 1 is a hot spare.

Sounds like I should have one or more of those 

Also sounds like it's going to be expensive

[Psi]

It has worked numerous times for me on drives with read errors.
It doesn't work if the drive is dead (obviously) or if the reallocation pool is full or if there's a hardware fault.

I've had it churning away on bad areas of many disks. Running dynostat for sometimes as long as a couple of days on end.
On most disks it eventually gets the bad areas reallocated with data intact.
Don't get more wrong, i'm not saying it works all the time, but it has definitely worked for me most of the time when dealing with drives with read errors. Where the drive keeps locking up or returning CRC/IO error when you tried to access a particular file/folder

[Psi]

They usually have 3 drives, 2 of which are kept in mirror and 1 is a hot spare.

Sounds like I should have one or more of those 

Also sounds like it's going to be expensive

I don't think it was ridiculously expensive. i remember the brand of one i used to work with. It was called accusys.

[amyk]

Swapping boards seems like the only kind of field-procedure that has any kind of reasonable expectation of success.  And even that method may be less promising with newer drive technology where so many factors are firmware and saved in NVRAM.

Indeed there is a lot of "adaptive parameters" as they're called, stored off the platters and different for each head stack. This is usually in an EEPROM on the board but I think I read somewhere that some models have that data stored as part of the head preamp.

But beyond that and platter runout, head alignment in modern drives is not such an issue as they're designed so the heads will find the right sectors as long as it can read them. The fact that thermal expansion will significantly change the absolute positions of tracks means that no fixed track positioning can be used, they just use the embedded servo info to find where the tracks are:

http://hddscan.com/doc/HDD_Tracks_and_Zones.html

[PlainName]

accusys

Thanks. I had a look at their webby and, ignoring the full size rack stuff (!) I think we'd be looking at around $2k. Not that bad, but far too expensive.

One of my requirements is that whatever I use is common and cheap. If I use $2K worth of kit then should it go titsup I will be stiffed with a $2K bill to get back to where I was, and I don't have a choice of whether to pay it or not. Further, and I've been hit a few times with this, the probability is that just when I need it most (i.e. to replace blown kit) it turns out they stopped making them the year before and they're like pixiedust.

For a non-critical system, a proprietary $2K thing is fine, but not for looking after my data. Against that, if the take-away drive is essentially a bootable, readable drive on any old disk controller, that mitigates the problem a lot so I could still be tempted

[retiredcaps]

re: the backblaze report: That 3TB drive is the same as my 2TB drives, the  7200.14  series.

and mine was deployed Nov 2012.  And also failed in Q3 2015.  Go figure.

I wondering if all these Seagate failures is related to the floods in Thailand back in Nov 2011 which affected the hard disk industry?

http://www.theverge.com/2011/11/23/2582677/thailand-flood-seagate-hard-drive-shortage

Maybe Seagate bought less than prime parts and/or the switch to higher density disk was released before it was fully tested?

"And while manufacturers are adapting to the shortage by including fewer parts in their drives by using a single higher-density disk with two heads instead of two disks with four heads, Luczo says price increases are inevitable."

[nctnico]

re: the backblaze report: That 3TB drive is the same as my 2TB drives, the  7200.14  series.

and mine was deployed Nov 2012.  And also failed in Q3 2015.  Go figure.

Seagate is very consistent... in failing HDDs

I've had previous failures with seagate too.  And I've had good luck with Western Digital, but I've had those fail before, so it's not one vs. the other.
I think that todays higher density HDDs are just too tightly packed bits on the platters now, and that's very difficult to read back consistently,  The drives have to go through hoops to get a good read, and the heads are getting smaller and smaller too. It's bound to fail, and it's also evidenced by the fact that the common disk drive warranty is only 1 year now.

This time around I thought I would try Toshiba, which I believe to be Fujitsu.  Fujitsu has had a great hard drive division, I remember big Fujitsu removable packs back in the 80's, so they are one of the early pioneers of the technology.

In my experience the biggest enemy of a hard drive is heat. Remember hard drives have been on the edge of technology for decades due to strong competition. If you keep a hard drive cooled properly (lots of airflow) you can easely get a decade out of it. However a lot of computer cases offer zero airflow to the hard drive bays and that could make a drive fail within a year. If a hard drive doesn't need to be fast you can also consider using Western Digital's green series. These use less power (rotate at 5400rpm) and therefore aren't thermally stressed like higher performance drives.

[wraper]

I wondering if all these Seagate failures is related to the floods in Thailand back in Nov 2011 which affected the hard disk industry?

They made a lot of shite drives even without floods, like famous 7200.11 series (have a few laying around, full of bad sectors). Edit: Nope, those are 7200.12, still shite. Edit2: turns out I have both dead 7200.11 and 7200.12 HDDs 
If looking on the past 7 years, overall they took a huge leadership in making the least reliable drives out of all manufacturers.
One more graph from backblaze:

At the opposite end of the spectrum are Seagate disks. Last year, the two 1.5TB Seagate models used by Backblaze had failure rates of 25.4 percent (for the Barracuda 7200.11) and 9.9 percent (for the Barracuda LP). Those units fared a little better this time around, with failure rates of 23.8 and 9.6 percent, even though they were the oldest disks in the test (average ages of 4.7 and 4.9 years, respectively). However, their poor performance was eclipsed by the 3TB Barracuda 7200.14 units, which had a whopping 43.1 percent failure rate, in spite of an average age of just 2.2 years.

[Mechanical Menace]

In my experience the biggest enemy of a hard drive is heat.

In my experience it's a dodgy power supply in one way or another. For some reason where I live is prone to sags and swells. Until I built myself a UPS of sorts my PC was resetting itself a couple of times a day through to them and I was going through a HDD every 6 months.

[sync]

In my experience the biggest enemy of a hard drive is heat.

In my experience it's a dodgy power supply in one way or another.

In my experience it's cheapness. Server hard drives last much longer. And heat is also an enemy.

[codeboy2k]

buy 2 more drives and run them in RAID10 or RAIDZ2

I think you should forget RAID. ... [ ... ]

What it won't do is protect you from disasters. [ ... ] So why bother?

Since you must have the backup anyway, all you're really interested in is circumventing the several hours of downtime whilst you restore from backup (plus the time taken to acquire new media). For that, a mirror is pretty good. There is a low risk that bother drives will fail as you're dealing with a problem, but that will screw your bigger and better RAID too, as you found.

Instead of shoving two more drives in that RAID array, and hope no real disaster occurs, just use them as two backup sets, alternating them and/or preferentially keeping one off-site. Just not having one plugged into a live PC will make your data MUCH safer.

I think I will follow this advice, and use a mirror when I rebuild it, instead of RAIDZ (which is a RAID5 workalike)
I like the idea of an external hotswap disk, so that I can just use the mirroring itself to make a backup. I can hotswap two or more to keep one live for performance and one backup out of the PC.  I can buy more in the future and rotate them.

With a system like that in place I might have been more diligent about the backups   As it turned out, I had TB's of RAID and no good way to back it up, so I got lazy and didn't do it before the inevitable failure.

[codeboy2k]

In my experience the biggest enemy of a hard drive is heat.

In my experience it's a dodgy power supply in one way or another.

In my experience it's cheapness. Server hard drives last much longer. And heat is also an enemy.

I don't know if heat was a factor here.  All three drives were in a small mini-atx case, but spaced well apart (about 5cm between each drive) and there was decent airflow.

The last available SMART data that I have for the drive just before it failed shows the max temperature at 45C .  No drive was above 45C in the max temp column, and I don't think 45C is too hot for a drive.  Maybe it was ?

I was running SMART daily short offline tests, and weekly long tests via cron.  SMART logs showed these tests always passed, except the one drive that failed first started to show short test failures at 90% complete then failed within 2 days after that. The second drive to fail never had any errors in the SMART log, except its reallocated sector count was increasing over the years.  It always seemed like normal to me, given that it was a high-density 2TB drive, I expected some reallocated sectors to be normal.  At the time of the second drive failure, there was 382 pending reallocated sectors (which doesn't seem too high). It just outright failed while running in the degraded RAID.  The second drive never failed any SMART offline tests, long or short.

Finally, being ZFS, I was also running a weekly data scrub (which, for non ZFS aware people, means that it reads back all the data, verifies all data on the members of the RAID array, and rewrites and logs errors).  That weekly scrub log NEVER showed any errors ever being repaired.

Overall, I'm pretty disappointed in this experience with RAID, but it's not the end of the world for me. Just more of a personal feeling of loss more than anything else.  And I'll rebuild it with a mirror set next time and a clear backup strategy using hot swap drives.

[codeboy2k]

In my experience it's a dodgy power supply in one way or another. For some reason where I live is prone to sags and swells. Until I built myself a UPS of sorts my PC was resetting itself a couple of times a day through to them and I was going through a HDD every 6 months.

Twice In the past I've had a power outage take out a desktop drive. This was back when 30GB was the normal desktop drive size Usually that happened when the grid browns-out or flicks off then on again really quickly (is that a grid switch? I think so).

We were also experiencing quite a few power failures here, especially during the winter storms. So I put in a UPS on every desktop and on the server, and they've saved me countless times already.  This RAID server at the topic of this thread was on a UPS.

[wraper]

The second drive to fail never had any errors in the SMART log, except its reallocated sector count was increasing over the years.  It always seemed like normal to me, given that it was a high-density 2TB drive

Normal HDDs don't have growing reallocated sector count. It means that HDD is crap and data from those sectors most likely was corrupted. For example, I have 1TB WD green with 38 000 hours power on time, 2 TB WD green with 22 000 hours, 3TB toshiba 4800 hours. All of them have zero reallocated sector count.

At the time of the second drive failure, there was 382 pending reallocated sectors (which doesn't seem too high)

This means almost dead HDD

[codeboy2k]

This means almost dead HDD

That's clear now

There is lots of misinformation on the net regarding reallocated sector counts.  It really is normal for large TB disks to have some reallocated sectors, in my opinion it's inevitable.  I think I might have dropped the ball when it started growing over the years. There was a huge growth at the beginning when it was first installed, and then it was stable for a few years, but recently would add one or two more sectors every month or two. 

There was no big red banner saying "CHANGE YOUR DRIVE NOW, DUFUS!" -- maybe there should be.

Also of note was that the first drive to fail had no such reallocated sectors, and only recently started to fail the short offline SMART test, and then failed completely 2 days after that. The second drive did have the reallocated sectors already, but has never failed any offline tests, and also failed 24 hours after the first drive.   The third drive of the RAID set currently has no reallocated sectors and has never failed the SMART offline tests.

All of which tells me there is no good indicator of pending drive failure.

[Mechanical Menace]

3TB toshiba 4800 hours.

That Toshiba is half way through it's life span if my experiences are at all common.

[codeboy2k]

3TB toshiba 4800 hours.

That Toshiba is half way through it's life span if my experiences are at all common.

  I sure hope not ..

I just bought 2 new Toshiba 2TB drives to replace the 2 dead Seagates. And based on my own 2 out of 3 failed in 2.5 yrs, and 43% of Backblazes's Seagates failed in 2.5 years, I am having my doubts about the last remaining Seagate that I own, and I just want to toss it out the door and get another disk.  Maybe my 3rd and 4th disk should be a different brand.

[wraper]

3TB toshiba 4800 hours.

That Toshiba is half way through it's life span if my experiences are at all common.

this is Toshiba DT01ACA300 aka rebadged Hitachi HDS723030BLE640 which proved to be quiet reliable.