Author Topic: modern storage is crazy  (Read 6105 times)

0 Members and 1 Guest are viewing this topic.

Offline james_s

  • Super Contributor
  • ***
  • Posts: 21611
  • Country: us
Re: modern storage is crazy
« Reply #25 on: April 28, 2022, 05:05:29 pm »
I don't see much advantage there. In my experience most failures are not the platters themselves but the head amplifier or other internal components. I don't think internal RAID would buy you anything, the whole point of RAID is that you can swap out an individual failed drive and rebuild the array.
 
The following users thanked this post: DiTBho

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7508
  • Country: va
Re: modern storage is crazy
« Reply #26 on: April 28, 2022, 05:19:23 pm »
Quote
Would there be any point in the drive manufacturers offering built in RAID?

Only for neophytes. Anyone worth their salt who thinks RAID is going to fix something will, of course, use drives from multiple sources.
 
The following users thanked this post: DiTBho

Offline eugene

  • Frequent Contributor
  • **
  • Posts: 497
  • Country: us
Re: modern storage is crazy
« Reply #27 on: April 28, 2022, 05:32:45 pm »
C'mon guys. I'm not suggesting that it would be as good as other solutions, I'm just wondering if a 2TB internal HD with built-in RAID would somehow protect the data better than a 10TB drive that's 95% empty.

It's a way to make use of those over sized drives in a typical laptop (for example.)
90% of quoted statistics are fictional
 
The following users thanked this post: DiTBho

Online mariush

  • Super Contributor
  • ***
  • Posts: 5170
  • Country: ro
  • .
Re: modern storage is crazy
« Reply #28 on: April 28, 2022, 07:34:06 pm »
No, because RAID is never for data protection. edit : I mean SHOULD never be used for data protection, or as the only data protection method.  There are RAID versions that offer some data protection, in case of a single or multiple drive failures.

A hard drive with internal raid would still have a single point of failure.. let's say a short circuit on the circuit board or over voltage event blows up the eeprom chip that stores the drive geometry info (number of platters, surfaces, tracks, lists of bad sectors, etc) and the controller can no longer access the data.

People/Companies often buy hard drives from multiple locations, or buy drives staggered, just so that they'll get the same model but with different batch numbers, different manufacturing dates.

For example, what if the factory had a small batch of components that were faulty and were all used on hard drives in week 10 of 2021? If you buy 6 drives from the same store to make a raid 5/6 chances are you're gonna get consecutive serial numbers, as the store probably ordered a box of 25-50 drives.  Then, you could have a situation where all six drives fail after a year of 24/7 usage, all within a few days or so from each other.

btw, a bit off topic... .. Seagate even had a firmware bug that cause drives to lock and no longer respond after some amount of time of operation : http://www.datarecoveryspecialists.co.uk/blog/firmware-bug-on-seagate-hard-drive
Had you bought a few drives at the same time, you'd be screwed with all these drives failing at nearly the same time

By making a RAID 5 / 6 or higher with drives with different production dates you reduce the risk of all drives in the raid failing within a few days from each other, or potentially while you spend 10-20 hours to repair the raid by replacing the faulty drive.


If you want data protection, you could have your raid with some parity / spare drive as hot spare, and you'd have some way of periodically (ex at 2am every day or every sunday night) clone the drives to another computer that's in another location (another room so in case of fire you won't have both computers damaged, or at least another power circuit in case you have some power event that blows both computers' power supplies)

Then ideally you'd also copy the critical data to a drive or tapes and put it in a safe, or upload the data to a remote location (a storage server in a datacenter is cheap, 50-100$ a month, you can just encrypt archives with some error correction info (quickpar for example) and upload them to a remote machine.
 
« Last Edit: April 28, 2022, 07:38:58 pm by mariush »
 
The following users thanked this post: DiTBho

Online DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 4367
  • Country: gb
Re: modern storage is crazy
« Reply #29 on: April 28, 2022, 07:59:11 pm »
Seagate even had a firmware bug that cause drives to lock and no longer respond after some amount of time of operation; Had you bought a few drives at the same time, you'd be screwed with all these drives failing at nearly the same time

d'oh, I have six of those in my 2009-2010 NAS  :palm:
The article says
Quote
For clients with Seagate Barrcuda 7200.11, DiamondMax 22 and ES.2 SATA hard drives manufactured before 2009, there is a well documented firmware 'bug'.
My disks are 7200.12, firmware CC44: I am not sure now if 7200.12/CC44 in the bug-blacklist.
Code: [Select]
2010-01-5385, qty=6, Seagate Barracuda 7200.12 SATA 500GB, 005-953, €44,90, €269,40
« Last Edit: April 28, 2022, 08:37:19 pm by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online jpanhalt

  • Super Contributor
  • ***
  • Posts: 4002
  • Country: us
Re: modern storage is crazy
« Reply #30 on: April 28, 2022, 08:07:25 pm »
No, because RAID is never for data protection. edit : I mean SHOULD never be used for data protection, or as the only data protection method.

Can you elaborate why RAID1 on separate drives should not be used for data protection?  I have been doing that for >20 years and have had at least 3 individual drive failures, but preserved my data.  (I keep OS, Desktop, and programs I use on a separate drive, which soon will be RAID1 too).

 
The following users thanked this post: DiTBho

Offline SpacedCowboy

  • Frequent Contributor
  • **
  • Posts: 319
  • Country: gb
  • Aging physicist
Re: modern storage is crazy
« Reply #31 on: April 28, 2022, 08:07:47 pm »
That wouldn't likely protect as much as you think it would, IMHO. A couple of things off the top of my head that could go wrong:

  • If there is a systemic issue with the device as a whole (eg: a power failure) then all the platters are simultaneously gone
  • If there's a head crash on one platter, it's probably going to impact all the others too - even the debris of that first head-crash could easily cause more, but if something goes so wrong that you have a head crash on one platter, that same event is probably going to affect others too

One of the key points of RAID is "Redundant". If your system is tightly-coupled (and the platters in a disk drive are *very* tightly coupled), you're likely not getting the 'redundant' part correct. The idea is that a failure in one part of the system doesn't affect the others - and that's very tough to do if they're tightly coupled together.

I'm not a RAID expert, just a satisfied customer, but even if you go with internal RAID on an SSD (to abrogate the latter point above), the power thing is still a relevant problem, IMHO.
 
The following users thanked this post: tooki, DiTBho

Offline james_s

  • Super Contributor
  • ***
  • Posts: 21611
  • Country: us
Re: modern storage is crazy
« Reply #32 on: April 28, 2022, 08:07:53 pm »
C'mon guys. I'm not suggesting that it would be as good as other solutions, I'm just wondering if a 2TB internal HD with built-in RAID would somehow protect the data better than a 10TB drive that's 95% empty.

It's a way to make use of those over sized drives in a typical laptop (for example.)

What oversized HDDs? I have a 1TB SSD in my laptop, it's getting more filled than I'd like it to be so I'm going to have to clean house soon. At the time I bought it though anything larger was drastically more expensive, I haven't looked lately. I can think of exactly 0% of hard drive failures I've experienced where some kind of internal RAID wasting half the space would have helped.
 
The following users thanked this post: DiTBho

Online mariush

  • Super Contributor
  • ***
  • Posts: 5170
  • Country: ro
  • .
Re: modern storage is crazy
« Reply #33 on: April 28, 2022, 08:32:18 pm »
No, because RAID is never for data protection. edit : I mean SHOULD never be used for data protection, or as the only data protection method.

Can you elaborate why RAID1 on separate drives should not be used for data protection?  I have been doing that for >20 years and have had at least 3 individual drive failures, but preserved my data.  (I keep OS, Desktop, and programs I use on a separate drive, which soon will be RAID1 too).

RAID 1 only helps with the most basic of hardware failures, when a hard drive dies on its own.  If you're lucky and you notice the faulty drive right away and you replace it fast, your data is safe... otherwise as I explained, if you have drives with consecutive serial numbers, made in same factory, same week, same year, there's increased chances the other drive could fail for the same reasons the other failed already so you need to rush to replace the broken drive. 

For any other reasons, RAID 1 doesn't help you keep your data safe.

For example, let's  say the power supply craps out and outputs 15-20v instead of 12v to the hard drives and kills the motor on both... or burns a hole through both drives circuit boards. Now BOTH drives in your RAID 1 are dead.   

I've also seen the SATA connector on one drive shorting out causing the sata housing to melt and even make flames, and luckily there was no hard drive immediately above the other drive otherwise that drive's sata connector would have caught fire as well.

If it's a work computer, let's say you fire an employee and he decides to kick the pc potentially causing both hard drives to fail at same time. Or maybe there's a small earthquake and the PC falls off the desk to the floor while it's working - few hard drives have acceleration sensors outside laptop models.
 
Or maybe something as basic as a heavy book or a flower pot falls from a shelve above the computer on the computer case, the mechanical shock could damage both drives.

Or let's say a pipe breaks and you have water up to your knees in your house and your computer is on the floor, below water.

Besides hardware failures... Let's say you accidentally download some ransomware that encrypts all your files ... both drives will have encrypted files, you're screwed.

 
The following users thanked this post: DiTBho

Offline Ranayna

  • Frequent Contributor
  • **
  • Posts: 986
  • Country: de
Re: modern storage is crazy
« Reply #34 on: April 28, 2022, 08:33:06 pm »
No, because RAID is never for data protection. edit : I mean SHOULD never be used for data protection, or as the only data protection method.

Can you elaborate why RAID1 on separate drives should not be used for data protection?  I have been doing that for >20 years and have had at least 3 individual drive failures, but preserved my data.  (I keep OS, Desktop, and programs I use on a separate drive, which soon will be RAID1 too).
It comes down to semantics.

RAID (except obviously RAID 0) is for data availability. Some RAID levels have speed advantages compared to others or single drives.
It protects you of failures of one or more of the actual drives. Thats it.

It does not protect you against a lot of things that you would want your data to be protected from, among others:
- accidental deletion
- a cryptolocker
- Controller failure
- data corruption due to faulty ram

RAID can complement backup, but can *never* replace a proper backup.
 
The following users thanked this post: tooki, DiTBho

Offline eugene

  • Frequent Contributor
  • **
  • Posts: 497
  • Country: us
Re: modern storage is crazy
« Reply #35 on: April 28, 2022, 08:48:19 pm »
OK. Now we all know some of the different ways that storing redundant copies of data on separate platters, or even separate drives in the same enclosure might not save one's data.

Still looking for creative ways to use a single drive that has 10 or 20 times the capacity I need in my laptop.
90% of quoted statistics are fictional
 

Offline KaneTW

  • Frequent Contributor
  • **
  • Posts: 811
  • Country: de
Re: modern storage is crazy
« Reply #36 on: April 28, 2022, 08:51:33 pm »
I was allways fascinated by the prices of Siemens Simatic memory cards. For example, list price for the 256MB (yes, megabytes) is 364€... 32GB model will cost you an arm and a leg with the price tag of 1038€. And those are active products and not some obsolete old parts...
I suspect that in the hardware they are all the same, they just make them look smaller... Oh, forgot to say, 4MB model is cheap it's only 53€  :-DD

You're paying for the usage license. They're just standard SD cards with some licensing stuff added.
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7508
  • Country: va
Re: modern storage is crazy
« Reply #37 on: April 28, 2022, 08:56:08 pm »
Quote
Can you elaborate why RAID1 on separate drives should not be used for data protection?

I will echo what mariush said in his earlier posts.

The advantage of RAID is that there should be no downtime for (relatively) trivial problems. If you have RAID1 and a disk fails, you can generally keep going (although the rebuild can be slow). With single drive you're stuffed until you've acquired the replacement and performed a restore, which can be from hours to days depending on how prepared you were.

Also, RAID can improve perfomance.

But RAID should never take the place of backups. There are many problems, from a disk faulting during a rebuild through an external actual disaster to a simple wrong finger issue. You need to do proper backups as well, and if you're going to do those then many of the RAID solutions aren't actually necessary. Do you really need real time protection from, say, two drives failing simultaneously or can you cope with an hour offline?

Personally I used to run RAID1 to save on downtime should a drive fail, but the slowness during rebuilds (which would occur if the OS didn't shut down cleanly) were too much aggro for the perceived gains, so I don't even do that nowadays. A realtime copy of changed important files (typically source code) pushed off-box as they change deals with inter-backup data loss (and adds version history to boot, which RAID1 didn't).

A client had a server running RAID5. I was called in when it wouldn't boot one day and found that the data was gone. Apparently, a drive had failed but no-one noticed (because, RAID) and then another drive failed. They realised the problem and bought two replacement drives,  but couldn't get them to rebuild... Of course, they didn't need backups because, RAID.

 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7508
  • Country: va
Re: modern storage is crazy
« Reply #38 on: April 28, 2022, 08:58:36 pm »
Quote
Still looking for creative ways to use a single drive that has 10 or 20 times the capacity I need in my laptop.

Crypto.

Torrent node.

Porn collection.

Cloud backups for someone else.
 

Online jpanhalt

  • Super Contributor
  • ***
  • Posts: 4002
  • Country: us
Re: modern storage is crazy
« Reply #39 on: April 28, 2022, 10:06:22 pm »
@mariush
Basically, you seem to be saying that if a nuke hits your house or wherever you PC is, how do you protect your data?  I'm not particularly worried about that, as the chance I would be a safe distance away is absolutely near zero.  If I am toast, I don't care about my data.
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 7508
  • Country: va
Re: modern storage is crazy
« Reply #40 on: April 28, 2022, 10:28:51 pm »
Quote
if a nuke hits your house or wherever you PC is, how do you protect your data?

That's rather extreme. Suppose a fire breaks out - do you shut your PC down properly, unplug it and carry it to safety, or do you save your pets, your spouse, perhaps even yourself instead?

'Disaster' can cover many things from "oops, didn't meant to type that" to, as you suggest, a direct hit from a nuke. Not all of them are disastrously disastrous, and not all of them let you mitigate the effects as they occur.
 

Offline amyk

  • Super Contributor
  • ***
  • Posts: 8526
Re: modern storage is crazy
« Reply #41 on: April 29, 2022, 02:21:50 am »
I suppose the OP is looking for some 4MB memory cards too.  :-DD

Just kidding around; I totally get his frustration. Ten years ago I bought a Synology NAS and stuffed two 2TB drives into it (RAID 1.) I thought 2 TB was ridiculous at the time and I was right; the drive is still only half full. The really sad thing is that the drives are 5.25". If one of them fails I'm not sure I could even find a replacement!
2TB 5.25" drives? :wtf:
 

Offline james_s

  • Super Contributor
  • ***
  • Posts: 21611
  • Country: us
Re: modern storage is crazy
« Reply #42 on: April 29, 2022, 03:54:51 am »
Still looking for creative ways to use a single drive that has 10 or 20 times the capacity I need in my laptop.

Movies for watching on airplane flights, RAW format photos from a DSLR, music collection, backups of your other machines. How do you have 20 times the space you need on your laptop? If you can get by with so little space then dump the spinning drive and pop put an SSD in there, the smaller ones are dirt cheap and it is a *massive* peformance improvement.
 

Offline eugene

  • Frequent Contributor
  • **
  • Posts: 497
  • Country: us
Re: modern storage is crazy
« Reply #43 on: April 29, 2022, 03:10:19 pm »
2TB 5.25" drives? :wtf:

They're 3.5" I remember them looking huge... I knew they weren't 8"  :-DD
90% of quoted statistics are fictional
 

Offline eugene

  • Frequent Contributor
  • **
  • Posts: 497
  • Country: us
Re: modern storage is crazy
« Reply #44 on: April 29, 2022, 03:23:17 pm »
How do you have 20 times the space you need on your laptop?

I don't. Both of my laptops have a 1.5TB total and are at less than 50% capacity (which is just about right.) But, this thread is about extremely large drives (14TB) that are bigger than most people can use.
90% of quoted statistics are fictional
 

Offline BradC

  • Super Contributor
  • ***
  • Posts: 2129
  • Country: au
Re: modern storage is crazy
« Reply #45 on: April 29, 2022, 03:58:43 pm »
Seagate even had a firmware bug that cause drives to lock and no longer respond after some amount of time of operation : http://www.datarecoveryspecialists.co.uk/blog/firmware-bug-on-seagate-hard-drive
Had you bought a few drives at the same time, you'd be screwed with all these drives failing at nearly the same time

10 in a RAID-6. Thankfully the fault only manifests itself after a power cycle and these were only cycled about once every 2 years. I fortunately found out about it before they were cycled and had the opportunity to do the firmware updates on a rotating basis with no issue. As Max would say "missed it by -> <- that much".

To echo what everyone else has said. RAID is for availability, not backup.
 
The following users thanked this post: tooki

Offline james_s

  • Super Contributor
  • ***
  • Posts: 21611
  • Country: us
Re: modern storage is crazy
« Reply #46 on: April 29, 2022, 06:01:08 pm »
I don't. Both of my laptops have a 1.5TB total and are at less than 50% capacity (which is just about right.) But, this thread is about extremely large drives (14TB) that are bigger than most people can use.

Nobody has a 14TB drive in their laptop.

There is a huge market for massive drives, I'd love to have a couple of 14TB drives for my media server but I get by with 8TB since they're so much cheaper. If I could get 16TB in a single drive for backup use that would be great, I could backup the whole thing onto a single drive. Tons of people have media collections these days, I ripped all of my DVDs, blurays and CDs years ago and packed them all away. Lots of people into photography store many TB of RAW format images. My security cameras record onto a hard drive too, currently 4TB gets me 2-3 months of archive footage but that decreases every time I add a camera or upgrade to a higher resolution.
 

Offline tooki

  • Super Contributor
  • ***
  • Posts: 13157
  • Country: ch
Re: modern storage is crazy
« Reply #47 on: April 29, 2022, 06:10:16 pm »
I suppose the OP is looking for some 4MB memory cards too.  :-DD

Just kidding around; I totally get his frustration. Ten years ago I bought a Synology NAS and stuffed two 2TB drives into it (RAID 1.) I thought 2 TB was ridiculous at the time and I was right; the drive is still only half full. The really sad thing is that the drives are 5.25". If one of them fails I'm not sure I could even find a replacement!
You definitely did not buy 5.25” hard disk drives for your NAS ten years ago, since they stopped making 5.25” hard disks over 22 years ago (and even then they were extraordinarily rare). Not even old stock, since the last one of those was just 19GB.

Are you sure you’re not measuring a removable  drive tray?
 

Offline tooki

  • Super Contributor
  • ***
  • Posts: 13157
  • Country: ch
Re: modern storage is crazy
« Reply #48 on: April 29, 2022, 06:12:16 pm »
2TB 5.25" drives? :wtf:

They're 3.5" I remember them looking huge... I knew they weren't 8"  :-DD
So… the standard size for desktop drives for the last 35 years? Yeah, that’s shocking…  ::)
 

Offline tooki

  • Super Contributor
  • ***
  • Posts: 13157
  • Country: ch
Re: modern storage is crazy
« Reply #49 on: April 29, 2022, 06:16:39 pm »
No, because RAID is never for data protection. edit : I mean SHOULD never be used for data protection, or as the only data protection method.

Can you elaborate why RAID1 on separate drives should not be used for data protection?  I have been doing that for >20 years and have had at least 3 individual drive failures, but preserved my data.  (I keep OS, Desktop, and programs I use on a separate drive, which soon will be RAID1 too).
It comes down to semantics.

RAID (except obviously RAID 0) is for data availability. Some RAID levels have speed advantages compared to others or single drives.
It protects you of failures of one or more of the actual drives. Thats it.

It does not protect you against a lot of things that you would want your data to be protected from, among others:
- accidental deletion
- a cryptolocker
- Controller failure
- data corruption due to faulty ram

RAID can complement backup, but can *never* replace a proper backup.
This. So much this.



RAID. 👏 Is. 👏 Not. 👏 A. 👏 Backup!!! 👏

RAID dutifully duplicates any deletion or modification  caused by any reason, including software errors, malware, user error, as well as deletions made on purpose that you didn’t realize were wrong until later.

Additionally, RAID doesn’t protect you from damage or loss of the entire array, as in fire or theft. That’s what off-site backups are for.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf