HDDs and badblocks corrupt downloaded files?

[wraper]

I do believe it is only supported by SCSI/SAS etc. drives, but not SATA or IDE/PATA drives, though.  Fixed; thanks for the correction!

Yep, not possible on consumer drives. Programs which can do remap are simply hammering bad sector until HDD decides to actually remap it. Probability of success highly depends on the drive in question. IME on modern drives generally does not work well.

[David Hess]

For spinny disks, I replace drives whenever there are any reallocated sectors (RAW reallocated sector count is nonzero).  This includes brand-new drives.  (And I don't trust Seagate drives, even as paperweights.)

...

In about half the cases you see reallocated sectors increasing a few hours (of use) before the drive dies completely, but then again, most drives do perfectly fine with a small number (say, up to a dozen) reallocated sectors – and many have a few reallocated sectors off the factory.

My experience is that until the wear-out part of their operating life, about half of the drives degrade quickly after the first couple of new bad sectors.  The others just continue to operate like normal.

Which means that no RAID or disk monitoring is as good as backups and keeping your important data on multiple physically separate media.

I currently have a 5 x 2TB RAID6 on my test system made up with all of my spare 2TB drives and one of them, a Samsung HD204UI, keeps returning bad sectors without marking any bad, and the RAID just handles it, which is a nice confidence booster that the RAID controller is working properly.

I recently moved my "portable" data to an external SSD (1) and I maintain an offline backup on a duplicate SSD.  Periodically, like every couple days, I hash the entire contents, update the offline backup, and then check the hashes on the offline backup.  The hash generation and checking also serves to read the entire contents of each SSD giving them the maximum opportunity to scrub a bad sector.  With USB 3.0, the external 250GB SSDs are fast enough to use directly and the backup procedure only takes a few minutes.

Do not all manufacturers have automatic relocation and isolation of defective sectors?

Yes, they do.

How it works in real life, is that when the drive detects that it failed to correctly write a sector,

nope. drives do not detect that. writing is blind , the read heads are off. verification would require waiting an entire revolution to read the sector and compare. this is not done by the drive. it is up to the operating system to do this.

That is right; the drive does not know the sector is bad until it is read later, at which point it marks it as "pending reallocation".  The reallocation then happens when the sector is written which indicates to the drive that the data is no longer needed.  Until then, it will make a best effort to recover the data every time it is read.

If you are running RAID, then when a bad sector is detected, the RAID should regenerate the data and write it back to the drive to force reallocation.  RAID controllers usually support periodic scrubbing where all data is read to look for bad sectors so they can be scrubbed.

What is the truth? HDDs have or not in the fiirmware the automatic function to relocate and isolate bad sectors badblocks? only server hdds? and the year 2000 hdds?

As far as I know, every hard drive with an integrated controller supports automatic reallocation as described above.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

That would be scrub-on-read versus scrub-on-write which we have been discussing.  I personally have never seen a scrub-on-read on a hard drive, but good SSDs are suppose to do this.  (2) On a hard drive scrub-on-read presents practical difficulties because typically no verification pass is done.

(1) An external SSD in an enclosure is my replacement for USB Flash sticks which I consider to be too unreliable.  Maybe somebody makes a reliable USB Flash stick, but I never found it.

(2) SSDs which perform idle time scrubbing *must* also support power loss protection because program and erase cycles can occur at any time, which implies that SSDs which do not support power loss protection, do not perform idle time scrubbing.

[Halcyon]

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

Actually, there are still drives which don't use ECC or have reduced ECC functionality (on purpose). You mostly see them in so-called "AV" or "surveillance" drives. ECC is different to simple read and verify. It's usually far more important to ensure video (and audio) is recorded in near-real time, than worrying about a corrupt frame or two.

"Hard disk controllers" usually refer to RAID or off-board SAS/SATA controllers, which are all quite common.

[wraper]

Actually, there are still drives which don't use ECC

Nonsense, with modern write density such drives simply could not possibly function.

recorded in near-real time, than worrying about a corrupt frame or two

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

[NiHaoMike]

A niche use for HDDs with much simplified/"weaker" ECC would be cache drives. There, corruption isn't a big deal as long as it's detected - worst case you just have to refetch the data. Not sure how much extra storage you would actually get doing that, however.

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

Back in the day, MJPEG compression was most commonly used and corruption really is isolated to a single frame. Not true nowadays, of course...

[Halcyon]

Actually, there are still drives which don't use ECC

Nonsense, with modern write density such drives simply could not possibly function.

recorded in near-real time, than worrying about a corrupt frame or two

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

I guess my years in video production and digital forensics meant nothing?

Video files are relatively easily corrupted with the vast majority of the files contents still being playable even if a small part is damaged, lost or read/decoded too slowly by the system. You often see this as digital artifacting and skipping when frames are lost. If you design a CCTV system any other way, you're doing it wrong.

CCTV usually uses AVC (or some varient thereof) and also records keyframes. This is part of the reason why an entire file isn't rendered unplayable if a small part is lost. I'm not sure how many CCTV systems you've recovered data from, but I would say for me, that number is "hundreds". Everything from high-end professional systems down to the cheapest, crappiest Chinese units.

I also never said CCTV drives were a "cheaper" version. They are just designed differently for a specific purpose. CCTV drives are designed primarily for 24/7, write-intensive operations. They usually have less aggressive error correction (so that read/write errors don't hold up recording data to disk) and generally larger caches (some of the WD Purples have 512 MB on-board). Some are also designed specifically for quiet operation. You only need to look as far as the product spec sheets.

It seems that you are just making things up.

[Monkeh]

Actually, there are still drives which don't use ECC

Nonsense, with modern write density such drives simply could not possibly function.

recorded in near-real time, than worrying about a corrupt frame or two

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

I guess my years in video production and digital forensics meant nothing?

Video files are relatively easily corrupted with the vast majority of the files contents still being playable even if a small part is damaged, lost or read/decoded too slowly by the system. You often see this as digital artifacting and skipping when frames are lost. If you design a CCTV system any other way, you're doing it wrong.

CCTV usually uses AVC (or some varient thereof) and also records keyframes. This is part of the reason why an entire file isn't rendered unplayable if a small part is lost. I'm not sure how many CCTV systems you've recovered data from, but I would say for me, that number is "hundreds". Everything from high-end professional systems down to the cheapest, crappiest Chinese units.

I also never said CCTV drives were a "cheaper" version. They are just designed differently for a specific purpose. CCTV drives are designed primarily for 24/7, write-intensive operations. They usually have less aggressive error correction (so that read/write errors don't hold up recording data to disk) and generally larger caches (some of the WD Purples have 512 MB on-board). Some are also designed specifically for quiet operation. You only need to look as far as the product spec sheets.

It seems that you are just making things up.

Being optimised for write performance and having TLER does not equate to not having any error correction whatsoever.

[wraper]

I also never said CCTV drives were a "cheaper" version. They are just designed differently for a specific purpose. CCTV drives are designed primarily for 24/7, write-intensive operations. They usually have less aggressive error correction (so that read/write errors don't hold up recording data to disk) and generally larger caches (some of the WD Purples have 512 MB on-board). Some are also designed specifically for quiet operation. You only need to look as far as the product spec sheets.

It seems that you are just making things up.

WD purple (surveillance) Non-recoverable read errors per bits read spec is either <1 in 10¹⁵ or <1 in 10¹⁴ for low capacity models.
Both WD green and black has <1 in 10¹⁴. So most of their surveillance models have stronger ECC than PC models.

WD purple datasheet: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-purple-hdd/data-sheet-wd-purple-series-2879-800012.pdf
WD black datasheet: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-black-hdd/product-brief-western-digital-wd-black-pc-hdd.pdf
WD green datasheet: https://cdn.cnetcontent.com/syndication/mediaserver/vcs/wd/inline-content/WD10EARX/12B19F75C2CF2BC269415BC368A375EF482D7C25_source.pdf

[wraper]

CCTV usually uses AVC (or some varient thereof) and also records keyframes. This is part of the reason why an entire file isn't rendered unplayable if a small part is lost. I'm not sure how many CCTV systems you've recovered data from, but I would say for me, that number is "hundreds". Everything from high-end professional systems down to the cheapest, crappiest Chinese units.

And you completely misrepresented what I wrote. That's is exactly because of low amount of keyframes you cannot simply lose a few frames unless corruption happens just before a keyframe. Since you will get corrupted everything till the next keyframe happens. And with relatively unchanging video and low framerate like in CCTV, it can be a very long time till the next keyframe happens.

You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video.

[Halcyon]

I think you seem to be the one misintepreting things.

One should not rely on the URE to determine drive "quality". It's like using price as an indicator of quality.

I'll leave it with you to do some further reading.

[wraper]

One should not rely on the URE to determine drive "quality". It's like using price as an indicator of quality.

You just claimed without any evidence they have weaker ECC which is exactly opposite of truth. Actual HDD specs show exactly the opposite. Now you talk about "quality".

I'll leave it with you to do some further reading.

Reading what? And I suggest you reading some datasheets.

[Monkeh]

I think you seem to be the one misintepreting things.

I think you seem to be the one making things up. Unless you'd care to evidence a drive with no ECC?

Encoding the data on the medium with Reed-Solomon, Turbo, LDPC, et al. is literally the definition of ECC, by the way, before you wriggle some more. What happens beyond that is an additional strategy - in some cases, that may amount to reporting a read error and calling it quits if the ECC fails on the first pass.

[wraper]

in some cases, that may amount to reporting a read error and calling it quits if the ECC fails on the first pass

It's in order of at least seconds before HDD gives up reading bad sector repeatedly. ECC fail on the first pass happens all of the time. If you run some HDD surface scan program, it will become obvious that quite a lot of sectors read slower, especially if HDD is old. Also if you try disturbing HDD position, you will get the same result.

[Monkeh]

in some cases, that may amount to reporting a read error and calling it quits if the ECC fails on the first pass

It's in order of at least seconds before HDD gives up reading bad sector repeatedly. ECC fail on the first pass happens all of the time. If you run some HDD surface scan program, it will become obvious that quite a lot of sectors read slower, especially if HDD is old. Also if you try disturbing HDD position, you will get the same result.

Normal HDDs, yes. What the strategy is on a write-loaded AV drive, on the other hand.. (honestly, I have no idea. They wouldn't tell the likes of us at gunpoint..)

'NAS' drives give up after a few seconds (I believe the classic WD number is 7), typical drives can sit there for minutes looking dumb, AV drives should be rather more abrupt to maintain write throughput. This is probably what Halcyon is mistaking for a complete lack of or crippling of ECC (.. but again, it's not, because ECC is always and error handling is higher level). Giving up and reporting a read error to the host also doesn't necessarily imply a reallocation. More undefined behaviour..

[Jeroen3]

I suspect AV drives use different write scheduling or platter mapping than normal disks.
Maybe some other tweaks that make it deal with 64 sequential data streams better than a general purpose disk

[Wuerstchenhund]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

That is a different mechanism. ECC operates during READ operations. Your data is encoded using a Reed-Solomon / Turbo or LDPC code that requires about 10% more bytes than your real data. this encoded block is written blind and not verified. ( you would need to wait an entire revolution to check the block just written ) .

- if an anomaly is detected during write (head strike , called a thermal asperity TA) the drive WILL retry. if repeat TA's happen in the same block the hardware ECC kicks in and marks that block as bad and relocates the sector to one of the spares. if no more spares are avaialble it will start using actual space. Unless a TA occurs the data is NOT read back to verify it. That is left up to the operating system. Some drives can actually do it but the drive needs ot be instructed by the OS to do it as it slows down the throughput of the drive. By default this function is off.

- if a TA happens during a read operation the sector is re-read. the drive tracks where TA's occur. if too many occur in a certain region then SMART will report this. if a read happens without a TA event there can still be 'soft' errors : the data block coming from the head contains errors. Your data can be reconstructed due to the encoding applied before writing. all this stuff is reported in the SMART system. running drive maintenance toolkits can retrieve this data and instruct the drive to grab the data and move it to different locations. I don't know to what level tools like microsoft chkdsk or scandisk do this. point is the DRIVE doesn't do ANYTHING unless instructed. The only time the drive takes action by itself is during a TA upon writing. in any other case the errors can be corrected due to the ECC encoding used. The errors are flagged , but it is up to the user ( or Os) to deal with the issue .

I don't see how that contradicts what I wrote. Some of your information is also a bit outdated (for example, modern hard drives do lots of things without being instructed).

You are right that ECC correction for the data on the platter kicks in during Read only unless the drive detects an issue with a block during write. But this doesn't change the fact that the physical data itself is ECC protected and unless the drive runs out of reserve sectors it will be able to maintain the integrity of the data that has been written. And that's the main point.

Because the drive maintaining the integrity of it's physical data also includes the often proclaimed "bit rot" scenario where a data bit in a sector changes sporadically even though the physical sector on the platter is good. Because when reading the data the drive will detect the error and re-write the sector and correct the error. As you stated correctly, for this to happen the drive must first read the affected sector, which is why RAID controllers usually perform regular patrol scan. For desktop applications, it's usually sufficient to wait 'til the actual sector is read (where it will be corrected by the drive). In any case, the drive will correct a "rotten bit" during the next read.

Now as to the OS verifying written data, that was indeed a thing back in the days of MSDOS (where you could enable verify reads with "verify on" command; I think the command didn't do anything in WindowsNT and successors, only in DOS and Win9x). That made sense when drives were using stepper motors and still needed to be low-level formatted (with a mandatory subsequent run through a separate surface scan to identify and mark bad sectors!). Because there was no real defect management in hard drives, defect sectors had to be dealt with at the filesystem level.

At least since the ATA days that's no longer the case. The drive geometry reported to the host has largely no resemblance to the actual physical drive geometry (aside from the fact that for the last 2 decades or so we use LBA instead of CHS anyways so the host only sees a number of blocks), and low-level formats are no longer executed as now they can't be done outside the factory. Since the actual disk controller moved into to the drive, having the drive deal with defect management internally was just the next logical step. ATA drives have a "defect-free interface", i.e. the drive does remapping of bad sectors internally and on the fly without showing them at the interface level, so that the drive always appears to the host as having no defects (hence the term "defect-free interface").

This of course only works as long as the drive has sufficient spare sectors to re-allocate, once it runs out it can no longer re-allocate and then will show errors at the interface. Once a drive reaches that stage it's considered "defect" and should be replaced. All newer drives also throw up SMART warnings once the spare sector count drops below a certain threshold, which is a warning that the drive has reached the end of its service life and should be replaced.

CHKDSK and SCANDISK are both legacy tools from that DOS era which have little use today aside from checking the file system for integrity (and that's at a much higher level than the data on a hard disk drive). They can no longer deal with bad sectors simply because on a working drive they can't see them, and once they see them then the drive has long passed the point where it really should have been replaced.

Which has made read-after-write operations by the OS utterly pointless.

[Wuerstchenhund]

If you are running RAID, then when a bad sector is detected, the RAID should regenerate the data and write it back to the drive to force reallocation.  RAID controllers usually support periodic scrubbing where all data is read to look for bad sectors so they can be scrubbed.

That's wrong. RAID controllers don't see disk defects. Sector re-allocation is completely transparent on any modern (i.e. since the days of ATA) drive.

For anything somewhat modern (i.e. SAS/SATA) the RAID controller only sees a bunch of blocks.

The patrol read only serves to trigger the drive's internal defect management (data is read so it will trigger an ECC check, and if there's an error the data will be rewritten or, if that fails, the block moved). The controller itself has no handling in disk errors, and if the controller starts seeing bad blocks then the drive is flagged as defect.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

That would be scrub-on-read versus scrub-on-write which we have been discussing.  I personally have never seen a scrub-on-read on a hard drive, but good SSDs are suppose to do this.  (2) On a hard drive scrub-on-read presents practical difficulties because typically no verification pass is done.[/quote]

All good modern RAID controllers perform Patrol *Reads*. Because this is enough to trigger the drive's internal defect management.

SSDs are a different matter as they should not have any patrol reads done on them, and less so any writes. SSDs also have internal defect management (which is part of it's Garbage Collection) and unlike with hard drives this normally works without having to be triggered (which for SSDs is done with the TRIM command).

(2) SSDs which perform idle time scrubbing *must* also support power loss protection because program and erase cycles can occur at any time, which implies that SSDs which do not support power loss protection, do not perform idle time scrubbing.

That's wrong. PLP on SSDs is required to make sure data that is in the drive's cache is written to the flash memory after a power loss occurs so that it doesn't get lost. If it does then this can affect the integrity of the OS level data layer.

This however has nothing to do with Garbage Collection, which runs in the background and doesn't need PLP.

[Wuerstchenhund]

I suspect AV drives use different write scheduling or platter mapping than normal disks.
Maybe some other tweaks that make it deal with 64 sequential data streams better than a general purpose disk

AV drives are different in that they priorize a sustained data rate over integrity. An AV drive uses different sector allocation strategies (deferred re-allocation), and normally doesn't re-try to read a defective sector.

[BravoV]

So you guys have been sinking this discussion into the details and technical aspect of HD error, while the OP probably will be back with the haunting question, what is the name ? 

So it can be used to google for the details, and then with wishful thinking to use that info to do some elite kung-fu HD hack trying to read that particular offending sector manually, with the hope that can salvage what ever data, even partially that is still readable from that single sector. 

This reminds me of the era using floppy, that was trying to read a floppy bad sector using low level disk editor utility program. 

[Wuerstchenhund]

So you guys have been sinking this discussion into the details and technical aspect of HD error, while the OP probably will be back with the haunting question, what is the name ? 

There isn't a name because there's nothing in a hard drive which "automatically allocates, isolates and prevents downloaded and transferred files from being saved in bad sectors (badblock) of HDDs". The question suggests an overly simplistic understanding of how a computer works.

The hard disk doesn't "know" where the data it is sent or which is requested from it comes from, no does it need to. As far as the hard drive is concerned, it's data, period.

Now what the automatic bad block re-allocation concerns, that is part of the ATA standard which replaced the old IDE standard more than two decades ago. So any EIDE or SCSI hard disk made in the year 2000 should have "defect-free interface", i.e. manage bad blocks internally and transparently to the interface.

So it can be used to google for the details, and then with wishful thinking to use that info to do some elite kung-fu HD hack trying to read that particular offending sector manually, with the hope that can salvage what ever data, even partially that is still readable from that single sector. 

The problem with that is that on a modern drive spare sectors are not accessible from the interface side unless they have been used to re-allocate a bad block.

Also, if a hard drive which was made in 2000 or later exhibits a bad block to the OS then that means it has run out of spare sectors and should have been replaced long ago.

This reminds me of the era using floppy, that was trying to read a floppy bad sector using low level disk editor utility program. 

Floppies never had any defect management on their own, that has always been up to the OS. So fiddling with sectors was both possible and necessary.

[BravoV]

There isn't a name ...<snip> ...

You tell that to the OP. 

Floppies never had any defect management on their own, that has always been up to the OS. So fiddling with sectors was both possible and necessary.

Yep, remembered I used disk editing software, that forced read the bad sector, and for text content, sometimes its valuable as we can still recognize human readable stream between the garbled content, not so much for binary data though.

[Halcyon]

I suspect AV drives use different write scheduling or platter mapping than normal disks.
Maybe some other tweaks that make it deal with 64 sequential data streams better than a general purpose disk

Yep, pretty much. Among other things. They serve a very specific purpose. I certainly wouldn't be using them in a high performance NAS.

There isn't a name ...<snip> ...

You tell that to the OP. 

Floppies never had any defect management on their own, that has always been up to the OS. So fiddling with sectors was both possible and necessary.

Yep, remembered I used disk editing software, that forced read the bad sector, and for text content, sometimes its valuable as we can still recognize human readable stream between the garbled content, not so much for binary data though.

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

[Monkeh]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order.

?! Again with this rubbish..

[wraper]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

LOL, to read backwards, you need to spin it backwards , though heads will stick to the platters if you try to do so and it will be destroyed. As of order at which you read sectors, reading in different order can help to retrieve data when HDD craps out at bad sectors so you can retrieve the data from both sides of bad sector before HDD craps out. Also FYI HDD is a random access device, it does not care about the order data is read at.

[BravoV]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

I'm curious, what exactly did you do ? Mind share the specific details, from brand of disk, model, interface, OS and also the software for doing that, really, I am so eagerly like to do that my self if possible.