HDDs and badblocks corrupt downloaded files?

[Nominal Animal]

Do not all manufacturers have automatic relocation and isolation of defective sectors?

Yes, they do.

How it works in real life, is that when the drive detects that it failed to correctly write a sector, it internally marks that sector as bad, and uses a reserved sector for it instead.  The drives have a fixed number of internally reserved sectors (that are impossible to access via the SATA/IDE/SCSI interfaces – except possibly for some specialized industrial-use SCSI/SAS drives only those with more money than sense can buy), and they do this mapping internally.  You may be able to detect such relocated sectors by timing reads and writes, but the fact that these drives have their own RAM caches makes it quite complicated.

For standard IDE, SATA, and SCSI drives, including both spinny and SSD variants, SMART is the interface drives provide for monitoring things like temperature and reallocated sector counts, et cetera.

On servers with spinny disks, you often have a SMART service/daemon (smartd on Linux machines) running.  The purpose of this one is to read and rewrite sectors on the disk using I/O idle time, without much impacting other use, so that media failures are detected early.  Remember, current drives only detect a failure when they try to write a sector; they do not usually notice errors on read.  Also, these service daemons can also run online and offline media checks (internal service checks the disk does itself, when not burdened by reads/writes; also exposed via SMART).

For example, here is the output of smartctl -a /dev/nvme0 of this particular HP EliteBook 840 G4 with a Samsung 512GB SSD:

Code: [Select]

smartctl 6.6 2016-05-31 r4324 [x86_64-linux-5.4.0-45-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, [url=http://www.smartmontools.org]www.smartmontools.org[/url]

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZVLW512HMJP-000H1
Serial Number:                      [omitted]
Firmware Version:                   CXY73H1Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 512,110,190,592 [512 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Utilization:            163,689,857,024 [163 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Sat Oct  3 19:52:57 2020 EEST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL *Other*
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Warning  Comp. Temp. Threshold:     68 Celsius
Critical Comp. Temp. Threshold:     71 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     7.60W       -        -    0  0  0  0        0       0
 1 +     6.00W       -        -    1  1  1  1        0       0
 2 +     5.10W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1500
 4 -   0.0050W       -        -    4  4  4  4     2200    6000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        32 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    1%
Data Units Read:                    3,296,452 [1.68 TB]
Data Units Written:                 19,111,259 [9.78 TB]
Host Read Commands:                 41,091,531
Host Write Commands:                182,926,857
Controller Busy Time:               809
Power Cycles:                       1,009
Power On Hours:                     2,058
Unsafe Shutdowns:                   36
Media and Data Integrity Errors:    0
Error Information Log Entries:      64
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               32 Celsius
Temperature Sensor 2:               34 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0         64     0  0x0018  0x4004  0x02c            0     0     -
  1         63     0  0x0017  0x4004  0x02c            0     0     -
  2         62     0  0x0018  0x4004  0x02c            0     0     -
  3         61     0  0x0017  0x4004  0x02c            0     0     -
  4         60     0  0x0018  0x4004  0x02c            0     0     -
  5         59     0  0x0017  0x4004  0x02c            0     0     -
  6         58     0  0x0018  0x4004  0x02c            0     0     -
  7         57     0  0x0017  0x4004  0x02c            0     0     -
  8         56     0  0x0018  0x4004  0x02c            0     0     -
  9         55     0  0x0017  0x4004  0x02c            0     0     -
 10         54     0  0x0018  0x4004  0x02c            0     0     -
 11         53     0  0x0017  0x4004  0x02c            0     0     -
 12         52     0  0x0018  0x4004  0x02c            0     0     -
 13         51     0  0x0017  0x4004  0x02c            0     0     -
 14         50     0  0x0018  0x4004  0x02c            0     0     -
 15         49     0  0x0017  0x4004  0x02c            0     0     -
... (48 entries not shown)
For spinny disks, I replace drives whenever there are any reallocated sectors (RAW reallocated sector count is nonzero).  This includes brand-new drives.  (And I don't trust Seagate drives, even as paperweights.)

For SSDs, SMART is not really the best interface, but the fact that there are 100% spares available, and zero media and data integrity errors, tells me I can trust this one.  (Otherwise I'd switch the SSD.)

If you use only better-quality spinny disks, like Hitachi and most Western Digitals, anecdotal evidence (and now-inaccessible Google statistics) show that spinny-disk HDDs work just fine up to about +40'C without degradation in service life; and that there is no way to detect when a drive is about to die.  In about half the cases you see reallocated sectors increasing a few hours (of use) before the drive dies completely, but then again, most drives do perfectly fine with a small number (say, up to a dozen) reallocated sectors – and many have a few reallocated sectors off the factory.
If you maintain a cluster (like I have in the past; a couple of clusters from a few dozen to a few hundred computing nodes), you'll find out that using "known good" off-the-self HDD models is more bang for the buck in the long term than any enterprise models et cetera.  (There was a point in time I used 15kRPM SCSI drives, but that was, oh, two decades ago.  Bog-standard SATA for the last decade or so for bulk storage, SSD drives for fast working storage that gets replaced when it wears out.)

Which means that no RAID or disk monitoring is as good as backups and keeping your important data on multiple physically separate media.

[helius]

On spinning-rust disks, it's true that there are specific reserved sectors (usually there are N reserved sectors per cylinder and M reserved cylinders per drive). With SSD it's more complicated because the externally visible sectors are remapped onto the physical storage blocks for wear leveling. So if you write data to sector 0x100f00 a thousand times, the physical block used each time can be different (the drive also has to read and rewrite the page containing that sector, i.e. write amplification). So instead of sectors that are mostly stored in order with a few damaged sectors reallocated to spares, all of the sectors on an SSD are moving around each time they get written. This has implications for bad block reporting and reallocation, since the underlying physical block that may be defective has no specific address visible to the host. There also does not need to be a specific region of the media reserved for spares, since written blocks are spread over the entire media anyway. SSD durabiliy is enhanced by reducing the volume size and "sparing" more of the media for wear leveling.

[free_electron]

Do not all manufacturers have automatic relocation and isolation of defective sectors?

Yes, they do.

How it works in real life, is that when the drive detects that it failed to correctly write a sector,

nope. drives do not detect that. writing is blind , the read heads are off. verification would require waiting an entire revolution to read the sector and compare. this is not done by the drive. it is up to the operating system to do this.
That's why we have things like journalling file systems. those can have an option to verify after write. but this slows down the drive access tremendously.
such security is typically the job of raid systems. there you don;t get a performance hit as the file is spooled to more than one drive at a time. then one of the drives is verified . the others keep spooling the next file. if the file is correct it is marked as the reference copy and the others will compare later.

this kind if stuff is not done in desktop computers with only one drive. you really need server class hardware with hardware raid engines and drive pools. on a single drive the throughput hit would be massive.
An intermediate solution is the hybrid drive. data is splooled to flash and then offloaded to rotating media with crosscheck. But it still is the task of the OS to do that. the drives don't do that on their own.

[classicsamus87]

What is the truth? HDDs have or not in the fiirmware the automatic function to relocate and isolate bad sectors badblocks? only server hdds? and the year 2000 hdds?

[Nominal Animal]

I'll be talking about spinny-disk drives here, because SSDs behave differently (due to wear leveling).

Let me be a bit more clear: Most drives detect bad blocks at read time, but all do reallocation only on writes.  (Edited to add: Even drives that detect a bad sector based on soft error on read, reallocate that sector by writing the corrected data to a reserved sector.)

Each written sector has some error correction data (ECC) associated with it.
When a sector is read, and the data and the ECC match, you have read a sector successfully without errors.
When a sector is read, but the data and the ECC do not match, but the original data can be reconstructed using the ECC data, you have a soft error.  The sector is read successfully.  (The drive records the soft error, and is reported via SMART.  Whether the soft error is reported to the OS, I do not know.)
When a sector is read, but the data does not match the ECC, and the original data cannot be reconstructed using the ECC data, you have an uncorrectable read error.  (I believe, but cannot verify, that most drives internally retry the read at least a few times, before accepting failure.)  This uncorrectable read error is reported to the OS.  Internally, the drive marks the sector bad, but does not try to reallocate it.
When a sector is written, according to the patents, some drives can detect anomalies at write time.  These do an internal reallocation immediately, without reporting an error to the OS.  There is no loss of data.
When a sector that has been marked bad internally by the drive is written (i.e., a previous read has failed with an uncorrectable error), the drive remaps the sector to one of the internally reserved ones.  There is no loss of data, and succeeding reads and writes of/to that sector cause the drive to actually read/write the relocated reserve sector instead.  This is transparent to the OS; the OS does not detect when this happens.

As to the SMART service/daemon you can run in most OSes, it continuously reads and writes back the same data to each sector of the drive.  No matter which phase the drive detects the error, this causes the drive to detect and reallocate bad sectors.  The bad sectors are also reported to the service daemon, and since it knows the sector, it knows the file/directory affected (if any; sometimes it occurs in unused sectors).  The drives themselves do not do this on their own, an OS-level service/daemon is needed.
Systems using RAID5/6 do not run SMART daemons, because the parity information stored for each data block is used to detect errors.  (It depends on the RAID controller and/or the OS RAID5/6 drivers, whether parity errors cause sector read-rewrite cycles to detect and reallocate bad blocks.)

Here's a list of PDFs you can read:

• Expired 1990 Compaq patent on Automatic hard disk bad sector remapping
• Expired 1991 Maxtor patent on Surface reallocation for improved manufacturing process
• Expired 1996/1997 Samsung patent on Improving access performance on track with re-allocation sector in a hard disk drive
• Expired 1997/1998 Samsung (now Seagate) patent on Reassigning data sectors by detecting soft defects
• Expired 1998 STMicroelectronics patent on Look-ahead reallocation disk defect management
• Bonus: the old Google report on Failure Trends in Large Disk Drive Population

As you can see, I picked a few spread over multiple manufacturers so you can be assured that these are techniques used by all manufacturers, not specialized stuff in "enterprise" drives only.

[classicsamus87]

ECC is the real name of the technology that automatically allocates and isolates defective sectors from the HDD and prevents files from being written to those sectors?

[Monkeh]

ECC is the real name of the technology that automatically allocates and isolates defective sectors from the HDD and prevents files from being written to those sectors?

No.

Please stop plucking random terms you don't understand. Either do actual research or stop.

[mariush]

Dude, you're like a broken record.

STOP it already with asking about names of a function. There's no answer to such question. There's no name for that technology.

ECC is  short for "Error Checking and Correction" - it's just a chunk of data that can be used to determine if a piece of data is potentially corrupted or not.

For example, let's say you have 1 BYTE (8 bits) which has the bits set to this 0111 0100
You can make your own 1 bit ECC code by making this rule: bit is 1 if the number of 1 bits in the byte is even, 0 otherwise.
So for the example above, our 1 bit ECC code would be 1, because there's 4 1 bits, so an even number.
This is a very poor ECC code, because it only gives you the ability to detect if the byte was read incorrectly SOMETIMES and gives you no information on how to determine which bit was bad, but nevertheless it's a basic ECC code.

On hard drives, old mechanical drives used to have around 320 Bytes of ECC code for 8 x 512 Bytes of data (4096 bytes)
On newer hard drives that use 4096 byte sectors, they can use only around 100 bytes of ECC code to get an even better error detection and ability to correct a few corrupted bits.

IF you want to learn more, see this: https://www.anandtech.com/show/2888

The hard drive reads the sectors AND the ECC data that's appended to the end of the sector and then the firmware calculates the ECC code for that sector and compares it with the already read ECC data, and if there's differences, it can use the ECC data to correct the data read.

SSDs also have ECC data, but the amount is different, and they use different ECC algorithms compared to mechanical drives.

[classicsamus87]

If you have no name for this automatic technology to allocate and isolate defective sectors of the HDD, how did you obtain this information? the manufacturer is very restricted so it is difficult to know if year 2000 HDDs have this technology

[Nominal Animal]

If you have no name for this automatic technology to allocate and isolate defective sectors of the HDD, how did you obtain this information?

LMGTFY.

[Siwastaja]

I think you've got your answer already:

It's "sector reallocation".

Unless some manufacturer is calling it Robert.

[BravoV]

I guess the OP being secretive, or just being thick, probably he/she is trying really-really hard to salvage a bitcoin that resides exactly at the bad sector in an old HD  , hence, even "partly" corrupt, as long it can be read at that particular sector, like in the old days of treating floppy disk bad sector say like text content, maybe there is still a chance. 

[Psi]

If OP has decided not to pay for a professional data recovery service then the only thing left to try is spinrite.
Maybe he will get lucky and it will get one good sector read.

[BravoV]

If OP has decided not to pay for a professional data recovery service then the only thing left to try is spinrite.
Maybe he will get lucky and it will get one good sector read.

Its already clear the OP doesn't even want to pay a dime, with hope I guess to get the inner detail on the HD mechanism on sector reallocation, hope to find the hack that able to read that particular bad sector, even just part of it, hence all this weird insisting question that being asked again and again. 

[helius]

You can't access the firmware.

What ever the name you're looking for, point is, no, you can not access those function as you wanted as its hidden from user, period.

It's a fundamental, unalterable component of how modern (read: since the mid 1980s) drives work, there's no reversing it or interfering with it.

There are more things in heaven and earth, Horatio,
Than are dreamt of in your philosophy...

https://forum.hddguru.com/viewtopic.php?f=13&t=27119

[Monkeh]

You can't access the firmware.

What ever the name you're looking for, point is, no, you can not access those function as you wanted as its hidden from user, period.

It's a fundamental, unalterable component of how modern (read: since the mid 1980s) drives work, there's no reversing it or interfering with it.

There are more things in heaven and earth, Horatio,
Than are dreamt of in your philosophy...

https://forum.hddguru.com/viewtopic.php?f=13&t=27119

I'm well aware of manipulating the firmware using interfaces we shouldn't be using, but that's not normal operation and there are good reasons I didn't bring it up. One of them being it's not universal.

[classicsamus87]

If the HDD has SMART then does it have the function of automatically isolating and allocating defective badblock sectors? I've seen Seagate HDD from 2000 with SMART but I don't know if it has this technology

[Halcyon]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

What you're talking about is commonly referred to in the industry as "reallocated sectors". There is a metric for it in SMART. In some cases, you don't want the hard disk to recover from an error (or limit the time taken before giving up), for instance with RAID controllers. Western Digital call this time-limited error recovery (TLER).

[Wuerstchenhund]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

[classicsamus87]

My HDD is IDE from the year 2000, its firmware supports SMART, but within SMART does it contain the automatic technology for isolating and relocating badblocks?

[Zbig]

CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C... oh bugger it...
You guys could either keep responding to a looped shell script till the COVID-zombie apocalypse or find some more pleasing pastime to fill the end of the world as we know it with; your call

[free_electron]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

That is a different mechanism. ECC operates during READ operations. Your data is encoded using a Reed-Solomon / Turbo or LDPC code that requires about 10% more bytes than your real data. this encoded block is written blind and not verified. ( you would need to wait an entire revolution to check the block just written ) .

- if an anomaly is detected during write (head strike , called a thermal asperity TA) the drive WILL retry. if repeat TA's happen in the same block the hardware ECC kicks in and marks that block as bad and relocates the sector to one of the spares. if no more spares are avaialble it will start using actual space. Unless a TA occurs the data is NOT read back to verify it. That is left up to the operating system. Some drives can actually do it but the drive needs ot be instructed by the OS to do it as it slows down the throughput of the drive. By default this function is off.

- if a TA happens during a read operation the sector is re-read. the drive tracks where TA's occur. if too many occur in a certain region then SMART will report this. if a read happens without a TA event there can still be 'soft' errors : the data block coming from the head contains errors. Your data can be reconstructed due to the encoding applied before writing. all this stuff is reported in the SMART system. running drive maintenance toolkits can retrieve this data and instruct the drive to grab the data and move it to different locations. I don't know to what level tools like microsoft chkdsk or scandisk do this. point is the DRIVE doesn't do ANYTHING unless instructed. The only time the drive takes action by itself is during a TA upon writing. in any other case the errors can be corrected due to the ECC encoding used. The errors are flagged , but it is up to the user ( or Os) to deal with the issue .

[Nominal Animal]

My HDD is IDE from the year 2000, its firmware supports SMART, but within SMART does it contain the automatic technology for isolating and relocating badblocks?

Use smartmontools to obtain the SMART attributes of the disk.
In Linux, install smartmontools package using the distro package management, then run sudo smartctl -x /dev/device
In Windows or Mac OS, use the graphical interface to smartmontools, GSmartControl.

If the attribute list contains attribute 5, Reallocated sector count ("Reallocated_sector_ct"), it does contain the bad sector reallocation technology.  The RAW value is the actual number of allocated sectors.
(There is no ATA or SCSI command to reallocate a sector, or to mark a sector "bad"; the drive does all that internally. Some drives do support "reassign physical sector" command, which basically tells the disk that the OS believes the sector to be "bad".)

The other values can be interesting if you e.g. read the Google report on failure trends.

[helius]

(There is no ATA or SCSI command to reallocate a sector, or to mark a sector "bad"; the drive does all that internally.)

!!!!

REASSIGN BLOCKS (Operation 07h)

[Nominal Animal]

(There is no ATA or SCSI command to reallocate a sector, or to mark a sector "bad"; the drive does all that internally.)

!!!!

REASSIGN BLOCKS (Operation 07h)

Aw crap.  I do believe it is only supported by SCSI/SAS etc. drives, but not SATA or IDE/PATA drives, though.  Fixed; thanks for the correction!