HDDs and badblocks corrupt downloaded files?

[classicsamus87]

What is the name of the function present in HDDS that automatically allocates, isolates and prevents downloaded and transferred files from being saved in bad sectors (badblock) of HDDs? is this function present in year 2000 HDDs?

[classicsamus87]

I have an HDD from the year 2000-2003 and this HDD has some badblock, so I downloaded some important files on that HDD and copied it to the DVD so I thought these files were saved and corrupted in these defective sectors of the HDD then copied to the DVD so I wanted to find out the name of this function, where this function is located

[sleemanj]

Both the OS and drive's firmware may flag sectors on the disk as bad, or rather in the case of the firmware remap them to spare sectors transparently to the OS.

S.M.A.R.T may expose the sector reallocation count from the firmware, S.M.A.R.T was developed in the late 90s, so your year 2000 drive may well have it.

If what you are asking is "can I scan this drive, find bad sectors, mark them as bad, and use the good sectors only", then, yes but don't bother, your drive is failing, replace it.

[classicsamus87]

I thought that SMART was only diagnostic and did not relocate and block bad sectors from being used to receive files downloaded from the internet

[sleemanj]

You are correct that S.M.A.R.T is diagnostic.

"Relocating" is done in the firmware.  You can't access the firmware.

[classicsamus87]

does this firmware relocation technology have a name? is it embedded in SMART?

[Psi]

I've seen it called a few things

"Sector reallocation"
"Reserved sectors"
"Reserved sector pooling"



Basically how it works is, it marks a sector as "pending" reallocation whenever it has trouble reading that sector.
You can see the number of sectors in this state in SMART, it's called Pending Reallocation Count.

If it ever gets a successful read of data at a sector that is marked "pending" (or your overwrite the sector) then it completes the reallocation and stores that data in a new sector that has been allocated to replace the bad one.

If you have a drive that has pending sectors and want to force them all to reallocate you can completely erase the drive with zeros. That will allow it to reallocate them.
If the drive has pending sectors that keep giving errors when you try access those files then drive will keep trying to read them over and over again hoping that one attempt will succeed so it can do the reallocation.

[BravoV]

What ever the name you're looking for, point is, no, you can not access those function as you wanted as its hidden from user, period.

[classicsamus87]

Is this function present in SMART or in the HDD firmware?

[Monkeh]

Is this function present in SMART or in the HDD firmware?

As was already explicitly stated, in the firmware. .. which is where SMART is, too - it's just a semi-standardised interface.

It's a fundamental, unalterable component of how modern (read: since the mid 1980s) drives work, there's no reversing it or interfering with it.

[classicsamus87]

I will not change anything in the firmware I just want to understand if this function has a name or is it just SMART

[Monkeh]

I will not change anything in the firmware I just want to understand if this function has a name or is it just SMART

Again, SMART is a diagnostics interface.

The function is simply called 'error handling' - block reallocation is just one outcome of a read or write error.

[BravoV]

I will not change anything in the firmware I just want to understand if this function has a name or is it just SMART

You can NOT change the firmware, even if you want to, except you're working at the HD manufacturer at the technical department that have the knowledge specific in that area.

What ever function that you're looking for like sector error, reallocation and etc, is just the generic terms in magnetic disk storage technology.

SMART is just a reporting, see it as like your car's dashboard, it tells you your fuel level, engine temperature, speed and etc.

What are you asking is like how the gear change and drive the wheels, or what the route of the fuel pipe from the tank to the engine, your dashboard (SMART) does NOT show you that information.

Even you know how the gear changes or how the fuel pipe route to engine, it does NOT mean you can access/repair that, unless you are QUALIFIED car technician specifically for that car model, in this case the HD manufacturer's technical staff only.

And again, NO ... you can not easily find that information on Google, that you think once you found it, you can do/fix/change the fuel routing yourself in your car, forget it.

I guess you have trouble understanding English ?

[Monkeh]

Even you know how the gear changes or how the fuel pipe route to engine, it does NOT mean you can access/repair that, unless you are QUALIFIED car technician specifically for that car model

 

[David Hess]

The way it currently works is that when the hard drive finds an unrecoverable bad sector, it tells the OS and internally flags that sector for "pending reallocation", but it does *not* reallocate that sector until it is written.  So to clear the bad sectors from a drive, you need to write to them which tells the drive that the data they contain is no longer required and a spare sector can be substituted.

If you were running RAID, then the RAID controller would recover the data from the other disks and then write that data back to the bad sector causing the drive to remap it.

[helius]

There are different policies: Reallocate On Write, Reallocate on Read, and options like Error Read Retries and Transfer Data On Error. The SCSI-3 standard specifies which parameter bits affect those policies (which applies to SAS drives).
P-ATA/S-ATA are simpler and tend to have fewer user-facing options. As a rule, they don't even have a write-protect capability, which is always annoying for data recovery.

[classicsamus87]

It is very bad HDD manufacturers do not show more information do not say the name of this function present in the firmware

[Siwastaja]

It is very bad HDD manufacturers do not show more information do not say the name of this function present in the firmware

Seems like a completely random fixation for you.

They don't publish names of any other internal implementation details either. That can be anything on their internal documentation and codebase, like Bob The Reallocator, a.k.a. The Good Ol' Robert, who knows, and why does it matter?

[BravoV]

It is very bad HDD manufacturers do not show more information do not say the name of this function present in the firmware

Agree, and remember that, do NOT buy any HDD from the manufacturer that does not provide that. 

[sleemanj]

It is very bad HDD manufacturers do not show more information do not say the name of this function present in the firmware

I really don't understand your question.

I think there must be some sort of X-Y problem here ( https://en.m.wikipedia.org/wiki/XY_problem ).

What exactly do you want to achieve with such information?  You can't access the firmware, even if you could it's not like knowing 'the name of the function' would be of any use to you.   If you are trying to recover data maybe google bad sector data recovery software.  Ultimately bad sectors are bad, if you manage to get data off it at all, it's not going to be in good shape.

For all you know the sectors went bad after the data was written,  it's a 20 year old hard drive which you knew already had badblocks, it was in it's death throes.

[David Hess]

It is very bad HDD manufacturers do not show more information do not say the name of this function present in the firmware

I have not had any problems finding that data in drives which support SMART.

[classicsamus87]

For HDD firmware to relocate and isolate defective sectors from receiving files, is it enough to have SMART? this function is inside SMART and SMART inside the firmware?

[BravoV]

Pick one ... troll or thick ? 

[Siwastaja]

SMART is the standardized interface for querying statistics etc. across disks of different manufacturers. The firmware implements SMART messaging functionality as one of the features. It also implements everything else to make the hard disk work, including sector relocation; this is all manufacturer-specific. Like if you want to build a car, you are going to decide what kind of materials you use to build the body with, make your drawings, and so on. But the steering wheel (SMART) is something every manufacturer does the same way so you know how to use it.

[classicsamus87]

Do not all manufacturers have automatic relocation and isolation of defective sectors? SMART is separate from this relocation function
I am doing a study on HDD and I take notes but this automatic relocation function does not have a technology name because I did not find it in research I find out if this technology has HDD in the year 2000 I searched in manual but it has no name

[Nominal Animal]

Do not all manufacturers have automatic relocation and isolation of defective sectors?

Yes, they do.

How it works in real life, is that when the drive detects that it failed to correctly write a sector, it internally marks that sector as bad, and uses a reserved sector for it instead.  The drives have a fixed number of internally reserved sectors (that are impossible to access via the SATA/IDE/SCSI interfaces – except possibly for some specialized industrial-use SCSI/SAS drives only those with more money than sense can buy), and they do this mapping internally.  You may be able to detect such relocated sectors by timing reads and writes, but the fact that these drives have their own RAM caches makes it quite complicated.

For standard IDE, SATA, and SCSI drives, including both spinny and SSD variants, SMART is the interface drives provide for monitoring things like temperature and reallocated sector counts, et cetera.

On servers with spinny disks, you often have a SMART service/daemon (smartd on Linux machines) running.  The purpose of this one is to read and rewrite sectors on the disk using I/O idle time, without much impacting other use, so that media failures are detected early.  Remember, current drives only detect a failure when they try to write a sector; they do not usually notice errors on read.  Also, these service daemons can also run online and offline media checks (internal service checks the disk does itself, when not burdened by reads/writes; also exposed via SMART).

For example, here is the output of smartctl -a /dev/nvme0 of this particular HP EliteBook 840 G4 with a Samsung 512GB SSD:

Code: [Select]

smartctl 6.6 2016-05-31 r4324 [x86_64-linux-5.4.0-45-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, [url=http://www.smartmontools.org]www.smartmontools.org[/url]

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZVLW512HMJP-000H1
Serial Number:                      [omitted]
Firmware Version:                   CXY73H1Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 512,110,190,592 [512 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Utilization:            163,689,857,024 [163 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Sat Oct  3 19:52:57 2020 EEST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL *Other*
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Warning  Comp. Temp. Threshold:     68 Celsius
Critical Comp. Temp. Threshold:     71 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     7.60W       -        -    0  0  0  0        0       0
 1 +     6.00W       -        -    1  1  1  1        0       0
 2 +     5.10W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1500
 4 -   0.0050W       -        -    4  4  4  4     2200    6000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        32 Celsius
Available Spare:                    100%
Available Spare Threshold:          5%
Percentage Used:                    1%
Data Units Read:                    3,296,452 [1.68 TB]
Data Units Written:                 19,111,259 [9.78 TB]
Host Read Commands:                 41,091,531
Host Write Commands:                182,926,857
Controller Busy Time:               809
Power Cycles:                       1,009
Power On Hours:                     2,058
Unsafe Shutdowns:                   36
Media and Data Integrity Errors:    0
Error Information Log Entries:      64
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               32 Celsius
Temperature Sensor 2:               34 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0         64     0  0x0018  0x4004  0x02c            0     0     -
  1         63     0  0x0017  0x4004  0x02c            0     0     -
  2         62     0  0x0018  0x4004  0x02c            0     0     -
  3         61     0  0x0017  0x4004  0x02c            0     0     -
  4         60     0  0x0018  0x4004  0x02c            0     0     -
  5         59     0  0x0017  0x4004  0x02c            0     0     -
  6         58     0  0x0018  0x4004  0x02c            0     0     -
  7         57     0  0x0017  0x4004  0x02c            0     0     -
  8         56     0  0x0018  0x4004  0x02c            0     0     -
  9         55     0  0x0017  0x4004  0x02c            0     0     -
 10         54     0  0x0018  0x4004  0x02c            0     0     -
 11         53     0  0x0017  0x4004  0x02c            0     0     -
 12         52     0  0x0018  0x4004  0x02c            0     0     -
 13         51     0  0x0017  0x4004  0x02c            0     0     -
 14         50     0  0x0018  0x4004  0x02c            0     0     -
 15         49     0  0x0017  0x4004  0x02c            0     0     -
... (48 entries not shown)
For spinny disks, I replace drives whenever there are any reallocated sectors (RAW reallocated sector count is nonzero).  This includes brand-new drives.  (And I don't trust Seagate drives, even as paperweights.)

For SSDs, SMART is not really the best interface, but the fact that there are 100% spares available, and zero media and data integrity errors, tells me I can trust this one.  (Otherwise I'd switch the SSD.)

If you use only better-quality spinny disks, like Hitachi and most Western Digitals, anecdotal evidence (and now-inaccessible Google statistics) show that spinny-disk HDDs work just fine up to about +40'C without degradation in service life; and that there is no way to detect when a drive is about to die.  In about half the cases you see reallocated sectors increasing a few hours (of use) before the drive dies completely, but then again, most drives do perfectly fine with a small number (say, up to a dozen) reallocated sectors – and many have a few reallocated sectors off the factory.
If you maintain a cluster (like I have in the past; a couple of clusters from a few dozen to a few hundred computing nodes), you'll find out that using "known good" off-the-self HDD models is more bang for the buck in the long term than any enterprise models et cetera.  (There was a point in time I used 15kRPM SCSI drives, but that was, oh, two decades ago.  Bog-standard SATA for the last decade or so for bulk storage, SSD drives for fast working storage that gets replaced when it wears out.)

Which means that no RAID or disk monitoring is as good as backups and keeping your important data on multiple physically separate media.

[helius]

On spinning-rust disks, it's true that there are specific reserved sectors (usually there are N reserved sectors per cylinder and M reserved cylinders per drive). With SSD it's more complicated because the externally visible sectors are remapped onto the physical storage blocks for wear leveling. So if you write data to sector 0x100f00 a thousand times, the physical block used each time can be different (the drive also has to read and rewrite the page containing that sector, i.e. write amplification). So instead of sectors that are mostly stored in order with a few damaged sectors reallocated to spares, all of the sectors on an SSD are moving around each time they get written. This has implications for bad block reporting and reallocation, since the underlying physical block that may be defective has no specific address visible to the host. There also does not need to be a specific region of the media reserved for spares, since written blocks are spread over the entire media anyway. SSD durabiliy is enhanced by reducing the volume size and "sparing" more of the media for wear leveling.

[free_electron]

Do not all manufacturers have automatic relocation and isolation of defective sectors?

Yes, they do.

How it works in real life, is that when the drive detects that it failed to correctly write a sector,

nope. drives do not detect that. writing is blind , the read heads are off. verification would require waiting an entire revolution to read the sector and compare. this is not done by the drive. it is up to the operating system to do this.
That's why we have things like journalling file systems. those can have an option to verify after write. but this slows down the drive access tremendously.
such security is typically the job of raid systems. there you don;t get a performance hit as the file is spooled to more than one drive at a time. then one of the drives is verified . the others keep spooling the next file. if the file is correct it is marked as the reference copy and the others will compare later.

this kind if stuff is not done in desktop computers with only one drive. you really need server class hardware with hardware raid engines and drive pools. on a single drive the throughput hit would be massive.
An intermediate solution is the hybrid drive. data is splooled to flash and then offloaded to rotating media with crosscheck. But it still is the task of the OS to do that. the drives don't do that on their own.

[classicsamus87]

What is the truth? HDDs have or not in the fiirmware the automatic function to relocate and isolate bad sectors badblocks? only server hdds? and the year 2000 hdds?

[Nominal Animal]

I'll be talking about spinny-disk drives here, because SSDs behave differently (due to wear leveling).

Let me be a bit more clear: Most drives detect bad blocks at read time, but all do reallocation only on writes.  (Edited to add: Even drives that detect a bad sector based on soft error on read, reallocate that sector by writing the corrected data to a reserved sector.)

Each written sector has some error correction data (ECC) associated with it.
When a sector is read, and the data and the ECC match, you have read a sector successfully without errors.
When a sector is read, but the data and the ECC do not match, but the original data can be reconstructed using the ECC data, you have a soft error.  The sector is read successfully.  (The drive records the soft error, and is reported via SMART.  Whether the soft error is reported to the OS, I do not know.)
When a sector is read, but the data does not match the ECC, and the original data cannot be reconstructed using the ECC data, you have an uncorrectable read error.  (I believe, but cannot verify, that most drives internally retry the read at least a few times, before accepting failure.)  This uncorrectable read error is reported to the OS.  Internally, the drive marks the sector bad, but does not try to reallocate it.
When a sector is written, according to the patents, some drives can detect anomalies at write time.  These do an internal reallocation immediately, without reporting an error to the OS.  There is no loss of data.
When a sector that has been marked bad internally by the drive is written (i.e., a previous read has failed with an uncorrectable error), the drive remaps the sector to one of the internally reserved ones.  There is no loss of data, and succeeding reads and writes of/to that sector cause the drive to actually read/write the relocated reserve sector instead.  This is transparent to the OS; the OS does not detect when this happens.

As to the SMART service/daemon you can run in most OSes, it continuously reads and writes back the same data to each sector of the drive.  No matter which phase the drive detects the error, this causes the drive to detect and reallocate bad sectors.  The bad sectors are also reported to the service daemon, and since it knows the sector, it knows the file/directory affected (if any; sometimes it occurs in unused sectors).  The drives themselves do not do this on their own, an OS-level service/daemon is needed.
Systems using RAID5/6 do not run SMART daemons, because the parity information stored for each data block is used to detect errors.  (It depends on the RAID controller and/or the OS RAID5/6 drivers, whether parity errors cause sector read-rewrite cycles to detect and reallocate bad blocks.)

Here's a list of PDFs you can read:

• Expired 1990 Compaq patent on Automatic hard disk bad sector remapping
• Expired 1991 Maxtor patent on Surface reallocation for improved manufacturing process
• Expired 1996/1997 Samsung patent on Improving access performance on track with re-allocation sector in a hard disk drive
• Expired 1997/1998 Samsung (now Seagate) patent on Reassigning data sectors by detecting soft defects
• Expired 1998 STMicroelectronics patent on Look-ahead reallocation disk defect management
• Bonus: the old Google report on Failure Trends in Large Disk Drive Population

As you can see, I picked a few spread over multiple manufacturers so you can be assured that these are techniques used by all manufacturers, not specialized stuff in "enterprise" drives only.

[classicsamus87]

ECC is the real name of the technology that automatically allocates and isolates defective sectors from the HDD and prevents files from being written to those sectors?

[Monkeh]

ECC is the real name of the technology that automatically allocates and isolates defective sectors from the HDD and prevents files from being written to those sectors?

No.

Please stop plucking random terms you don't understand. Either do actual research or stop.

[mariush]

Dude, you're like a broken record.

STOP it already with asking about names of a function. There's no answer to such question. There's no name for that technology.

ECC is  short for "Error Checking and Correction" - it's just a chunk of data that can be used to determine if a piece of data is potentially corrupted or not.

For example, let's say you have 1 BYTE (8 bits) which has the bits set to this 0111 0100
You can make your own 1 bit ECC code by making this rule: bit is 1 if the number of 1 bits in the byte is even, 0 otherwise.
So for the example above, our 1 bit ECC code would be 1, because there's 4 1 bits, so an even number.
This is a very poor ECC code, because it only gives you the ability to detect if the byte was read incorrectly SOMETIMES and gives you no information on how to determine which bit was bad, but nevertheless it's a basic ECC code.

On hard drives, old mechanical drives used to have around 320 Bytes of ECC code for 8 x 512 Bytes of data (4096 bytes)
On newer hard drives that use 4096 byte sectors, they can use only around 100 bytes of ECC code to get an even better error detection and ability to correct a few corrupted bits.

IF you want to learn more, see this: https://www.anandtech.com/show/2888

The hard drive reads the sectors AND the ECC data that's appended to the end of the sector and then the firmware calculates the ECC code for that sector and compares it with the already read ECC data, and if there's differences, it can use the ECC data to correct the data read.

SSDs also have ECC data, but the amount is different, and they use different ECC algorithms compared to mechanical drives.

[classicsamus87]

If you have no name for this automatic technology to allocate and isolate defective sectors of the HDD, how did you obtain this information? the manufacturer is very restricted so it is difficult to know if year 2000 HDDs have this technology

[Nominal Animal]

If you have no name for this automatic technology to allocate and isolate defective sectors of the HDD, how did you obtain this information?

LMGTFY.

[Siwastaja]

I think you've got your answer already:

It's "sector reallocation".

Unless some manufacturer is calling it Robert.

[BravoV]

I guess the OP being secretive, or just being thick, probably he/she is trying really-really hard to salvage a bitcoin that resides exactly at the bad sector in an old HD  , hence, even "partly" corrupt, as long it can be read at that particular sector, like in the old days of treating floppy disk bad sector say like text content, maybe there is still a chance. 

[Psi]

If OP has decided not to pay for a professional data recovery service then the only thing left to try is spinrite.
Maybe he will get lucky and it will get one good sector read.

[BravoV]

If OP has decided not to pay for a professional data recovery service then the only thing left to try is spinrite.
Maybe he will get lucky and it will get one good sector read.

Its already clear the OP doesn't even want to pay a dime, with hope I guess to get the inner detail on the HD mechanism on sector reallocation, hope to find the hack that able to read that particular bad sector, even just part of it, hence all this weird insisting question that being asked again and again. 

[helius]

You can't access the firmware.

What ever the name you're looking for, point is, no, you can not access those function as you wanted as its hidden from user, period.

It's a fundamental, unalterable component of how modern (read: since the mid 1980s) drives work, there's no reversing it or interfering with it.

There are more things in heaven and earth, Horatio,
Than are dreamt of in your philosophy...

https://forum.hddguru.com/viewtopic.php?f=13&t=27119

[Monkeh]

You can't access the firmware.

What ever the name you're looking for, point is, no, you can not access those function as you wanted as its hidden from user, period.

It's a fundamental, unalterable component of how modern (read: since the mid 1980s) drives work, there's no reversing it or interfering with it.

There are more things in heaven and earth, Horatio,
Than are dreamt of in your philosophy...

https://forum.hddguru.com/viewtopic.php?f=13&t=27119

I'm well aware of manipulating the firmware using interfaces we shouldn't be using, but that's not normal operation and there are good reasons I didn't bring it up. One of them being it's not universal.

[classicsamus87]

If the HDD has SMART then does it have the function of automatically isolating and allocating defective badblock sectors? I've seen Seagate HDD from 2000 with SMART but I don't know if it has this technology

[Halcyon]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

What you're talking about is commonly referred to in the industry as "reallocated sectors". There is a metric for it in SMART. In some cases, you don't want the hard disk to recover from an error (or limit the time taken before giving up), for instance with RAID controllers. Western Digital call this time-limited error recovery (TLER).

[Wuerstchenhund]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

[classicsamus87]

My HDD is IDE from the year 2000, its firmware supports SMART, but within SMART does it contain the automatic technology for isolating and relocating badblocks?

[Zbig]

CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C, CTRL-C... oh bugger it...
You guys could either keep responding to a looped shell script till the COVID-zombie apocalypse or find some more pleasing pastime to fill the end of the world as we know it with; your call

[free_electron]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

That is a different mechanism. ECC operates during READ operations. Your data is encoded using a Reed-Solomon / Turbo or LDPC code that requires about 10% more bytes than your real data. this encoded block is written blind and not verified. ( you would need to wait an entire revolution to check the block just written ) .

- if an anomaly is detected during write (head strike , called a thermal asperity TA) the drive WILL retry. if repeat TA's happen in the same block the hardware ECC kicks in and marks that block as bad and relocates the sector to one of the spares. if no more spares are avaialble it will start using actual space. Unless a TA occurs the data is NOT read back to verify it. That is left up to the operating system. Some drives can actually do it but the drive needs ot be instructed by the OS to do it as it slows down the throughput of the drive. By default this function is off.

- if a TA happens during a read operation the sector is re-read. the drive tracks where TA's occur. if too many occur in a certain region then SMART will report this. if a read happens without a TA event there can still be 'soft' errors : the data block coming from the head contains errors. Your data can be reconstructed due to the encoding applied before writing. all this stuff is reported in the SMART system. running drive maintenance toolkits can retrieve this data and instruct the drive to grab the data and move it to different locations. I don't know to what level tools like microsoft chkdsk or scandisk do this. point is the DRIVE doesn't do ANYTHING unless instructed. The only time the drive takes action by itself is during a TA upon writing. in any other case the errors can be corrected due to the ECC encoding used. The errors are flagged , but it is up to the user ( or Os) to deal with the issue .

[Nominal Animal]

My HDD is IDE from the year 2000, its firmware supports SMART, but within SMART does it contain the automatic technology for isolating and relocating badblocks?

Use smartmontools to obtain the SMART attributes of the disk.
In Linux, install smartmontools package using the distro package management, then run sudo smartctl -x /dev/device
In Windows or Mac OS, use the graphical interface to smartmontools, GSmartControl.

If the attribute list contains attribute 5, Reallocated sector count ("Reallocated_sector_ct"), it does contain the bad sector reallocation technology.  The RAW value is the actual number of allocated sectors.
(There is no ATA or SCSI command to reallocate a sector, or to mark a sector "bad"; the drive does all that internally. Some drives do support "reassign physical sector" command, which basically tells the disk that the OS believes the sector to be "bad".)

The other values can be interesting if you e.g. read the Google report on failure trends.

[helius]

(There is no ATA or SCSI command to reallocate a sector, or to mark a sector "bad"; the drive does all that internally.)

!!!!

REASSIGN BLOCKS (Operation 07h)

[Nominal Animal]

(There is no ATA or SCSI command to reallocate a sector, or to mark a sector "bad"; the drive does all that internally.)

!!!!

REASSIGN BLOCKS (Operation 07h)

Aw crap.  I do believe it is only supported by SCSI/SAS etc. drives, but not SATA or IDE/PATA drives, though.  Fixed; thanks for the correction!

[wraper]

I do believe it is only supported by SCSI/SAS etc. drives, but not SATA or IDE/PATA drives, though.  Fixed; thanks for the correction!

Yep, not possible on consumer drives. Programs which can do remap are simply hammering bad sector until HDD decides to actually remap it. Probability of success highly depends on the drive in question. IME on modern drives generally does not work well.

[David Hess]

For spinny disks, I replace drives whenever there are any reallocated sectors (RAW reallocated sector count is nonzero).  This includes brand-new drives.  (And I don't trust Seagate drives, even as paperweights.)

...

In about half the cases you see reallocated sectors increasing a few hours (of use) before the drive dies completely, but then again, most drives do perfectly fine with a small number (say, up to a dozen) reallocated sectors – and many have a few reallocated sectors off the factory.

My experience is that until the wear-out part of their operating life, about half of the drives degrade quickly after the first couple of new bad sectors.  The others just continue to operate like normal.

Which means that no RAID or disk monitoring is as good as backups and keeping your important data on multiple physically separate media.

I currently have a 5 x 2TB RAID6 on my test system made up with all of my spare 2TB drives and one of them, a Samsung HD204UI, keeps returning bad sectors without marking any bad, and the RAID just handles it, which is a nice confidence booster that the RAID controller is working properly.

I recently moved my "portable" data to an external SSD (1) and I maintain an offline backup on a duplicate SSD.  Periodically, like every couple days, I hash the entire contents, update the offline backup, and then check the hashes on the offline backup.  The hash generation and checking also serves to read the entire contents of each SSD giving them the maximum opportunity to scrub a bad sector.  With USB 3.0, the external 250GB SSDs are fast enough to use directly and the backup procedure only takes a few minutes.

Do not all manufacturers have automatic relocation and isolation of defective sectors?

Yes, they do.

How it works in real life, is that when the drive detects that it failed to correctly write a sector,

nope. drives do not detect that. writing is blind , the read heads are off. verification would require waiting an entire revolution to read the sector and compare. this is not done by the drive. it is up to the operating system to do this.

That is right; the drive does not know the sector is bad until it is read later, at which point it marks it as "pending reallocation".  The reallocation then happens when the sector is written which indicates to the drive that the data is no longer needed.  Until then, it will make a best effort to recover the data every time it is read.

If you are running RAID, then when a bad sector is detected, the RAID should regenerate the data and write it back to the drive to force reallocation.  RAID controllers usually support periodic scrubbing where all data is read to look for bad sectors so they can be scrubbed.

What is the truth? HDDs have or not in the fiirmware the automatic function to relocate and isolate bad sectors badblocks? only server hdds? and the year 2000 hdds?

As far as I know, every hard drive with an integrated controller supports automatic reallocation as described above.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

That would be scrub-on-read versus scrub-on-write which we have been discussing.  I personally have never seen a scrub-on-read on a hard drive, but good SSDs are suppose to do this.  (2) On a hard drive scrub-on-read presents practical difficulties because typically no verification pass is done.

(1) An external SSD in an enclosure is my replacement for USB Flash sticks which I consider to be too unreliable.  Maybe somebody makes a reliable USB Flash stick, but I never found it.

(2) SSDs which perform idle time scrubbing *must* also support power loss protection because program and erase cycles can occur at any time, which implies that SSDs which do not support power loss protection, do not perform idle time scrubbing.

[Halcyon]

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

Actually, there are still drives which don't use ECC or have reduced ECC functionality (on purpose). You mostly see them in so-called "AV" or "surveillance" drives. ECC is different to simple read and verify. It's usually far more important to ensure video (and audio) is recorded in near-real time, than worrying about a corrupt frame or two.

"Hard disk controllers" usually refer to RAID or off-board SAS/SATA controllers, which are all quite common.

[wraper]

Actually, there are still drives which don't use ECC

Nonsense, with modern write density such drives simply could not possibly function.

recorded in near-real time, than worrying about a corrupt frame or two

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

[NiHaoMike]

A niche use for HDDs with much simplified/"weaker" ECC would be cache drives. There, corruption isn't a big deal as long as it's detected - worst case you just have to refetch the data. Not sure how much extra storage you would actually get doing that, however.

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

Back in the day, MJPEG compression was most commonly used and corruption really is isolated to a single frame. Not true nowadays, of course...

[Halcyon]

Actually, there are still drives which don't use ECC

Nonsense, with modern write density such drives simply could not possibly function.

recorded in near-real time, than worrying about a corrupt frame or two

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

I guess my years in video production and digital forensics meant nothing?

Video files are relatively easily corrupted with the vast majority of the files contents still being playable even if a small part is damaged, lost or read/decoded too slowly by the system. You often see this as digital artifacting and skipping when frames are lost. If you design a CCTV system any other way, you're doing it wrong.

CCTV usually uses AVC (or some varient thereof) and also records keyframes. This is part of the reason why an entire file isn't rendered unplayable if a small part is lost. I'm not sure how many CCTV systems you've recovered data from, but I would say for me, that number is "hundreds". Everything from high-end professional systems down to the cheapest, crappiest Chinese units.

I also never said CCTV drives were a "cheaper" version. They are just designed differently for a specific purpose. CCTV drives are designed primarily for 24/7, write-intensive operations. They usually have less aggressive error correction (so that read/write errors don't hold up recording data to disk) and generally larger caches (some of the WD Purples have 512 MB on-board). Some are also designed specifically for quiet operation. You only need to look as far as the product spec sheets.

It seems that you are just making things up.

[Monkeh]

Actually, there are still drives which don't use ECC

Nonsense, with modern write density such drives simply could not possibly function.

recorded in near-real time, than worrying about a corrupt frame or two

, You cannot simply corrupt a frame or two. As if you have completely no clue how video compression works. You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video. BTW surveillance HDD are not some cheapened version, they are exactly opposite.

I guess my years in video production and digital forensics meant nothing?

Video files are relatively easily corrupted with the vast majority of the files contents still being playable even if a small part is damaged, lost or read/decoded too slowly by the system. You often see this as digital artifacting and skipping when frames are lost. If you design a CCTV system any other way, you're doing it wrong.

CCTV usually uses AVC (or some varient thereof) and also records keyframes. This is part of the reason why an entire file isn't rendered unplayable if a small part is lost. I'm not sure how many CCTV systems you've recovered data from, but I would say for me, that number is "hundreds". Everything from high-end professional systems down to the cheapest, crappiest Chinese units.

I also never said CCTV drives were a "cheaper" version. They are just designed differently for a specific purpose. CCTV drives are designed primarily for 24/7, write-intensive operations. They usually have less aggressive error correction (so that read/write errors don't hold up recording data to disk) and generally larger caches (some of the WD Purples have 512 MB on-board). Some are also designed specifically for quiet operation. You only need to look as far as the product spec sheets.

It seems that you are just making things up.

Being optimised for write performance and having TLER does not equate to not having any error correction whatsoever.

[wraper]

I also never said CCTV drives were a "cheaper" version. They are just designed differently for a specific purpose. CCTV drives are designed primarily for 24/7, write-intensive operations. They usually have less aggressive error correction (so that read/write errors don't hold up recording data to disk) and generally larger caches (some of the WD Purples have 512 MB on-board). Some are also designed specifically for quiet operation. You only need to look as far as the product spec sheets.

It seems that you are just making things up.

WD purple (surveillance) Non-recoverable read errors per bits read spec is either <1 in 10¹⁵ or <1 in 10¹⁴ for low capacity models.
Both WD green and black has <1 in 10¹⁴. So most of their surveillance models have stronger ECC than PC models.

WD purple datasheet: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-purple-hdd/data-sheet-wd-purple-series-2879-800012.pdf
WD black datasheet: https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/product/internal-drives/wd-black-hdd/product-brief-western-digital-wd-black-pc-hdd.pdf
WD green datasheet: https://cdn.cnetcontent.com/syndication/mediaserver/vcs/wd/inline-content/WD10EARX/12B19F75C2CF2BC269415BC368A375EF482D7C25_source.pdf

[wraper]

CCTV usually uses AVC (or some varient thereof) and also records keyframes. This is part of the reason why an entire file isn't rendered unplayable if a small part is lost. I'm not sure how many CCTV systems you've recovered data from, but I would say for me, that number is "hundreds". Everything from high-end professional systems down to the cheapest, crappiest Chinese units.

And you completely misrepresented what I wrote. That's is exactly because of low amount of keyframes you cannot simply lose a few frames unless corruption happens just before a keyframe. Since you will get corrupted everything till the next keyframe happens. And with relatively unchanging video and low framerate like in CCTV, it can be a very long time till the next keyframe happens.

You will get severely corrupt file, and with how rare keyframes are in surveillance video, a single error could irrecoverably corrupt minutes of video.

[Halcyon]

I think you seem to be the one misintepreting things.

One should not rely on the URE to determine drive "quality". It's like using price as an indicator of quality.

I'll leave it with you to do some further reading.

[wraper]

One should not rely on the URE to determine drive "quality". It's like using price as an indicator of quality.

You just claimed without any evidence they have weaker ECC which is exactly opposite of truth. Actual HDD specs show exactly the opposite. Now you talk about "quality".

I'll leave it with you to do some further reading.

Reading what? And I suggest you reading some datasheets.

[Monkeh]

I think you seem to be the one misintepreting things.

I think you seem to be the one making things up. Unless you'd care to evidence a drive with no ECC?

Encoding the data on the medium with Reed-Solomon, Turbo, LDPC, et al. is literally the definition of ECC, by the way, before you wriggle some more. What happens beyond that is an additional strategy - in some cases, that may amount to reporting a read error and calling it quits if the ECC fails on the first pass.

[wraper]

in some cases, that may amount to reporting a read error and calling it quits if the ECC fails on the first pass

It's in order of at least seconds before HDD gives up reading bad sector repeatedly. ECC fail on the first pass happens all of the time. If you run some HDD surface scan program, it will become obvious that quite a lot of sectors read slower, especially if HDD is old. Also if you try disturbing HDD position, you will get the same result.

[Monkeh]

in some cases, that may amount to reporting a read error and calling it quits if the ECC fails on the first pass

It's in order of at least seconds before HDD gives up reading bad sector repeatedly. ECC fail on the first pass happens all of the time. If you run some HDD surface scan program, it will become obvious that quite a lot of sectors read slower, especially if HDD is old. Also if you try disturbing HDD position, you will get the same result.

Normal HDDs, yes. What the strategy is on a write-loaded AV drive, on the other hand.. (honestly, I have no idea. They wouldn't tell the likes of us at gunpoint..)

'NAS' drives give up after a few seconds (I believe the classic WD number is 7), typical drives can sit there for minutes looking dumb, AV drives should be rather more abrupt to maintain write throughput. This is probably what Halcyon is mistaking for a complete lack of or crippling of ECC (.. but again, it's not, because ECC is always and error handling is higher level). Giving up and reporting a read error to the host also doesn't necessarily imply a reallocation. More undefined behaviour..

[Jeroen3]

I suspect AV drives use different write scheduling or platter mapping than normal disks.
Maybe some other tweaks that make it deal with 64 sequential data streams better than a general purpose disk

[Wuerstchenhund]

Error detection (and recovery) isn't always performed by the hard drive, sometimes it's the file system/operating system itself that does it or the hard disk controller.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

We also don't really have "hard disk controllers" any more (the real controllers are part of the disk, and have been so since the IDE days). What can be found on mainboards is nothing more than a pimped-up interface converter, and even hardware RAID controllers are merely operating at block level (they know nothing about the physical disk properties).

All errors are, in general, dealt with on a hardware level so that the logical interface at the OS level is defect-free. The only exceptions to this are software RAID implementations, which of course must deal with errors at the array level.

That is a different mechanism. ECC operates during READ operations. Your data is encoded using a Reed-Solomon / Turbo or LDPC code that requires about 10% more bytes than your real data. this encoded block is written blind and not verified. ( you would need to wait an entire revolution to check the block just written ) .

- if an anomaly is detected during write (head strike , called a thermal asperity TA) the drive WILL retry. if repeat TA's happen in the same block the hardware ECC kicks in and marks that block as bad and relocates the sector to one of the spares. if no more spares are avaialble it will start using actual space. Unless a TA occurs the data is NOT read back to verify it. That is left up to the operating system. Some drives can actually do it but the drive needs ot be instructed by the OS to do it as it slows down the throughput of the drive. By default this function is off.

- if a TA happens during a read operation the sector is re-read. the drive tracks where TA's occur. if too many occur in a certain region then SMART will report this. if a read happens without a TA event there can still be 'soft' errors : the data block coming from the head contains errors. Your data can be reconstructed due to the encoding applied before writing. all this stuff is reported in the SMART system. running drive maintenance toolkits can retrieve this data and instruct the drive to grab the data and move it to different locations. I don't know to what level tools like microsoft chkdsk or scandisk do this. point is the DRIVE doesn't do ANYTHING unless instructed. The only time the drive takes action by itself is during a TA upon writing. in any other case the errors can be corrected due to the ECC encoding used. The errors are flagged , but it is up to the user ( or Os) to deal with the issue .

I don't see how that contradicts what I wrote. Some of your information is also a bit outdated (for example, modern hard drives do lots of things without being instructed).

You are right that ECC correction for the data on the platter kicks in during Read only unless the drive detects an issue with a block during write. But this doesn't change the fact that the physical data itself is ECC protected and unless the drive runs out of reserve sectors it will be able to maintain the integrity of the data that has been written. And that's the main point.

Because the drive maintaining the integrity of it's physical data also includes the often proclaimed "bit rot" scenario where a data bit in a sector changes sporadically even though the physical sector on the platter is good. Because when reading the data the drive will detect the error and re-write the sector and correct the error. As you stated correctly, for this to happen the drive must first read the affected sector, which is why RAID controllers usually perform regular patrol scan. For desktop applications, it's usually sufficient to wait 'til the actual sector is read (where it will be corrected by the drive). In any case, the drive will correct a "rotten bit" during the next read.

Now as to the OS verifying written data, that was indeed a thing back in the days of MSDOS (where you could enable verify reads with "verify on" command; I think the command didn't do anything in WindowsNT and successors, only in DOS and Win9x). That made sense when drives were using stepper motors and still needed to be low-level formatted (with a mandatory subsequent run through a separate surface scan to identify and mark bad sectors!). Because there was no real defect management in hard drives, defect sectors had to be dealt with at the filesystem level.

At least since the ATA days that's no longer the case. The drive geometry reported to the host has largely no resemblance to the actual physical drive geometry (aside from the fact that for the last 2 decades or so we use LBA instead of CHS anyways so the host only sees a number of blocks), and low-level formats are no longer executed as now they can't be done outside the factory. Since the actual disk controller moved into to the drive, having the drive deal with defect management internally was just the next logical step. ATA drives have a "defect-free interface", i.e. the drive does remapping of bad sectors internally and on the fly without showing them at the interface level, so that the drive always appears to the host as having no defects (hence the term "defect-free interface").

This of course only works as long as the drive has sufficient spare sectors to re-allocate, once it runs out it can no longer re-allocate and then will show errors at the interface. Once a drive reaches that stage it's considered "defect" and should be replaced. All newer drives also throw up SMART warnings once the spare sector count drops below a certain threshold, which is a warning that the drive has reached the end of its service life and should be replaced.

CHKDSK and SCANDISK are both legacy tools from that DOS era which have little use today aside from checking the file system for integrity (and that's at a much higher level than the data on a hard disk drive). They can no longer deal with bad sectors simply because on a working drive they can't see them, and once they see them then the drive has long passed the point where it really should have been replaced.

Which has made read-after-write operations by the OS utterly pointless.

[Wuerstchenhund]

If you are running RAID, then when a bad sector is detected, the RAID should regenerate the data and write it back to the drive to force reallocation.  RAID controllers usually support periodic scrubbing where all data is read to look for bad sectors so they can be scrubbed.

That's wrong. RAID controllers don't see disk defects. Sector re-allocation is completely transparent on any modern (i.e. since the days of ATA) drive.

For anything somewhat modern (i.e. SAS/SATA) the RAID controller only sees a bunch of blocks.

The patrol read only serves to trigger the drive's internal defect management (data is read so it will trigger an ECC check, and if there's an error the data will be rewritten or, if that fails, the block moved). The controller itself has no handling in disk errors, and if the controller starts seeing bad blocks then the drive is flagged as defect.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

That would be scrub-on-read versus scrub-on-write which we have been discussing.  I personally have never seen a scrub-on-read on a hard drive, but good SSDs are suppose to do this.  (2) On a hard drive scrub-on-read presents practical difficulties because typically no verification pass is done.[/quote]

All good modern RAID controllers perform Patrol *Reads*. Because this is enough to trigger the drive's internal defect management.

SSDs are a different matter as they should not have any patrol reads done on them, and less so any writes. SSDs also have internal defect management (which is part of it's Garbage Collection) and unlike with hard drives this normally works without having to be triggered (which for SSDs is done with the TRIM command).

(2) SSDs which perform idle time scrubbing *must* also support power loss protection because program and erase cycles can occur at any time, which implies that SSDs which do not support power loss protection, do not perform idle time scrubbing.

That's wrong. PLP on SSDs is required to make sure data that is in the drive's cache is written to the flash memory after a power loss occurs so that it doesn't get lost. If it does then this can affect the integrity of the OS level data layer.

This however has nothing to do with Garbage Collection, which runs in the background and doesn't need PLP.

[Wuerstchenhund]

I suspect AV drives use different write scheduling or platter mapping than normal disks.
Maybe some other tweaks that make it deal with 64 sequential data streams better than a general purpose disk

AV drives are different in that they priorize a sustained data rate over integrity. An AV drive uses different sector allocation strategies (deferred re-allocation), and normally doesn't re-try to read a defective sector.

[BravoV]

So you guys have been sinking this discussion into the details and technical aspect of HD error, while the OP probably will be back with the haunting question, what is the name ? 

So it can be used to google for the details, and then with wishful thinking to use that info to do some elite kung-fu HD hack trying to read that particular offending sector manually, with the hope that can salvage what ever data, even partially that is still readable from that single sector. 

This reminds me of the era using floppy, that was trying to read a floppy bad sector using low level disk editor utility program. 

[Wuerstchenhund]

So you guys have been sinking this discussion into the details and technical aspect of HD error, while the OP probably will be back with the haunting question, what is the name ? 

There isn't a name because there's nothing in a hard drive which "automatically allocates, isolates and prevents downloaded and transferred files from being saved in bad sectors (badblock) of HDDs". The question suggests an overly simplistic understanding of how a computer works.

The hard disk doesn't "know" where the data it is sent or which is requested from it comes from, no does it need to. As far as the hard drive is concerned, it's data, period.

Now what the automatic bad block re-allocation concerns, that is part of the ATA standard which replaced the old IDE standard more than two decades ago. So any EIDE or SCSI hard disk made in the year 2000 should have "defect-free interface", i.e. manage bad blocks internally and transparently to the interface.

So it can be used to google for the details, and then with wishful thinking to use that info to do some elite kung-fu HD hack trying to read that particular offending sector manually, with the hope that can salvage what ever data, even partially that is still readable from that single sector. 

The problem with that is that on a modern drive spare sectors are not accessible from the interface side unless they have been used to re-allocate a bad block.

Also, if a hard drive which was made in 2000 or later exhibits a bad block to the OS then that means it has run out of spare sectors and should have been replaced long ago.

This reminds me of the era using floppy, that was trying to read a floppy bad sector using low level disk editor utility program. 

Floppies never had any defect management on their own, that has always been up to the OS. So fiddling with sectors was both possible and necessary.

[BravoV]

There isn't a name ...<snip> ...

You tell that to the OP. 

Floppies never had any defect management on their own, that has always been up to the OS. So fiddling with sectors was both possible and necessary.

Yep, remembered I used disk editing software, that forced read the bad sector, and for text content, sometimes its valuable as we can still recognize human readable stream between the garbled content, not so much for binary data though.

[Halcyon]

I suspect AV drives use different write scheduling or platter mapping than normal disks.
Maybe some other tweaks that make it deal with 64 sequential data streams better than a general purpose disk

Yep, pretty much. Among other things. They serve a very specific purpose. I certainly wouldn't be using them in a high performance NAS.

There isn't a name ...<snip> ...

You tell that to the OP. 

Floppies never had any defect management on their own, that has always been up to the OS. So fiddling with sectors was both possible and necessary.

Yep, remembered I used disk editing software, that forced read the bad sector, and for text content, sometimes its valuable as we can still recognize human readable stream between the garbled content, not so much for binary data though.

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

[Monkeh]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order.

?! Again with this rubbish..

[wraper]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

LOL, to read backwards, you need to spin it backwards , though heads will stick to the platters if you try to do so and it will be destroyed. As of order at which you read sectors, reading in different order can help to retrieve data when HDD craps out at bad sectors so you can retrieve the data from both sides of bad sector before HDD craps out. Also FYI HDD is a random access device, it does not care about the order data is read at.

[BravoV]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

I'm curious, what exactly did you do ? Mind share the specific details, from brand of disk, model, interface, OS and also the software for doing that, really, I am so eagerly like to do that my self if possible.

[BravoV]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order.

?! Again with this rubbish..

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

LOL, to read backwards, you need to spin it backwards , though heads will stick to the platters if you try to do so and it will be destroyed. As of order at which you read sectors, reading in different order can help to retrieve data when HDD craps out at bad sectors so you can retrieve the data from both sides of bad sector before HDD craps out. Also FYI HDD is a random access device, it does not care about the order data is read at.

Guys, just cut Halcyon a slack will you ? At least for him to share the details as I find his claim is extraordinary in this age.

If we were living in ST506 era, or in PC floppy, these kind of hack was norm, as they were well known to be documented and accessible by public, so user can drill down to the tiny details even down to single bit read/written like at floppy disk controller. Oldies like Spinrite was well known to exploit these kind of low level features at ST506 HD, like at alter sector's interleave at the tracks forming to boost performance and etc.

Damn, I'm exposing my age by typing that, am I ?   

[classicsamus87]

HDD has SMART technology is it an indication that it automatically remaps and isolates bad sectors (badblocks)?

[wraper]

HDD has SMART technology is it an indication that it automatically remaps and isolates bad sectors (badblocks)?

SMART is simply an instrument for external monitoring of HDD statistics/health. All modern HDD remap sectors by themselves.

[free_electron]

I don't see how that contradicts what I wrote. Some of your information is also a bit outdated (for example, modern hard drives do lots of things without being instructed).

It doesn't contradict. there are two different mechanisms.

ECC is encoding of the user data into a pattern that ends up as a magnetic stream. if , the magnetic stream gets damaged ( surface impurity , physical damage ) the data can be reconstructed using software.
During write the only thing that can be detected is a TA event ( headstrike ). Most drives these days do a cleanup and attempt the write again. if it fails again the data is moved to a different block and area where the strike happened is marked bad. i do not know if they attempt to  read to verify. for throughput purposes they may just immediately retry the same block and see if another TA happens. if the write completes without a TA the drive moves on to write the next block. writing data is fire and forget. They don't even bother reading it back, reliability is that high.
If during a read a TA event happens the ECC engine will attempt a recovery. if data is recovered : move on. if it fails they will automatically retry the read. in case of another TA the cycle repeats. (with timeouts of course)
TA events  are typically non repetitive. it's not like there is something stuck to one particular spot. a TA is caused by a loose particle passing through. it is very hard for anything to stick to the media due to the lubrication process ( platters are coated )

So TA events are transient and typically do not lead to data loss. the drive autorecovers , during write and read without the OS even be aware.
but, a TA event can disturb the magnetic stream enough so that ECC cannot do its work. there are simply too many bits lost in the stream ( the stream on the platter is safe , the stream as captured by the head has a long trail of 'garbage in it', so long that ECC cannot recover.

so the mechanisms serve different purposes :
ECC corrects magnetic errors on the platters  : tiny disturbances affecting a few bits
TA forces the drive to do a retry as the disturbance is so large there is no way ECC can recover from that.upon write the stream is almost certianly damaged beyond recovery so we rewrite immediately. upon read we can try ECC since we have the time of one revolution to decide if w e move on , or re-attempt the write based on the outcome of the ECC decode.

Some of my stuff may be outdate now. i've been out of the harddisk business for 6 years now , so things have moved on. When i left we just finished shingled recording and dual actuator technology. what they do now in terms of dark voodoo i don't know.

[Halcyon]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

LOL, to read backwards, you need to spin it backwards , though heads will stick to the platters if you try to do so and it will be destroyed. As of order at which you read sectors, reading in different order can help to retrieve data when HDD craps out at bad sectors so you can retrieve the data from both sides of bad sector before HDD craps out. Also FYI HDD is a random access device, it does not care about the order data is read at.

It's clear by your comments that you have never done data recovery on a professional level. There are a handful of hardware tools which do exactly this in order to read data off a damaged disk. You're basically reading data from the last LBA back to the first. They also allow you to (for example) completely disable one or more of the read heads and only read out certain platters using the good ones.

I'll point you to the Atola manual (one of such devices) if you wanted to see how it works. Scott Moulton is one of the lead experts in the field, so if you have a spare $3500 to spend, you can partake in one of his 5-day training courses (well worth the money).

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order.

?! Again with this rubbish..

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

LOL, to read backwards, you need to spin it backwards , though heads will stick to the platters if you try to do so and it will be destroyed. As of order at which you read sectors, reading in different order can help to retrieve data when HDD craps out at bad sectors so you can retrieve the data from both sides of bad sector before HDD craps out. Also FYI HDD is a random access device, it does not care about the order data is read at.

Guys, just cut Halcyon a slack will you ? At least for him to share the details as I find his claim is extraordinary in this age.

If we were living in ST506 era, or in PC floppy, these kind of hack was norm, as they were well known to be documented and accessible by public, so user can drill down to the tiny details even down to single bit read/written like at floppy disk controller. Oldies like Spinrite was well known to exploit these kind of low level features at ST506 HD, like at alter sector's interleave at the tracks forming to boost performance and etc.

Thank you. You just can't educate some people. Whether people choose to accept what I'm saying is a matter for them. It's all easily researched and available out there in the public domain.

I do this kind of stuff for a living (among other things). We are seeing less and less of this kind of work as many people switch to SSDs. It's a bit of a dying craft. I've even managed to recover CCTV from fire damaged hard disks before. It's a slow process, but it can be done as long as the disk isn't completely incinerated.

[Monkeh]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

LOL, to read backwards, you need to spin it backwards , though heads will stick to the platters if you try to do so and it will be destroyed. As of order at which you read sectors, reading in different order can help to retrieve data when HDD craps out at bad sectors so you can retrieve the data from both sides of bad sector before HDD craps out. Also FYI HDD is a random access device, it does not care about the order data is read at.

It's clear by your comments that you have never done data recovery on a professional level. There are a handful of hardware tools which do exactly this in order to read data off a damaged disk. You're basically reading data from the last LBA back to the first. They also allow you to (for example) completely disable one or more of the read heads and only read out certain platters using the good ones.

I'll point you to the Atola manual (one of such devices) if you wanted to see how it works. Scott Moulton is one of the lead experts in the field, so if you have a spare $3500 to spend, you can partake in one of his 5-day training courses (well worth the money).

Anyone can change the order they ask for sectors in. Doesn't disable ECC.

I do this kind of stuff for a living

The guy who plastered the wall under my new lounge window does it for a living. Doesn't change the fact he did a shit job.

[Halcyon]

I do this kind of stuff for a living

The guy who plastered the wall under my new lounge window does it for a living. Doesn't change the fact he did a shit job.

I'm beginning to see a bit of a trend here.

Do you judge other experts on their abilities and skills as well? Perhaps you know better than your mechanic, or you can make Duck à l'Orange better than a trained chef. Maybe you can EE better than Dave as well?

Next time I have a difficult job, I'll come and seek advice from you first.

[Monkeh]

I do this kind of stuff for a living

The guy who plastered the wall under my new lounge window does it for a living. Doesn't change the fact he did a shit job.

I'm beginning to see a bit of a trend here.

Do you judge other experts on their abilities and skills as well? Perhaps you know better than your mechanic, or you can make Duck à l'Orange better than a trained chef. Maybe you can EE better than Dave as well?

Next time I have a difficult job, I'll come and seek advice from you first.

What, exactly, should I judge people on? How much they talk themselves up? Pieces of paper? Bold claims?

Stop deflecting for a change. How, exactly, does changing the order of a series of read operations disable ECC? Can you evidence it? Can you point me to any remotely modern HDD which has no ECC functionality whatsoever and writes totally raw data to the platters in the hope it can get it back?

[wraper]

You can also read a disk backwards which yields some pretty reasonable results. It essentially disables any ECC and literally reads the data stream in reverse order. I found it handy with some damaged and problematic drives. It can be slow, but when the data is valuable, sometimes it's worth the days or weeks to read the data out reliably.

LOL, to read backwards, you need to spin it backwards , though heads will stick to the platters if you try to do so and it will be destroyed. As of order at which you read sectors, reading in different order can help to retrieve data when HDD craps out at bad sectors so you can retrieve the data from both sides of bad sector before HDD craps out. Also FYI HDD is a random access device, it does not care about the order data is read at.

It's clear by your comments that you have never done data recovery on a professional level. There are a handful of hardware tools which do exactly this in order to read data off a damaged disk. You're basically reading data from the last LBA back to the first. They also allow you to (for example) completely disable one or more of the read heads and only read out certain platters using the good ones.

I'll point you to the Atola manual (one of such devices) if you wanted to see how it works. Scott Moulton is one of the lead experts in the field, so if you have a spare $3500 to spend, you can partake in one of his 5-day training courses (well worth the money).

I don't know if I need to cry or laugh. Also it seems you preferer to ignore or straw man what is not convenient to you in my posts. Reading starting from last LBA is not reading HDD in reverse. It's simply starting reading in order from higher LBA to lower LBA. There is no preferred reading order to begin with. It's just a normal HDD operation, not some special mode FFS. As I already said, HDD is a random access device. And I already said in quoted post what's the reason for doing so. If you really did data recovery professionally, I'm amazed by level of ignorance and misunderstanding of what you were actually doing. And being proud if it rather that even think of possibility of being wrong and actually checking if what you are saying is true. Not to mention ever quoting any information that supports your claims.
I guess you mean this by mentioning Atola manual:

There are three methods of scanning:
Linear — from start LBA to end LBA
Backward — from end LBA to start LBA (in reverse)

And proudly claim magical ECC ignoring while completely misunderstanding what it actually does.

[David Hess]

If you are running RAID, then when a bad sector is detected, the RAID should regenerate the data and write it back to the drive to force reallocation.  RAID controllers usually support periodic scrubbing where all data is read to look for bad sectors so they can be scrubbed.

That's wrong. RAID controllers don't see disk defects. Sector re-allocation is completely transparent on any modern (i.e. since the days of ATA) drive.

For anything somewhat modern (i.e. SAS/SATA) the RAID controller only sees a bunch of blocks.

The patrol read only serves to trigger the drive's internal defect management (data is read so it will trigger an ECC check, and if there's an error the data will be rewritten or, if that fails, the block moved). The controller itself has no handling in disk errors, and if the controller starts seeing bad blocks then the drive is flagged as defect.

You are going to have a hard time convincing me of that since I have watched it in real time.  Maybe you have only used crap RAID controllers, of which there are many.

Hard drives do *not* perform scrub-on-read operation for uncorrectable errors; instead they attempt to read the data until a timeout is reached, and then return the bad data and indicate an error.  "RAID" drives allow a shorter timeout to be set so they do not waste time trying to recover data which can be regenerated anyway.

When the RAID controller receives the bad data and an error, good ones regenerate the data and write it back to the drive, which then performs a scrub-on-write.  If this did not occur, then there would be no reason for the RAID controller to perform idle time scrubbing.

It would be dumb to discard an entire drive because of a single bad read when the data can be recovered and written back to force the drive to reallocation that sector; of course firmware based RAID controllers often are this dumb.

Actually, every hard drive made since we moved from plain old IDE to ATA employs ECC (and it was common even before then). No exceptions. All data on a hard drive is ECC protected, and the drive will know if there's an error (the idea behind "bit rot"). And unless the bit error happens to be in a defective sector *and* the drive has exhausted its spares then the drive will correct the error the next time the sector is read.

That would be scrub-on-read versus scrub-on-write which we have been discussing.  I personally have never seen a scrub-on-read on a hard drive, but good SSDs are suppose to do this.  (2) On a hard drive scrub-on-read presents practical difficulties because typically no verification pass is done.

All good modern RAID controllers perform Patrol *Reads*. Because this is enough to trigger the drive's internal defect management.

The internal defect management could operate on a correctable error, but I have never seen happen.  I would have noticed if the defect list grew without errors being reported.  Many times I have done extended SMART surface scans and watched for this very thing and it never happened.  Most recently I have done it multiple times in the past couple of weeks on a pair of 1TB WD Greens, but doing an external surface scan which includes writing had some results.

SSDs are a different matter as they should not have any patrol reads done on them, and less so any writes. SSDs also have internal defect management (which is part of it's Garbage Collection) and unlike with hard drives this normally works without having to be triggered (which for SSDs is done with the TRIM command).

The difference is that SSD must perform verification after every write as part of the write process, and they also scrub-on-read of sectors which are still correctable if they have too many soft errors.  Many perform idle time scrubbing because retention of modern Flash memory is abysmal, or at least their specifications say they do.  I sure know that many do not and will happily suffer bit rot while powered.

(2) SSDs which perform idle time scrubbing *must* also support power loss protection because program and erase cycles can occur at any time, which implies that SSDs which do not support power loss protection, do not perform idle time scrubbing.

That's wrong. PLP on SSDs is required to make sure data that is in the drive's cache is written to the flash memory after a power loss occurs so that it doesn't get lost. If it does then this can affect the integrity of the OS level data layer.

No, but there are two levels of power loss protection.

What you are describing is protection of data in transit which includes buffered and data in write cache.  Not all drives have this nor is it required if the filesystem and drive handle synchronous writes properly.  In the worst case, file contents are not updated but no metadata or other contents should be lost.

However power loss protection is also required for *any* write operation, and possibly any erase operation.  The reason for this is that interrupted writes can not only corrupt the data which is being written, but also data in other sectors, including the flash translation table, which can result in a non-recoverable situation.

Their are two reasons data can be so easily corrupted with an incomplete write.  With multi-level flash, the existing data when a sector is updated is at risk during the update and it is easy to see why.  It seem to me like this is avoidable by only storing the data from one sector with multiple levels but apparently Flash chips are not organized that way.

The other reason is more insidious; the state machine which controls the write operation can glitch during power loss, which is how drives which lack power loss protection got bricked so easily in those published tests several years ago.  Some drives lost no data, not even data in transit, some drives lost only data in transit, and most suffered either unrelated data loss or complete failure, but that was before the problem was understood well.

[David Hess]

Also, if a hard drive which was made in 2000 or later exhibits a bad block to the OS then that means it has run out of spare sectors and should have been replaced long ago.

I recovered a RAID set not long ago which had exactly that problem.  The drive reported an error during a RAID rebuild, because I stupidly did not run a scrub of the array before swapping a drive, which stopped the RAID rebuild.  The drive actually had accumulated several bad sectors but it only takes one to stop the rebuild.  The solution was to scan the RAID at the level of the file system to determine which files were damaged, and then deliberately run a read-write of the entire drive outside of the array to force reallocation of the bad sectors, albeit with corrupted data.  Once that was done, the drive reported no bad sectors during the RAID rebuild and the damaged files could be restored.

So a drive can report a bad sector to the OS, or RAID controller in this case, but I have had both recently, while not being out of spare sectors because scrub-on-read deliberately never happened, and the drive was waiting for a write to the bad sector to perform scrub-on-write.  It would be much worse for the drive to return bad data and no error while performing a scrub-on-read operation.

[helius]

What is special about "AV" drives is their firmware handling of thermal recalibration. As the temperature inside the drive changes, parts like the head armature and the platters change dimension, and since they have different thermal coefficients of expansion, throws off their tracking. The drive firmware needs to search for the embedded servo signals to form a model of how far apart the tracks are for generating the seek impulses, and periodically re-calibrates in case the temperature has changed. This re-calibration interrupts any data transfer and makes it wait, which would be a problem for real-time media capture or playback. AV firmware just uses a different re-calibration strategy that postpones the re-cal until the disk is idle.

This also means that there is no benefit for AV drives for surveillance recording, since by definition it is never idle. They were made for video and film post-production and multi-track recording studios.

[Monkeh]

This also means that there is no benefit for AV drives for surveillance recording, since by definition it is never idle. They were made for video and film post-production and multi-track recording studios.

You should tell WD that, they're marketing their drives to the wrong people, apparently.

[Halcyon]

This also means that there is no benefit for AV drives for surveillance recording, since by definition it is never idle. They were made for video and film post-production and multi-track recording studios.

A lot of those "AV" drives are primarily for CCTV recording, where you have multiple, relatively low bitrate streams. Any half-decent video or film recording studio who are working with large amounts of high-bitrate recordings will be using proper NASs or SANs with enterprise drives or SSD. When I worked in video production in a few Australian TV studios, read speed was just as important as write speed. Those consumer AV drives are designed primarily for writes where as read speed can vary significantly.

[MK14]

Anyone can change the order they ask for sectors in. Doesn't disable ECC.

And proudly claim magical ECC ignoring while completely misunderstanding what it actually does.

As far as I can tell, I completely (>99% as regards reading it backwards, and at least some agreement on ECC) agree with Halcyon here.

I'm beginning to see a bit of a trend here.

Me too.
Simple googling of the concept, brings complete (>99%, as regards reading it backwards, and at least some agreement as regards the ECC situation), agreement, with what you have been saying.

In summary, it seems that if you try and read a set of files, on a hard disk, with one or more faulty sectors. Using the 'normal' forward reading methods. As well as reading the requested sector, e.g. sector 100,000.
A modern drive (HDD), also tries to partly fill up its (often) massive cache buffer, e.g. 128 MB worth.
Using the many heads (on the often multiple platters), which (given modern very high density HDDs), will be a huge number of sectors.
If any of them have read errors (even if those bad sectors, have nothing whatsoever to do with the sector/file, you were originally trying to read), then the drive will tend to go into attempting to continually rereading the 'faulty' sectors, rather than at least supplying the working sectors, to the OS.

But if you read the HDD 'backwards', i.e. request the sectors of the files of interest in reverse order. It will only read the sectors that you request (although potentially extremely slowly), ignoring the bad sectors, either fully, or until a bad sector is part of the file you are trying to read.

Hence the ECC is only being calculated for the sectors you are trying to actually read, rather than a huge number of other ones, which could be bad, and hence causing the read to be aborted or badly delayed.

Many links found, example(s):
https://www.forensicfocus.com/forums/general/reading-disk-areas-backwards-does-it-help/

https://www.linkedin.com/pulse/imaging-reverse-reduce-data-loss-risk-hdd-failure-serge-shirobokov

[Halcyon]

I will clarify, you don't have to disable ECC. You can also manually set the number of times the drive attempts to re-read a sector. It can be zero or it can be more. It will depend on the drive firmware, but tools like Atola and DeepSpar allow you to really get into the low level of the firmware to change how the drive would normally perform when connected to a host. As I said, an example is that individual drive heads can be completely turned off if you want to (I've used this method a few times when one head is completely stuffed, rather than doing a head swap).

But yeh, when you're talking about damaged drives, read speed can get real slow with each subsequent pass as it tried to read out more data than the pass before. The last drive I imaged got to about Pass 7 and was reading at between 1 and 7 KB/sec. Needless to say, I canned the process at that point, but most of the data was recovered and readable. It just depends on the job and cost vs. benefit.

[Wuerstchenhund]

That's wrong. RAID controllers don't see disk defects. Sector re-allocation is completely transparent on any modern (i.e. since the days of ATA) drive.

For anything somewhat modern (i.e. SAS/SATA) the RAID controller only sees a bunch of blocks.

The patrol read only serves to trigger the drive's internal defect management (data is read so it will trigger an ECC check, and if there's an error the data will be rewritten or, if that fails, the block moved). The controller itself has no handling in disk errors, and if the controller starts seeing bad blocks then the drive is flagged as defect.

You are going to have a hard time convincing me of that since I have watched it in real time.

You have seen what, exactly?

Maybe you have only used crap RAID controllers, of which there are many.

I'm not sure I'd call HPE's Smart Array range (or Dell's PERC, or Microsemi's Adaptec) hardware RAID controllers "crap", as they represent the upper end of the market.

But I'm certainly not talking about fake RAID stuff if that's what you have in mind.

Hard drives do *not* perform scrub-on-read operation for uncorrectable errors; instead they attempt to read the data until a timeout is reached, and then return the bad data and indicate an error.  "RAID" drives allow a shorter timeout to be set so they do not waste time trying to recover data which can be regenerated anyway.

That is correct when it comes to *uncorrectable* errors.

But as far as *correctable* errors are concerned, they (as the name implies) are corrected inside the drive when found during read. The typical "bit rot" scenario is exactly that, a "flipped bit" on an otherwise healthy sector. Which, when detected, results in a re-write of that sector.

Now, as to *uncorrectable* errors, it is correct that the drive eventually returns corrupted data. However, when a hard drive reports an uncorrectable error then that's because the defect exceeds into the ECC area so there is not enough information to restore the original information. Which commonly means the drive is defective and should be replaced. It's as simple as that.

When the RAID controller receives the bad data and an error, good ones regenerate the data and write it back to the drive, which then performs a scrub-on-write.  If this did not occur, then there would be no reason for the RAID controller to perform idle time scrubbing.

Nope. What you wrote would be (mostly) correct if we were talking about MFM/RLL/ESDI or early IDE (or SCSI-1 drives) from some 30 years ago. But we're not.

Modern (i.e. made in 2000 or later) drives don't represent their physical layout to the host. They report some artificial CHS layout which has nothing to do with the actual physical layout to the host for backward compatibility reasons (so that antique systems still CHS for addressing can boot from these drives). Even the sector size is often fake, as most modern hard drives use 4k sectors internally while reporting 512byte sectors on the interface.

But for most part the CHS layout isn't even used. LBA (Logical Block Adressing) has been a thing even before the year 2000 (it was first used with SCSI drives long before then), and it's been the standard way of addressing disks for many many years. With LBA, the host only sees a device which has a certain number of blocks. There's no CHS involved. LBA has been supported at least since Windows 98 and NT 4.0, and became the standard format with Windows 2000. And while LBA was an extension for IDE and ATA, LBA is the defined addressing standard for SATA, SAS and NVMe storage.

You RAID controller, whatever type/model that is, would have to be really old to not use LBA (and if that controller is so old then I guess the disks are, too). And even then it would only see the fake geometry reported by the drive, not the real one.

If a modern RAID controller encounters a bad block (unrecoverable error), it will try to reconstruct the data from the redundancy disks and then may attempt to re-write the block to the affected disk. If this disk has sufficient spare sectors, it may revert the write to a spare block, after which the block will be fine and the integrity of the data is restored. If that was a one-off, the drive may well be fine for years. However, if that happens more often (as it's the case on a dying drive), the disk will eventually run out of spare sectors after which the attempt by the RAID controller to re-write the block will fail, and then the disk is failed and the array goes into contingency mode.

It would be dumb to discard an entire drive because of a single bad read when the data can be recovered and written back to force the drive to reallocation that sector; of course firmware based RAID controllers often are this dumb.

In a professional environment, if a drive shows bad sectors at the interface it's scrapped, period. Any decent RAID controller will immediately flag a drive as soon as unrecoverable errors start to appear. Because for a modern drive unrecoverable errors are usually a clear sign that the drive is defective, and the only thing that would be stupid is to not scrap the drive and risk the integrity of the host data. It's a simple as that. Because at the end of the day the host data (and the hourly rate for the admin who has to deal with it and the potential fall-out should the drive remain in service) is worth a lot more than that stupid hard drive.

Now I accept that for hobbyist use that may well be different, and if you can't afford to replace a drive it's certainly tempting to work around the problem of a defect sector. But that is only a viable option if your data (and your time) isn't worth much (because if it was you'd not try to cheap-skate backup). And it doesn't change the fact that the drive is telling you that you can no longer rely on it.

The internal defect management could operate on a correctable error, but I have never seen happen.

Well, yes, that's because it's supposed to be *transparent* to the host. Which is the whole point of a "defect-free interface".

I would have noticed if the defect list grew without errors being reported.  Many times I have done extended SMART surface scans and watched for this very thing and it never happened.  Most recently I have done it multiple times in the past couple of weeks on a pair of 1TB WD Greens, but doing an external surface scan which includes writing had some results.

Because you don't seem to fully understand what you are seeing. Any high level surface scan tool only scans the area that the drive is reporting as good (it has no access to the whole drive area - "defect-free Interface", remember?). Defects are hidden because defect management is completely *transparent*. The only way you can check what's actually going on in the drive is through SMART data.

When you're at the point where your tool can "see" defects then that means the drive has developed unrecoverable errors and is defective and should be discarded, but at the very least should not be used to store anything of importance.

SSDs are a different matter as they should not have any patrol reads done on them, and less so any writes. SSDs also have internal defect management (which is part of it's Garbage Collection) and unlike with hard drives this normally works without having to be triggered (which for SSDs is done with the TRIM command).

The difference is that SSD must perform verification after every write as part of the write process, and they also scrub-on-read of sectors which are still correctable if they have too many soft errors.  Many perform idle time scrubbing because retention of modern Flash memory is abysmal, or at least their specifications say they do.  I sure know that many do not and will happily suffer bit rot while powered.

There's nothing in SSDs that matches "scrubbing". Which, on flash, would be completely counterproductive because every read of a flash cell reduces the amount of charge that is stored in that cell, so if a SSD did regular "read-scrubs" then the charge level would quickly become so low that the sector would have to be re-written to maintain its information, a process which causes wear on the flash cell.

In reality, SSDs employ ECC (although slightly more sophisticated then on spinning rust) for error correction, Garbage Collection (which erases blocks that are marked for deletion) and Wear Leveling (where the SSD keeps track of flash use and tries to balance the load evenly across all cells).

(2) SSDs which perform idle time scrubbing *must* also support power loss protection because program and erase cycles can occur at any time, which implies that SSDs which do not support power loss protection, do not perform idle time scrubbing.

That's wrong. PLP on SSDs is required to make sure data that is in the drive's cache is written to the flash memory after a power loss occurs so that it doesn't get lost. If it does then this can affect the integrity of the OS level data layer.

No, but there are two levels of power loss protection.

What you are describing is protection of data in transit which includes buffered and data in write cache.  Not all drives have this nor is it required if the filesystem and drive handle synchronous writes properly.  In the worst case, file contents are not updated but no metadata or other contents should be lost.

That is correct. PLP exists to protect host data.

However power loss protection is also required for *any* write operation, and possibly any erase operation.  The reason for this is that interrupted writes can not only corrupt the data which is being written, but also data in other sectors, including the flash translation table, which can result in a non-recoverable situation.

Their are two reasons data can be so easily corrupted with an incomplete write.  With multi-level flash, the existing data when a sector is updated is at risk during the update and it is easy to see why.  It seem to me like this is avoidable by only storing the data from one sector with multiple levels but apparently Flash chips are not organized that way.

First of all, SSDs internally are physically organized in blocks, not sectors. Remember the LBA I mentioned above? LBA is *exactly* how SSDs are structured internally. On SSDs, Sectors are an artificial construct which has zero relation to which flash cell the information is physically located.

Garbage Collection and Wear Levelling also don't need PLP. If power is interrupted during the deletion of a block then the block will remain marked as for deletion and deletion will be repeated after power comes back up. For Wear Levelling, when data is moved from one block to another then the data from the old block is copied to a new block and then the old block is marked for deletion. If that process is interrupted then the old data is still there and the block shift is simply repeated.

The other reason is more insidious; the state machine which controls the write operation can glitch during power loss,

The part in a SSD which controls write operations (and everything else) is the SSD controller and that is not a State Machine, it's a micro-controller running specific software (the drive's firmware) to perform it's duties.

which is how drives which lack power loss protection got bricked so easily in those published tests several years ago.  Some drives lost no data, not even data in transit, some drives lost only data in transit, and most suffered either unrelated data loss or complete failure, but that was before the problem was understood well.

I know that early consumer SSDs killed themselves (particularly those made by OCZ) for a number of reasons, most of which were related to the lack of automatic GC (the drive needed GC triggered by the OS via TRIM, which the back then still common WindowsXP didn't support) and a wide range of firmware errors.

But the point remains that the only "data in transit" in a SSD is host data, and that SSDs do not work the way you think they do.

[Wuerstchenhund]

Also, if a hard drive which was made in 2000 or later exhibits a bad block to the OS then that means it has run out of spare sectors and should have been replaced long ago.

I recovered a RAID set not long ago which had exactly that problem.  The drive reported an error during a RAID rebuild, because I stupidly did not run a scrub of the array before swapping a drive, which stopped the RAID rebuild.  The drive actually had accumulated several bad sectors but it only takes one to stop the rebuild.  The solution was to scan the RAID at the level of the file system to determine which files were damaged, and then deliberately run a read-write of the entire drive outside of the array to force reallocation of the bad sectors, albeit with corrupted data.  Once that was done, the drive reported no bad sectors during the RAID rebuild and the damaged files could be restored.

So in short, you put a knowingly defective drive in a RAID array, and when the rebuild process falls over the drive's dead sectors you fix it by manually relocating corrupt host data to another sector so the rebuild process can be fooled into completing? Seriously??? 

At least where I am working doing such stunts would very likely get you fired for negligience and incompetence.

As I said in my last post I understand that for hobbyist use the criteria may well be different and re-using a drive that shows defects may or may not make sense, but the fact remains that this drive is defective and really should be replaced, not forced into use.

So a drive can report a bad sector to the OS, or RAID controller in this case, but I have had both recently, while not being out of spare sectors because scrub-on-read deliberately never happened, and the drive was waiting for a write to the bad sector to perform scrub-on-write.  It would be much worse for the drive to return bad data and no error while performing a scrub-on-read operation.

You're not understanding the issues here. The drive shows bad sectors at the interface because it can't recover the data in these sectors because of the large defect area on the platter. And when the drive can't recover the data (because the ECC information is affected as well) then it has to report an unrecoverable error as without correct data it makes no sense to relocate the sector.

The fact that the rebuild failed on the drive should have already been a warning that the best place for this drive would be the electronics recycling dumpster (or the shredder if the data was sensitive).

This really is RAID taken ad absurdum.

[Monkeh]

As far as I can tell, I completely (>99% as regards reading it backwards, and at least some agreement on ECC) agree with Halcyon here.But if you read the HDD 'backwards', i.e. request the sectors of the files of interest in reverse order. It will only read the sectors that you request (although potentially extremely slowly), ignoring the bad sectors, either fully, or until a bad sector is part of the file you are trying to read.

Hence the ECC is only being calculated for the sectors you are trying to actually read, rather than a huge number of other ones, which could be bad, and hence causing the read to be aborted or badly delayed.

Yes, changing the way you request sectors from the drive may change its read-ahead strategy. That doesn't mean ECC is disabled.

Changing the order of reads is absolutely a valid tactic, one I've used, and I've never argued it isn't. But the claim is that it disables ECC, the claim is ECC doesn't even exist on some drives, and the only proof so far offered is "I'm a professional". ECC and error recovery are not the same thing.

[Halcyon]

 

[MK14]

the only proof so far offered is "I'm a professional". ECC and error recovery are not the same thing.

https://atola.com/products/insight/bad-sector-recovery.html

What the above link seems to be saying (quick summary), is that if there are unrecoverable ECC errors, then the HDD will simply return an error message/number, NOT any actual data (which might be garbled/corrupt).

So, as an absolute last resort, if you really MUST try and recover the data, or at least some of it. E.g. A text file.
You can read the disk using another read method, which WILL return data, even if the ECC tests/corrections fail.

That at least means you can copy over the required data, onto a good HDD drive, even if there are bad ECC error(s) present. Then later, process what you have been able to extract, and possibly recover some or all of the data.

E.g. A fire/water/etc damaged HDD, where the data is extremely valuable, possibly found at a big business, or the scene of a crime, or crashed airplane, or other high value data situation.
Or just simply someones entire digital photograph collection, over the last 12 years, and they want to recover as many photographs as possible. Even though windows refuses to read the faulty HDD.

[Monkeh]

the only proof so far offered is "I'm a professional". ECC and error recovery are not the same thing.

https://atola.com/products/insight/bad-sector-recovery.html

What the above link seems to be saying (quick summary), is that if there are unrecoverable ECC errors, then the HDD will simply return an error message/number, NOT any actual data (which might be garbled/corrupt).

So, as an absolute last resort, if you really MUST try and recover the data, or at least some of it. E.g. A text file.
You can read the disk using another read method, which WILL return data, even if the ECC tests/corrections fail.

That at least means you can copy over the required data, onto a good HDD drive, even if there are bad ECC error(s) present. Then later, process what you have been able to extract, and possibly recover some or all of the data.

E.g. A fire/water/etc damaged HDD, where the data is extremely valuable, possibly found at a big business, or the scene of a crime, or crashed airplane, or other high value data situation.
Or just simply someones entire digital photograph collection, over the last 12 years, and they want to recover as many photographs as possible. Even though windows refuses to read the faulty HDD.

Hey, some actual discourse. Refreshing.

READ LONG isn't quite as clear cut as suggested, but it does indeed exist and may be useful (.. of course, it also may be as good as random noise). It can still fail (you can ask for no retries).

The READ LONG command performs similarly to the READ SECTOR(S) command except
that it returns the data and a number of vendor specific bytes appended to the data field of the desired
sector. During a READ LONG command, the device does not check to determine if there has been a data
error. Only single sector READ LONG operations are supported.

The transfer of the vendor specific bytes shall be 16 bit transfers with the vendor specific byte in bits 7
through 0. Bits 15 through 8 shall be ignored by the host. The host shall use PIO mode 0 when using this
command.

Error recovery performed by the device either with or without retries is vendor specific.

Of course, this is from a standard superseded 22 years ago, so it's anyone's guess what this really does on a modern drive if it responds to it. What's actually contained in the (variable length) vendor bytes is, of course, vendor specific.

Still, data was written with ECC and, obviously, this command is entirely useless in normal operation.

[MK14]

Of course, this is from a standard superseded 22 years ago, so it's anyone's guess what this really does on a modern drive if it responds to it. What's actually contained in the (variable length) vendor bytes is, of course, vendor specific.

Still, data was written with ECC and, obviously, this command is entirely useless in normal operation.

I remember in the old days using Spinrite (decades ago). Apparently it is still for sale, and there is an interesting (at least for me) 12 minute (approx), video about how it works in 2012. The video explains about 'bit rot', and how modern drives, with the ever smaller data bits (i.e. tiny physical dimensions, as stored on disks), because of the modern massive data capacities. Are susceptible, to errors, because of their tiny dimensions.
It also explains about how Spinrite using different strategies, to attempt to read (accurately) data, even if the ECC has gone too bad, to recover the data, via normal reads.

https://www.grc.com/sr/whatitdoes.htm   (Part of Gibson Research Corporation, which also does the ShieldsUP! open port tests, which some people, might of also heard of or used).

[David Hess]

When the RAID controller receives the bad data and an error, good ones regenerate the data and write it back to the drive, which then performs a scrub-on-write.  If this did not occur, then there would be no reason for the RAID controller to perform idle time scrubbing.

Nope. What you wrote would be (mostly) correct if we were talking about MFM/RLL/ESDI or early IDE (or SCSI-1 drives) from some 30 years ago. But we're not.

Modern (i.e. made in 2000 or later) drives don't represent their physical layout to the host. They report some artificial CHS layout which has nothing to do with the actual physical layout to the host for backward compatibility reasons (so that antique systems still CHS for addressing can boot from these drives). Even the sector size is often fake, as most modern hard drives use 4k sectors internally while reporting 512byte sectors on the interface.

But for most part the CHS layout isn't even used. LBA (Logical Block Adressing) has been a thing even before the year 2000 (it was first used with SCSI drives long before then), and it's been the standard way of addressing disks for many many years. With LBA, the host only sees a device which has a certain number of blocks. There's no CHS involved. LBA has been supported at least since Windows 98 and NT 4.0, and became the standard format with Windows 2000. And while LBA was an extension for IDE and ATA, LBA is the defined addressing standard for SATA, SAS and NVMe storage.

You RAID controller, whatever type/model that is, would have to be really old to not use LBA (and if that controller is so old then I guess the disks are, too). And even then it would only see the fake geometry reported by the drive, not the real one.

The only person here bringing up pre-IDE drives is you.  What part of "scrub-on-write" do you not understand?

If a modern RAID controller encounters a bad block (unrecoverable error), it will try to reconstruct the data from the redundancy disks and then may attempt to re-write the block to the affected disk. If this disk has sufficient spare sectors, it may revert the write to a spare block, after which the block will be fine and the integrity of the data is restored. If that was a one-off, the drive may well be fine for years. However, if that happens more often (as it's the case on a dying drive), the disk will eventually run out of spare sectors after which the attempt by the RAID controller to re-write the block will fail, and then the disk is failed and the array goes into contingency mode.

So I am wrong, but you agree while contradicting what you said earlier?

It would be dumb to discard an entire drive because of a single bad read when the data can be recovered and written back to force the drive to reallocation that sector; of course firmware based RAID controllers often are this dumb.

In a professional environment, if a drive shows bad sectors at the interface it's scrapped, period. Any decent RAID controller will immediately flag a drive as soon as unrecoverable errors start to appear. Because for a modern drive unrecoverable errors are usually a clear sign that the drive is defective, and the only thing that would be stupid is to not scrap the drive and risk the integrity of the host data. It's a simple as that. Because at the end of the day the host data (and the hourly rate for the admin who has to deal with it and the potential fall-out should the drive remain in service) is worth a lot more than that stupid hard drive.

Now I accept that for hobbyist use that may well be different, and if you can't afford to replace a drive it's certainly tempting to work around the problem of a defect sector. But that is only a viable option if your data (and your time) isn't worth much (because if it was you'd not try to cheap-skate backup). And it doesn't change the fact that the drive is telling you that you can no longer rely on it.

That is a policy decision and not inherent to RAID or sector reallocation.  Why even bring it up?

The internal defect management could operate on a correctable error, but I have never seen happen.

Well, yes, that's because it's supposed to be *transparent* to the host. Which is the whole point of a "defect-free interface".

It cannot be transparent because the visible reallocated sector count would increase.  Did you even read what I wrote?

I would have noticed if the defect list grew without errors being reported.  Many times I have done extended SMART surface scans and watched for this very thing and it never happened.  Most recently I have done it multiple times in the past couple of weeks on a pair of 1TB WD Greens, but doing an external surface scan which includes writing had some results.

Because you don't seem to fully understand what you are seeing. Any high level surface scan tool only scans the area that the drive is reporting as good (it has no access to the whole drive area - "defect-free Interface", remember?). Defects are hidden because defect management is completely *transparent*. The only way you can check what's actually going on in the drive is through SMART data.

When you're at the point where your tool can "see" defects then that means the drive has developed unrecoverable errors and is defective and should be discarded, but at the very least should not be used to store anything of importance.

Did I say "high level surface scan" or "SMART surface scan"?

And only the visible sectors need to be scanned anyway for the reason you identify.  The question was whether hard drives do scrub-on-read of bad sectors.  They do not.  And that has nothing to do with physical sectors which have been mapped out.

SSDs are a different matter as they should not have any patrol reads done on them, and less so any writes. SSDs also have internal defect management (which is part of it's Garbage Collection) and unlike with hard drives this normally works without having to be triggered (which for SSDs is done with the TRIM command).

The difference is that SSD must perform verification after every write as part of the write process, and they also scrub-on-read of sectors which are still correctable if they have too many soft errors.  Many perform idle time scrubbing because retention of modern Flash memory is abysmal, or at least their specifications say they do.  I sure know that many do not and will happily suffer bit rot while powered.

There's nothing in SSDs that matches "scrubbing". Which, on flash, would be completely counterproductive because every read of a flash cell reduces the amount of charge that is stored in that cell, so if a SSD did regular "read-scrubs" then the charge level would quickly become so low that the sector would have to be re-written to maintain its information, a process which causes wear on the flash cell.

Reads disturbance is observable and needs to be taken into account, but is minor compared to other factors.

Read disturbance and write disturbance also affect cells which are not being accessed, making idle time scrubbing in some form more important.

However power loss protection is also required for *any* write operation, and possibly any erase operation.  The reason for this is that interrupted writes can not only corrupt the data which is being written, but also data in other sectors, including the flash translation table, which can result in a non-recoverable situation.

Their are two reasons data can be so easily corrupted with an incomplete write.  With multi-level flash, the existing data when a sector is updated is at risk during the update and it is easy to see why.  It seem to me like this is avoidable by only storing the data from one sector with multiple levels but apparently Flash chips are not organized that way.

First of all, SSDs internally are physically organized in blocks, not sectors. Remember the LBA I mentioned above? LBA is *exactly* how SSDs are structured internally. On SSDs, Sectors are an artificial construct which has zero relation to which flash cell the information is physically located.

That distinction is irrelevant for this discussion, which is not about write amplification.

Garbage Collection and Wear Levelling also don't need PLP. If power is interrupted during the deletion of a block then the block will remain marked as for deletion and deletion will be repeated after power comes back up. For Wear Levelling, when data is moved from one block to another then the data from the old block is copied to a new block and then the old block is marked for deletion. If that process is interrupted then the old data is still there and the block shift is simply repeated.

*Any* interrupted write can destroy data which is not in the current block.

The other reason is more insidious; the state machine which controls the write operation can glitch during power loss,

The part in a SSD which controls write operations (and everything else) is the SSD controller and that is not a State Machine, it's a micro-controller running specific software (the drive's firmware) to perform it's duties.

Oh good, they implemented the state machine in a micro-controller so all is solved!

which is how drives which lack power loss protection got bricked so easily in those published tests several years ago.  Some drives lost no data, not even data in transit, some drives lost only data in transit, and most suffered either unrelated data loss or complete failure, but that was before the problem was understood well.

I know that early consumer SSDs killed themselves (particularly those made by OCZ) for a number of reasons, most of which were related to the lack of automatic GC (the drive needed GC triggered by the OS via TRIM, which the back then still common WindowsXP didn't support) and a wide range of firmware errors.

But the point remains that the only "data in transit" in a SSD is host data, and that SSDs do not work the way you think they do.

It was not only OCZ drives which failed, but it was *every* drive which did not have power loss protection.  Of particular note is that all of the drives with Sandvine controllers, which were advertised as not requiring power loss protection, failed.

[David Hess]

Also, if a hard drive which was made in 2000 or later exhibits a bad block to the OS then that means it has run out of spare sectors and should have been replaced long ago.

I recovered a RAID set not long ago which had exactly that problem.  The drive reported an error during a RAID rebuild, because I stupidly did not run a scrub of the array before swapping a drive, which stopped the RAID rebuild.  The drive actually had accumulated several bad sectors but it only takes one to stop the rebuild.  The solution was to scan the RAID at the level of the file system to determine which files were damaged, and then deliberately run a read-write of the entire drive outside of the array to force reallocation of the bad sectors, albeit with corrupted data.  Once that was done, the drive reported no bad sectors during the RAID rebuild and the damaged files could be restored.

So in short, you put a knowingly defective drive in a RAID array, and when the rebuild process falls over the drive's dead sectors you fix it by manually relocating corrupt host data to another sector so the rebuild process can be fooled into completing? Seriously??? 

No, I replaced a good drive in the RAID array with a different good drive and bad sectors on one of the remaining drives stopped the rebuild.  Had I scrubbed the RAID array before doing the drive swap, then there would have been no problem.

So a drive can report a bad sector to the OS, or RAID controller in this case, but I have had both recently, while not being out of spare sectors because scrub-on-read deliberately never happened, and the drive was waiting for a write to the bad sector to perform scrub-on-write.  It would be much worse for the drive to return bad data and no error while performing a scrub-on-read operation.

You're not understanding the issues here. The drive shows bad sectors at the interface because it can't recover the data in these sectors because of the large defect area on the platter. And when the drive can't recover the data (because the ECC information is affected as well) then it has to report an unrecoverable error as without correct data it makes no sense to relocate the sector.

I agree; hard drives do not scrub-on-read.  I said that.

The fact that the rebuild failed on the drive should have already been a warning that the best place for this drive would be the electronics recycling dumpster (or the shredder if the data was sensitive).

The drive was good; see above.  The only failure was your reading comprehension.

[BradC]

I remember in the old days using Spinrite (decades ago). Apparently it is still for sale, and there is an interesting (at least for me) 12 minute (approx), video about how it works in 2012. The video explains about 'bit rot', and how modern drives, with the ever smaller data bits (i.e. tiny physical dimensions, as stored on disks), because of the modern massive data capacities. Are susceptible, to errors, because of their tiny dimensions.
It also explains about how Spinrite using different strategies, to attempt to read (accurately) data, even if the ECC has gone too bad, to recover the data, via normal reads.

Don't bother reading or watching any of the Spinrite propaganda. It does precisely one thing. It reads each sector until it finds one it can't read, then it engages "dynastat". This performs repeated read-long commands on the sector and builds a statistic about what it thinks each bit should be. So for example, 4096 bits in a "sector", and it reads each sector 100 times. Any bit that was a 1 for more than 50 reads is flagged as a 1 and so on. It then writes the sector back to the disk. If at any time it gets a good read, it just writes that straight back.

Now, it does a credible job at that, but it really is a "one trick pony" and only "recovers" data in-place.

I haven't used it for a few years but it was still working on 4TB drives and they responded to read-long.

The rest of the purported features such as performing periodic maintenance to avoid bit-rot bordes on homeopathy, but he still sells enough copies to keep the lights on, and he certainly has a pack of religious followers.

I keep it as a tool in the box, but it gets used a lot less than it did in the days of ST-506 drives and floppies.

[MK14]

Don't bother reading or watching any of the Spinrite propaganda. It does precisely one thing. It reads each sector until it finds one it can't read, then it engages "dynastat". This performs repeated read-long commands on the sector and builds a statistic about what it thinks each bit should be. So for example, 4096 bits in a "sector", and it reads each sector 100 times. Any bit that was a 1 for more than 50 reads is flagged as a 1 and so on. It then writes the sector back to the disk. If at any time it gets a good read, it just writes that straight back.

Now, it does a credible job at that, but it really is a "one trick pony" and only "recovers" data in-place.

I haven't used it for a few years but it was still working on 4TB drives and they responded to read-long.

The rest of the purported features such as performing periodic maintenance to avoid bit-rot bordes on homeopathy, but he still sells enough copies to keep the lights on, and he certainly has a pack of religious followers.

I keep it as a tool in the box, but it gets used a lot less than it did in the days of ST-506 drives and floppies.

Thanks, that makes sense.
Spinrite, has been available for a huge amount of time, so I'm not surprised by your comments. Hard disks, themselves, are gradually becoming ever so slightly more obsolete, as time goes on.
I think once SSDs, commonly/cheaply/reliably exceed, even the biggest hard disk drives, in terms of capacity (per $/£), HDDs, will disappear even sooner.

The two big things which concern me about the increasingly popular SSD drives, are their limited write endurance life, which potentially spoils their usage for some applications.
The other, is the (reportedly/specified), relatively short/limited (unpowered/'refreshed') data life. If the drive is left unpowered (especially), for long periods of time.
E.g. Some reports, seem to say the data life, can be as short as 1 year, on some modern SSD drives.

Unlike other parameters of an SSD (e.g. data capacity size, access speed and total data bandwidth), the data life is difficult for users to test themselves.

Some of the later HDDs, also seem to have these new technologies (which I haven't been keeping up with), which also seem to potentially limit the data write endurance and/or data lifetime. Because of the techniques, which allow modern HDDs, to have huge capacities.

In general user reviews/reports, seem to be rather/very negative about these new types of HDD drives, and annoyed/angry with how the hard drive manufactures, have relatively silently introduced these technologies, into their drive ranges. Without making it at all clear to the end users, exactly what type of drive they are really getting.
E.g. Saying this hard drive is meant for data archiving ONLY, or ONLY for limited data writes, per year.

I.e. making it more of a nightmare, to choose a decent/reliable massive capacity HDD, for backup or other important uses.

E.g. I somewhat recently bought some huge (to me) capacity external HDD units, WD (if I remember correctly). But apparently/reportedly, you have to actually open them up physically (which I'm reluctant to do). In order to find out if you have got the 'proper' HDD types, or these newer, less well respected HDD types.
From memory, WD Reds = Good, WD Whites = Not so good.

You use to be able to just simply splash out and buy quality Hitachi drives, but these days, most of the separate HDD manufactures, have (due to takeovers and mergers), become only a handful of very big HDD manufactures.

[helius]

every read of a flash cell reduces the amount of charge that is stored in that cell

Is that actually true? The structure of a bit in flash (or EPROM) is a floating gate insulated from the underlying transistor. Because it's floating, charges cannot leak away except by quantum tunneling. I thought it was pretty well established that the speed of tunneling was independent of read activity or indeed whether there is even power applied.

[Halcyon]

I remember in the old days using Spinrite (decades ago). Apparently it is still for sale, and there is an interesting (at least for me) 12 minute (approx), video about how it works in 2012. The video explains about 'bit rot', and how modern drives, with the ever smaller data bits (i.e. tiny physical dimensions, as stored on disks), because of the modern massive data capacities. Are susceptible, to errors, because of their tiny dimensions.
It also explains about how Spinrite using different strategies, to attempt to read (accurately) data, even if the ECC has gone too bad, to recover the data, via normal reads.

Don't bother reading or watching any of the Spinrite propaganda. It does precisely one thing. It reads each sector until it finds one it can't read, then it engages "dynastat". This performs repeated read-long commands on the sector and builds a statistic about what it thinks each bit should be. So for example, 4096 bits in a "sector", and it reads each sector 100 times. Any bit that was a 1 for more than 50 reads is flagged as a 1 and so on. It then writes the sector back to the disk. If at any time it gets a good read, it just writes that straight back.

Now, it does a credible job at that, but it really is a "one trick pony" and only "recovers" data in-place.

I haven't used it for a few years but it was still working on 4TB drives and they responded to read-long.

The rest of the purported features such as performing periodic maintenance to avoid bit-rot bordes on homeopathy, but he still sells enough copies to keep the lights on, and he certainly has a pack of religious followers.

I keep it as a tool in the box, but it gets used a lot less than it did in the days of ST-506 drives and floppies.

Ahhh Spinrite! Thanks for reminding me. I think they put more effort into the ASCII animations and various technical-looking screens than actually doing anything meaningful.

[Wuerstchenhund]

every read of a flash cell reduces the amount of charge that is stored in that cell

Is that actually true? The structure of a bit in flash (or EPROM) is a floating gate insulated from the underlying transistor. Because it's floating, charges cannot leak away except by quantum tunneling. I thought it was pretty well established that the speed of tunneling was independent of read activity or indeed whether there is even power applied.

It's been a while when I read about this (and I can't find the paper that discussed this right now) but from what I remember the issue was that during the read process of a NAND cell (where the flash controller applies a range of voltages to detect the cell's threshold voltage) charges trapped in the oxide breaks out, thereby reducing the cell's threshold voltage. It has no practical impact on older SSD types using SLC or (two-level) MLC flash but it's an issue with modern small 3+ level NAND cells (which, considering that a modern TLC cell stores less than 100 electrons, isn't surprising).

On top of that, with NAND flash there's also an effect called "Read Disturbance" where a read operation of on one cell shifts the threshold voltages of neighboring cells ("victim cells") in the same block so that they can reach a different logical state than the one that was written.

Reading doesn't wear out NAND flash but it far from being impact-less.

[Wuerstchenhund]

So in short, you put a knowingly defective drive in a RAID array, and when the rebuild process falls over the drive's dead sectors you fix it by manually relocating corrupt host data to another sector so the rebuild process can be fooled into completing? Seriously??? 

No, I replaced a good drive in the RAID array with a different good drive and bad sectors on one of the remaining drives stopped the rebuild.

I see. Not that this is much better

Had I scrubbed the RAID array before doing the drive swap, then there would have been no problem.

The bad drive would still have been bad and should have been discarded.

The fact that the rebuild failed on the drive should have already been a warning that the best place for this drive would be the electronics recycling dumpster (or the shredder if the data was sensitive).

The drive was good; see above.  The only failure was your reading comprehension.

The drive you may have put in originally was good (that's the bit I got wrong), but the existing drive which exhibited bad sectors certainly wasn't.

Again, the rebuild failure should have been a warning sign that this drive is defective and should have been discarded.

As I said for hobbyist use there aren't any hard rules and everything goes, but if this was for data that really mattered then keeping a knowingly defective drive in the array would be grossly negligient.

[Wuerstchenhund]

Nope. What you wrote would be (mostly) correct if we were talking about MFM/RLL/ESDI or early IDE (or SCSI-1 drives) from some 30 years ago. But we're not.

Modern (i.e. made in 2000 or later) drives don't represent their physical layout to the host. They report some artificial CHS layout which has nothing to do with the actual physical layout to the host for backward compatibility reasons (so that antique systems still CHS for addressing can boot from these drives). Even the sector size is often fake, as most modern hard drives use 4k sectors internally while reporting 512byte sectors on the interface.

But for most part the CHS layout isn't even used. LBA (Logical Block Adressing) has been a thing even before the year 2000 (it was first used with SCSI drives long before then), and it's been the standard way of addressing disks for many many years. With LBA, the host only sees a device which has a certain number of blocks. There's no CHS involved. LBA has been supported at least since Windows 98 and NT 4.0, and became the standard format with Windows 2000. And while LBA was an extension for IDE and ATA, LBA is the defined addressing standard for SATA, SAS and NVMe storage.

You RAID controller, whatever type/model that is, would have to be really old to not use LBA (and if that controller is so old then I guess the disks are, too). And even then it would only see the fake geometry reported by the drive, not the real one.

The only person here bringing up pre-IDE drives is you.

Yes, because that's the *only* generation where what you state could even be relevant for.

Your general understanding of this topic also appears to be rooted around that timeframe.

What part of "scrub-on-write" do you not understand?

What part of "it doesn't work the way you think you do" do you not understand?

If a modern RAID controller encounters a bad block (unrecoverable error), it will try to reconstruct the data from the redundancy disks and then may attempt to re-write the block to the affected disk. If this disk has sufficient spare sectors, it may revert the write to a spare block, after which the block will be fine and the integrity of the data is restored. If that was a one-off, the drive may well be fine for years. However, if that happens more often (as it's the case on a dying drive), the disk will eventually run out of spare sectors after which the attempt by the RAID controller to re-write the block will fail, and then the disk is failed and the array goes into contingency mode.

So I am wrong, but you agree while contradicting what you said earlier?

Nope. You just don't seem to understand that the checks and activities a RAID controller performs is at a different level than what the internal defect management of a hard drive does.

Handling of media defects is up to the drive's internal defect management, which (again) is *transparent* to the host. When it's not (i.e. the drive encounters an unrecoverable error) then that means the drive is telling you to that it is on its way out. When that happens in a disk that is part of a RAID array in redundant configuration (i.e. anything except RAID0 or JBOD) then the controller, in an attempt to maintain integrity of the RAID array, will reconstruct the data from it's redunancy elements and send the block to the drive for writing (where, if the drive's defect management will write it to a space physical sector if there are some left). If that fails (because there are no spare sectors left) then the drive is flagged for replacement.

Now what you seem to believe is that the RAID controller's handling of defective sectors is synonymous with the hard drive's internal defect management. But that is not the case. The RAID controller has zero control over how the drive handles defects, all it can do is to ask the drive to write a certain block, and whatever the RAID controller does it does so only to maintain the integrity of the RAID array (it's not to handle drive defects, which, again, is what the drive is responsible for).

It would be dumb to discard an entire drive because of a single bad read when the data can be recovered and written back to force the drive to reallocation that sector; of course firmware based RAID controllers often are this dumb.

In a professional environment, if a drive shows bad sectors at the interface it's scrapped, period. Any decent RAID controller will immediately flag a drive as soon as unrecoverable errors start to appear. Because for a modern drive unrecoverable errors are usually a clear sign that the drive is defective, and the only thing that would be stupid is to not scrap the drive and risk the integrity of the host data. It's a simple as that. Because at the end of the day the host data (and the hourly rate for the admin who has to deal with it and the potential fall-out should the drive remain in service) is worth a lot more than that stupid hard drive.

Now I accept that for hobbyist use that may well be different, and if you can't afford to replace a drive it's certainly tempting to work around the problem of a defect sector. But that is only a viable option if your data (and your time) isn't worth much (because if it was you'd not try to cheap-skate backup). And it doesn't change the fact that the drive is telling you that you can no longer rely on it.

That is a policy decision and not inherent to RAID or sector reallocation.  Why even bring it up?

Because its *not* a "policy decision", these are established procedures based on how storage systems actually work.

A drive that shows defects on the interface is no longer reliable and should be discarded. It's not rocket science to understand why this is established practice.

The internal defect management could operate on a correctable error, but I have never seen happen.

Well, yes, that's because it's supposed to be *transparent* to the host. Which is the whole point of a "defect-free interface".

It cannot be transparent because the visible reallocated sector count would increase.

"Transparent to the interface" means that drive defects are hidden to the host (i.e. the drive appears with zero defects).

The *only* exception are SMART data, because SMART data is supposed to show a drive's health status.

"Re-allocated Sectors" indicate how many times the drive encountered a defective sector, was able to reconstruct the data and moved the data to a spare sector.

"Recoverable Errors" are soft errors where the drive encountered an error in a sector but was able to restore the data and successfully rewrite the sector (no re-allocation required).

Both are "correctable errors"

"Unrecoverable errors" are when the drive encounters bad data and is incapable of restoring the original data.

I would have noticed if the defect list grew without errors being reported.  Many times I have done extended SMART surface scans and watched for this very thing and it never happened.  Most recently I have done it multiple times in the past couple of weeks on a pair of 1TB WD Greens, but doing an external surface scan which includes writing had some results.

Because you don't seem to fully understand what you are seeing. Any high level surface scan tool only scans the area that the drive is reporting as good (it has no access to the whole drive area - "defect-free Interface", remember?). Defects are hidden because defect management is completely *transparent*. The only way you can check what's actually going on in the drive is through SMART data.

When you're at the point where your tool can "see" defects then that means the drive has developed unrecoverable errors and is defective and should be discarded, but at the very least should not be used to store anything of importance.

Did I say "high level surface scan" or "SMART surface scan"?

It doesn't matter as SMART Short Test and SMART Extended Tests are both tests that only read the user area (not the extended area that contains the reserve blocks or any other surface area, e.g. the firmware areas some drives have). The only difference is where the results go (to the host PC on a regular surface scan, to the disk controller on a SMART Test).

On top of that, both SMART Short Test and Long Test are generally read-only. And they are so because on a healthy drive there shouldn't be any unrecoverable errors. Also, sector re-allocation normally only works for the user area, if there's a defect in the firmware area which can't be corrected by ECC then the drive is toast.

With modern drives, there is no way to perform a low-level format or even low-leve surface scan. Simple as that.

SSDs are a different matter as they should not have any patrol reads done on them, and less so any writes. SSDs also have internal defect management (which is part of it's Garbage Collection) and unlike with hard drives this normally works without having to be triggered (which for SSDs is done with the TRIM command).

The difference is that SSD must perform verification after every write as part of the write process, and they also scrub-on-read of sectors which are still correctable if they have too many soft errors.  Many perform idle time scrubbing because retention of modern Flash memory is abysmal, or at least their specifications say they do.  I sure know that many do not and will happily suffer bit rot while powered.

There's nothing in SSDs that matches "scrubbing". Which, on flash, would be completely counterproductive because every read of a flash cell reduces the amount of charge that is stored in that cell, so if a SSD did regular "read-scrubs" then the charge level would quickly become so low that the sector would have to be re-written to maintain its information, a process which causes wear on the flash cell.

Reads disturbance is observable and needs to be taken into account, but is minor compared to other factors.

Correct, although the effects become more pronounced the smaller the flash structures and the more levels a cell has to store.

Read disturbance and write disturbance also affect cells which are not being accessed, making idle time scrubbing in some form more important.

No, it doesn't (Read Disturbance is captured by ECC). Also, read cycles are monitored as is cell aging and Wear Leveling moves data to other blocks when ECC detects an error or a certain number of P/E cycles or read cycles have been performed, which alleviates the issue completely.

No patrol reads required.

However power loss protection is also required for *any* write operation, and possibly any erase operation.  The reason for this is that interrupted writes can not only corrupt the data which is being written, but also data in other sectors, including the flash translation table, which can result in a non-recoverable situation.

Their are two reasons data can be so easily corrupted with an incomplete write.  With multi-level flash, the existing data when a sector is updated is at risk during the update and it is easy to see why.  It seem to me like this is avoidable by only storing the data from one sector with multiple levels but apparently Flash chips are not organized that way.

First of all, SSDs internally are physically organized in blocks, not sectors. Remember the LBA I mentioned above? LBA is *exactly* how SSDs are structured internally. On SSDs, Sectors are an artificial construct which has zero relation to which flash cell the information is physically located.

That distinction is irrelevant for this discussion, which is not about write amplification.

It's relevant because there are no sectors in an SSD (there are blocks) while you talk about sectors which have no relevance for the location of the physical data.

Garbage Collection and Wear Levelling also don't need PLP. If power is interrupted during the deletion of a block then the block will remain marked as for deletion and deletion will be repeated after power comes back up. For Wear Levelling, when data is moved from one block to another then the data from the old block is copied to a new block and then the old block is marked for deletion. If that process is interrupted then the old data is still there and the block shift is simply repeated.

*Any* interrupted write can destroy data which is not in the current block.

No, it can't. Because the original block is erased only after the write process in the new block has completed. If that process is interrupted then the original data is still there.

The other reason is more insidious; the state machine which controls the write operation can glitch during power loss,

The part in a SSD which controls write operations (and everything else) is the SSD controller and that is not a State Machine, it's a micro-controller running specific software (the drive's firmware) to perform it's duties.

Oh good, they implemented the state machine in a micro-controller so all is solved!

I'm not sure you really understand the concept of "State Machine".

The flash controller in a typical SSD is not much different than the main processor of the host it is connected to. Many SSD controllers are ARM based.

It was not only OCZ drives which failed, but it was *every* drive which did not have power loss protection.  Of particular note is that all of the drives with Sandvine controllers, which were advertised as not requiring power loss protection, failed.

I really would like to see some evidence for that claim. Because it sounds bogus to me. Of course SSDs without PLP can lose data but they certainly shouldn't fail.

Besides, I don't know any "Sandvine" SSD controllers (I know SandForce controllers, part of LSI) but maybe I just missed it.

[Nominal Animal]

CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

I am not aware of any SATA or SCSI drives (talking about spinning disks) manufactured in this millenium that are particularly sensitive to unexpected power loss.  (If power loss is associated with physical acceleration, like dropping, then physical damage may occur; but it is difficult to say how much if any that damage probability is due to power loss.)  Data loss is to be expected, depending on the file system used, but I am talking about drive functionality here.

I suspect that a big reason for this is the availability and price of supercaps, or ordinary high-capacitance capacitors, that can retain enough power to do a controlled shutdown even when supply cuts off unexpectedly.

If so, a particular batch of drives could be particularly vulnerable to sudden power loss, not because of firmware issues, but because of substandard capacitors, or cost-cutting at the OEM.  This kind of weakness often skews experiences at data centers and clusters, where you get dozens to hundreds of the same component from the same (or a few different) manufacturing batches.  (I do not have enough experience on SSDs to have an opinion; the clusters I admin'd used spinning rust.)

This means that even if you think you know something cannot happen, does not mean it is true; the underlying reason can be different, but the overall failure pattern the same.

[helius]

Even hard drives from the 1960s were designed to survive power cuts without damage. A common design was a head-retract solenoid that could take power from the collapsing field of the spindle motor in case power was disrupted, pulling the heads away from the surface before loss of lift.

Voice-coil drives can unload the heads in tens of milliseconds, and the energy required is easily stored in a MLCC.

[Nominal Animal]

Even hard drives from the 1960s were designed to survive power cuts without damage.

My first hard drive in 1989 or so, a 32 MByte MFM drive (which came formatted at 20 MBytes, on a Hyundai '286 clone with ATi EGA Wonder graphics card), had something like that implemented, but in real life, it would not survive too many cycles of that: one had to use the hdpark utility (if I recall the DOS command right) before shutdown.  I think it was just a cheap implementation in that drive... (Within three years, the motherboard gave up the ghost; then the ADI 14" monitor, and finally the HDD.  And boy was that machine loud, like a turbine engine whenever it was powered on.)

I mean, while something is supposed to be designed in, does not mean the bean counters etc. haven't insisted on so much cheapness it does not actually work that well in practice.  It is common in all fields of tools, really: given enough cost-cutting and striving for short-term gains over long-term ones, everything turns to shit.

[helius]

Oh, the small stepper-motor drives of the 1980s were definitely not as robust. In particular, they mostly didn't have the ability to unload the heads on power failure: the "Landing Zone" BIOS parameter and "PARK.COM" utilities worked by seeking to a dedicated non-data cylinder and then tying the poles of the stepper together so they resisted movement from that spot. MFM drives had no onboard controller, so they basically had no "brains" when it came to protecting themselves. If you forgot to run "PARK.COM" before cutting power, the heads would land on the data area* and might scrape off magnetic material. This was compounded by computers without soft power/shutdown capabilities, where cutting power was always needed to power off.

Related to this evolution and different levels of technology, Seagate has been the foremost company in most people's mental landscape for magnetic media, but the drives they developed in-house were the slow stepper-type described above. It was only when they acquired Imprimis from Control Data that they gained high speed, voice-coil operated, advanced drives with integrated controllers. The famous Seagate Hawk and Barracuda were Imprimis designs.

*This is the defining feature of a "Winchester" drive. The IBM Winchester project developed the first drives that didn't need emergency head-retract solenoids, because the heads had the capability to land on the surface without crashing and destroying themselves.

[BradC]

Ahhh Spinrite! Thanks for reminding me. I think they put more effort into the ASCII animations and various technical-looking screens than actually doing anything meaningful.

It *is* very pretty. Back in the old days from memory it would also re-interleave your HDD to optimise the track layout for your machine. I always used Norton for that.

The "dynastat" recovery harks way back, and it is pretty clever when you think about it. Take many, many reads of the same sector with the ECC "disabled" and build a statistical version of what it thinks the sector is. That worked particularly well for floppies.

One of the other tricks it *had* up its sleeve was intermittent seeks between reads which would make interesting use of the backlash in the old stepper mechanisms (both FDD and HDD) to centerthe head over different parts of the same track. Of course that means nothing these days, but it was another neat trick.

When you re-read that, it is very much "take a pretty informed, but ultimately speculative guess at the real sector contents and then write it back as authoritative". All the other "maintenance" stuff placebo at best.

[Wuerstchenhund]

CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense. Removable storage like USB sticks or CF cards have no "power down" function (as for example SSDs have), in fact CF cards are widely used in devices which don't perform some kind of a shutdown but just remove power (and even those that do are susceptible to sporadic power loss).

It doesn't kill the CF card, no should it.

For removable devices, the only reason they should be "removed" via the appropriate OS functions is tell the OS to write out any data that is not written out and then stop writing to the device to avoid corrupting the file system on that drive. It has nothing to do with sudden power loss (which is and never was a problem for removable flash storage devices).

[Nominal Animal]

CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense.

Sudden power loss during a write operation – say, updating directory access timestamps or similar; anything that requires an update of the internal state.  They just don't have the circuitry to deal with it.  Technically, it would be trivial: just add some capacitors, that provide enough current to finish any pending write operation.

Removable storage like USB sticks or CF cards have no "power down" function (as for example SSDs have), in fact CF cards are widely used in devices which don't perform some kind of a shutdown but just remove power (and even those that do are susceptible to sporadic power loss).

See what happens when you remove the device battery when it is writing to the CF card.  At least a few years ago, the chances were the card was bricked.

It has nothing to do with sudden power loss (which is and never was a problem for removable flash storage devices).

Hogwash.  CF cards maintain their own wear leveling tables (and other internal state), and for whatever reason, tend to get bricked whenever power loss occurs during the update of that state.  It has absolutely nothing to do with the host accessible data (filesystem et cetera), and everything to do with the device internal state.

Early USB memory sticks suffered from exactly the same problems.  I had several 64Mbyte and 128Mbyte sticks that died this way.  They soon fixed the issue; I do not know exactly how, but I believe by using capacitors to hold enough charge so that the internal state could always be updated successfully.

(One way to brick USB memory sticks is to short the device-side VUSB and GND via a small value bleed resistor immediately after the device side is disconnected from the host side, and do so when writing to the stick.  The bleed resistor depletes the capacitors, so the management chip is unable to correctly update its internal state.
Again, this has nothing to do with flash memory tech per se, and everything to do with the devices internal non-exposed state being only partially updated.)

Anecdotally, many CF cards still lack protection against sudden power loss.  One reason could be that since these use the ATA (PATA aka parallel-ATA) interface, the charge needed for a full internal state update is so large that many manufacturers skimp on the capacitors.  I suspect, but cannot verify, that CF Revision 4.1 (Power Enhanced CF) is related.

[Siwastaja]

Oh, flash memory corrupting at power loss does make sense.

Flash memory works by erasing into a certain state (usually logical '1' but in some implementations, logical '0' has been seen) a larger block (variably called "block", "sector", or something else) at a time; this operation is quite slow; then only the opposite state (usually logic '0') can be written bit-by-bit, as a relatively fast operation.

As a result, obviously, power loss is always a consideration when modifying anything, including any internal data. Any modification of the erasable unit which may contain changes from '0' to '1' requires read-the-entire-unit - erase - modify - write-back-the-entire-unit operation.

Such operations should be obviously avoided, but it's equally obvious that when there is a chance for a problem, in the field of quickly developing, cheap products, buggy or poor implementations are inevitable, but tend to get better over time.

But the big question is: What is the name of the function in HDDs that automatically prevents downloaded files from being saved in bad sectors?

[classicsamus87]

What is the relationship of S.M.A.R.T with the automatic technology present in firmware for reallocation and isolation HDD defect sectors (badblocks)?

[Monkeh]

What is the relationship of S.M.A.R.T with the automatic technology present in firmware for reallocation and isolation HDD defect sectors (badblocks)?

Well, SMART took it out for dinner once, but it's purely platonic.

[Nominal Animal]

But the big question is: What is the name of the function in HDDs that automatically prevents downloaded files from being saved in bad sectors?

I do not believe there is a name for it, only the description – reallocation of bad/failed blocks/sectors.

I do believe all the most relevant papers and patents can be found using that description (those terms and their variants in English).  None of the patents I linked to earlier seem to use any common name for the operation, but they do describe the approaches to quite some detail.  (Including heuristic reallocation before actual failure based on soft errors/correctable ECC error rate per block, which is the only approach I know of that preserves existing data; unfortunately, since Samsung sold its hard drive unit to Seagate, it is impossible to tell if it is used in any commercially available spinny-disk drives.)

(Your suggestion for the name, "sector reallocation", works for me though.)

[Siwastaja]

What is the relationship of S.M.A.R.T with the automatic technology present in firmware for reallocation and isolation HDD defect sectors (badblocks)?

Joking aside, serious answer to this is, SMART maintains and reports statistics about the reallocation (among many other things) to the user.

[BradC]

Joking aside, serious answer to this is, SMART maintains and reports statistics about the reallocation (among many other things) to the user.

Well, SMART is designed to maintain and report statistics about the reallocation (among many other things) to the user. What it does in reality is entirely what the firmware writer tells it to do, and it often "misrepresents the truth". Papers over imperfections, downright lies about other statistics and/or reports them in a "vendor specific" and mostly undocumented form.

The only thing SMART is reliably good for is making statistical comparisons between drives in a large population of identical or mostly similar drives.

[MK14]

What is the relationship of S.M.A.R.T with the automatic technology present in firmware for reallocation and isolation HDD defect sectors (badblocks)?

https://www.mjm.co.uk/articles/bad-sector-remapping.html

Quick Summary:
Once the bad-sectors, pool of spare resources has been so used up (via the HDD's firmware), the drive has been determined to be in need of replacement. The "backup and replacement is recommended" FLAG section of S.M.A.R.T. is set. To warn the user, that the disk is now considered "bad" and needs replacing.

[classicsamus87]

HDDs of the year 2000 have this automatic function in the firmware to reallocate and isolate defective sectors (bad blocks) of the HDD? I searched and didn't find much information because this function has no name


I downloaded important files on the HDD and I thought they were saved in the bad sectors

[MK14]

this function has no name

If you search for "P-LIST G-LIST T-LIST", you should find some details.
They are the Production/Primary/Permanent, Growing and Track, (P/G/T) DEFECT LISTS.

E.g. Amongst the many links, you can find:
https://www.pcmag.com/encyclopedia/term/hard-disk-defect-management

[classicsamus87]

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

[Monkeh]

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

No. The function is called Jim.

[BravoV]

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

No. The function is called Jim.

Having fun, don't you ? 

[BradC]

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

No. The function is called Jim.

I think I got into an argument with him in a bar once. He was being a SMART arse and wouldn't give me back my data. Said it had been "reallocated". Irritating sod really got my backup so much I wanted to take a byte out of him and serve his head on a platter.

[classicsamus87]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

[BravoV]

 
 
 
 

[Monkeh]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Please stop trying to use a forum to write a paper for you.

[BradC]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

The SMART answer is no. Of course the real answer is maybe, or perhaps.

[DrG]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Please stop trying to use a forum to write a paper for you.

Correction, forums.

I read this thread and analogous ones elsewhere and I could not figure it out at all. At first, I thought it was a bot, but then, with the edits and all, I got off that possibility and just figured it was for the kicks of seeing responses. Your explanation is plausible.

I have to add that it says something about me that I continue to read a long thread to try to understand the OP's motivation when my interest in the original subject matter has long since waned.

[Wuerstchenhund]

CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense.

Sudden power loss during a write operation – say, updating directory access timestamps or similar; anything that requires an update of the internal state.  They just don't have the circuitry to deal with it.  Technically, it would be trivial: just add some capacitors, that provide enough current to finish any pending write operation.

It still doesn't make any sense. You claimed a power loss would "brick" the card (i.e. render it unusable). Also, Timestamps are something a file system deals with, not the card (which is pretty oblivious to the data that is written to it).

Removable storage like USB sticks or CF cards have no "power down" function (as for example SSDs have), in fact CF cards are widely used in devices which don't perform some kind of a shutdown but just remove power (and even those that do are susceptible to sporadic power loss).

See what happens when you remove the device battery when it is writing to the CF card.  At least a few years ago, the chances were the card was bricked.

No, it wasn't. A power loss during write operation may well corrupt your data but it doesn't "brick" the card.

We have thousands of CF cards in devices around the world, and have been so for over two decades, and haven't seen a single card "bricked" because of powerloss (when they die they usually do so because of write exhaustions, and a few cards failed sporadically).

There are also literally millions of CF cards in routers and other networking gear, all susceptible to sudden power loss. And still, your claim of cards dying because of power loss remains unheard of.

It has nothing to do with sudden power loss (which is and never was a problem for removable flash storage devices).

Hogwash.  CF cards maintain their own wear leveling tables (and other internal state), and for whatever reason, tend to get bricked whenever power loss occurs during the update of that state.  It has absolutely nothing to do with the host accessible data (filesystem et cetera), and everything to do with the device internal state.

That's nonsense.

First of all, not all CF cards have wear levelling (before CF Rev 4.1 only a few industrial CF cards had any Wear Leveling).

Second, Wear Leveling doesn't work the way you think it does. If a block is marked for shifting, it's content is copied to the new block first, and only if that copy process is completed then the original block is erased. If this process is interrupted during the copying phase, the old block is still marked for shifting and the copy process will simply be repeated after the power is restored.

Early USB memory sticks suffered from exactly the same problems.  I had several 64Mbyte and 128Mbyte sticks that died this way.  They soon fixed the issue; I do not know exactly how, but I believe by using capacitors to hold enough charge so that the internal state could always be updated successfully.

Emphasis mine, because that is your problem. Your whole arguments are based on imagination and guesswork.

Memory sticks die, a lot. They do so because USB sticks are the most price-conscious market for flash memory. Many sticks use reclaimed (i.e. used) flash memory which may or may not be close to the end of its life. Most of them have no Wear Leveling either. They are the equivalent to the floppy disk and should be treated as such, i.e. as consumables.

I still have a box full of old Samsung, SanDisk and HP 16-64MB USB sticks from the old days, and they all were taken out of service due to their capacity, not because they died.

(One way to brick USB memory sticks is to short the device-side VUSB and GND via a small value bleed resistor immediately after the device side is disconnected from the host side, and do so when writing to the stick.  The bleed resistor depletes the capacitors, so the management chip is unable to correctly update its internal state.

That's nonsense (besides, all shorting VUSB and GND during writing does is to interrupt the write process and trigger the USB port's over-current protection).

If you do that to a disconnected USB stick then nothing happens.

As I said before, things don't work the way you imagine them.

Again, this has nothing to do with flash memory tech per se, and everything to do with the devices internal non-exposed state being only partially updated.)

No, it's mostly a thing of your imagination.

Anecdotally, many CF cards still lack protection against sudden power loss.  One reason could be that since these use the ATA (PATA aka parallel-ATA) interface, the charge needed for a full internal state update is so large that many manufacturers skimp on the capacitors.  I suspect, but cannot verify, that CF Revision 4.1 (Power Enhanced CF) is related.

"Power Enhanced CF" is a CF card which supports power management modes (i.e. idle modes) to reduce power draw during phases of inactivity, which is important for battery operated devices like cameras to extend battery life.

It has nothing to do with power loss protection.

Maybe you should not inform yourself through anecdotes.

[magic]

I like you believe that it is possible to create a fault tolerant flash translation layer, although I have never done one myself.

I know for sure it is possible to create one which shits itself upon power loss because that has been done by others

edit
I also have a pendrive which reliably passes testing with sequential files to 100% capacity but turns random blocks into 0xff when it's loaded with lots of random I/O (root filesystem of Linux). Even with syncing and unmounting. I have another flash drive which works fine under such conditions.
Not every problem in those things can be blamed on raw flash. The controllers have their share of suckage too.

[Wuerstchenhund]

I like you believe that it is possible to create a fault tolerant flash translation layer, although I have never done one myself.

I know for sure it is possible to create one which shits itself upon power loss because that has been done by others

Well, dynamic FTLs are part of Wear Leveling, and together with ECC it protects the drive from faults caused by soft errors or defective flash cells.

But I agree that there are some shitty implementations (*cough* Kingspec *cough).

edit
I also have a pendrive which reliably passes testing with sequential files to 100% capacity but turns random blocks into 0xff when it's loaded with lots of random I/O (root filesystem of Linux). Even with syncing and unmounting. I have another flash drive which works fine under such conditions.
Not every problem in those things can be blamed on raw flash. The controllers have their share of suckage too.

Tell me about it. I have another box full with Kingston Traveller USB3 sticks which in principle are fine, aside from the small problem that the flash controller quickly overheats during R/W operations which take the drive offline sporadically.

Thanks to Kingston's long warranty I was able to replace a few of them for new ones which suffer from the exact same problem 

[Nominal Animal]

CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense.

Sudden power loss during a write operation – say, updating directory access timestamps or similar; anything that requires an update of the internal state.  They just don't have the circuitry to deal with it.  Technically, it would be trivial: just add some capacitors, that provide enough current to finish any pending write operation.

It still doesn't make any sense. You claimed a power loss would "brick" the card (i.e. render it unusable). Also, Timestamps are something a file system deals with, not the card (which is pretty oblivious to the data that is written to it).

Wear leveling means that when you update a data sector, a completely different data sector gets updated; it is not that the flash memory containing the old data is updated.  (Wear leveling that lets the data be updated in place is useless.)

If the flash management chip loses power when updating the wear leveling data, it can brick itself.  See?  The internal state includes the wear leveling data, which is not exposed to the OS in any way.  The flash management chip assumes the wear leveling data is okay; but if it loses power in the middle of updating the wear leveling data, that data will be inconsistent, and brick the stick.

We have thousands of CF cards in devices around the world, and have been so for over two decades, and haven't seen a single card "bricked" because of powerloss (when they die they usually do so because of write exhaustions, and a few cards failed sporadically).

Well, I have, although it was a decade ago.  Are the cards you use all from a specific manufacturer?  I wanna know, so I can use those too.

There are also literally millions of CF cards in routers and other networking gear, all susceptible to sudden power loss.

Routers?  I've never seen one in a router; care to name one?
The ones I've seen are in cameras, which have OSes that shut down gracefully when the battery is low.

Also note that simply adding bulk capacitance to the supply going to the CF card, can completely avoid the death-on-power-loss situation.
This is because then the CF card always has enough power available to complete the last write operation the host sent before losing power, including the internal (wear leveling and whatever data).

SBCs really do not have a lot of bulk capacitance on their bus voltages.  For example, early 3G USB modems used with SBCs had "spiky" current requirements, the solution being adding a couple of milliFarads of low-ESR bulk capacitance between USB +5V and GND for these devices.  On an SBC, the CF card supply would basically drop as soon as the main processor lost power, thus making it likelier for the CF card to be bricked due to powerloss during a write (as internal wear-leveling etc. data was only partially updated).

Emphasis mine, because that is your problem. Your whole arguments are based on imagination and guesswork.

And yours is based on "me Wuerstchenhund, you wrong".

If you do that to a disconnected USB stick then nothing happens.

You think/believe; you obviously haven't tried it in practice.

Perhaps you should take your own advice, and stop claiming truths based on your own beliefs?

Because of you, my ignore list is no longer empty.

[magic]

Wear leveling means that when you update a data sector, a completely different data sector gets updated; it is not that the flash memory containing the old data is updated.  (Wear leveling that lets the data be updated in place is useless.)

If the flash management chip loses power when updating the wear leveling data, it can brick itself.  See?  The internal state includes the wear leveling data, which is not exposed to the OS in any way.  The flash management chip assumes the wear leveling data is okay; but if it loses power in the middle of updating the wear leveling data, that data will be inconsistent, and brick the stick.

That's why every semi-decent storage system based on flash should use some sort of B trees with copy on write updates and timestamps on every revision of the superblock so you know which is the latest one. IMO it should be possible to build a bulletproof flash drive and some SSDs are known to handle power loss well, it's just that many cheap ones either don't bother or are buggy.

Another problem is DRAM write back caches. Some SSDs supposedly solve this problem by having enough capacitance to write the full cache back in case of power loss. If you castrate their cap bank then bad things will happen.

[MK14]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Starting from the beginning again. What exactly are you trying to do/achieve and/or find out ?

Recover data from old disks ?
Or technical details on how old hard disk drives worked, for your own curiosity.
Or something else, such as a patent dispute/claim, court case or for other research reasons ?

I've apparently seen you with threads on a number of different forums. Such as:

https://community.wd.com/t/questions-bad-block-and-reallocated-sectors-count/255399/8

https://forum.allaboutcircuits.com/threads/question-hdd-and-badblock.173159/

https://www.electro-tech-online.com/threads/hdds-and-badblocks-corrupt-downloaded-files.159990/

Actually, I'm finding too many (assuming there are many more out there, I didn't find). It seems suspiciously like spamming or other bad stuff.

[Nominal Animal]

That's why every semi-decent storage system based on flash should use some sort of B trees with copy on write updates and timestamps on every revision of the superblock so you know which is the latest one. IMO it should be possible to build a bulletproof flash drive and some SSDs are known to handle power loss well, it's just that many cheap ones either don't bother or are buggy.

Exactly – although B-trees aren't the only option; there are a lot of known-good methods –; it's that because the memory controller is usually implemented in silicon, it (and the associated passives that cost next to nothing) is an attractive choice for cost-cutting.

Some use heuristics (to avoid storing the number of block writes, saving memory), and depending on the filesystem (block patterns), can die very quickly because of superblock updates hit the same physical flash blocks exhausting them.  But, because it is dependent on the filesystem, users blame the filesystem (for being "bad" on flash media), instead of the wear leveling.  This is particularly common among people who have high brand awareness, and consider the flash manufacturer more reliable than e.g. open-source software (open sores, they often call it), as non-Windows/non-Mac filesystems are rarely even considered by the manufacturers, but widely used on (especially embedded) devices based on Linux, Minix, or BSD variants.  They believe that if the device wear leveling behaves well on FAT/exFAT, it must be the software's fault if it dies rapidly on any other filesystem (or in raw block use, without any filesystem at all).

(Compare to e.g. Apple devices liquid sensitivity, which Louis Rossmann often talks about. Most other laptop/tablet boards are not nearly as sensitive to tiny amounts of moisture, because it is easy and cheap to achieve.  For Apple, it is an internal design choice, extending to color-changing moisture detection stickers on their boards...  Stuff like putting 24V backlight power next to a logic pin going directly to a low-voltage processor in a connector prone to moisture and dust ingress, near the display hinge.  There is no technical reason to do that, if the same connector has half a dozen ground pins just a mm or two away.)

We've seen effects of similar cost-cutting on the spinning rust side, too.  While WD drives have a good reputation, they have quite a few series that had serious issues with RAID, or performance in general; things that they disclosed later only because users were too upset (upset enough to harm their brand if they did nothing).

I fully know there is no technical or technological reason for the weaknesses I've claimed in this thread; it's just stupidity or cost-cutting.

Starting from the beginning again. What exactly are you trying to do/achieve and/or find out ?

Very good point.

I personally was kind-of assuming the underlying question was something like "if I download a file, is it possible a spinny-disk HDD manufactured after 2000 stores it in blocks that turn out to be bad?"

The answer is yes and no.  No, because the drives do not knowingly store anything to the blocks it knows to be bad.  Yes, because while the block may be considered good when the data is written to it, it can be detected as bad when trying to read the data later on.  SMART reports these as 'uncorrectable errors'.  Heck, the block might be good at that time, but the media surface degrade later on (physically or magnetically, for whatever reason from manufacturing issue to temperature gradients to radiation damage) becoming 'bad'.  This applies even if there happens to be a drive that knows nothing about bad blocks or sector reallocation.

There is a technology that detects blocks going bad by analysing the error correction frequency (amount of bit errors corrected with ECC data), and reallocates the block before any data loss happens.  (This is described in a patent I linked to earlier.)  However, we don't know which, if any, drives implement this; I think most manufacturers would consider this "too costly", wrt. to their profit margins.

Actually, I'm finding too many (assuming there are many more out there, I didn't find). It seems suspiciously like spamming or other bad stuff.

That reminded me of certain efforts of renaming "daemons" used in Unix systems to "angels" or "agents", and similar more recent efforts wrt. "master" and "slave"... Perhaps some people just have too much idle time right now?

[Mechatrommer]

i think OP is chatbot, mission accomplished to keep you guys going...

[classicsamus87]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

[LeonR]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

A sledgehammer. Isolates all the bad blocks at the first slam.

 

[BravoV]

i think OP is chatbot, mission accomplished to keep you guys going...

... or just a retard.

[MK14]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

A Banhammer needs to be used.

[Simon]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

A Banhammer needs to be used.

Leave him to his misery

[classicsamus87]

how did you obtain this information if this automatic technology for relocating defective sectors of the HDD has no name?