HDDs and badblocks corrupt downloaded files?

[BravoV]

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

No. The function is called Jim.

Having fun, don't you ? 

[BradC]

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

No. The function is called Jim.

I think I got into an argument with him in a bar once. He was being a SMART arse and wouldn't give me back my data. Said it had been "reallocated". Irritating sod really got my backup so much I wanted to take a byte out of him and serve his head on a platter.

[classicsamus87]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

[BravoV]

 
 
 
 

[Monkeh]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Please stop trying to use a forum to write a paper for you.

[BradC]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

The SMART answer is no. Of course the real answer is maybe, or perhaps.

[DrG]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Please stop trying to use a forum to write a paper for you.

Correction, forums.

I read this thread and analogous ones elsewhere and I could not figure it out at all. At first, I thought it was a bot, but then, with the edits and all, I got off that possibility and just figured it was for the kicks of seeing responses. Your explanation is plausible.

I have to add that it says something about me that I continue to read a long thread to try to understand the OP's motivation when my interest in the original subject matter has long since waned.

[Wuerstchenhund]

CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense.

Sudden power loss during a write operation – say, updating directory access timestamps or similar; anything that requires an update of the internal state.  They just don't have the circuitry to deal with it.  Technically, it would be trivial: just add some capacitors, that provide enough current to finish any pending write operation.

It still doesn't make any sense. You claimed a power loss would "brick" the card (i.e. render it unusable). Also, Timestamps are something a file system deals with, not the card (which is pretty oblivious to the data that is written to it).

Removable storage like USB sticks or CF cards have no "power down" function (as for example SSDs have), in fact CF cards are widely used in devices which don't perform some kind of a shutdown but just remove power (and even those that do are susceptible to sporadic power loss).

See what happens when you remove the device battery when it is writing to the CF card.  At least a few years ago, the chances were the card was bricked.

No, it wasn't. A power loss during write operation may well corrupt your data but it doesn't "brick" the card.

We have thousands of CF cards in devices around the world, and have been so for over two decades, and haven't seen a single card "bricked" because of powerloss (when they die they usually do so because of write exhaustions, and a few cards failed sporadically).

There are also literally millions of CF cards in routers and other networking gear, all susceptible to sudden power loss. And still, your claim of cards dying because of power loss remains unheard of.

It has nothing to do with sudden power loss (which is and never was a problem for removable flash storage devices).

Hogwash.  CF cards maintain their own wear leveling tables (and other internal state), and for whatever reason, tend to get bricked whenever power loss occurs during the update of that state.  It has absolutely nothing to do with the host accessible data (filesystem et cetera), and everything to do with the device internal state.

That's nonsense.

First of all, not all CF cards have wear levelling (before CF Rev 4.1 only a few industrial CF cards had any Wear Leveling).

Second, Wear Leveling doesn't work the way you think it does. If a block is marked for shifting, it's content is copied to the new block first, and only if that copy process is completed then the original block is erased. If this process is interrupted during the copying phase, the old block is still marked for shifting and the copy process will simply be repeated after the power is restored.

Early USB memory sticks suffered from exactly the same problems.  I had several 64Mbyte and 128Mbyte sticks that died this way.  They soon fixed the issue; I do not know exactly how, but I believe by using capacitors to hold enough charge so that the internal state could always be updated successfully.

Emphasis mine, because that is your problem. Your whole arguments are based on imagination and guesswork.

Memory sticks die, a lot. They do so because USB sticks are the most price-conscious market for flash memory. Many sticks use reclaimed (i.e. used) flash memory which may or may not be close to the end of its life. Most of them have no Wear Leveling either. They are the equivalent to the floppy disk and should be treated as such, i.e. as consumables.

I still have a box full of old Samsung, SanDisk and HP 16-64MB USB sticks from the old days, and they all were taken out of service due to their capacity, not because they died.

(One way to brick USB memory sticks is to short the device-side VUSB and GND via a small value bleed resistor immediately after the device side is disconnected from the host side, and do so when writing to the stick.  The bleed resistor depletes the capacitors, so the management chip is unable to correctly update its internal state.

That's nonsense (besides, all shorting VUSB and GND during writing does is to interrupt the write process and trigger the USB port's over-current protection).

If you do that to a disconnected USB stick then nothing happens.

As I said before, things don't work the way you imagine them.

Again, this has nothing to do with flash memory tech per se, and everything to do with the devices internal non-exposed state being only partially updated.)

No, it's mostly a thing of your imagination.

Anecdotally, many CF cards still lack protection against sudden power loss.  One reason could be that since these use the ATA (PATA aka parallel-ATA) interface, the charge needed for a full internal state update is so large that many manufacturers skimp on the capacitors.  I suspect, but cannot verify, that CF Revision 4.1 (Power Enhanced CF) is related.

"Power Enhanced CF" is a CF card which supports power management modes (i.e. idle modes) to reduce power draw during phases of inactivity, which is important for battery operated devices like cameras to extend battery life.

It has nothing to do with power loss protection.

Maybe you should not inform yourself through anecdotes.

[magic]

I like you believe that it is possible to create a fault tolerant flash translation layer, although I have never done one myself.

I know for sure it is possible to create one which shits itself upon power loss because that has been done by others

edit
I also have a pendrive which reliably passes testing with sequential files to 100% capacity but turns random blocks into 0xff when it's loaded with lots of random I/O (root filesystem of Linux). Even with syncing and unmounting. I have another flash drive which works fine under such conditions.
Not every problem in those things can be blamed on raw flash. The controllers have their share of suckage too.

[Wuerstchenhund]

I like you believe that it is possible to create a fault tolerant flash translation layer, although I have never done one myself.

I know for sure it is possible to create one which shits itself upon power loss because that has been done by others

Well, dynamic FTLs are part of Wear Leveling, and together with ECC it protects the drive from faults caused by soft errors or defective flash cells.

But I agree that there are some shitty implementations (*cough* Kingspec *cough).

edit
I also have a pendrive which reliably passes testing with sequential files to 100% capacity but turns random blocks into 0xff when it's loaded with lots of random I/O (root filesystem of Linux). Even with syncing and unmounting. I have another flash drive which works fine under such conditions.
Not every problem in those things can be blamed on raw flash. The controllers have their share of suckage too.

Tell me about it. I have another box full with Kingston Traveller USB3 sticks which in principle are fine, aside from the small problem that the flash controller quickly overheats during R/W operations which take the drive offline sporadically.

Thanks to Kingston's long warranty I was able to replace a few of them for new ones which suffer from the exact same problem 

[Nominal Animal]

CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense.

Sudden power loss during a write operation – say, updating directory access timestamps or similar; anything that requires an update of the internal state.  They just don't have the circuitry to deal with it.  Technically, it would be trivial: just add some capacitors, that provide enough current to finish any pending write operation.

It still doesn't make any sense. You claimed a power loss would "brick" the card (i.e. render it unusable). Also, Timestamps are something a file system deals with, not the card (which is pretty oblivious to the data that is written to it).

Wear leveling means that when you update a data sector, a completely different data sector gets updated; it is not that the flash memory containing the old data is updated.  (Wear leveling that lets the data be updated in place is useless.)

If the flash management chip loses power when updating the wear leveling data, it can brick itself.  See?  The internal state includes the wear leveling data, which is not exposed to the OS in any way.  The flash management chip assumes the wear leveling data is okay; but if it loses power in the middle of updating the wear leveling data, that data will be inconsistent, and brick the stick.

We have thousands of CF cards in devices around the world, and have been so for over two decades, and haven't seen a single card "bricked" because of powerloss (when they die they usually do so because of write exhaustions, and a few cards failed sporadically).

Well, I have, although it was a decade ago.  Are the cards you use all from a specific manufacturer?  I wanna know, so I can use those too.

There are also literally millions of CF cards in routers and other networking gear, all susceptible to sudden power loss.

Routers?  I've never seen one in a router; care to name one?
The ones I've seen are in cameras, which have OSes that shut down gracefully when the battery is low.

Also note that simply adding bulk capacitance to the supply going to the CF card, can completely avoid the death-on-power-loss situation.
This is because then the CF card always has enough power available to complete the last write operation the host sent before losing power, including the internal (wear leveling and whatever data).

SBCs really do not have a lot of bulk capacitance on their bus voltages.  For example, early 3G USB modems used with SBCs had "spiky" current requirements, the solution being adding a couple of milliFarads of low-ESR bulk capacitance between USB +5V and GND for these devices.  On an SBC, the CF card supply would basically drop as soon as the main processor lost power, thus making it likelier for the CF card to be bricked due to powerloss during a write (as internal wear-leveling etc. data was only partially updated).

Emphasis mine, because that is your problem. Your whole arguments are based on imagination and guesswork.

And yours is based on "me Wuerstchenhund, you wrong".

If you do that to a disconnected USB stick then nothing happens.

You think/believe; you obviously haven't tried it in practice.

Perhaps you should take your own advice, and stop claiming truths based on your own beliefs?

Because of you, my ignore list is no longer empty.

[magic]

Wear leveling means that when you update a data sector, a completely different data sector gets updated; it is not that the flash memory containing the old data is updated.  (Wear leveling that lets the data be updated in place is useless.)

If the flash management chip loses power when updating the wear leveling data, it can brick itself.  See?  The internal state includes the wear leveling data, which is not exposed to the OS in any way.  The flash management chip assumes the wear leveling data is okay; but if it loses power in the middle of updating the wear leveling data, that data will be inconsistent, and brick the stick.

That's why every semi-decent storage system based on flash should use some sort of B trees with copy on write updates and timestamps on every revision of the superblock so you know which is the latest one. IMO it should be possible to build a bulletproof flash drive and some SSDs are known to handle power loss well, it's just that many cheap ones either don't bother or are buggy.

Another problem is DRAM write back caches. Some SSDs supposedly solve this problem by having enough capacitance to write the full cache back in case of power loss. If you castrate their cap bank then bad things will happen.

[MK14]

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Starting from the beginning again. What exactly are you trying to do/achieve and/or find out ?

Recover data from old disks ?
Or technical details on how old hard disk drives worked, for your own curiosity.
Or something else, such as a patent dispute/claim, court case or for other research reasons ?

I've apparently seen you with threads on a number of different forums. Such as:

https://community.wd.com/t/questions-bad-block-and-reallocated-sectors-count/255399/8

https://forum.allaboutcircuits.com/threads/question-hdd-and-badblock.173159/

https://www.electro-tech-online.com/threads/hdds-and-badblocks-corrupt-downloaded-files.159990/

Actually, I'm finding too many (assuming there are many more out there, I didn't find). It seems suspiciously like spamming or other bad stuff.

[Nominal Animal]

That's why every semi-decent storage system based on flash should use some sort of B trees with copy on write updates and timestamps on every revision of the superblock so you know which is the latest one. IMO it should be possible to build a bulletproof flash drive and some SSDs are known to handle power loss well, it's just that many cheap ones either don't bother or are buggy.

Exactly – although B-trees aren't the only option; there are a lot of known-good methods –; it's that because the memory controller is usually implemented in silicon, it (and the associated passives that cost next to nothing) is an attractive choice for cost-cutting.

Some use heuristics (to avoid storing the number of block writes, saving memory), and depending on the filesystem (block patterns), can die very quickly because of superblock updates hit the same physical flash blocks exhausting them.  But, because it is dependent on the filesystem, users blame the filesystem (for being "bad" on flash media), instead of the wear leveling.  This is particularly common among people who have high brand awareness, and consider the flash manufacturer more reliable than e.g. open-source software (open sores, they often call it), as non-Windows/non-Mac filesystems are rarely even considered by the manufacturers, but widely used on (especially embedded) devices based on Linux, Minix, or BSD variants.  They believe that if the device wear leveling behaves well on FAT/exFAT, it must be the software's fault if it dies rapidly on any other filesystem (or in raw block use, without any filesystem at all).

(Compare to e.g. Apple devices liquid sensitivity, which Louis Rossmann often talks about. Most other laptop/tablet boards are not nearly as sensitive to tiny amounts of moisture, because it is easy and cheap to achieve.  For Apple, it is an internal design choice, extending to color-changing moisture detection stickers on their boards...  Stuff like putting 24V backlight power next to a logic pin going directly to a low-voltage processor in a connector prone to moisture and dust ingress, near the display hinge.  There is no technical reason to do that, if the same connector has half a dozen ground pins just a mm or two away.)

We've seen effects of similar cost-cutting on the spinning rust side, too.  While WD drives have a good reputation, they have quite a few series that had serious issues with RAID, or performance in general; things that they disclosed later only because users were too upset (upset enough to harm their brand if they did nothing).

I fully know there is no technical or technological reason for the weaknesses I've claimed in this thread; it's just stupidity or cost-cutting.

Starting from the beginning again. What exactly are you trying to do/achieve and/or find out ?

Very good point.

I personally was kind-of assuming the underlying question was something like "if I download a file, is it possible a spinny-disk HDD manufactured after 2000 stores it in blocks that turn out to be bad?"

The answer is yes and no.  No, because the drives do not knowingly store anything to the blocks it knows to be bad.  Yes, because while the block may be considered good when the data is written to it, it can be detected as bad when trying to read the data later on.  SMART reports these as 'uncorrectable errors'.  Heck, the block might be good at that time, but the media surface degrade later on (physically or magnetically, for whatever reason from manufacturing issue to temperature gradients to radiation damage) becoming 'bad'.  This applies even if there happens to be a drive that knows nothing about bad blocks or sector reallocation.

There is a technology that detects blocks going bad by analysing the error correction frequency (amount of bit errors corrected with ECC data), and reallocates the block before any data loss happens.  (This is described in a patent I linked to earlier.)  However, we don't know which, if any, drives implement this; I think most manufacturers would consider this "too costly", wrt. to their profit margins.

Actually, I'm finding too many (assuming there are many more out there, I didn't find). It seems suspiciously like spamming or other bad stuff.

That reminded me of certain efforts of renaming "daemons" used in Unix systems to "angels" or "agents", and similar more recent efforts wrt. "master" and "slave"... Perhaps some people just have too much idle time right now?

[Mechatrommer]

i think OP is chatbot, mission accomplished to keep you guys going...

[classicsamus87]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

[LeonR]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

A sledgehammer. Isolates all the bad blocks at the first slam.

 

[BravoV]

i think OP is chatbot, mission accomplished to keep you guys going...

... or just a retard.

[MK14]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

A Banhammer needs to be used.

[Simon]

What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

A Banhammer needs to be used.

Leave him to his misery

[classicsamus87]

how did you obtain this information if this automatic technology for relocating defective sectors of the HDD has no name?