Author Topic: HDDs and badblocks corrupt downloaded files?  (Read 10159 times)

0 Members and 1 Guest are viewing this topic.

Offline BravoV

  • Super Contributor
  • ***
  • Posts: 7366
  • Country: 00
  • +++ ATH1
Re: HDDs and badblocks corrupt downloaded files?
« Reply #125 on: October 13, 2020, 03:33:15 pm »

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

No. The function is called Jim.

Having fun, don't you ?  :-DD
 
The following users thanked this post: hexreader

Offline BradC

  • Super Contributor
  • ***
  • Posts: 1890
  • Country: au
Re: HDDs and badblocks corrupt downloaded files?
« Reply #126 on: October 14, 2020, 10:16:17 am »

Is this the name of the automatic reallocation function of defective sectors of the HDD present in the firmware?

No. The function is called Jim.

I think I got into an argument with him in a bar once. He was being a SMART arse and wouldn't give me back my data. Said it had been "reallocated". Irritating sod really got my backup so much I wanted to take a byte out of him and serve his head on a platter.

 
The following users thanked this post: BravoV, hexreader, tooki, Ysjoelfir

Offline classicsamus87

  • Regular Contributor
  • *
  • !
  • Posts: 97
  • Country: br
Re: HDDs and badblocks corrupt downloaded files?
« Reply #127 on: October 14, 2020, 12:51:45 pm »

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?
 

Offline BravoV

  • Super Contributor
  • ***
  • Posts: 7366
  • Country: 00
  • +++ ATH1
Re: HDDs and badblocks corrupt downloaded files?
« Reply #128 on: October 14, 2020, 12:53:27 pm »
:palm: :palm: :palm:
 :scared: :scared:
 :-DD
 :scared: :scared:
 :palm: :palm: :palm:
 
The following users thanked this post: tooki

Offline Monkeh

  • Super Contributor
  • ***
  • Posts: 7402
  • Country: gb
Re: HDDs and badblocks corrupt downloaded files?
« Reply #129 on: October 14, 2020, 02:08:23 pm »

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Please stop trying to use a forum to write a paper for you.
 

Offline BradC

  • Super Contributor
  • ***
  • Posts: 1890
  • Country: au
Re: HDDs and badblocks corrupt downloaded files?
« Reply #130 on: October 14, 2020, 02:15:44 pm »

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

The SMART answer is no. Of course the real answer is maybe, or perhaps.
 

Offline DrG

  • Super Contributor
  • ***
  • !
  • Posts: 1203
  • Country: us
Re: HDDs and badblocks corrupt downloaded files?
« Reply #131 on: October 14, 2020, 04:53:08 pm »

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Please stop trying to use a forum to write a paper for you.

Correction, forums.

I read this thread and analogous ones elsewhere and I could not figure it out at all. At first, I thought it was a bot, but then, with the edits and all, I got off that possibility and just figured it was for the kicks of seeing responses. Your explanation is plausible.

I have to add that it says something about me that I continue to read a long thread to try to understand the OP's motivation when my interest in the original subject matter has long since waned.
- Invest in science - it pays big dividends. -
 
The following users thanked this post: hexreader

Offline Wuerstchenhund

  • Super Contributor
  • ***
  • Posts: 3088
  • Country: gb
  • Able to drop by occasionally only
Re: HDDs and badblocks corrupt downloaded files?
« Reply #132 on: October 15, 2020, 08:22:02 am »
CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense.

Sudden power loss during a write operation – say, updating directory access timestamps or similar; anything that requires an update of the internal state.  They just don't have the circuitry to deal with it.  Technically, it would be trivial: just add some capacitors, that provide enough current to finish any pending write operation.

It still doesn't make any sense. You claimed a power loss would "brick" the card (i.e. render it unusable). Also, Timestamps are something a file system deals with, not the card (which is pretty oblivious to the data that is written to it).

Quote
Removable storage like USB sticks or CF cards have no "power down" function (as for example SSDs have), in fact CF cards are widely used in devices which don't perform some kind of a shutdown but just remove power (and even those that do are susceptible to sporadic power loss).
See what happens when you remove the device battery when it is writing to the CF card.  At least a few years ago, the chances were the card was bricked.

No, it wasn't. A power loss during write operation may well corrupt your data but it doesn't "brick" the card.

We have thousands of CF cards in devices around the world, and have been so for over two decades, and haven't seen a single card "bricked" because of powerloss (when they die they usually do so because of write exhaustions, and a few cards failed sporadically).

There are also literally millions of CF cards in routers and other networking gear, all susceptible to sudden power loss. And still, your claim of cards dying because of power loss remains unheard of.

Quote
It has nothing to do with sudden power loss (which is and never was a problem for removable flash storage devices).

Hogwash.  CF cards maintain their own wear leveling tables (and other internal state), and for whatever reason, tend to get bricked whenever power loss occurs during the update of that state.  It has absolutely nothing to do with the host accessible data (filesystem et cetera), and everything to do with the device internal state.

That's nonsense.

First of all, not all CF cards have wear levelling (before CF Rev 4.1 only a few industrial CF cards had any Wear Leveling).

Second, Wear Leveling doesn't work the way you think it does. If a block is marked for shifting, it's content is copied to the new block first, and only if that copy process is completed then the original block is erased. If this process is interrupted during the copying phase, the old block is still marked for shifting and the copy process will simply be repeated after the power is restored.

Quote
Early USB memory sticks suffered from exactly the same problems.  I had several 64Mbyte and 128Mbyte sticks that died this way.  They soon fixed the issue; I do not know exactly how, but I believe by using capacitors to hold enough charge so that the internal state could always be updated successfully.

Emphasis mine, because that is your problem. Your whole arguments are based on imagination and guesswork.

Memory sticks die, a lot. They do so because USB sticks are the most price-conscious market for flash memory. Many sticks use reclaimed (i.e. used) flash memory which may or may not be close to the end of its life. Most of them have no Wear Leveling either. They are the equivalent to the floppy disk and should be treated as such, i.e. as consumables.

I still have a box full of old Samsung, SanDisk and HP 16-64MB USB sticks from the old days, and they all were taken out of service due to their capacity, not because they died.

Quote
(One way to brick USB memory sticks is to short the device-side VUSB and GND via a small value bleed resistor immediately after the device side is disconnected from the host side, and do so when writing to the stick.  The bleed resistor depletes the capacitors, so the management chip is unable to correctly update its internal state.

That's nonsense (besides, all shorting VUSB and GND during writing does is to interrupt the write process and trigger the USB port's over-current protection).

If you do that to a disconnected USB stick then nothing happens.

As I said before, things don't work the way you imagine them.

Quote
Again, this has nothing to do with flash memory tech per se, and everything to do with the devices internal non-exposed state being only partially updated.)

No, it's mostly a thing of your imagination.

Quote
Anecdotally, many CF cards still lack protection against sudden power loss.  One reason could be that since these use the ATA (PATA aka parallel-ATA) interface, the charge needed for a full internal state update is so large that many manufacturers skimp on the capacitors.  I suspect, but cannot verify, that CF Revision 4.1 (Power Enhanced CF) is related.

"Power Enhanced CF" is a CF card which supports power management modes (i.e. idle modes) to reduce power draw during phases of inactivity, which is important for battery operated devices like cameras to extend battery life.

It has nothing to do with power loss protection.

Maybe you should not inform yourself through anecdotes.
« Last Edit: October 15, 2020, 08:37:12 am by Wuerstchenhund »
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 4551
  • Country: pl
Re: HDDs and badblocks corrupt downloaded files?
« Reply #133 on: October 15, 2020, 08:27:59 am »
I like you believe that it is possible to create a fault tolerant flash translation layer, although I have never done one myself.

I know for sure it is possible to create one which shits itself upon power loss because that has been done by others :P

edit
I also have a pendrive which reliably passes testing with sequential files to 100% capacity but turns random blocks into 0xff when it's loaded with lots of random I/O (root filesystem of Linux). Even with syncing and unmounting. I have another flash drive which works fine under such conditions.
Not every problem in those things can be blamed on raw flash. The controllers have their share of suckage too.
« Last Edit: October 15, 2020, 08:36:13 am by magic »
 

Offline Wuerstchenhund

  • Super Contributor
  • ***
  • Posts: 3088
  • Country: gb
  • Able to drop by occasionally only
Re: HDDs and badblocks corrupt downloaded files?
« Reply #134 on: October 15, 2020, 08:48:18 am »
I like you believe that it is possible to create a fault tolerant flash translation layer, although I have never done one myself.

I know for sure it is possible to create one which shits itself upon power loss because that has been done by others :P

Well, dynamic FTLs are part of Wear Leveling, and together with ECC it protects the drive from faults caused by soft errors or defective flash cells.

But I agree that there are some shitty implementations (*cough* Kingspec *cough).

Quote
edit
I also have a pendrive which reliably passes testing with sequential files to 100% capacity but turns random blocks into 0xff when it's loaded with lots of random I/O (root filesystem of Linux). Even with syncing and unmounting. I have another flash drive which works fine under such conditions.
Not every problem in those things can be blamed on raw flash. The controllers have their share of suckage too.

Tell me about it. I have another box full with Kingston Traveller USB3 sticks which in principle are fine, aside from the small problem that the flash controller quickly overheats during R/W operations which take the drive offline sporadically.

Thanks to Kingston's long warranty I was able to replace a few of them for new ones which suffer from the exact same problem  |O
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 3596
  • Country: fi
    • My home page and email address
Re: HDDs and badblocks corrupt downloaded files?
« Reply #135 on: October 15, 2020, 10:00:45 am »
CompactFlash cards (as used on many portable devices like cameras) are very sensitive to power loss.  As in, there is a good chance a sudden power bricks the entire card.  Early USB memory sticks used to be very sensitive to that too, but I guess users are so careless at removing them that they had to make them more robust against power loss.

That doesn't make any sense.

Sudden power loss during a write operation – say, updating directory access timestamps or similar; anything that requires an update of the internal state.  They just don't have the circuitry to deal with it.  Technically, it would be trivial: just add some capacitors, that provide enough current to finish any pending write operation.

It still doesn't make any sense. You claimed a power loss would "brick" the card (i.e. render it unusable). Also, Timestamps are something a file system deals with, not the card (which is pretty oblivious to the data that is written to it).

Wear leveling means that when you update a data sector, a completely different data sector gets updated; it is not that the flash memory containing the old data is updated.  (Wear leveling that lets the data be updated in place is useless.)

If the flash management chip loses power when updating the wear leveling data, it can brick itself.  See?  The internal state includes the wear leveling data, which is not exposed to the OS in any way.  The flash management chip assumes the wear leveling data is okay; but if it loses power in the middle of updating the wear leveling data, that data will be inconsistent, and brick the stick.



We have thousands of CF cards in devices around the world, and have been so for over two decades, and haven't seen a single card "bricked" because of powerloss (when they die they usually do so because of write exhaustions, and a few cards failed sporadically).
Well, I have, although it was a decade ago.  Are the cards you use all from a specific manufacturer?  I wanna know, so I can use those too.

There are also literally millions of CF cards in routers and other networking gear, all susceptible to sudden power loss.
Routers?  I've never seen one in a router; care to name one?
The ones I've seen are in cameras, which have OSes that shut down gracefully when the battery is low.

Also note that simply adding bulk capacitance to the supply going to the CF card, can completely avoid the death-on-power-loss situation.
This is because then the CF card always has enough power available to complete the last write operation the host sent before losing power, including the internal (wear leveling and whatever data).

SBCs really do not have a lot of bulk capacitance on their bus voltages.  For example, early 3G USB modems used with SBCs had "spiky" current requirements, the solution being adding a couple of milliFarads of low-ESR bulk capacitance between USB +5V and GND for these devices.  On an SBC, the CF card supply would basically drop as soon as the main processor lost power, thus making it likelier for the CF card to be bricked due to powerloss during a write (as internal wear-leveling etc. data was only partially updated).

Emphasis mine, because that is your problem. Your whole arguments are based on imagination and guesswork.
And yours is based on "me Wuerstchenhund, you wrong".

If you do that to a disconnected USB stick then nothing happens.
You think/believe; you obviously haven't tried it in practice.

Perhaps you should take your own advice, and stop claiming truths based on your own beliefs?

Because of you, my ignore list is no longer empty.
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 4551
  • Country: pl
Re: HDDs and badblocks corrupt downloaded files?
« Reply #136 on: October 15, 2020, 11:01:31 am »
Wear leveling means that when you update a data sector, a completely different data sector gets updated; it is not that the flash memory containing the old data is updated.  (Wear leveling that lets the data be updated in place is useless.)

If the flash management chip loses power when updating the wear leveling data, it can brick itself.  See?  The internal state includes the wear leveling data, which is not exposed to the OS in any way.  The flash management chip assumes the wear leveling data is okay; but if it loses power in the middle of updating the wear leveling data, that data will be inconsistent, and brick the stick.
That's why every semi-decent storage system based on flash should use some sort of B trees with copy on write updates and timestamps on every revision of the superblock so you know which is the latest one. IMO it should be possible to build a bulletproof flash drive and some SSDs are known to handle power loss well, it's just that many cheap ones either don't bother or are buggy.

Another problem is DRAM write back caches. Some SSDs supposedly solve this problem by having enough capacitance to write the full cache back in case of power loss. If you castrate their cap bank then bad things will happen.
 

Online MK14

  • Super Contributor
  • ***
  • Posts: 3371
  • Country: gb
Re: HDDs and badblocks corrupt downloaded files?
« Reply #137 on: October 15, 2020, 10:57:56 pm »

If the old HDD contains SMART then does it have automatic reallocated technology from defective sectors? does he only need to have SMART?

Starting from the beginning again. What exactly are you trying to do/achieve and/or find out ?

Recover data from old disks ?
Or technical details on how old hard disk drives worked, for your own curiosity.
Or something else, such as a patent dispute/claim, court case or for other research reasons ?

I've apparently seen you with threads on a number of different forums. Such as:

https://community.wd.com/t/questions-bad-block-and-reallocated-sectors-count/255399/8

https://forum.allaboutcircuits.com/threads/question-hdd-and-badblock.173159/

https://www.electro-tech-online.com/threads/hdds-and-badblocks-corrupt-downloaded-files.159990/

Actually, I'm finding too many (assuming there are many more out there, I didn't find). It seems suspiciously like spamming or other bad stuff.
« Last Edit: October 16, 2020, 12:04:45 am by MK14 »
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 3596
  • Country: fi
    • My home page and email address
Re: HDDs and badblocks corrupt downloaded files?
« Reply #138 on: October 16, 2020, 12:53:04 am »
That's why every semi-decent storage system based on flash should use some sort of B trees with copy on write updates and timestamps on every revision of the superblock so you know which is the latest one. IMO it should be possible to build a bulletproof flash drive and some SSDs are known to handle power loss well, it's just that many cheap ones either don't bother or are buggy.
Exactly – although B-trees aren't the only option; there are a lot of known-good methods –; it's that because the memory controller is usually implemented in silicon, it (and the associated passives that cost next to nothing) is an attractive choice for cost-cutting.

Some use heuristics (to avoid storing the number of block writes, saving memory), and depending on the filesystem (block patterns), can die very quickly because of superblock updates hit the same physical flash blocks exhausting them.  But, because it is dependent on the filesystem, users blame the filesystem (for being "bad" on flash media), instead of the wear leveling.  This is particularly common among people who have high brand awareness, and consider the flash manufacturer more reliable than e.g. open-source software (open sores, they often call it), as non-Windows/non-Mac filesystems are rarely even considered by the manufacturers, but widely used on (especially embedded) devices based on Linux, Minix, or BSD variants.  They believe that if the device wear leveling behaves well on FAT/exFAT, it must be the software's fault if it dies rapidly on any other filesystem (or in raw block use, without any filesystem at all).

(Compare to e.g. Apple devices liquid sensitivity, which Louis Rossmann often talks about. Most other laptop/tablet boards are not nearly as sensitive to tiny amounts of moisture, because it is easy and cheap to achieve.  For Apple, it is an internal design choice, extending to color-changing moisture detection stickers on their boards...  Stuff like putting 24V backlight power next to a logic pin going directly to a low-voltage processor in a connector prone to moisture and dust ingress, near the display hinge.  There is no technical reason to do that, if the same connector has half a dozen ground pins just a mm or two away.)

We've seen effects of similar cost-cutting on the spinning rust side, too.  While WD drives have a good reputation, they have quite a few series that had serious issues with RAID, or performance in general; things that they disclosed later only because users were too upset (upset enough to harm their brand if they did nothing).

I fully know there is no technical or technological reason for the weaknesses I've claimed in this thread; it's just stupidity or cost-cutting.

Starting from the beginning again. What exactly are you trying to do/achieve and/or find out ?
Very good point.

I personally was kind-of assuming the underlying question was something like "if I download a file, is it possible a spinny-disk HDD manufactured after 2000 stores it in blocks that turn out to be bad?"

The answer is yes and no.  No, because the drives do not knowingly store anything to the blocks it knows to be bad.  Yes, because while the block may be considered good when the data is written to it, it can be detected as bad when trying to read the data later on.  SMART reports these as 'uncorrectable errors'.  Heck, the block might be good at that time, but the media surface degrade later on (physically or magnetically, for whatever reason from manufacturing issue to temperature gradients to radiation damage) becoming 'bad'.  This applies even if there happens to be a drive that knows nothing about bad blocks or sector reallocation.

There is a technology that detects blocks going bad by analysing the error correction frequency (amount of bit errors corrected with ECC data), and reallocates the block before any data loss happens.  (This is described in a patent I linked to earlier.)  However, we don't know which, if any, drives implement this; I think most manufacturers would consider this "too costly", wrt. to their profit margins.

Actually, I'm finding too many (assuming there are many more out there, I didn't find). It seems suspiciously like spamming or other bad stuff.
That reminded me of certain efforts of renaming "daemons" used in Unix systems to "angels" or "agents", and similar more recent efforts wrt. "master" and "slave"... Perhaps some people just have too much idle time right now?
 
The following users thanked this post: MK14

Offline Mechatrommer

  • Super Contributor
  • ***
  • Posts: 10444
  • Country: my
  • reassessing directives...
Re: HDDs and badblocks corrupt downloaded files?
« Reply #139 on: October 16, 2020, 01:00:36 am »
i think OP is chatbot, mission accomplished to keep you guys going...
It's extremely difficult to start life.. one features of nature.. physical laws are mathematical theory of great beauty... You may wonder Why? our knowledge shows that nature is so constructed. We simply have to accept it. One could describe the situation by saying that... (Paul Dirac)
 
The following users thanked this post: hexreader, tooki, MK14

Offline classicsamus87

  • Regular Contributor
  • *
  • !
  • Posts: 97
  • Country: br
Re: HDDs and badblocks corrupt downloaded files?
« Reply #140 on: October 16, 2020, 10:09:43 am »
What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs
 
The following users thanked this post: Siwastaja

Offline LeonR

  • Regular Contributor
  • *
  • Posts: 150
  • Country: br
  • PC hardware enthusiast
Re: HDDs and badblocks corrupt downloaded files?
« Reply #141 on: October 16, 2020, 12:43:34 pm »
What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs

A sledgehammer. Isolates all the bad blocks at the first slam.

 :-+
 
The following users thanked this post: hexreader, Ysjoelfir

Offline BravoV

  • Super Contributor
  • ***
  • Posts: 7366
  • Country: 00
  • +++ ATH1
Re: HDDs and badblocks corrupt downloaded files?
« Reply #142 on: October 16, 2020, 12:50:46 pm »
i think OP is chatbot, mission accomplished to keep you guys going...

... or just a retard.
 
The following users thanked this post: hexreader, tooki

Online MK14

  • Super Contributor
  • ***
  • Posts: 3371
  • Country: gb
Re: HDDs and badblocks corrupt downloaded files?
« Reply #143 on: October 16, 2020, 01:38:58 pm »
What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs
A Banhammer needs to be used.
 
The following users thanked this post: hexreader

Online Simon

  • Global Moderator
  • *****
  • Posts: 16652
  • Country: gb
  • Did that just blow up? No? might work after all !!
    • Simon's Electronics
Re: HDDs and badblocks corrupt downloaded files?
« Reply #144 on: October 16, 2020, 04:43:18 pm »
What technology does the HDD need to automatically isolate and reallocate bad sectors? I have 2000 year IDE HDDs
A Banhammer needs to be used.

Leave him to his misery
 

Offline classicsamus87

  • Regular Contributor
  • *
  • !
  • Posts: 97
  • Country: br
Re: HDDs and badblocks corrupt downloaded files?
« Reply #145 on: October 24, 2020, 02:59:02 pm »
how did you obtain this information if this automatic technology for relocating defective sectors of the HDD has no name?
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf