Author Topic: let's face it: usb-sticks are not reliable  (Read 8071 times)

0 Members and 1 Guest are viewing this topic.

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
let's face it: usb-sticks are not reliable
« on: April 11, 2022, 07:00:52 am »
Back to 2011, I bought a couple of usb-sticks from Amazon, TDK and Sandisk brand new sticks, supposed to be "decent quality" for a kind of backup kind of backup you carefully store inside a locker and don't touch for years.

Yesterday night I mounted those usb-sticks in my Linux box to read some old files and ...
Code: [Select]
sd 7:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 7:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
sd 7:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
sd 7:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 1c 42 1e 00 00 f0 00
blk_update_request: critical medium error, dev sdc, sector 1851934
sd 7:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 7:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
sd 7:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
sd 7:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 1c 42 a8 00 00 02 00
blk_update_request: critical medium error, dev sdc, sector 1852072
Buffer I/O error on dev sdc1, logical block 926005, async page read
 sdc: sdc1 sdc2 sdc3
sd 7:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 7:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
sd 7:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
sd 7:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 7a e0 60 00 00 f0 00
blk_update_request: critical medium error, dev sdc, sector 8052832
sd 7:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 7:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
sd 7:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
sd 7:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 7a e1 60 00 00 f0 00
blk_update_request: critical medium error, dev sdc, sector 8053088
sd 7:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 7:0:0:0: [sdc] tag#0 Sense Key : Medium Error [current]
sd 7:0:0:0: [sdc] tag#0 Add. Sense: Unrecovered read error
sd 7:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 7a e1 00 00 00 08 00
blk_update_request: critical medium error, dev sdc, sector 8052992
Buffer I/O error on dev sdc3, logical block 394064, async page read
Quick test
Code: [Select]
dd of=/dev/null if=/dev/sdc3
dd: reading '/dev/sdc3': Input/output error
3152512+0 records in
3152512+0 records out
1614086144 bytes (1.6 GB) copied, 1521.2 s, 1.1 MB/s

Thanks god, I also saved all those files in DVD-ram cartridges and even a copy on an hard-disk, which are all in perfect working condition.

Not a single file was lost.

Moral of the story: don't trust USB-sticks!
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 
The following users thanked this post: Trader

Offline james_s

  • Super Contributor
  • ***
  • Posts: 21611
  • Country: us
Re: let's face it: usb-sticks are not reliable
« Reply #1 on: April 11, 2022, 07:05:20 am »
Based on a sample of just two that you bought 11 years ago?

I've had dozens of USB sticks over the years, so far I've never had a single one of them fail. They're as reliable as just about any other consumer storage.
 
The following users thanked this post: edavid, tooki

Offline wraper

  • Supporter
  • ****
  • Posts: 16796
  • Country: lv
Re: let's face it: usb-sticks are not reliable
« Reply #2 on: April 11, 2022, 07:39:16 am »
Back to 2011, I bought a couple of usb-sticks from Amazon, TDK and Sandisk brand new sticks, supposed to be "decent quality" for a kind of backup kind of backup you carefully store inside a locker and don't touch for years.
And here is your mistake. NAND cells discharge over time and in case of MLC, TLC, QLC it becomes hard to distinguish actual data within the cell as basically it's stored in analog voltage levels. For long term storage you need SLC flash, or one which is rewritten from time to time (decent SSD do this automatically). Type of error correction used and if it can adjust to diminishing voltage levels matters a lot too. I have even worse, Patriot 64GB SD card purchased about 10 years ago which starts corrupting the data after about 1.5 years.
 
The following users thanked this post: boB, george.b, DiTBho, MazeFrame

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #3 on: April 11, 2022, 07:46:23 am »
Based on a sample of just two that you bought 11 years ago?

Based on several similar episodes happened in the last eleven years.

Just, if in the other cases I have used and handle without care the USB that I brought around - and you think - ups, doesn't it no more work? d'oh, must be your fault - in this case the two sticks were securely stored in a locker for years.
« Last Edit: April 11, 2022, 07:53:11 am by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #4 on: April 11, 2022, 07:49:52 am »
Back to 2011, I bought a couple of usb-sticks from Amazon, TDK and Sandisk brand new sticks, supposed to be "decent quality" for a kind of backup kind of backup you carefully store inside a locker and don't touch for years.
And here is your mistake. NAND cells discharge over time and in case of MLC, TLC, QLC it becomes hard to distinguish actual data within the cell as basically it's stored in analog voltage levels. For long term storage you need SLC flash, or one which is rewritten from time to time (decent SSD do this automatically). Type of error correction used and if it can adjust to diminishing voltage levels matters a lot too. I have even worse, Patriot 64GB SD card purchased about 10 years ago which starts corrupting the data after about 1.5 years.

Yup, precisely the point, lesson learned  :D
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #5 on: April 11, 2022, 08:06:40 am »
However, the GPS-watch Garmin 7X Sapphire Solar costs 1000 Euros, and guess what? Its flash is NAND cell, so you can have it with a sapphire-glass that will last 100 years and a titanium ferrule that will last even more years, but its flash, the flash that stores its firmware ... will go crazy in 10 years.

Would you still buy it now that you know how things are?

The same applies to the most of the routers I have here: the firmware is hosted in a mini USB-stick plugged to a hidden port (not accessible until you open the plastic case), or NAND-flash chips.

All things that will go crazy in 10 years.

Certain products should really use SLC-flash. Are there any SLC-flash USB-sticks?
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline magic

  • Super Contributor
  • ***
  • Posts: 6733
  • Country: pl
Re: let's face it: usb-sticks are not reliable
« Reply #6 on: April 11, 2022, 09:19:21 am »
I have a USB stick which works normally until I put a writable Linux root filesystem on it - then it randomly loses data, replacing blocks in the middle of files with 0xFF.

I suspect some controller firmware bug ::)

Other than that, yes, all flash media are fairly unreliable. And when they fail, they may easily take all data with them, not just a sector or two.
 

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #7 on: April 11, 2022, 09:35:04 am »
Yup, and if the damage is located in the first block ... bye bye partition-table  :o :o :o
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #8 on: April 11, 2022, 09:36:58 am »
Code: [Select]
dd if=/dev/sda of=/dev/null
30266496+0 records in
30266496+0 records out
15496445952 bytes (15 GB) copied, 1022.07 s, 15.2 MB/s

this usb-pendrive is still alive, but it's only a temporary state you shouldn't trust for the future ;D
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline david77

  • Frequent Contributor
  • **
  • Posts: 934
  • Country: de
Re: let's face it: usb-sticks are not reliable
« Reply #9 on: April 11, 2022, 10:16:37 am »
Why would you ever think a USB stick is suitable for any kind of long term data storage?  :-//
 
The following users thanked this post: DiTBho

Offline golden_labels

  • Super Contributor
  • ***
  • Posts: 1183
  • Country: pl
Re: let's face it: usb-sticks are not reliable
« Reply #10 on: April 11, 2022, 10:23:48 am »
You are totally right, but… that shouldn’t come as a surprise. USB sticks were always disposable transfer storage Their purpose is to carry data between devices and that’s all. Similar to (Micro-)SD cards nowadays.

Vendors were conveniently (for themselves) silent about it, profiting on whatever assumptions customers might’ve came up with. So I do not blame anyone, who did not realize this. But low capacity is already a hint it’s not a long-term storage. And anyone using them frequently would know it’s a expected for them to die rather quickly.

They can still be utilized for backup in specific cases, though. Backup of the most important personal data. That’s a dozen megabytes, which can be appended monthly to a bunch of cheap USB sticks. Chances of more than one of them failing at the same time are low enough, but you have multiple copies that can be kept cheaply and effortlessly spread physically.
« Last Edit: April 11, 2022, 10:29:59 am by golden_labels »
People imagine AI as T1000. What we got so far is glorified T9.
 

Online Someone

  • Super Contributor
  • ***
  • Posts: 4510
  • Country: au
    • send complaints here
Re: let's face it: usb-sticks are not reliable
« Reply #11 on: April 11, 2022, 10:24:59 am »
This has always been a trade off in data storage from the "floppy" days. Do you want more space, or more robust storage? Keep extending the ECC/forward error correction:
https://en.wikipedia.org/wiki/Error_correction_code
in theory you can make it as robust as you want. But thats all set down at the controller level, so you have to buy something suitable rather than configure it.
 

Offline golden_labels

  • Super Contributor
  • ***
  • Posts: 1183
  • Country: pl
Re: let's face it: usb-sticks are not reliable
« Reply #12 on: April 11, 2022, 10:45:08 am »
ECC is a normal part of data encoding on all commonly used storage media nowdays. I stress “part of data encoding”. It’s no longer a mechanism to secure data against a malfunction, but an inherent part of how the signal is stored. That’s how we can both utilize media otherwise unsuitable for reliable data storage and push the limits of data density. I would say that for optical disks read errors are a normal mode of operation.

And ECC would be of minimal help in this case. USB sticks usually either work or become completely dead. Parchive may be useful for media that may experience partial data loss, while the access is retained. Optical media are one of the examples: unless deformed, making them unreadable requires more effort than most people imagine.
« Last Edit: April 11, 2022, 10:52:48 am by golden_labels »
People imagine AI as T1000. What we got so far is glorified T9.
 

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #13 on: April 11, 2022, 10:47:18 am »
Code: [Select]
end_request: I/O error, dev sda, sector 5652192
quiet_error: 51 callbacks suppressed
Buffer I/O error on device sda, logical block 5652192
Buffer I/O error on device sda, logical block 5652193
Buffer I/O error on device sda, logical block 5652194
Buffer I/O error on device sda, logical block 5652195
Buffer I/O error on device sda, logical block 5652196
Buffer I/O error on device sda, logical block 5652197
Buffer I/O error on device sda, logical block 5652198
Buffer I/O error on device sda, logical block 5652199
Buffer I/O error on device sda, logical block 5652200
Buffer I/O error on device sda, logical block 5652201
sd 4:0:0:0: [sda] Unhandled error code
sd 4:0:0:0: [sda]  Result: hostbyte=0x01 driverbyte=0x00
sd 4:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 56 3f d0 00 00 10 00
end_request: I/O error, dev sda, sector 5652432
usb 1-1: new high speed USB device number 7 using ehci_hcd
usb 1-1: device descriptor read/64, error -110
usb 1-1: device descriptor read/64, error -110
usb 1-1: new high speed USB device number 8 using ehci_hcd
usb 1-1: device descriptor read/64, error -110
usb 1-1: device descriptor read/64, error -110
usb 1-1: new high speed USB device number 9 using ehci_hcd
usb 1-1: device descriptor read/8, error -110
usb 1-1: device descriptor read/8, error -110
usb 1-1: new high speed USB device number 10 using ehci_hcd
usb 1-1: device descriptor read/8, error -110
usb 1-1: device descriptor read/8, error -110
hub 1-0:1.0: unable to enumerate USB device on port 1
usb 3-1: new full speed USB device number 2 using ohci_hcd
usb 3-1: device descriptor read/64, error -110
usb 3-1: device descriptor read/64, error -110
usb 3-1: new full speed USB device number 3 using ohci_hcd
usb 3-1: device descriptor read/64, error -110
usb 3-1: device descriptor read/64, error -110
usb 3-1: new full speed USB device number 4 using ohci_hcd
usb 3-1: device descriptor read/8, error -110
usb 3-1: device descriptor read/8, error -110
usb 3-1: new full speed USB device number 5 using ohci_hcd
usb 3-1: device descriptor read/8, error -110
usb 3-1: device descriptor read/8, error -110
hub 3-0:1.0: unable to enumerate USB device on port 1

booooom, found a third usb-pendrive with dead blocks  :o :o :o
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #14 on: April 11, 2022, 11:01:05 am »
The last defective one USB-stick was included in a MIPS64r2 Cavium Octeon router with 512 MB of RAM, which uses a removable USB pendrive for storage(1).

Code: [Select]
Bus 002 Device 012: ID 13fe:3e00 Kingston Technology Company Inc. Flash Drive

Dead after 5 years from the purchase of my EdgeRouter Lite ERLite-3.

It's not a problem, I have a backup, just ... first I have to find where the router is located - on which floor? and on which corner of the lab? and under which heavy equipment?!? - ... access the router, and replace the pendrive.

bug life  :o :o :o :o


edit:
(1) the pendrive contained a copy of the rootfs, but was never mounted rw, the rootfs was always loaded from the pendrive into the ram and executed in ram, therefore, here it is a another example of flash that dies not because it consumes its writing-cycles but rather only because NAND-flash-cells age.
« Last Edit: April 11, 2022, 11:08:22 am by DiTBho »
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online Someone

  • Super Contributor
  • ***
  • Posts: 4510
  • Country: au
    • send complaints here
Re: let's face it: usb-sticks are not reliable
« Reply #15 on: April 11, 2022, 12:50:52 pm »
And ECC would be of minimal help in this case. USB sticks usually either work or become completely dead.
Not my experience at all, having "ancient" (20+ years old) flash drives sitting around here that still work perfectly, and have been able to hold data for 10+ years ok. The only complete failures I have seen with flash storage has been mechanical failure of the connector/enclosure/board. On the other hand, files (and occasionally file systems) do go corrupt with annoying regularity. They would be protected with better ECC codes, that file corruption failure mode is a trade-off between space/performance and reliability.

(1) the pendrive contained a copy of the rootfs, but was never mounted rw, the rootfs was always loaded from the pendrive into the ram and executed in ram, therefore, here it is a another example of flash that dies not because it consumes its writing-cycles but rather only because NAND-flash-cells age.
Yep, it happens. Hence "high" reliability systems reading through the (duplicate/ECC diverse) flash and refreshing/moving the data to maintain life.
 
The following users thanked this post: DiTBho

Offline cdev

  • Super Contributor
  • ***
  • !
  • Posts: 7350
  • Country: 00
Re: let's face it: usb-sticks are not reliable
« Reply #16 on: April 12, 2022, 02:06:46 am »
Cuneiform documents are incredibly long lived, giving us amazing levels of detail on what was happening in the Middle East, say, 5000 years ago.

Do you know what I mean? Ive been getting into history a lot recently and its endlessly fascinating. 
"What the large print giveth, the small print taketh away."
 
The following users thanked this post: Circlotron, evb149

Offline golden_labels

  • Super Contributor
  • ***
  • Posts: 1183
  • Country: pl
Re: let's face it: usb-sticks are not reliable
« Reply #17 on: April 12, 2022, 02:59:40 am »
Not my experience at all, having "ancient" (20+ years old) flash drives sitting around here that still work perfectly, and have been able to hold data for 10+ years ok.
I am surprised. Not even by the mention of failure modes, but that they survived 20 years. Were they actually used in that period? Because, from what I observed, they die within months to years of active use. Twenty years for an USB stick sounds incredible. And I am saying that as a person, whose main HDD has so many hours on its clock, that timestamps in the SMART test log wrapped around.

Cuneiform documents are incredibly long lived, giving us amazing levels of detail on what was happening in the Middle East, say, 5000 years ago.
A the cost of having extremely low information density and bandwidth!  ;)
« Last Edit: April 12, 2022, 03:02:28 am by golden_labels »
People imagine AI as T1000. What we got so far is glorified T9.
 

Offline twospoons

  • Regular Contributor
  • *
  • Posts: 224
  • Country: nz
Re: let's face it: usb-sticks are not reliable
« Reply #18 on: April 12, 2022, 03:25:15 am »
To be fair, when you think about what a flash memory cell actually is, its pretty incredible it works at all. We're talking femtoFarad capacitors holding a few thousand electrons for years!
 
The following users thanked this post: tooki

Offline nigelwright7557

  • Frequent Contributor
  • **
  • Posts: 689
  • Country: gb
    • Electronic controls
Re: let's face it: usb-sticks are not reliable
« Reply #19 on: April 12, 2022, 03:39:18 am »
If you buy a decent make they work ok. I use Lexar and never had a problem with them.
I found some of the cheap Chinese drives dont work well.
 

Offline Berni

  • Super Contributor
  • ***
  • Posts: 4922
  • Country: si
Re: let's face it: usb-sticks are not reliable
« Reply #20 on: April 12, 2022, 06:43:57 am »
All NAND flash cells eventually loose charge and cause bit rot, even SLC ones. So flash is fundamentally unsuitable for archival storage.

Optical media is a bit better but still has bitrot issues. The marks on a CD-R can fade over time much like a thermal paper printout while the aluminum of factory stamped CDs can sometimes start corroding away if given just the right conditions.

Magnetic storage tends to be the most resilient to degradation. Even today data centers use magnetic tape cartridges for storing the worst case scenario backups. The modern tapes can hold many TB per cartridge, but are too niche for home use. For home use the only magnetic format remaining is a hard drive. Those work pretty well since hard drives have pretty long lifespans.

But ultimately data archival is a process not just a task. Resources have to be put in to regularly make and verify backups.
 
The following users thanked this post: david77, Someone, DiTBho

Offline david77

  • Frequent Contributor
  • **
  • Posts: 934
  • Country: de
Re: let's face it: usb-sticks are not reliable
« Reply #21 on: April 12, 2022, 07:53:57 am »
I get laughed at when people see I still use DLT for backing up my important data like projects, personal data and images.
 

Offline Berni

  • Super Contributor
  • ***
  • Posts: 4922
  • Country: si
Re: let's face it: usb-sticks are not reliable
« Reply #22 on: April 12, 2022, 09:54:51 am »
Ah didn't know people actually use data tape cartridges at home.

I personally use a NAS server full of hard drives for storing data long term. It is running software raid with pairty to require 2 drive failures before i loose data and since its Unraid i can still recover data from the remaining drives individually if the raid array goes belly up. My PCs then get weekly backups done onto that NAS server.(Using veeam agent)

I do plan to also start making yearly backups to external USB hard drives, as those can be easily physically separated to protect from lightning or ransomware. But i haven't got around to that.

An even better plan is to make an agreement with a friend to provide a few TB of backup space to each other. Then have the two NAS servers periodically backup each other over the internet. Never got around to that either.
 

Online Halcyon

  • Global Moderator
  • *****
  • Posts: 5631
  • Country: au
Re: let's face it: usb-sticks are not reliable
« Reply #23 on: April 12, 2022, 10:36:10 am »
Consumer-grade flash memory should never be relied on for a standalone backup solution. It's fine it you have other identical copies, but you should go into it with a mindset that one day, without warning, it will fail.

That being said, for the most part, I've found consumer flash memory pretty reliable. I have had some issues with Verbatim drives before, but brand alone is not a defining factor of quality. I've used an old Lacie flash drive for the past 13 years or so and despite it being USB 2.0, it's still reliable and never had an issue with it. The only reason I still use it is because it has an all metal casing and looks like a normal key, so it fits nicely into my key holder.

Backup often and backup across multiple devices.
 
The following users thanked this post: DiTBho

Offline DiTBhoTopic starter

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #24 on: April 12, 2022, 10:59:10 am »
yes, backups, but ... what about devices like my MIPS64r2 Cavium Octeon router? It uses a removable USB pendrive for its rootfs, and yesterday when I checked it demonstrated it's not reliable.

Thanks god the flash is not soldered inside the Router, it's a common USB pendrive, which I will probably replace with a USB 1.6" Hard-disk, but what about the soldered flash used in GPS watch like the Garmin Fenix?

We are talking about 32GB of flash for
- its firmware (you can update it)
- its maps (you pay for them, and you download them into the watch)
- your fitness data, ok, here you upload everything to the computer

but, you buy the watch and back-up everything weekly and if the flash on the watch dies ... will you be ready to trash the watch and buy a new one?

This morning I checked the flash of my other old router (2009). In this case the rootfs is inside an 8MByte flash soldered on the PCB. Found three dead blocks, they make the splash filesystem corrupted and the boot block is also no good, thank god the flash has a Soic package and the PCB allows you to unsolder and replace the flash chip, but what about the BGA packages and PCBs dense?!?

The same for smartphones, your iPhone, your iPad, Android-phones, etc ... uhm, it's like saying no more than 4-5 years


The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf