Author Topic: let's face it: usb-sticks are not reliable  (Read 4823 times)

0 Members and 1 Guest are viewing this topic.

Offline Berni

  • Super Contributor
  • ***
  • Posts: 4301
  • Country: si
Re: let's face it: usb-sticks are not reliable
« Reply #25 on: April 12, 2022, 11:09:40 am »
Yeah flash for firmware storage is a genuine concern.

I do have a old Android phone that likely had its flash rot away. Pretty sure it worked fine when it stopped being used, but once i dug it up figuring it would be a good backup phone, i turned it on and it just would not boot. Didn't try to reflash it or anything since its so old anyway.

Some of the Keysight X2000 and X3000 scopes had an issue where the firmware would rot away enough to make the scope not boot. The fix being sending it back to the factory for a warranty repair.

Tho that brings up the question how reliable is the firmware memory on hard drive controllers? If that rots away then all your data on the hard drive is also effectively gone.
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #26 on: April 12, 2022, 11:44:48 am »
Tho that brings up the question how reliable is the firmware memory on hard drive controllers? If that rots away then all your data on the hard drive is also effectively gone.

brrrr, I feel chills on the back, as if there was a cat ready to claw.
Those flash-things look like walking corpses walking on tiptoes.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #27 on: April 12, 2022, 11:49:40 am »
A photo of my MIPS64 router, kernel 3.* era, the corpse of its dead USB-Pendrive lies there, on the left.
[attachimg=1]
 

Offline Halcyon

  • Global Moderator
  • *****
  • Posts: 4903
  • Country: au
Re: let's face it: usb-sticks are not reliable
« Reply #28 on: April 12, 2022, 11:54:24 am »
yes, backups, but ... what about devices like my MIPS64r2 Cavium Octeon router? It uses a removable USB pendrive for its rootfs, and yesterday when I checked it demonstrated it's not reliable.

Short-sighted design at-best. Avoid at all costs. If I see that, it screams "failure" to me.
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #29 on: April 12, 2022, 12:07:46 pm »
Short-sighted design at-best. Avoid at all costs. If I see that, it screams "failure" to me.

Eh, your suspicions are well founded even on the firmware side: since kernel 2.6.* there always has been a high chance of kernel panic during soft reboots of device due to some nasty bugs in Octeon USB driver, and I've spent weeks of my life fixing things.

When I bought it, a device like that router was the only MIPS64 device available on the market, I have a couple of projects that require that hardware profile and for other reasons I couldn't use Qemu so I had no choice.
 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 14758
  • Country: us
  • DavidH
Re: let's face it: usb-sticks are not reliable
« Reply #30 on: April 12, 2022, 05:52:00 pm »
I have had the same experience with USB Flash sticks.  I tested various almost new units several years ago and all displayed retention times of less than a year whether powered or unpowered.  Since then, I rely on SATA SSDs mounted in a USB enclosure so that I can rely on scrub on read and scrub on write to at least periodically refresh the data.
 
The following users thanked this post: Nominal Animal, DiTBho

Online Ed.Kloonk

  • Super Contributor
  • ***
  • Posts: 3267
  • Country: au
  • Cat video aficionado
Re: let's face it: usb-sticks are not reliable
« Reply #31 on: April 13, 2022, 01:59:06 pm »
I'll just leave this here.
 
The following users thanked this post: tooki, DiTBho

Offline Halcyon

  • Global Moderator
  • *****
  • Posts: 4903
  • Country: au
Re: let's face it: usb-sticks are not reliable
« Reply #32 on: April 13, 2022, 10:36:16 pm »
Short-sighted design at-best. Avoid at all costs. If I see that, it screams "failure" to me.

Eh, your suspicions are well founded even on the firmware side: since kernel 2.6.* there always has been a high chance of kernel panic during soft reboots of device due to some nasty bugs in Octeon USB driver, and I've spent weeks of my life fixing things.

When I bought it, a device like that router was the only MIPS64 device available on the market, I have a couple of projects that require that hardware profile and for other reasons I couldn't use Qemu so I had no choice.

Ubiquiti was one company that learned the hard way with some of their products, I think from memory it was the EdgeRouter Lite and some of their other "consumer" grade devices that used a normal USB flash drive for firmware and there were all kinds of problems with corruption. Since then, they've mostly done away with them (I don't know any of their products that still use flash storage for system software).

 
The following users thanked this post: thm_w, DiTBho

Online thm_w

  • Super Contributor
  • ***
  • Posts: 4167
  • Country: ca
  • Non-expert
Re: let's face it: usb-sticks are not reliable
« Reply #33 on: April 13, 2022, 11:54:19 pm »
Even ignoring corruption, we used thousands of consumer grade USB drives (sandisk, kingston, etc.) and enough would just flat out die that it was irritating. Plug into a computer and it could not be formatted.
Switched to SLC USB drives, and no issues as of yet: https://industrial.apacer.com/en-ww/SSD/EH353

There is a lot more competition in the microSD space due to dashcams, with samsung/sandisk/transcend all having high endurance cards.
https://ripitapart.com/2020/07/16/reverse-engineering-and-analysis-of-sandisk-high-endurance-microsdxc-card/

It looks like in these cards they use 3D NAND and pseudo SLC to improve reliability. I would definitely consider using one in a consumer grade product over soldered in NAND.
Profile -> Modify profile -> Look and Layout ->  Don't show users' signatures
 
The following users thanked this post: DiTBho

Offline Berni

  • Super Contributor
  • ***
  • Posts: 4301
  • Country: si
Re: let's face it: usb-sticks are not reliable
« Reply #34 on: April 14, 2022, 05:53:33 am »
SD Cards have another problem in that they like to garble large areas due to unexpected power loss.

The cards controller chip shuffles pages around as a form of wear leveling. Unfortunately it does it in a way that destroys data if the process is interrupted. This includes writes to the area containing the filesystems file tables.

The effect is a common cause of bricking in RaspberryPis as the default linux image directly writes to the card. Eventually you manage to pull power from a RPi at a crucial moment, the card eats up the file table and on next power on the Pi can't boot anymore from the corrupt filesystem.

What makes it even worse is that SD cards generally won't have TRIM support, so when swapping out blocks for wear leveling it might swap out block from different parts of the card. So if you tried to get around the problem by having 2 completely separate partition tables on a card, carrying completely separate filesystems, using the first one as ROM and second one as storage, then you could still get burned. The card might wear level by swapping blocks between the two filesystems and then garble them upon power loss.
 
The following users thanked this post: thm_w, DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #35 on: April 14, 2022, 06:07:25 am »
@Halcyon
yup.

Ironically I did a similar mistake to resurrect a consumer AR23 router made in 2008.
It has only 32MByte of ram and only 8Myte of soldered NAND-flash, which contains
- bootloader (64Kbyte)
- booting settings (32Kbyte)
- linux kernel 2.6.1*   (~2MByte)
- splash rootfs, uclibc-diet based (~5MByte)
- wifi calibration settings (2Kbyte)

After 10 years, the flash-partition containing the splash rootfs gonna corrupted because a couple of NAND-blocks died.

(no matter what you write to those blocks, you will always read-back 0xff --> bye bye flash cells)

So I desoldered the SPI-flash chip for a fresh one, but since the firmware was unable to address more than 8Mbyte of space (d'oh), then I made a different setup:
- (first-stage) bootloader (64Kbyte) <------------ closed source, I cannot touch it
- booting settings (32Kbyte)
- (second-stage) usb-boot-loader (800Kbyte) <------------ my project, to bootstrap from USB
- wifi calibration settings (2Kbyte)


(first-stage) bootloader --->  (second-stage) usb-boot-loader ---> kernel boot

the first stage loads the second stage, which comes with a partial USB2-EHCI-bulk implementation able to
- understand partitions on a USB-bulk device (usually a pendrive, but here you could attach a usb-disk)
- load into ram the first partition (there is no filesystem, it's a wild and raw-read-and-load-into-ram)
- clean the cache and reset the EHCI controller
- understand the elf header of the Linux kernel, at least understand which address to jump to
- jump into that address in ram to start the kernel

So now there is a common USB-pendrive attached to the router, and it's from where the rooter bootstraps and then mounts the rootfs.

Ugly design, prone to failure, no doubts about  :-//
« Last Edit: April 14, 2022, 06:11:56 am by DiTBho »
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #36 on: April 14, 2022, 06:31:06 am »
Code: [Select]
exec
Base address 0x80030000 Entry 0x803a4210

second stage
initializing usb-EHCI ... success
found usb-bulk device, reading kernel.elf@/usb/disk0/partition0 ... done
24944 Kb available
_9541 Kb required
checking elf-header ... success
jumping into kernel_start
 

Offline Berni

  • Super Contributor
  • ***
  • Posts: 4301
  • Country: si
Re: let's face it: usb-sticks are not reliable
« Reply #37 on: April 14, 2022, 06:32:49 am »
That sounds like a LOT of effort to fix a little home router.

I probably wouldn't have gone much further than measuring all the supply rails before declaring it an uneconomical repair.
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #38 on: April 14, 2022, 07:04:29 am »
That sounds like a LOT of effort to fix a little home router.

yup, the whole resurrection took five months of work, scheduled during weekends when the weather was too bad to go outside; for sure an uneconomical repair, even because the CPU has no FPU, no crypt- instructions, and it's clocked at 150Mhz, so it also has little computing-power.

But it's my wheelbarrow-router, I mean a toy if compared to what you can buy and use today, but a nice toy to play with.

In 2008, I paid 50 euro for that router, and it was the only cheap - but - hack-able router around, also my first true experience with an embedded-Linux device when things like RPI were not yet existing, ... it's no more supported by the Linux Kernel, so I had to spend months to resurrect the kernel, but it's a simple SoC, simpler than everything is today used, that's why I like it, because I know it very well and it's simple.
« Last Edit: April 14, 2022, 06:38:02 pm by DiTBho »
 
The following users thanked this post: Ed.Kloonk, cdev

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 10177
  • Country: fr
Re: let's face it: usb-sticks are not reliable
« Reply #39 on: April 14, 2022, 05:31:26 pm »
Anyway, do no blindly trust *any* storage support for long-term data retention. Your best bet for long-term retention is data replication. So, replicate your data on a regular basis.

Flash-based storage can be anywhere from good to very bad. The Flash chips themselves can sometimes be obsolete tech (especially in those cheap usb sticks), the controllers can be crap and buggy as hell (indeed again especially in those cheap usb sticks). Using an external SSD is a better option, but do no trust it blindly especially if you don't access it very often.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #40 on: April 17, 2022, 06:45:14 am »
Speaking of "flash not reliable", Amiga has several accelerating cards, one of these adds a Motorola 68060 CPU. It's commercially called "CyberStorm Mk2", made by the Germany company Phase 5 Digital Products in 1996 for Amiga 3000 and 4000, it has a Flash ROM which contains the updatable  firmware needed to update for the 66 MHz design, and for installing the SCSI module ...

... and do you guess what?

It was re-designed in the early 2000s and twenty years later that flash is died ... you need to replace the whole chip with a fresh one.

Thanks god I had a copy of the flash-image. Last night I finished repairing for a friend of mine who was literally crying when he saw his Amiga fail to boot and realized the problem was the Cyberstorm-060.

I can understand him, you cry because the card is rare, and it has always been very very expensive from the start.

I'm seriously starting to * HATE * flash  :o :o :o
 

Offline nigelwright7557

  • Frequent Contributor
  • **
  • Posts: 550
  • Country: gb
    • Electronic controls
Re: let's face it: usb-sticks are not reliable
« Reply #41 on: May 07, 2022, 12:18:33 am »
I recently bought a 256GB flash drive.
It was cheap so wasnt expecting miracles.
It sort of worked but kept coming up with an error on a USB 3.0 port near the end of saving many GB of data.
So I swapped it to a USB 2.0 port and the problems went away.
It now reliably saves many GB no problem.

 

Offline David Hess

  • Super Contributor
  • ***
  • Posts: 14758
  • Country: us
  • DavidH
Re: let's face it: usb-sticks are not reliable
« Reply #42 on: May 07, 2022, 01:00:55 am »
Flash-based storage can be anywhere from good to very bad. The Flash chips themselves can sometimes be obsolete tech (especially in those cheap usb sticks), the controllers can be crap and buggy as hell (indeed again especially in those cheap usb sticks).

I just had another "new" USB Flash stick die.  I replaced it with a Samsung unit and will see how that works.

Quote
Using an external SSD is a better option, but do no trust it blindly especially if you don't access it very often.

I have a pair of Crucial M500 SSDs in USB3 enclosures.  I periodically replicate to the backup, and then do a complete comparison using hashes.  Reading both drives to generate the hashes should force a scrub on read before any data is lost.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #43 on: May 07, 2022, 10:26:15 am »
Yesterday night I powered on my old MIPS-laptop, and bOOOOm cannot boot!

is it the RTC? no, it's not
is it the NVRAM? no, settings are all rights
is it the PSU? no it's working
so what is it?

... oh, see, there is no bootoader on the SSD
... oh, let's netboot a first-aid kernel + rootfs
... oh, see, there is not even a partition map
... oh, see, fdisk says the disk has a capacity of zero Kbytes
... oh, WTF?!?

Why was I so surprised?!?  Why am I still so surprised!?! Because the SSD is not disappeared, it's still there to respond to queries but it's recognized on the sATA line with _ZERO_ Mbyte ... as if the content has evaporated, which is what makes it insanely crazy to me.

What did it happened in the last seven months? I didn't even use the laptop, flash suffers data-leak-aging, but what happened looks rather data drastically fast evaporation, I mean a puddle evaporates in a week, it's how Chemical works on Earth, but here it's like if you see an ocean evaporated in half a year.

64Gbyte of data gone, it shouldn't be normal  :o :o :o

Anyway, no problem there is a backup, and there is also a new SSD drive (SAMSUNG EVO, this time), but hey? I really have a crazy bad luck with flash  :-//
 

Offline cdev

  • Super Contributor
  • ***
  • !
  • Posts: 7350
  • Country: 00
Re: let's face it: usb-sticks are not reliable
« Reply #44 on: May 07, 2022, 12:18:24 pm »
That has happened to me too.. not a lot but enough to make me wary of flash memory. Ive been able to recover the data however.. with much difficulty.

SSDs are a cut above.. though.. usually.
"What the large print giveth, the small print taketh away."
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #45 on: May 07, 2022, 09:53:30 pm »
Lesson learned here, flash-sticks and SSDs failure can happen at any time even without warning, and a complete failure could come in a few minutes, a few months, or, in some cases, even a few years.

Even if you don't use the drive for months.

SMART doesn't even report an error. The registry is clean. PowerOnHours reports around 9,000 hours, not so impressive value, but the flash is fried like french fries.

Still, capacity of ZERO byte  :o :o :o

Probably crappy flash, or crappy algorithms managing that crappy flash. Who knows. For sure, a bad SSD.

RIP!
 

Offline nigelwright7557

  • Frequent Contributor
  • **
  • Posts: 550
  • Country: gb
    • Electronic controls
Re: let's face it: usb-sticks are not reliable
« Reply #46 on: May 15, 2022, 12:56:11 am »
I recently bought a 256GB flash drive.
Had all sorts of problems with m new pc with it.
Lots of errors.
So out of interested swapped it from usb 3 port to usb 2.0 port and errors magically disappeared.
Been ok since.
 

Offline xrunner

  • Super Contributor
  • ***
  • Posts: 6517
  • Country: us
  • hp>Agilent>Keysight>???
Re: let's face it: usb-sticks are not reliable
« Reply #47 on: May 15, 2022, 01:33:28 am »
I needed to put a bootable Linux image on a USB stick. Had some El Cheepo 16 GB sticks that came in a pack of five. I tested them and they worked fine copying files to them. Plenty big enough for the Linux image but I couldn't make a bootable image with any of them to save my life. Bought a new Sandisk USB stick and it worked the first time.

 :-//
[hp] Hewlett . Packard
 

Online AndyBeez

  • Regular Contributor
  • *
  • Posts: 218
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #48 on: May 17, 2022, 08:46:00 pm »
My very first 'floppy-buster' in the Windows 98 era was a meagre 128Mb and cost a week's salary. Now 16Gb drives are so last decade they reside in the discount aisle with the $1 phone chargers. Moores Law probably assumes that a density increase will equal static reliabilty values. But when chip houses are trying to meet stupid low retail price points, this might not be the case. Maybe the first generation of gigabyte drives will have the shortest shelf life, because no-one assumed they would outlast the one year warranty period? I'm not going to get into an argument about NAND versus NOR, but you get the idea. My 128Mb still works but later 4GB devices from 2005, went under the engineering hammer of doom a long while ago.
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 2093
  • Country: gb
Re: let's face it: usb-sticks are not reliable
« Reply #49 on: May 18, 2022, 04:54:00 am »
My 128Mb still works

I remember them, as well as iREV and DVD-ram.
Both still works here, other things are pretty dead.

But this damn flash is a bloody epic failure, not only the one used for cheap massive-storage (USB pendrives), but also the one used for firmware storage.

That's what I cannot accept: things like the Amiga Cyberstorm with dead firmware-flash!
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf