EEVblog Electronics Community Forum
Electronics => Projects, Designs, and Technical Stuff => Topic started by: DanInvents on September 21, 2023, 07:06:19 am
-
Hi!
Currently I'm thinking about upgrading my Rockit flight computer (https://www.tindie.com/products/25820/ (https://www.tindie.com/products/25820/)). Rockit is a compact flight computer that logs altitude, acceleration, and deploys parachutes for rocket recovery.
Back on June 15 (video https://youtu.be/GfuPt991mbA?si=3wq6zAegmOSN-e0R&t=186 (https://youtu.be/GfuPt991mbA?si=3wq6zAegmOSN-e0R&t=186)) my rocket crashed and the power from the board was cut-off while data was being saved. The microSD card ended up being corrupted and unrecoverable.
Is there a clever circuit or solution that could prevent this from happening? I would like to keep the board size small and this would rule out adding backup batteries.
I am also considering using a solid-state memory to replace the microSD card thus reducing the board size. If power is interrupted when data is being written, would it also corrupt the memory IC?
Thanks!
Daniel
-
Zero out and reformat the card before every use, then it would be easier to use forensic tools to recover the data.
-
How are you storing the data, and what filesystem are you using? Choosing an appropriate (journeling, not FAT) filesystem and file format (I would recommend sqlite) could make a big difference to the recoverability of the data. Alternatively, if the data is small enough, you might want to consider using an append only log in EEPROM, which hs small enough write sizes that you could avoid having more than one row per write.
-
Rolling your own system to save the data which does not require a filesystem is one approach.
There's nothing to recover if you know exactly where and how the data is stored on the flash chip. It's either there or its not.
There's no filesystem tables to get corrupted
-
If the data block that's being written to suffers a power loss while the data is being written then it will be corrupted. This is going to happen with any technology. So what you want is something that writes to small blocks. No simple circuit will prevent this as the simple circuit (or even a battery) will also eventually lose power. You best option might to be transmit the data and receive the data on the ground by something with a reliable power supply.
In your case it could also help you find the rocket in case of another mishap.
See this: https://www.reddit.com/r/rocketry/comments/fi3omt/long_range_communications_on_model_rockets/ (https://www.reddit.com/r/rocketry/comments/fi3omt/long_range_communications_on_model_rockets/)
-
A lot depends on what microSD is used. Some are quite robust in this regard. IME Sandisk cards were very good at this. Others will often utterly corrupt the data after a just few power losses.
-
"Corrupted and unrecoverable" would probably mean that the directory entry or the FAT tables were corrupted. But even so, was the data unrecoverable even with a hex editor?
Well as Psi suggested, you could write directly to sectors on the card, and not have a file system as such. Or you could use FAT32 and create a big file in advance, then write directly to the sectors of the data area of that file. The file would still read as a file on your computer, and only the last bit might be lost if power fails. That would require that the file's sectors be sequential, so you would want to erase the card, then create the new file. Your MCU should be able to do that. The Arduino SdFat library has examples that do that.
-
You might want to consider ferroelectric RAM (https://www.mouser.be/c/semiconductors/memory-ics/f-ram/). Compared to EEPROM or flash, it has faster write speed and orders of magnitude better endurance.
It normally comes with I2C or SPI interfaces, so can easily replace µSD.
As was said, you don't need a file system for this, but nothing could stop you from adding another, mostly unnecessary, level of complexity.
-
Thank you all for the comments!
To put things in context, I am using the SD.FAT library and the microSD card is formatted as FAT32. I am currently writing in blocks and this way I optimize the writing speed to write "as fast as possible". I also think that it is worth mentioning that I was using a cheap microSD card from Aliexpress and that might be the reason why it is unrecoverable.
When I plug the microSD card into my PC the device is not even recognized by the device manager. Even using a recovery program has not helped. I am not technically savvy when it comes to microSD card recovery, and I think that it is better to move on and try to avoid the issue from happening again.
I like the simplicity of having on-board data storage and I'd prefer to avoid beaming data to a ground station. Future customers should find the flight computer easy to use and this means ease to visualize flight data with a PC.
-
It would probably be practical to have enough capacitance to backup the supply to ensure a write completes, but you'd ideally need some co-operation between the filesystem software and the backup hardware, so it doesn't attempt to start a write when insufficient power is available.
-
It would probably be practical to have enough capacitance to backup the supply to ensure a write completes, but you'd ideally need some co-operation between the filesystem software and the backup hardware, so it doesn't attempt to start a write when insufficient power is available.
Yes, I was thinking along those lines. I was thinking about having the microcontroller monitor the supply voltage and once it drops below a certain threshold close the file. The current draw when the board is active is about 70 mA. In the current design the microcontroller has only a few microseconds to close the file before the power is cut out.
-
Backup power rails with a supercapacitor, probably around 0.22F, and have the MCU regularly measure power supply voltage, or use a supervisor IC that can generate an interrupt that the microcontroller uses to flush all filesystem writes, and then which then disables writing to the card when the voltage drops low enough, so that your card has time to finish erasing blocks and writing the data.
A tip is to take that card and erase it, and then use something like DD to write every flash block with FF, so that the flash controller does not have to erase blocks before write, though you will need to also use only SLC flash, so that you do not have the unknown response time of the SD card controller waiting to write the hidden cache inside the controller, and then it deciding to switch from writing SLC data for speed and small cells, and do the background task of converting the SLC blocks into MLC blocks. That is a very variable time, and the card will need constant power to do this, as the only thing it shows to the external interface is a slow write rate, as it spends lots of time shuffling blocks internally.
-
A tip is to take that card and erase it, and then use something like DD to write every flash block with FF, so that the flash controller does not have to erase blocks before write,
If you erase the card, you should end up with all FFs, and there should be no need to erase again before writing. It's not clear what state the sector is left in if you overwrite it with FFs. I suspect it will show as occupied with data, and require an erase before writing again. The formatting example in SdFat does the full erase, then formats the card, but does not overwrite. I suspect that is the optimum practice.
I'm not sure that any added capacitance would have made a difference in this case. After all, the rocket crashed into the rooftop. If that happens, there's no way to guarantee the processor would have time to react even if power is still available. It sounds like this card was destroyed in the crash if it isn't even recognized by the computer, but it's hard to say that happened because the processor was trying to write to it then. It could have been some other cause.
I'm not sure it makes sense to try to mitigate this possibility. Maybe if it crashes, all bets are off. But I suspect it would still survive most of the time. I'm just not sure there's anything you can do to affect the outcome when the rocket crashes. Maybe your time would be better spent preventing the crash.
-
I like the simplicity of having on-board data storage and I'd prefer to avoid beaming data to a ground station. Future customers should find the flight computer easy to use and this means ease to visualize flight data with a PC.
There are these industrial SD cards from Mouser, which claim Sudden power-off (SPO) protection as one of the features.
https://www.mouser.com/new/micron-technology/micron-i400-microsd-cards/ (https://www.mouser.com/new/micron-technology/micron-i400-microsd-cards/)
Built-in features for defect and error management
- LDPC error correction code implemented
- Global wear leveling
- Bad block management
- Refresh mechanism for UECC prevention
- Sudden power-off (SPO) protection
Operating voltage of 2.7 to 3.6V
-
I like the simplicity of having on-board data storage and I'd prefer to avoid beaming data to a ground station. Future customers should find the flight computer easy to use and this means ease to visualize flight data with a PC.
There are these industrial SD cards from Mouser, which claim Sudden power-off (SPO) protection as one of the features.
https://www.mouser.com/new/micron-technology/micron-i400-microsd-cards/ (https://www.mouser.com/new/micron-technology/micron-i400-microsd-cards/)
Built-in features for defect and error management
- LDPC error correction code implemented
- Global wear leveling
- Bad block management
- Refresh mechanism for UECC prevention
- Sudden power-off (SPO) protection
Operating voltage of 2.7 to 3.6V
The card itself may be able to protect against the effects of poweroff on its own internal operations, but it has no knowledge of what the filesystem is doing, so can't protect against filesystem corruption.
-
It would probably be practical to have enough capacitance to backup the supply to ensure a write completes, but you'd ideally need some co-operation between the filesystem software and the backup hardware, so it doesn't attempt to start a write when insufficient power is available.
This is exacly how I see it implemented in a few systems. Take a 10F supercapacitor, or two in series and use power OR-ing. And have a power loss signal.
Maybe quality SD card helps as well. Swissbit ie. Could be cheaper than re-rolling your system.
-
Just don't change to using an IBM Microdrive in your rocket. :-DD :-DD
I joke, but they were actually more reliable than you'd think. I think because material strength doesn't scale. So by making the drive so small they became significantly more robust than large drives are.
(https://www.eevblog.com/forum/projects/circuit-to-protect-microsd-cards-from-getting-corrupted-on-power-cuts/?action=dlattach;attach=1880971;image)
-
I joke, but they were actually more reliable than you'd think. I think because material strength doesn't scale. So by making the drive so small they became significantly more robust than large drives are.
Really? ??? That's not my experience. Back in those days I used to work adjacent to a bunch of freelance photographers, and we found them horribly unreliable. Often they'd send them in and they'd died in transit. Because of their (relatively for the time) large capacity, photographers were in the habit of using one drive for an entire shoot. The only copy of some high-value photography was lost one too many times and so management banned the photographers from using them - flash cards or DVD-Rs only.
-
I suspect what happened is the NAND FTL itself got corrupted, and possibly with that the SD firmware too. The only way to prevent that is with more robust SD cards (e.g. industrial ones mentioned above) or enough backup power to finish the last write.
How are you storing the data, and what filesystem are you using? Choosing an appropriate (journeling, not FAT) filesystem and file format (I would recommend sqlite) could make a big difference to the recoverability of the data. Alternatively, if the data is small enough, you might want to consider using an append only log in EEPROM, which hs small enough write sizes that you could avoid having more than one row per write.
FAT is very good for recoverability as allocation is likely going to be completely linear in this use-case and the filesystem structures are simple enough that not many writes are required.
-
try testdisk, that's a great tool for recovery
-
Best way to keep all your data safe is to transmit all the data via a down-link during the flight to your ground station. Provided your rocket is not flying hundreds of miles away it should be easily doable..
-
YAFFS, use a 5V power supply, in your "blackbox" feed a big cpacitor with a diode, then have a linear regulator for the sd card, have low voltage detection on the 5V input, if it goes low do whatever the standard says to initiate shutdown.
-
Thank you all for the comments!
To put things in context, I am using the SD.FAT library and the microSD card is formatted as FAT32. I am currently writing in blocks and this way I optimize the writing speed to write "as fast as possible". I also think that it is worth mentioning that I was using a cheap microSD card from Aliexpress and that might be the reason why it is unrecoverable.
Your main problem is that SD cards as well as SSDs and USB Sticks are Managed Flash. Managed Falsh does no expose the flash blocks but only an interface of an internal microcontroller with its own firmware and RAM. It's basically a black box that you can't control. Data written to the device is buffered ininzternal ram and only written to actual flash when the internal controller thinks it's a good time now. There can be literally yseconds between receiving data and writing them to flash - you simply don't get to know it.
The other kind is raw flash - either throgh SPI or parallel interface. With raw flash you have to do more on your controller (e.g. bad block management and wear levelling), but you have full control. There are no hidden buffers except for a row buffer. When the flash says a block is written it is safe. This might be more suitable for some applications.
-
If you're making the circuit board from scratch and writing the code yourself, I would use a non-volatile memory chip and have the data go into predefined memory locations, then there is no chance of corrupting it, only the chance of the last piece of data having a value that is not correct. It would be as small and light as a micro-SD (possibly smaller and lighter if you include the SD interface) and more robust as there would be less chance of vibration dislodging it. I had a similar issue with a Raspberry Pi, so I made a UPS with an 18650 and an LTC4040 that will power the pi and screen for 1/2 an hour.
Of course, if you only need a second or two you could consider using a supercap...
-
If you're making the circuit board from scratch and writing the code yourself, I would use a non-volatile memory chip and have the data go into predefined memory locations, then there is no chance of corrupting it
There is a chance corrupting it. IC may write to wrong address when voltage is not sufficient for guaranteed correct operation but high enough that IC still operates.
-
I joke, but they were actually more reliable than you'd think. I think because material strength doesn't scale. So by making the drive so small they became significantly more robust than large drives are.
Really? ??? That's not my experience. Back in those days I used to work adjacent to a bunch of freelance photographers, and we found them horribly unreliable. Often they'd send them in and they'd died in transit. Because of their (relatively for the time) large capacity, photographers were in the habit of using one drive for an entire shoot. The only copy of some high-value photography was lost one too many times and so management banned the photographers from using them - flash cards or DVD-Rs only.
Not reliable in general, they were terrible compared with solid state storage.
But they were quite a bit more reliable than you'd expect a spinning disk with read/write heads to be while in motion in a portable device.
Carrying around something and subjecting it to movement while it's spinning and moving the heads and wiring data onto nanometer scale tracks seems like a terrible terrible idea that would never work. But they actually made a product out of it. Not a good product, but the fact that it worked at all in a portable environment is impressive.
-
If you're making the circuit board from scratch and writing the code yourself, I would use a non-volatile memory chip and have the data go into predefined memory locations, then there is no chance of corrupting it
There is a chance corrupting it. IC may write to wrong address when voltage is not sufficient for guaranteed correct operation but high enough that IC still operates.
In that case you would have a memory address with incorrect data which should be easy to discard, when I said there's no chance of corrupting it, I meant corrupting the whole device and rendering the whole thing unreadable (without using data recovery software) as has happened with the SD card. That low voltage situation should be easily avoidable by having a reset IC that prevents operation below a set voltage (UVLO).