Help me select an embedded filesystem....

[cgroen]

I need help....
Thru the years I have been using commercial filesystems for embedded systems and products. Now has the time come to use "something" else for many reasons (the system we have used until now is pretty old and not optimum).

What I'm looking for is something that will allow me to use NAND devices (f.ex MT29F1G01 etc), so it should include wear leveling etc. I have been looking at littlefs among others but I'm afraid that littlefs is "too little", there are a number of restrictions as I read it as well as some potential problems (f.ex changing the header of a large file etc). I have of course also used FATFs for projects, this is ok as long as one requires FAT support (on SD cards for example) but for embedded systems that must be resilient to sudden poweroff conditions, its a no-go.
I have also looked at SPIFFS, but again I'm not sure this is perfect.

Regarding memory footprint, I don't care if it occupies 20 or 30 KB, and maybe 10 KB of RAM, I'm not after super small sized. Important is that it is well tested, can handle power down situations and be reasonable easy to port (and that it can support different storage types, NOR, NAND etc)

I know its a long shot, but perhaps you have the right suggestion ??

[krho]

F2FS from google.. but that's a linux filesystem, so this is probably not what you are after.

[cgroen]

Thanks krho,
sorry I forgot to mention that, you are correct, what I'm after is "standalone" system, something that has no requirement for linux etc.
I'm usually using a RTOS on my products, but I would rather find a system thats not relying on anything

I know its a tough one

[legacy]

FATFs for projects, this is ok as long as one requires FAT support (on SD cards for example) but for embedded systems that must be resilient to sudden poweroff conditions, its a no-go

why?

[legacy]

a lot of time ago, I developed this tiny project, then I switched to a completely different design 

Have a look, and feel free to reuse.

[cgroen]

FATFs for projects, this is ok as long as one requires FAT support (on SD cards for example) but for embedded systems that must be resilient to sudden poweroff conditions, its a no-go

why?

FAT is not very good at handling sudden powerdown, and FAT on a NAND device is also no good idea if I'm not mistaken (FAT table will be rewritten, wearing the flash)
FAT with some sort of journaling is another possibility maybe...

(funny little video and simulation for FATFs and littlefs here: https://os.mbed.com/blog/entry/littlefs-high-integrity-embedded-fs/)

[legacy]

FAT is not very good at handling sudden powerdown

well, we are using FAT in avionics. You are probably talking about a weird implementation, but the design of the FAT filesystem itself is so easy that you can complete every transaction without the need of cashing anything.

Novel NetWare is an example of FAT implementation where between a request and a IO-sync ... there might be up to 10 seconds (this was chosen in the 90s to speed up performances), which means ... you seriously needed a UPS.

[legacy]

NAND device

if you need this, you'd better look into the MTD/FlashDevices (NAND/NOR) subtree of Linux.

Anyway, you'd also better use a Linux router (10 Euro) as filesystem server, attached to the ethernet or USB-on-the-go so you can still use your RTOS without the problem of choosing and implementing a filesystem.

Personally, it's exactly what I did for my project: I moved the problem to Linux, for which we are plenty of cheap boards.

edit:
LOLLL, the old Commodore C128 implemented something similar, in fact, the 5.25" harddrive is an external standalone minicomputer that accepts requests on a special custom cable and provides answers.
Something like
C128: Hey? List all the files on the floppy
FloppyDriver: ok, here they are .... list[]={ ... }
C128: hey? Open this $filename, and move the pointer to the first block
FloppyDriver: ok, done
C128: hey? read 100 blocks of data
FloppyDriver: ok, here they are, block[]={block0...block98, EOL}, sorry there is no block#99
C128: ah, ok! Close the file.

and so on

Funny, because even some AS/400 computers by IBM do something similar 

[cgroen]

Its not an option to include a Linux router in these systems, its after all "embedded" (and some of the products are sold in 50K+ numbers
Some of the products are battery driven and left alone for years, some are connected to the internet using lwIP (and 4G modems) etc etc. I know it might be a long shot to find something that will "cure all" problems....

Regarding FAT, according to many sources, this is a no-go if you need something that can cope with powerloss (and still keep the filesystem intact _always_). At least I have not seen any that would do that, however I have seen a lot fail in this regard.

There is also this one: https://www.freertos.org/FreeRTOS-Plus/Fail_Safe_File_System/Reliance_Edge_Fail_Safe_File_System.shtml
Which is open source, and there is a possibility of a commercial license (which I have absolutely no problem in), it is more that I would like to be able to play around with it before handing over the money

[SiliconWizard]

https://yaffs.net/

[cgroen]

https://yaffs.net/

Thats not bad! Thanks for the pointer, I will have a very close look at that !

EDIT: although the "low memory footprint" is somewhat "high": https://yaffs.net/archives/yaffs1-memory-footprint

[JVR]

The solution is easy, use NOR.
I've used FAT systems on NAND, and paid the price. Unless there is a very specific need for large files, I'd much rather stick to smaller NOR devices. If you have to go NAND, read the documentation of the filesystem and wear leveling algorithms very closely, edgecases can bite you in the ass.

I ended up writing a pseudo-static filesystem with dual redundancy on NOR, and the test was 200 units on a contactor that was reset randomly within 100s for a week, not a single device with the NOR failed, every single FAT device (with logging) was dead.

[legacy]

well, supercapacitors are not so expensive nowadays and they can be used as a source of emergency energy to drive the MPU in order to correctly shutdown the filesystem.

sort of: ops, the main power supply has just failed, don't panic and let's raise an interrupt with the highest priority possible so the MPU will stop everything it's doing and use the emergency energy (which is limited to a couple of seconds before the voltage drops on the supercapacitor) to correctly write things on the media, write things on the journal, flush things, and shut down the filesystem.

why not?

[JVR]

True, and I've seen Linux based systems do this. I had the problem of not having the luxury of being able to solder to the ~9k devices installed in cars in the field. We used an expansion header to host the mezzanine board for the fix, so techs could just plug and play.

You make a good point though, but I'd still stay away from NAND unless I had too. Random bits going dark FTL.

[cgroen]

Thanks JVR,
I'm (besides NAND) currently using Dataflash (atmel/adesto) with ok results (using a commercial filesystem). I can though see a demand for larger filesystems in the near future so I'm not sure that the NOR route is the right one to go, although the higher stability is a very nice bonus....

The "super capacitor solution" is a no-go. Some of the devices this is going to be used in, has no room for stuff like that (especially size). I also have been bitten during the last 30 years a couple of times thinking that the hardware takes care of the powerfailure just perfectly, and then there was some corner case where it did not (temperature, batteries behaving strangely etc etc). I would rather have a filesystem that would not break regardless what the powerswitch was doing...

[JVR]

Dataflash behaves much the same as NOR, its stable, but small. You van get NOR up to around 64MiB for rather cheap these days. Well not cheap when compared to an SD card, but cheap for what you get.
Do note that SD cards and eMMC are not the way out, they were not designed to work with small files that change often, the wear leveling will kill you. If you really need more than 64MiB, I'd second yaffs, and say have  look at Micrium's file system, its very well written, but you do pay the price for it.

[NorthGuy]

If you're not going to move your storage device to other places, why do you ever need a filesystem. Just treat your device as a flat memory. You can use this memory to organize circular buffers, which is very handy for logs, or other storage schemes. If you have several tasks, divide your device into different areas and give each task its own area. This way there will be easy to make sure the data is intact in case of crash.

[nctnico]

FATFs for projects, this is ok as long as one requires FAT support (on SD cards for example) but for embedded systems that must be resilient to sudden poweroff conditions, its a no-go

why?

FAT is not very good at handling sudden powerdown, and FAT on a NAND device is also no good idea if I'm not mistaken (FAT table will be rewritten, wearing the flash)
FAT with some sort of journaling is another possibility maybe...

(funny little video and simulation for FATFs and littlefs here: https://os.mbed.com/blog/entry/littlefs-high-integrity-embedded-fs/)

You have to make a distinction here. SD cards, CF cards, SSDs, eMMC, etc are managed flash devices which do internal wear leveling. Using FAT on these is perfectly OK. Although you'd still won't have any journalling and recovery besides that shadow FAT table which should allow to roll back a mishap to some extend.

If you have an unmanaged (raw) flash device you'll need a layer which does error correction and wear levelling. You'd be looking at BCH error detection & correction and a wear levelling layer like UBI. On top of UBI you could use FAT or EXT4. There are several bootloaders (uboot, barebox) which have implementations of UBI and a fileystem in an embedded environment.

And easy solution would be to use FATfs on an eMMC device.

[cgroen]

Thanks for all the ideas so far
I need a "real filesystem" and not something that is too specialized. The products are used by customers as platforms for their own products (that they program them self) as well as for our own in-house products. Because of that I need a "normal" filesystem (open, close, write/read, subdirs etc).
The eMMC I have looked at before, but if I remember correctly, it was a relatively expensive solution? yaffs2 seems like a candidate as does reliance from datalight. But maybe there are others "out there"....

[nctnico]

How many units do you expect to sell? Getting the wear levelling layer and the filesystem on top of it running will take some effort which costs money (engineering hours). You'll also need to do extensive testing to see how a solution actually behaves. UBI combined with ext4 are quite resillient and have been thouroughly tested.

[cgroen]

How many units do you expect to sell? Getting the wear levelling layer and the filesystem on top of it running will take some effort which costs money (engineering hours). You'll also need to do extensive testing to see how a solution actually behaves. UBI combined with ext4 are quite resillient and have been thouroughly tested.

We are talking 100K+ per year (products are currently selling at that rate, this is for a new line of products/projects). But, at the same time I would also like this to work on small projects, basically I need a system that can be used in as many places as possible. I will have a look at UBI also, thanks for pointing that out!

[JVR]

What is the file read/write size and frequency? What is the total amount of data you expect to store?

[cgroen]

What is the file read/write size and frequency? What is the total amount of data you expect to store?

That is a good question! As the products are used by customers for their own application (on top of our API/operating system) it can be hard to say (some of the projects the products are used in we don't know the details of). But for the end products we do on the system ourself, it is typically a couple of updates (maybe 1 KByte) every 10 minutes or so. Space is typically in the 10 to 20 MBytes. But this can vary.

[Jeroen3]

I'm interested in what you will find out. FAT is generally a poor filesystem anyway. The only reason it's still around is for compatibility.

Yaffs looks promising. But there are also ext projects out there for bare metal systems.

[legacy]

I'm interested in what you will find out. FAT is generally a poor filesystem anyway. The only reason it's still around is for compatibility.

FAT32 has the great advantage of having low complexity, and it's the reason why it's chosen for embedded stuff.