The obligatory file system question...

[SiliconWizard]

Yeah this is a recurring topic. But let's see if there are new things to learn/experience to share.

So, what are good file systems these days to consider for a small system? Primary storage would be Flash - either QSPI, or SD cards - and typical code size would be under 50 KB, with RAM size use under 32 KB or so.

Options I considered so far: FatFs, littlefs, Yaffs.

littlefs looked promising but I've heard a few bad things about it. Now if you have more to share...
FatFs has the benefit of making the storage - in particular if using SD cards - easy to read and write on any platform.
Yaffs looks good, but a bit high on resource use.

[bson]

I use FatFs.  It's very simple to implement, designed for resource-constrained 8088 PCs to begin with, and for a use case with specific limited needs often simplifies greatly.  For example, if you read config from it but only ever write logs and never delete anything, and only ever write one file (the current log), then sector allocation is a simple incremental operation and you don't even need to cache any portion of the FAT (the contents of any FAT sector becomes deterministic).  It's been a while, but there's a header value somewhere that effectively can be used to track the next available sector.  (If not there you will likely have to do a FAT scan at mount time.)  Just generate and write updated FAT sectors whenever needed.  This makes it easy to implement, easy to test, and very light on its feet.

[cfbsoftware]

Is 4868 *bytes* small enough?  We had to impose a few constraints to achieve this with HCFiler. It was inspired by the way mag tapes were used in the 1960's / 1970's computers

• On the embedded system the filesystem is write-only.
• Disk writes are purely sequential.
• Only one file can be open for writing at any one time.
• Once a file is closed it cannot be written to again.
• New files are always located immediately after the last file to be created.

For more info:
http://www.astrobe.com/forum/viewtopic.php?f=4&t=534

[nctnico]

I use FatFs.  It's very simple to implement, designed for resource-constrained 8088 PCs to begin with, and for a use case with specific limited needs often simplifies greatly.  For example, if you read config from it but only ever write logs and never delete anything, and only ever write one file (the current log), then sector allocation is a simple incremental operation and you don't even need to cache any portion of the FAT (the contents of any FAT sector becomes deterministic).  It's been a while, but there's a header value somewhere that effectively can be used to track the next available sector.  (If not there you will likely have to do a FAT scan at mount time.)  Just generate and write updated FAT sectors whenever needed.  This makes it easy to implement, easy to test, and very light on its feet.

I second using FatFs. I have no experience with the other filesystems mentioned but I have used FatFs in several projects with good results.

[eutectique]

The client I am working for now had selected littlefs for their products. All of them use FOTA, if that matters. Haven't spotted anything suspicious so far.

I'm curious, what bad things have you heard?

[SiliconWizard]

The client I am working for now had selected littlefs for their products. All of them use FOTA, if that matters. Haven't spotted anything suspicious so far.

Thanks for the feedback!

I'm curious, what bad things have you heard?

Like 1/ still a number of bugs making it not very reliable, and 2/ maintainer(s) not being very responsive. But this was not first-hand feedback, so your input is appreciated.

[SiliconWizard]

Use cases may be useful, because it's often all in the context.

So, at the moment I have to main types of use cases from the POV of the embedded system:
- Mostly read-only: requires a lot of reads, but infrequent writes. Most files read in this context will have been written to outside of the system itself;
- Mostly write-only: frequent writes (typical of data logging and/or recording, such as audio), but infrequent reads. Writes should be efficient but also minimally prone to corruption in case of power loss.

In the first case, any file system/implementation that is not efficient for writes is not a problem, but reads must be efficient. Conversely for reads and writes in the second case. So both cases could be handled by a different file system or implementation if needed.

[eutectique]

Like 1/ still a number of bugs making it not very reliable, and 2/ maintainer(s) not being very responsive. But this was not first-hand feedback, so your input is appreciated.

This is good to know.

The way we use LFS is not very stressful. Infrequent write and read of parameters, occasional FOTAs. Perhaps, not much chance to face an issue.

[SiliconWizard]

Like 1/ still a number of bugs making it not very reliable, and 2/ maintainer(s) not being very responsive. But this was not first-hand feedback, so your input is appreciated.

This is good to know.

The way we use LFS is not very stressful. Infrequent write and read of parameters, occasional FOTAs. Perhaps, not much chance to face an issue.

Really so far I have no personal, or at least direct feedback. But you may just have a look at the project on github and look at the issues and see what kind of issues there are and how they are handled by maintainers. I still suggest you do some stress testing on it to be on the safe side.

[JOEBOBSICLE]

I've done well with little FS before. You need to make sure you're doing large writes instead of bytes at a time.

 I'm not sure if the amount of GitHub issues is a sign of success or failure, if you want a power safe filesystem then I think this is your only free choice tbh.

[SiliconWizard]

Is 4868 *bytes* small enough?  We had to impose a few constraints to achieve this with HCFiler. It was inspired by the way mag tapes were used in the 1960's / 1970's computers

• On the embedded system the filesystem is write-only.
• Disk writes are purely sequential.
• Only one file can be open for writing at any one time.
• Once a file is closed it cannot be written to again.
• New files are always located immediately after the last file to be created.

For more info:
http://www.astrobe.com/forum/viewtopic.php?f=4&t=534

Well, given the use cases I mentioned - surely not for more general-purpose use - this kind of approach could actually be the way to go.

Years ago, I designed a small read-only FS for Flash memory and SD cards. Would be created with a PC tool. Worked quite well. It was 400 odd lines of C code (.c and .h included) and about 6 KB of object code size (for a 16-bit target), with data integrity checking. Without it, it would have taken significantly less.

Likewise, a "data logging" kind of FS could be relatively simple too if data / files are guaranteed to be written in sequence with a few constraints. You'd get wear leveling for "free" as well.

Although the opposite of general-purpose, those approaches are certainly usable in the appropriate contexts.

For more general-purpose use, FatFs is certainly on my short list. It's been widely used, is quite stable, and supports FAT/FAT32/exFAT - a bonus if you need easy data exchange. One concern though is that FAT file systems are not robust and get corrupted easily. There are ways to use FatFs to minimize your chances of corruption, but it's not ideal.

[brucehoult]

Yeah this is a recurring topic. But let's see if there are new things to learn/experience to share.

So, what are good file systems these days to consider for a small system? Primary storage would be Flash - either QSPI, or SD cards - and typical code size would be under 50 KB, with RAM size use under 32 KB or so.

How big a "disk" do you want to use?

I don't know what was the best filesystem in the 1977-1981 time period out of Apple DOS, CP/M, FAT, or the UCSD P System (or others). They were all very basic and had low limits on the number of files, the size of an individual file, file name lengths, maximum disk size (without huge clusters) and so forth. Generally they didn't support subdirectories. Many of them didn't record the file length in the directory, so text files needed hacks such as storing a ^Z in the actual file.

Of all of those, FAT may or may not be the best, but it's definitely the only one that is still in widespread use and supported everywhere.

I imagine that you could make a really small but upward compatible version by dropping features such as directories and only supporting FAT12 (ideally under 2 MB, but up to 64 MB with 64 KB clusters) or FAT16 (fine for 2 MB to 2 GB).

No doubt someone already has. For example a FAT16 only library at https://github.com/greiman/Fat16/tree/master/Fat16

But maybe the full-fat FatFS library is fine. I don't know what size it compiles to. Internet sees to think people are using it on AVRs.

As much as I hate FAT in general, it doesn't seem worth investing the time to build or use something that isn't FAT-compatible.

[T3sl4co1l]

FYI, I've given FATFS a spin on AVR and it takes something like 10kB (including support for 64-bit integers because 4GB+ SD cards).
http://www.roland-riegel.de/sd-reader/doc/index.html

I... y'know, I'd say exactly what it is, I have the project here with object files sitting around, but because everything is spammed into a zillion sections other than plain-vanilla .text, .data, .bss etc., size just tells me it's empty. So, unless you have an idea how to calculate totals without linking it(..?), we'll just have to go on my recollection of what it added... which isn't great as I can't remember if it was 10 or 12k, or maybe I was thinking of the project total, or what it is down to the byte. Anyway, total on the link is close to 10k; YMMV with what version of GCC you're using.

Tim

[magic]

Writes should be efficient but also minimally prone to corruption in case of power loss.

Is there a risk of losing a whole page of data if you cut-off power to an SD card during writing?

Because if there is then updating FAT metadata will risk nuking a whole bunch of files, or at least a big chunk of the file being updated, if it's large enough and not fragmented.

edit
That being said, FAT is the default filesystem for SD cards and lots of cameras etc use it everyday with some degree of success. I think I have even heard that the initial sectors that typically hold FAT metadata may be made to be more durable than the bulk of the card, but don't quote me on that.

[brucehoult]

FYI, I've given FATFS a spin on AVR and it takes something like 10kB (including support for 64-bit integers because 4GB+ SD cards).

Yeah, so it says there the FAT16 code uses 7928 bytes of AVR code, and 188 bytes of static data.

That's on top of whatever is needed to physically read and write data to the storage, but that's independent of filesystem. I'd imagine the code would be smaller for ARM or RISC-V.

It says the author is using an AtMega168 with FAT16 but FAT32 needs a 328.

[NorthGuy]

SD and QSPI flash are totally different.

If you use SD, you most likely will have to read/write it on PC, so you typically would use a flavour of FAT which will be compatible with PC or with other devices where you're going to stick your SD in. The choice is not really yours.

If you use QSPI flash, you often don't need any file system at all. Say, for a data logger you may just write fixed-size records circularly. This way you always have your flash filled with the most recent records. Using file system will only add overhead and make everything less reliable.

Other time, you may need a file system in your QSPI flash. E.g. if you want a web server, you may use some sort of home-made read-only file system to fetch the appropriate pages. Or, if you device pretends to be MSD, FAT is a good choice.

There's no one-fit-all solution for using with QSPI flash.

[SiliconWizard]

Yeah this is a recurring topic. But let's see if there are new things to learn/experience to share.

So, what are good file systems these days to consider for a small system? Primary storage would be Flash - either QSPI, or SD cards - and typical code size would be under 50 KB, with RAM size use under 32 KB or so.

How big a "disk" do you want to use?

Up to several GB. I don't want a FS that is restricted to 2 GB, for instance.

But maybe the full-fat FatFS library is fine. I don't know what size it compiles to. Internet sees to think people are using it on AVRs.

For code size, I made a few experiments depending on configuration. Compiled it for x86_64 and also for ARM Cortex-M(7). Interestingly, code size for both targets is pretty similiar.

With the FatFs options I would need, I get ~18 KB (no exFAT support), and ~38 KB with exFAT enabled. That's absolutely not a problem. Compiled with the "read-only" option, this further cuts down size.
(Of course depends on optimization levels. Using '-Os', it's down to 24 KB with exFAT.)

For the record, compiling it with various options using "-Wall -Wextra" yields zero warning, which to me is a good point. Also ran Cppcheck on the code and it yields only a few, very minor warnings.

[SiliconWizard]

Writes should be efficient but also minimally prone to corruption in case of power loss.

Is there a risk of losing a whole page of data if you cut-off power to an SD card during writing?

Yes of course. The term used for SD cards is "blocks" rather than "pages", but it's just a matter of terms. The default block size is 512 on modern SD cards.

A few vendors claim - as some have said above - to have protection against data loss in case of power loss - but I don't know how effective that could be. We could only imagine that for that to work, the card would need to have sufficient capacitance on the power rail to allow completing a block write. I don't know how realistic that is given the size of the cards - in particular for microSD cards. SD cards (not microSD) usually have a whole PCB with bigger passives on it, so that looks already more doable.

Because if there is then updating FAT metadata will risk nuking a whole bunch of files, or at least a big chunk of the file being updated, if it's large enough and not fragmented.

There sure is.

edit
That being said, FAT is the default filesystem for SD cards and lots of cameras etc use it everyday with some degree of success. I think I have even heard that the initial sectors that typically hold FAT metadata may be made to be more durable than the bulk of the card, but don't quote me on that.

As you said, there's a difference between "some degree of success" and "reliable". Cameras and cell phones are big enough that they can embed all the capacitance that would be needed to successfully complete writes, and decent software does the rest. Now if you physically remove the card when it's writing blocks, that's something else.

FAT has been used with "some degree of success" in a very large number of applications, including computers for several decades, but that's definitely not a robust file system. It can corrupt very easily.

SD cards used as a commodity in consumer products is not a good example of reliability. As we pointed out, it's not even possible, apart from a few exceptions, to know if the card's controller does any kind of wear leveling, whereas it's an information you always get (and anyway, they all do) with SSDs.

[brucehoult]

SD and QSPI flash are totally different.

If you use SD, you most likely will have to read/write it on PC, so you typically would use a flavour of FAT which will be compatible with PC or with other devices where you're going to stick your SD in. The choice is not really yours.

That's not really true.

You can always put the SD card into your PC, tell it to ignore (not try to mount) it, and use dd to do a binary copy into a disk file. Then you can read it with probably the same C library you used to write it on the microcontroller.

[NorthGuy]

You can always put the SD card into your PC, tell it to ignore (not try to mount) it, and use dd to do a binary copy into a disk file. Then you can read it with probably the same C library you used to write it on the microcontroller.

Good idea.

[SiliconWizard]

SD and QSPI flash are totally different.

If you use SD, you most likely will have to read/write it on PC, so you typically would use a flavour of FAT which will be compatible with PC or with other devices where you're going to stick your SD in. The choice is not really yours.

That's not really true.

You can always put the SD card into your PC, tell it to ignore (not try to mount) it, and use dd to do a binary copy into a disk file. Then you can read it with probably the same C library you used to write it on the microcontroller.

I didn't first comment this NorthGuy point because it looked like unncessary digression, but you're of course right, here. I even explicitely mentioned that if you needed easy data exchange using SD cards, FAT/exFAT was the way to go, but there are many use cases in which you don't care about that. SD cards still have many benefits over QSPI flash in a range of applications; including an impossible to beat capacity/cost ratio, and the ability to replace them easily. And yes of course, as you mentioned, you can always write your own tools for reading and writing to them on a PC if needed - which is what I mentioned I did in one project. Not being able to easily read from and write to them without using the right tools can even be a plus in some applications.

[Berni]

ElmChans FatFs has always been my goto filesystem library. It is very well tested over the years and supports a pretty good set of features. Lots of the fancy features can be turned off easily with defines to compact it down some more.

But if you want it tiny he has a special Petit FatFs library that is cut down even more and designed to run on 8bit MCUs:
http://elm-chan.org/fsw/ff/00index_p.html

It might be a bit more annoying to use, but it is tiny for sure. Any smaller than that that likely means resorting to a raw format.

SD Cards indeed do not like having there power taken away when writing. It is the most common way SBCs like the raspberry get the filesystem destroyed and fail to boot after a bunch of power cycles. You can get more control in QSPI. The QSPI can also be advantageous for logging applications because you can write to them sequentially without the need to erase flash pages. This gives faster and more predictable write performance and is safer to power loss since only the last written chunk of data might get garbeled.

[cfbsoftware]

You can always put the SD card into your PC, tell it to ignore (not try to mount) it, and use dd to do a binary copy into a disk file. Then you can read it with probably the same C library you used to write it on the microcontroller.

You can also access the SD card as a raw device / physical device directly from your software on Windows in Administrator mode. We wrote a C# module to do this which we then call from Component Pascal using the following tip:

https://www.arsenal-of-wisdom.org/?p=766

You can always put a small dummy FAT partition at the beginning of the SD Card if you want to stop the PC complaining when you insert the card 

[SiliconWizard]

You can always put the SD card into your PC, tell it to ignore (not try to mount) it, and use dd to do a binary copy into a disk file. Then you can read it with probably the same C library you used to write it on the microcontroller.

You can also access the SD card as a raw device / physical device directly from your software on Windows in Administrator mode. We wrote a C# module to do this which we then call from Component Pascal using the following tip:

https://www.arsenal-of-wisdom.org/?p=766

You can always put a small dummy FAT partition at the beginning of the SD Card if you want to stop the PC complaining when you insert the card 

Yes and yes.

[Berni]

Actually that is a pretty good idea because you can use the small FAT partition at the beginning to hold an exe application for reading the data.

You get a readme,txt with instructions along with a exe that upon running checks what drive letter it is running from, checks what physical drive that letter is assigned to then opens that drive for raw access. That exe then gives you a way to browse trough the data and export it into some PC friendly file format like a CSV or something.

[abyrvalg]

But does your embedded application really need all “desktop” FS features? Directories, human-readable file names, timestamps etc. Many quite big systems (i.e. cellular basebands) running FSs stripped down on this side (numeric ids for file names, no dirs/timestamps/access rights), but with improved reliability/power loss tolerance (things like “write new record then mark old one as deleted” etc). Can’t recommend any name unfortunately, the only one I remember is ancient Intel’s FDI.

[SiliconWizard]

Of course not, as I clearly mentioned in this thread. There are many applications for which you absolutely don't need the FS to be easily accessible from a "PC" by the end-user. And just because we may be using SD cards (for reasons I mentioned earlier, there may be others) doesn't mean we do that to let the end-user easily read and write data to it directly.

But in cases for which the SD card is at least physically accessible to the end-user - and thus you know they may be tempted to take it out and read it on a PC - having a small FAT partition on it is a good idea. It avoids them accidentally formatting the card if they ever do this, and yes, you can take advantage of it to put some Readme file, license, whatever... including a mention that the user can't do anything useful with the card themselves.

[peter-h]

I am working with FatFS and it is very good.

The only issues I found where where I needed file system access before the RTOS (FreeRTOS) got started. I don't remember what the problems were exactly but this was achievable only by compiling it in the non-reentrant mode. The reentrant mode could not be used pre-RTOS-start. I solved it by using the FreeRTOS mutexes around the file ops, once RTOS code was running and potentially multiple threads could be reading/writing files.

On this project the file system is also accessible via USB as a removable block device with 512-byte sectors, and this works great.

Date/time stamps are easy but obviously do need a running RTC to be meaningful.

[SiliconWizard]

Speaking of RTOS, I'm currently considering how to use FatFs (or another lib) in a non-blocking way. That precisely doesn't seem easy if at all possible unless you use an RTOS - in which case, FatFs calls are still blocking, but can be preempted...

[Berni]

Yeah non blocking use of FatFs is tricky.

The FatFs code is fast and won't really execute much on its own. Its mostly the storage device driver that will be waiting for data. So what you can do is make the driver call a function during its wait loop. This function could do some background processing or just do an iteration of a "Hot loop" (the cooperative way of multitasking where you infinitely loop over calls to various functions and expect them to return in a timely manner).

If you are mostly writing to file then another solution might be to introduce a cache in the storage driver. That way writes go into RAM instead and so return instantly while the data slowly gets flushed to disk in the background.

Tho i don't see an issue of FatFs not working until you start FreeRTOS. That OS pretty much "boots" in an instant since it just has to initialize some structures as part of the boot process. If the application should not start doing things straight away then you can just simply start up those threads later. If this is for logging use so that you could log errors during boot then this is the wrong way to do it in the first place. Since even once the system is running a crash may leave the system in any weird state (including a locked up disk driver) so the crash handler function has to reinitialize all hardware required to write the error report, so typicaly this would spit the error out of the UART and save it to flash so that it can be written to disk the next time the system comes up. If you want to write it straight to disk then you might have to stick that tiny PetitFatFs inside your crash handler, init it from scratch and do the log write.

[SiliconWizard]

Yeah non blocking use of FatFs is tricky.

The FatFs code is fast and won't really execute much on its own. Its mostly the storage device driver that will be waiting for data.

Yep. But that will be triggered by calls to FatFs functions, making them in turn blocking... A related point is that if you want to use non-blocking IO in your driver, you're a bit doomed.

So what you can do is make the driver call a function during its wait loop. This function could do some background processing or just do an iteration of a "Hot loop" (the cooperative way of multitasking where you infinitely loop over calls to various functions and expect them to return in a timely manner).

Yes. I've seen that kind of stuff done too, and I do not like this approach much at all. It will make your cooperative multitasking look like spaghetti.
But if you don't have a choice...

That raises an interesting issue though. Depending on the way you want to implement "tasking", some libraries are less than ideal. I'm still giving it some thought, but going for my own implementation is still not out of the question...

That said, I think this would be an excellent example of how using a preemptive RTOS can have clear benefits - since this is a commonly asked question.

[SiliconWizard]

So, I decided to implement my own library for supporting FAT32 and exFAT, with non-blocking functions. Those file systems are relatively simple.

One annoying thing, IMO, is the case-insensitive part. I'm curious what you guys' opinion is about that. I know there are a few points in favor of that from a UI's perspective, but that could have been sorted at the UI level, rather than at the file system level. (Oh yeah, I know this is one of those endless debates...)

From a software POV, up-casing ASCII only is trivial. But up-casing Unicode, as needed for FAT32 long file names, and for exFAT file names, is a nightmare. exFAT has at least added "up-case tables", that can be tailored for the supported language(s). But the downside is that such tables take a lot of memory. If you want to cache them in RAM to have fast up-casing, they take 128 KBytes of memory (the whole 16-bit unicode set). exFAT has added the provision to store the table as compressed (RLE basically), but most implementations I've seen read the table when mounting the volume and then uncompress it in memory - so you basically need 128 KBytes just to be able to up-case. I'm thinking of keeping the table compressed in memory and scanning it for up-casing, but that will be kinda inefficient.

All in all, that's a lot of mess for implementations, for something that should have been handled at the UI level IMO - if that.

Any thoughts on this up-casing mess?

[lucazader]

This suggestion comes with two caveats:
-> you are or can use cpp in the project
-> you are ok with adding locale to the project (as this can/will bloat the output binary but quite a bit)

Feel free to ignore this suggestion if either of these are a dealbreaker.

You should be able to use std::towupper to perform this conversion quite easily:
https://gcc.godbolt.org/z/3o757d6fa

Code: [Select]

std::wstring capitalize_string(std::wstring s)
{
    std::transform(s.begin(), s.end(), s.begin(),
                   [](wchar_t c){ return std::towupper(c); });
    return s;
}

[SiliconWizard]

Those would be deal-breakers indeed. But apart from the language/dependencies/and bloat considerations, this solution would still be a problem, at least with exFAT.

exFAT volumes embed their own up-case table. Implementations are supposed to use it for up-casing, and nothing else. The problem here, is that if you use another means of up-casing that happens to have a few differences, even if minor, that could lead to a wrong "up-casing" for a given exFAT volume, and make it unusable on other systems.

You can't use a "locale" per se, as the conversion table is stored in the volume, but the locale identification itself is not provided. Thus, when mounting a given exFAT volume, you have no clue what locale it uses. All you have is a conversion table. It's flexible, but it's pretty annoying to use if you have limited memory, due to the large size of the required table.

Of course, you could ignore all that, and even implement exFAT as case-sensitive, if you're exclusively using the volume on your own system. But if you want to share the volume with other systems (which is one of the points of using exFAT), that would make the volume non-conforming and would cause all sorts of issues, from a given OS not mounting it, to mounting it read-only, to mounting it read-write but not "seeing" some of the files in the volume, etc.

The exFAT spec defines a minimal up-case table (which is basically just the ASCII upcasing) that you are allowed to use for a given volume, but if you use it, you'll need to format the volume yourself. No OS I know of, that supports exFAT, will format say an SD card in exFAT with the minimal up-case table. And for the fun fact: if you try mounting an exFAT volume that has only the minimal up-case table, on Windows, it will mount it read-only.

[magic]

 

One could argue you deserve it for attempting interoperability with Microsoft

I wonder if it would be feasible to maintain a reasonably-sized cache of symbols that you have actually encountered on the given FS and fall back to scanning the data from disk on a miss. Or maybe create a map of ranges that are affected by upcasing or not - I suspect they are quite continuous and sparse.

[SiliconWizard]

One could argue you deserve it for attempting interoperability with Microsoft

exFAT may have been designed at MS, but it has become the de-facto standard for SD cards. So, it's not just a matter of interoperabiity with "Microsoft" here. As I mentioned, if I used exFAT in a purely case-sensitive way, you're right, it would NOT be usable on any Windows system, but it would likely cause problems on any other OS compliant with the exFAT spec anyway...

And, the question is more general too. This is the same for all file systems that are case-insensitive. Of course all FAT versions are, but AFAIK, all Apple file systems are too.
NTFS supports both. While it's used in the case-insensitive mode in Windows, it can be set for case-sensitivity on a per-directory basis. But I certainly don't intend on implementing NTFS.
And, of course, most Unix/Linux file systems are case-sensitive.

But just saying that, case-insensitivity has been considered much easier on the average user (so again it's purely an UI problem), but back when pretty much only ASCII was supported, this was not a problem. Once Unicode came into play, that suddenly became a nightmare.

I wonder if it would be feasible to maintain a reasonably-sized cache of symbols that you have actually encountered on the given FS and fall back to scanning the data from disk on a miss. Or maybe create a map of ranges that are affected by upcasing or not - I suspect they are quite continuous and sparse.

I've thought about some kind of cache, yes. I suppose this should be effective in practice.
As I mentioned, I've also considered storing the table in memory in a compressed form (RLE). Sure it would make upcasing inefficient, but that's typically not something that is heavily done either in a typical use case, so could be considered too. I'm thinking of a mix of the two: storing the table compressed in memory, and adding some cache as well for a few tens of characters or so.

[brucehoult]

And, the question is more general too. This is the same for all file systems that are case-insensitive. Of course all FAT versions are, but AFAIK, all Apple file systems are too.

Apple file systems are case-insensitive by default, but Apple's tools support formatting a partition with HFS+ or APFS in either case-sensitive or case-insensitive versions. I often format external/backup drives to be case-sensitive and you can do it for your boot drive to if you want. It is required for building certain *nix software, including for example gcc. The OS and Apple applications work fine on a case-sensitive drive, as of course do all command-line tools, things you port from Linux etc.

However there are some applications known not to run on case-sensitive file systems. Steam and Photoshop are two of them. Presumably they generate or hard-code the names for certain files internally in multiple places and didn't take care to make the names the same case in each place. It should be easily fixed, but as neither Mac nor Windows is case-sensitive by default they somehow haven't bothered to.

[DiTBho]

I don't like filesystems that allow filenames to have spaces or weird characters. They are a pain to be handled by scripts.

One year ago I made a filename-filter program to handle the filenames of internet downloads.

-> unicode for me is only a thing I don't want to have.

[brucehoult]

I don't like filesystems that allow filenames to have spaces or weird characters. They are a pain to be handled by scripts.

Only if you use a scripting language with variables based on text substitution and re-parsing. Even then, you're fine if you properly quote uses of variables.

In a scripting language such as Perl, Python, Ruby, or JavaScript there is no problem at all with "weird" file names.

[SiliconWizard]

Yep. use proper tools. Filenames with whitespace need to be properly either put inside quotes, or escaped (with a backslash, for instance). Not rocket science.

For the purpose of generality, while not making it too horribly complex to implement (which *always* makes the probability of having bugs much higher), case-sensitive and character-agnostic, except for the very few path separator characters, is IMO the way to go. So, basically the unix-like approach.

Now 'whitespace' can still be questioned. The problem again if you exclude whitespace, is that, if we want to support Unicode (and not doing so is absolutely impossible if you live outside of the USA or maybe UK too, or otherwise you get back to the infamous code pages, *vomits*), whitespace could come in other forms than just the 0x20 ASCII character... Now granted that most command-line tools for most OSs, and in particular shells, only see the ASCII whitespace characters as whitespace, and the rest as regular characters - well, that is, if they even support Unicode. Sometimes that's a bit iffy.

But as with the case-insensitivity matter, it's in the end all in the UI. The filenames in "clear" are ONLY for the purpose of UI. Otherwise, a filesystem could just use a GUID for identifying a file, and move on.

But case-insensitivity is the worst thing to handle IMO. There are just too many possibilities if you want to support Unicode as a whole (and if you don't, you're just ignoring a big part of the planet), and it's a big mess, while, as I said, if you mess up the upcasing, it would not just be a cosmetic issue: it would render accessing some files impossible. Awful.

So all in all, while all of this is trivial using pure ASCII, it becomes almost intractable if you want to do it 100% correctly in Unicode. It's horrible.

[DiTBho]

Only if you use a scripting language with variables based on text substitution and re-parsing. Even then, you're fine if you properly quote uses of variables.

Yes, you can properly quote, but
with Bash scripts -> it's easy to break stuff.
with Php code -> it's easy to break stuff
with Curl scripts -> it's easy to break stuff

[brucehoult]

Don't forget TCL.

(though I'd really prefer to -- for some unknown reason it's still heavily used in EDA workflows)

[SiliconWizard]

It's easy to break stuff using any language in general. And this is kinda off topic =)