Funny filename problem with windows versus "unix"

[peter-h]

I have an embedded target which has a 2MB FAT12 USB MSC drive which is visible to the internal firmware with FatFS which has LFN (long filenames) disabled so all files are 8.3 only. The firmware can be updated by dropping a file factory0.dat into it and rebooting. It looks for a file factory0.dat.

From windows, you can drop in a file

factory0.dat
FACTORY0.DAT

and it works. The file is written into the directory as FACTORY0.DAT which is right for 8.3. The dir entry is uppercase.

I know windows is "not case sensitive" but it is still an interesting observation that windows maintains the case in its filesystem, but when asked to write the file into another disk which is FAT, the destination directory entry is uppercased. I tested this with a FAT32 flash stick. But somehow windows displays that filename in lowercase - how does it know?

Now the fun bit. When I use android (USB OTG) to write that file, it ends up as FACTOR~1.DAT. I am using the MLUSB app to mount FAT12 drives. In other words, even though the file can fit into 8.3, the host wrote it as LFN. And this is true regardless of its case on the source drive.



Is this a bug in MLUSB or even android? I thought that if a filename can fit into 8.3 it should be thus written, for FAT drives. It should not be written as a LFN.

It might mean that for these kinds of firmware upgrades to work, the device (FatFS) needs to look for both the 8.3 filename and its 8.3-compacted LFN version.

[Nominal Animal]

This behaviour is explained in man 8 mount under Mount options for vfat, shortname= mount option.  The default is shortname=mixed, but you expect shortname=lower behaviour.

This also means that from Linux/Android, you'll want to name the file as FACTORY0.DAT before dropping in to the VFAT, unless you change the default mount options for VFAT volumes.

But somehow windows displays that filename in lowercase - how does it know?

Since MS-DOS times, filename mangling.

The charset used for the shortnames is defined using the codepage=nnn mount option, with codepage=437 (MS-DOS Latin US) being the default.  The charset used for long names is defined using iocharset=charset mount option, with iocharset=iso8859-1 being the default.  Long file names are stored in Unicode format (UCS-2) on-disk.  Windows uses its own rules, currently active character set I believe.

In Linux, one can also mount the volume as msdos (instead of vfat), if you wish to avoid the long filename issues completely, and use the shortnames only.

[peter-h]

This also means that from Linux/Android, you'll want to name the file as FACTORY0.DAT before dropping in to the VFAT

I did that but it still ended up as FACTOR~1.DAT in the directory sector. And no other directory entry, which I think is wrong because if you have an LFN you should also have the mangled 8.3 version.

FatFS will open a file with any case filename, for sure if LFN is disabled. I have tested this many times.

The above unix mount link suggests that a FACTORY0.DAT (i.e. all uppercase) should not get mangled, but it does.

Anyway, I solved this with a rename of any factor~1.dat to factory0.dat at startup

[Nominal Animal]

I did that but it still ended up as FACTOR~1.DAT in the directory sector.

Can you show me the mount options (the relevant line in /proc/mounts), when the drive is mounted?

In Linux, you can change the automounted drive type (until unmount) via e.g.
    mount mountpoint-or-device -t msdos -o remount,rw,uid=$(id -u),gid=$(id -g),flush,errors=remount-ro,X-uhelper=udisks2
(where udisks2 is the standard D-Bus storage device manager; you can omit that part but you may need to umount it afterwards from the command line).
The msdos mount type uses short names only; the long name support is in vfat.

Also remember that if you already have a file named FACTOR~1.DAT with a long name factory0.dat, Linux VFAT will always use the long name if it exists.  Thus, you need to delete any existing file before dropping in a new one.

[peter-h]

Can you show me the mount options (the relevant line in /proc/mounts), when the drive is mounted?

No idea where this might be, on a non rooted android phone

Also the context is this
https://www.eevblog.com/forum/microcontrollers/any-way-to-do-a-fat16-usb-msc-drive-in-2mb-flash/
where android does not see FAT12 at all and I have to use the MLUSB app to access the drive.

Also remember that if you already have a file named FACTOR~1.DAT with a long name factory0.dat, Linux VFAT will always use the long name if it exists.  Thus, you need to delete any existing file before dropping in a new one.

Interesting... but actually I did delete existing file before the write.

There was another issue which I suspected might be doing this: I was using dropbox to transfer the file to the phone (to storage/temp or some such path; this is not a rooted phone because a lot of apps check for that) and when downloaded from dropbox it ended up as factory0.dat.pac which I then renamed (in a proper file explorer, not the crippled MyFiles) to remove the .pac, and yeah you can guess... So I repeated this by copying the file to the phone via USB and this prevented the .pac rename. But same result in the FAT12 drive.

The more basic point is that when I am seeing FACTOR~1DAT in sector 13 (above screenshot from HXD) which presumably is an attempt to create an LFN, I am not seeing any other entry for that file. There should always be the 2nd entry, FACTORY0DAT, no?

[Nominal Animal]

on a non rooted android phone

Ah.  Yeah, maybe not the best choice of a device for setting up a new firmware file...

Anyways, on Linux, you can explicitly mount the filesystem as fat12 via
    mount -t msdos -o rw,uid=$(id -u),gid=$(id -g),fat=12,showexec,errors=remount-ro /dev/device directory/
It will not generate long filenames, but may use them if they already exist; I've not verified this.

The more basic point is that when I am seeing FACTOR~1DAT in sector 13 (above screenshot from HXD) which presumably is an attempt to create an LFN, I am not seeing any other entry for that file. There should always be the 2nd entry, FACTORY0DAT, no?

No, FACTOR~1.DAT is the short FAT file name entry.  It is always 11 bytes, padded with spaces, with implicit . after the eighth character.  If I understand correctly, you want only the FACTORY0DAT entry.

The LFN uses UCS-2, so the long filename entry for factory0.dat should be
    66 00 61 00 63 00 74 00 6f 00 72 00 79 00 30 00 2e 00 64 00 61 00 74 00 00 00
as a single (not split) 13-character/UCS-2 slot.

I wonder if renaming the file to FACTORY0.DAT in Android, after transferring it, makes any difference?

Also, if you have the time, could you check if using (incorrect) name FACTORYO.DAT, i.e. use letter O instead of zero, makes a difference?

It is possible the logic for uppercase/lowercase-only name detection is wonky in the filesystem driver, so that all filenames with non-letter characters are treated as neither uppercase nor lowercase, b0rking the long filename selection logic.  If so, I can see if I can find and fix the bug in the filesystem driver causing this.  Otherwise, it is a bug in MLUSB application, specifically long filename support — but I do believe it simply uses the kernel mount interface, instead of implementing a full filesystem driver.

[peter-h]

If I understand correctly, you want only the FACTORY0DAT entry.

Yes.

I wonder if renaming the file to FACTORY0.DAT in Android, after transferring it, makes any difference?

Renaming the file (in MLUSB) didn't help with the filename, but it created what looks like a Unicode name


Under windows the directory looks like this

Also, if you have the time, could you check if using (incorrect) name FACTORYO.DAT, i.e. use letter O instead of zero, makes a difference?

No difference in that FACTOR~1DAT is still produced. How can it be the same, when 0 was changed to O?


The point is that windows and linux both produce FACTORY0DAT in there and no other entries...

This could be quite a rabbit hole. I always thought that if a filename can be represented as 8.3 it is stored (in FATxx) as 8.3 only, and only when LFN is required then you get LFN and the crunched 8.3 version.

[Nominal Animal]

This could be quite a rabbit hole. I always thought that if a filename can be represented as 8.3 it is stored (in FATxx) as 8.3 only, and only when LFN is required then you get LFN and the crunched 8.3 version.

No, it's not that simple.

Long filenames are simply "magic" VFAT directory entries (compare to FAT directory entries) that encode the long name, with up to 20 directory entries chained for a single name.  Their type is volume label (and filesystem drivers will create a volume label if there is none and you create a long filename in the root directory).  There are some rules on how filesystem drivers need to handle missing and deleted names, but that is about it.

Thus, whether a filesystem driver creates long filenames or not, is up to the driver.  It can choose to create a LFN for a file whose name is perfectly okay 8.3 name, or not; the use rules (always use LFN first if it exists) allow it both ways.

No difference in that FACTOR~1DAT is still produced. How can it be the same, when 0 was changed to O?

Because the issue is that your Android filesystem driver/app always creates long filenames, unlike the Linux fat/msdos/umsdos/vfat driver.

If the standard Linux fat/msdos/umsdos/vfat driver was used, the shortname= mount option controls this, and the only possibility is that somehow, the filename is not recognized as uppercase 8.3 format.  Again, when the default shortname=mixed is used, this driver creates only the 8.3 name if it is uppercase and fits, and a LFN in all other cases.  I supposed it was possible that a zero, not being a letter, throws it off, but looking at the Linux kernel sources (fs/fat/namei*.c), I don't think so.  Ergo, I think this is a bug in the Android MLUSB Mounter you're using. 

[peter-h]

I think so too, and have reported it. MLUSB are responsive on email.

No, it's not that simple.

OK I do recall the volume label hack. Yet, that (always creating an uppercase 8.3 if it fits) is what windows does and always has done, and linux likewise (with both all-uppercase and lowercase filenames).

I still don't get how the FAT12 (or FAT32) can remember the 8.3 filename case (as displayed under windows) when an examination of its directory shows an uppercase name. Probably a clue is in that internally generated names (with FatFS) display all-uppercase



As a funny aside, on my RNS510 car box, mp3 tracks played from a DVD show lowercase filenames (on the LCD) while the same tracks played from an SD card show uppercase filenames

[Nominal Animal]

I still don't get how the FAT12 (or FAT32) can remember the 8.3 filename case

Since Windows NT, bits 3 and 4 at offset 0xC in the short name directory entry encode the case.  If bit 3 is set (0x08), the base name is displayed in lower case.  If bit 4 is set (0x10), the extension is displayed in lower case.  Otherwise the shortname is displayed in uppercase.

So, if the factory0.dat directory entry is
    46 41 43 54 4f 52 59 30  44 41 54 20 00 ?? ?? ??
it will be shown as FACTORY0.DAT, but
    46 41 43 54 4f 52 59 30  44 41 54 20 18 ?? ?? ??
as factory0.dat.

As a funny aside, on my RNS510 car box, mp3 tracks played from a DVD show lowercase filenames (on the LCD) while the same tracks played from an SD card show uppercase filenames

Not all filesystem drivers are created equal!

[peter-h]

Well this has been a great learning experience.

It's funny how something can work for years and then you discover something big when you try to do it differently; in this case upgrade firmware from a phone instead of a laptop

Given how many people "live" on just a phone, I bet this has come up before.

No solution found for an Apple phone though.

[Nominal Animal]

It is very true that a specific version of Windows generates a very specific pattern of shortnames, but because different versions (at least MS-DOS pre-7 vs. MS-DOS 7 and later vs. NT) have done it in different ways, the rules on what they accept are pretty funky and relaxed.

Funnily enough, there is also no requirement for the LFN entry for that file to be FACTOR~1.DAT; even Microsoft/Windows does not specify any algorithm that ought to be used to construct the corresponding short name (but the one 2000 and later appear to use is known).  It suffices that the short name is unique; it does not actually need to match in any sense.  You can see the logic the Linux driver uses in fs/fat/namei_vfat.c:vfat_create_shortname().  The previous fifty or so lines prior to that also contain useful info.

[SiliconWizard]

It is very true that a specific version of Windows generates a very specific pattern of shortnames, but because different versions (at least MS-DOS pre-7 vs. MS-DOS 7 and later vs. NT) have done it in different ways, the rules on what they accept are pretty funky and relaxed.

Funnily enough, there is also no requirement for the LFN entry for that file to be FACTOR~1.DAT; even Microsoft/Windows does not specify any algorithm that ought to be used to construct the corresponding short name (but the one 2000 and later appear to use is known).  It suffices that the short name is unique; it does not actually need to match in any sense.  You can see the logic the Linux driver uses in fs/fat/namei_vfat.c:vfat_create_shortname().  The previous fifty or so lines prior to that also contain useful info.

Indeed. It should just be unique and there's obviously no way to reconstruct the LFN from the short name, so they are completely unrelated.
Common sense algorithms usually take the beginning of the LFN, "normalize" it in some way (to use only ASCII characters) and then complete the short name with a suffix that attempts to be unique compared to potentially other existing short names with the same beginning. In practice, this is a pretty annoying "feature" to have to handle in FAT filesystems as you have to list all matching short names in the same directory to be able to complete the new short name. Not ultra efficient.

[peter-h]

My requirement here is not creating short names. It is just copying an 8.3 file from some portable device with USB to my target's 2MB FAT12 filesystem, where I want the same 8.3 file (doesn't matter whether uppercase or lowercase since non-LFN FatFS doesn't care) to be written.

Windows and Linux seem to do this as expected.

[Nominal Animal]

My requirement here is not creating short names. It is just copying an 8.3 file from some portable device with USB to my target's 2MB FAT12 filesystem, where I want the same 8.3 file (doesn't matter whether uppercase or lowercase since non-LFN FatFS doesn't care) to be written.

Technically, to use the same shortname, is to create the correct expected shortname.  Windows and Linux do so, MLUSB Mounter apparently does not (and instead always creates a VFAT-style LFN directory entry).  I don't know how iOS (13 and later) or MacOS behave, but I'd expect them to be like Windows and Linux (due to BSD heritage if nothing else).

For the MLUSB Mounter problem report, we can describe the desired logic (same logic as linux fs/fat/namei_vfat.c) as:

Code: [Select]

When the file name contains only
    A B C D E F G H I J K L M N O P Q R S T U V W X Y Z
    0 1 2 3 4 5 6 7 8 9 $ % ' ` - @ { } ~ ! # ( ) & _ ^
with 1-8 characters in the base name, and 0-3 characters in the extension, only a 8.3 shortname is created; no LFN.
Alternatively, it can also accept all-lowercase characters for the base name and/or for the extension, and use the two bits in the directory entry to denote uppercase/lowercase (with the 8.3 shortname always in uppercase).

[peter-h]

Yes.

Especially as FAT12 is reportedly supported even by win11 and linux because win11 and linux still support floppy disks (!) and these can be 360k, 1.2M (in 5.25" size) or 1.44M, 2.88M (in 3.5" size) and all these are FAT12.

Regarding the Ipad, I have not been able to test this because my Ipad Mini, latest OS, does nothing whatsoever when my FAT12 target is connected to it It also does nothing whatsoever upon connection of the STLINK V3 which has that fake 3MB FAT12 USB MSC volume discussed in the other thread.

[Nominal Animal]

Regarding the Ipad, I have not been able to test this because my Ipad Mini, latest OS, does nothing whatsoever when my FAT12 target is connected to it

But, I assume, auto-mounts FAT32 without issues?  Ah, the OS would be IPadOS.  Do you have a-Shell installed on it?

Although, I personally don't think it would be unreasonable to require a Windows, Linux, or MacOS desktop or laptop for firmware updates.  Phones and iOS tablets and Chromebooks would be nice, yes, but I'm not sure one can expect them to be suitable for firmware updates.  But again, this is just my personal opinion, nothing more.

[peter-h]

Yes it works with FAT32 flash sticks. However I use it for just one job (running Foreflight) and know little about it.

I agree but a phone is arguably quite convenient for field upgrades where you just load a file. This box of mine could also do OTA but that a) requires internet access and b) has the potential of bricking a large installed base

Thank you for your outstanding explanations

[peter-h]

I got a reply from MLUSB:

As a quick follow-up, I checked the directory sector dump (sector 32).

The long filename entry for "factory0.dat" is present in the directory, but
the generated short filename is "FACTOR~1.DAT".

So your observation appears to be correct. The file itself is written
correctly, but the short filename generation behavior differs from Windows
(which generates "FACTORY0.DAT"). This is likely something that should be
improved.

I had not checked further along the disk to spot the LFN. And LFN is disabled in FatFS in my case.