FAT32 getting started

[Jan Audio]

Hi, i like to write a file to a FAT32 SDcard.
To make it simpler i format the card in windows with FAT32.
Now all i want is to write to 1 file in the root.

What do i have to do ?
btw : i am using a 8-bit PIC.
The read/write to SD card is functioning.
I also need to download all the librarys that exist for example, so far i have : sdfat.

thanks

[ucanel]

I wrote an sd card library for dspic30f,
fat16 and fat32 supported,
wrote it in assembler, library size was 1k.
I do not recommend you to write your own library
it takes unnecessaryly long time to understand and implement it.

But it is easy to communicate with an sd card
if we do not think about Fat.
So if it is suitable for you there is an workaround about Fat:
You will create a space filled long  txt file with a pc then
using winhex Hxd or similar
you will learn the start address of this  empty file
and at the mcu side
using this address you can easly write and read this file.

There are so many tutorials about physically reaching to sd card.
As a hint you will use SD SPI mode,
and there is SD Card initialisation sequence
after that there is Sd read and Sd write commands you will need.
ELM Chan's site is a very good starting point.
So many mcu manufacturer uses his FatFS library.
http://elm-chan.org/docs/mmc/mmc_e.html

http://elm-chan.org/docs/mmc/i/sdinit.png


And there is one more hint
sd cards has at least  4 or 5 different types
it is hard to support all of them with just one small library.

[Jan Audio]

Yes, i support V1 and V2 SD cards, i focus on SD-HC only.
The SD card is already initialized, thank you.

And yes i am going to write my own FAT32, with only 1 file to write.

[methods]

A bit off topic but my experience has been that the Arduino libraries handle this beautifully.

I have wired the SPI lines directly to a MicroSD card slot and data logged gigs as easy as writing the terminal.

I strongly encourage all engineers to support the Arduino paradigm.  I got my start with pic, specifically the 16f84 and later the 16F877.  This is back when we did everything and pick asm leveraging the true RISC.

Today I find myself struggling to get an stm32 to even do the most basic of tasks. . .  where for the last decade I've used arduino's to great success.

down selection of a microcontroller should be based on how well supported it is by a community. 

...  Unless it's part of a learning program were you actually wanted to be difficult.

-methods

[Jan Audio]

Yes i downloaded that code also.
The problem is reading the bootsector "bytes per sector" gives zero.
What can i do ?, is this normal ?

[Jan Audio]

...  Unless it's part of a learning program were you actually wanted to be difficult.

I want it to be simple and small, for use with small 8 bitter.
Most important : understandable for myself.

[ucanel]

If i remember correctly bytes per sector is 512 bytes standart for FAT32.
I will check my notes and write it again.

[PlainName]

down selection of a microcontroller should be based on how well supported it is by a community

Having community support is a useful plus, but surely the selection of a micro (indeed, anything) should be on whether it will do the job better than something else. The availability of a ton of code written by someone else is just one factor, and not necessarily the most important.

[Jan Audio]

I attatched a 20x4 LCD, and write hex adress and hex data values 4 in 1 row like this : 00000000 00 00 00 00

If i read the bootsector ( 512 bytes ), it has only some values at the end, it has the 0x55 and 0xAA, and before that some more.
The rest all zeros.

The size in use after formatting is 2K, i overwritten 4 blocks and the card is still working in windows.
btw : i dont know if it matters, i format it with the SD to USB converter from my 3D-printer.

It is formatted FAT32, if i look in those library codes, it gives "invalid card" if the "bytes per sector" reads zero.

[Nominal Animal]

Apologies for inserting Linux in this discussion, but if one just writes data to the SD card sequentially without any file system, it is rather easy to read in Linux: you can treat the block device as a file, and read it directly.  All you need to do is make sure you have the necessary privileges to do so.  (I can show a privileged "opener" written in C, though, that simply verifies the first 512 bytes have an acceptable fingerprint, and if so, pass the descriptor (file handle) back to a normal, unprivileged program, so they can treat the entire device as a single file.  This is safe, and only requires that you leave the first write block intact, containing a fingerprint that does not match any existing filesystem or partition table structure.  (I recommend some initial and final bytes, plus the rest as ASCII text explaining the format.)  It can be used in conjunction with e.g. Graphical User Interface written in Python3 and Qt5, for portability.)

Unfortunately, I do not know of any way to do the equivalent in Windows.

For Linux users, it means you do not need the overhead of any file system, as long as you have a way to determine the beginning and ending of data.  One method I have used, is to have a 64-bit monotonically increasing generation counter in every block (whatever block size you use to write to the device), and use the blocks on the device in sequential order.  In general, it does mean you need to "initialize" the device to all zeroes, unless it is already sufficiently initialized (all counters zeros – it can be easily, albeit slowly, verified).

On a device with N blocks, you then only need to read 2 + log₂ N or so blocks, to find the most recent or oldest block.  On a 8 GB SD card with 512 byte sectors/blocks, or 16,777,216 sectors, this means you only need to read 26 sectors to find the oldest or newest block; and when oldest and newest blocks are known, all other blocks are directly accessible without searching.  So, it is quite fast and efficient, too.  All this is easily written in library form – especially if you have a fixed structure for each block, say containing fixed-size records, or variable-sized records that do not cross a block boundary –, much easier than any filesystem driver.

[magic]

You optimally "initialize" an SD card by erasing it with a special command, though I'm not sure if it's guaranteed whether the result will be all 0x00 or all 0xff or perhaps something else. It tends to be all 0x00 IME.

[Jan Audio]

I need those "bytes per sector".

[Nominal Animal]

You optimally "initialize" an SD card by erasing it with a special command, though I'm not sure if it's guaranteed whether the result will be all 0x00 or all 0xff or perhaps something else. It tends to be all 0x00 IME.

Yeah, CMD38, but issuing it is nontrivial. All zeros or all ones works fine, as long as both 0 and 2⁶⁴-1 are considered invalid generation numbers.  A mix (0xFF00..) is technically not allowed, but one can fiddle a bit with the generation counter to avoid any of the eight bytes ever having zero or one.  It makes the math a bit slower on 8-bit MCUs, though.  So, just writing all zeros to the SD card is easier, but takes a while.  One can optimize that a bit by only clearing sectors whose eight first bytes are nonzero or non-2⁶⁴-1.

I need those "bytes per sector".

What do you mean?  Using a generation counter per block written is at most 1.5625% overhead (8 bytes out of every 512 byte sector; 0.2% at 8 bytes per 4096 byte block), which is definitely less than using any filesystem.

[Jan Audio]

What do you mean?  Using a generation counter per block written is at most 1.5625% overhead (8 bytes out of every 512 byte sector; 0.2% at 8 bytes per 4096 byte block), which is definitely less than using any filesystem.

Talk about What do you mean?, now you got me confused.
Should i ignore the bytespersector and use 512 ?
Should i ignore the whole bootsector and only validate 0x55 and 0xAA ?
I dont think it works.

[Nominal Animal]

Talk about What do you mean?, now you got me confused.

Since you didn't quote the message you answered, I assumed you were responding to me.

I now see you were continuing your previous post, i.e.

I attatched a 20x4 LCD, and write hex adress and hex data values 4 in 1 row like this : 00000000 00 00 00 00

If i read the bootsector ( 512 bytes ), it has only some values at the end, it has the 0x55 and 0xAA, and before that some more.
The rest all zeros.

The size in use after formatting is 2K, i overwritten 4 blocks and the card is still working in windows.
btw : i dont know if it matters, i format it with the SD to USB converter from my 3D-printer.

It is formatted FAT32, if i look in those library codes, it gives "invalid card" if the "bytes per sector" reads zero.

I need those "bytes per sector".

There are two layers here.  One is the physical layer at which the SecureDigital card works.  The other is the filesystem layer, FAT32 in your case.

At the hardware level, when writing to a SD cards, you write in 512-byte aligned blocks.  This is often called a physical sector, because back when storage media had spinning disks, the smallest unit they could read and write was a physical sector (or more properly, a circular arc on the rotating media), usually also 512 bytes.  (Nowadays hard drives with 4096 byte sectors are common, though; and on solid-state media like Flash, things are more complicated.)

At the filesystem level, FAT32 specifies a logical sector size.  This is is a power of two, and at least 512.
FAT32 also specifies a cluster size.  This is a power of two multiple of the logical sector size.

The typical logical sector sizes are 512 and 4096.  The cluster size varies, but 32768 is most common.  You can set either or both when formatting the filesystem, or let the utility implement some heuristic.  Either way, these two numbers are described in the file system boot sector.

I don't know about Windows tools, but in Linux, if I want to find them out, I connect the media (via an USB card reader for SD cards), and then run sudo umount /dev/partition && sudo fsck.ext -nv /dev/partition, which tells me stuff like

Code: [Select]

fsck.fat 4.1 (2017-01-24)
Checking we can access the last sector of the filesystem
Boot sector contents:
System ID "MSDOS5.0"
Media byte 0xf8 (hard disk)
       512 bytes per logical sector
     16384 bytes per cluster
     16448 reserved sectors
First FAT starts at byte 8421376 (sector 16448)
         2 FATs, 32 bit entries
   7577600 bytes per FAT (= 14800 sectors)
Root directory start at cluster 2 (arbitrary size)
Data area starts at byte 23576576 (sector 46048)
   1892901 data clusters (31013289984 bytes)
128 sectors/track, 63 heads
      8064 hidden sectors
  60618880 sectors total
Checking for unused clusters.
Checking free cluster summary.
/dev/sda1: 1064 files, 57794/1892901 clusters
which tells me that this is FAT32 with 512 bytes per logical sector, and 32768 bytes (64 logical sectors) per cluster.  The smallest block this FAT32 filesystem uses is therefore 32768 bytes; even a 1-byte file will reserve 32768 bytes on the media.

Hope this clarifies things a bit.

[Nominal Animal]

What I was telling above, was that when the SD card data is collated using a Linux machine, it is easy to not have any kind of filesystem on it at all, as in Linux you can read the entire SD card as if it was a single continuous file.

Of course, it won't mount, so you won't be able to see its contents on the desktop, but you can write simple tools in C, C++, Python, etc. that let the user examine and export the data to ordinary files – XML and CSV formats are very common.  There are Python modules for exporting the data in other file formats, like Excel or OpenOffice/LibreOffice, or PDF.  Essentially, this moves the complexity from the microcontroller to the management utilities on a computer.

The only "issue" in Linux (and *BSDs) is the permission model.  Ordinary users do not normally have direct access to the block devices, mounted or not.  For this, I have developed a privileged, secure C helper, that examines the underlying block device, and if it looks (has initial/final sector contents matching specific pattern which is not a normal filesystem or partition table, i.e. specifically designed for use with microcontrollers), passes the open file descriptor, file handle, back to the unprivileged ordinary program.  This relies on Unix domain sockets, and passing file descriptors over such a socket, as specified in the POSIX.1 standard.  Basically all except Windows is enough POSIXy today so that the basic idea works.

Note that this is also secure: the unprivileged program does not have privileged access to anything else.  It has access only to the verified-safe-to-access SD card or USB memory stick or whatever media has the correct "signature".

Unfortunately, I have no idea how you would do anything like this in Windows.  I know it is possible in Macs, but I am not familiar enough with Mac OS access restrictions to say for sure whether my C program would work there as well as it does in Linux, or whether it would need some Mac-specific additions; I'd need to check it on a Mac development machine first.  I also have no idea whether Microsoft WSL supports such access at all, because I haven't used Windows in over a decade.  I just don't know.

[nctnico]

Don't bother writing your own FAT implementation. I've used this one in several projects:
http://elm-chan.org/fsw/ff/00index_e.html

It can be tailored to include more or less features depending on what you need versus the amount of flash space you have.

[ucanel]

Can you open sd card with pc Winhex Hxd and add the screenshot of boot sector,
so we may check it for you.
Maybe you are doing something wrong,
BPS information is two bytes long and located at boot sector 0x0C,0x0B

[Jan Audio]

All the sudden it works.
I wrote all zeros to the card, then formatted,
after formatting i found a valid boot sector.

Now i can get started with the fun part.

Don't bother writing your own FAT implementation. I've used this one in several projects:
http://elm-chan.org/fsw/ff/00index_e.html

It can be tailored to include more or less features depending on what you need versus the amount of flash space you have.

Yes i have that and i find it unreadable.
I be using that for inspiration a bit if i,m stuck.

[Jan Audio]

My version allows block to write smaller then 512 ( useless with only 1 SPI port, only i plan reading parallel data )
End of the week i be writing my file, else i be posting here.

[Jan Audio]

I found the root folder, that was easy.
Now i have the file.

All i have to do now is search for how it handles free clusters, my file is bigger then 1 cluster.

[Jan Audio]

All done, it was really simple, i took a few shortcuts.

Read boot-sector.
Find fatsize.
Find root-directory.
I filled a file text with the ammount of bytes i am writing.
Then look up in the FAT for the next cluster in the file.

[coppercone2]

is it possible to make a fat128?

I don't like how they make these marketing names like ORANGE and HAMMER2 that have nothing to do with computer things

[vinayak]

hi Jan audio,

Could you please provide the code which you have written for implementing the FAT32.

Regards,
Vinayak

[darkspr1te]

You didnt really say exactly which pic you are using , but maybe this will help
https://simple-circuit.com/pic18f46k22-sd-card-fat32-mikroc/


darkspr1te

[vinayak]

Hi darkspr1te,

 I dont know exactly i just need some idea how i can implement the FAT32 file system IP without kernel involved

Regards,
Vinayak

[Psi]

Do you mean you want to do it all with DMA?

[vinayak]

 yes DMA only
why is that not possible?

[pcprogrammer]

just something like DMA won't cut it. Why, because it is a file system you want to setup, and that means making decisions based on what is already on the disk (card). It would require an intricate state machine to get it working.

If you want to know about FAT32 and how it works you should checkout the already mentioned FATFS code. http://elm-chan.org/fsw/ff/

[vinayak]

Ok... Thanks for replying

[5U4GB]

Now all i want is to write to 1 file in the root.

If you're only going to write 1 file to the root then format the card and write the file on the system of your choice, record which blocks on the card correspond to that file, and then overwrite only those blocks from the PIC code with the data you're saving.  The PIC code doesn't need to know anything about FAT, just which blocks to write to.

[Psi]

Now all i want is to write to 1 file in the root.

If you're only going to write 1 file to the root then format the card and write the file on the system of your choice, record which blocks on the card correspond to that file, and then overwrite only those blocks from the PIC code with the data you're saving.  The PIC code doesn't need to know anything about FAT, just which blocks to write to.

+1 for this being a simple solution.

Another options is to consider if you actually need a FS on the card.  If you need to remove the card and access it in a PC then you obviously do.
But sometimes you can simply think of the card as permanent memory for your device and just write wherever you want to, basically you roll your own super simple file system on the card for your use case and give up on it ever being readable in a PC.

[DiTBho]

Don't bother writing your own FAT implementation. I've used this one in several projects:
elm-chan

It can be tailored to include more or less features depending on what you need versus the amount of flash space you have.

People simply don't care, for me the Elm-Chan's FAT is very poor code quality.

[pcprogrammer]

People simply don't care, for me the Elm-Chan's FAT is very poor code quality.

I agree, but it works. Used it in the FNIRSI 1013D firmware and noticed that it allowed NULL pointers to be passed for output buffers. 

But judging on some of the responses people seem to miss that this is a 4 year old thread revived by vinayak to ask for the code. The OP has not been back since February 2022, so the change of him reading the posts are slim.

[Psi]

sigh,  it would be nice if SMF would color old threads different or something.
Or maybe a cobweb background image for any old thread that suddenly got revived. That would be cool.

[SiliconWizard]

FatFs "works" and is used by many, but similarly, when I had to use FAT in a project, I rejected it eventually and wrote my own implementation. I only needed FAT32 and exFAT as well, so didn't bother to implement FAT16 or FAT12, which are supported by FatFs.

"Very poor code quality" may be a bit harsh, but it's definitely not very good - and one of the main reasons, beyond that, that made me reject it, was that it only implemented blocking calls for all operations, which turns out very inconvenient if you're not using some kind of preemptive OS.

[5U4GB]

But judging on some of the responses people seem to miss that this is a 4 year old thread revived by vinayak to ask for the code. The OP has not been back since February 2022, so the change of him reading the posts are slim.

Even if the OP never reads it, it's still here as a reference for anyone else who ends up here, and there's valuable advice in the postings.

[DiTBho]

I rejected it eventually and wrote my own implementation

"very poor code quality" may seem very cynical, however ...

Years ago, I designed and implemented a filesystem for a job request. A few months earlier, the two super project managers had asked to find a solution, some of us started thinking about it and "ran for office" (to get the task), bringing a solution.

I broght a personal project I had started at university, cleaned it up in order to present it as a "candidate project". It had nice featues, everything was B+tree based, and the whole module it could easily be written to run as userspace service (microkernel approach).

I presented a demo, running it on the top of the XINU uckernel on a Sequoia PowerPC board.

The code quality was for sure more than decent as it was already passing MISRA and DO178B-level D. However, being a personal project of mine, started during my university days, rewritten several years after graduation and carried out only in moments of free time, it needed some revision and modification. Nothing serious, I had estimated less than 3 weeks of work, allocating two people full time.

So, I presented it with a list of "Pros and Cons" and ... got then rejected because

• an experimental filesystem design? Sounds too high complexity and too exspensive!
• people (who?!?) want to use a Windows laptop

Then a colleague of mine showed up, who had downloaded the code from the Elm-Chan FAT website, and ... I really don't understand how this happened, but the two project managers thought it was exactly the right solution and ordered us to use that code, which I had then to split into 4 parts in order to have 4 teams work in parallel to be able to clean and make it "decent" for an avionics allocation that passed at least DO178B level C.

... and ... it was a massacre, a damn bloody bloodbath.

Did it make sense? ...

... well, in the end, if you consider the consumed "human power" of 4 teams of 3 dudes each, 8h/24 for 3 months, it costed more than 4x times the cost I had estimated for myfs, but hey? "people want to use a Windows Laptop to copy GPS flight maps from a folder on the Desktop to the SD card/USB-stick that goes into the flight module on the airplane or helicopter" - said one of the two project manager ...

Computer science is sometime *VERY* frustrating

[SiliconWizard]

It's pretty much always the same thing: managers usually favor solutions that look "well established", because they appear as the path of least resistance.
What seems to matter in the end is not the end result, but how people have perceived it.
If choosing a well known (or one that appears a bit 'de facto') solution, managers will rarely get the blame, even if the project takes ages to complete with a lot of hiccups.
FAT has been around for decades now, and FatFs is rather popular, so it's definitely in the 'de facto' category.

If choosing an alternative, new or "obscure" solution, managers will often get the blame even just for the project being one week late. I've seen this endlessly.
That's unfortunately how it works. Humans need conformity. And while it has probably had many evolutionary benefits in our distant past, this is also what is currently dragging us down IMHO. End of rant.

[T3sl4co1l]

Did it make sense? ...

... well, in the end, if you consider the consumed "human power" of 4 teams of 3 dudes each, 8h/24 for 3 months, it costed more than 4x times the cost I had estimated for myfs, but hey? "people want to use a Windows Laptop to copy GPS flight maps from a folder on the Desktop to the SD card/USB-stick that goes into the flight module on the airplane or helicopter" - said one of the two project manager ...

Computer science is sometime *VERY* frustrating

Part of it I think is the lack of scientific knowledge, going on feel rather than fact.  Many times have I presented an idea, including potential pros and cons, in the interest of making a more informed decision about it; many times, alternative ideas have been given as just basic ideas without the cons, and probably the impression of simplicity plus lack of explicit cons wins out in the decision maker's mind.  It's bad psychology, it's dumb psychology, but human psychology it is.

Or maybe it's me being a 'sperg and that's the bigger turn-off in general, not so much my ideas necessarily.  That's always a possibility.

But real ideas are complex, real solutions have up and down sides, real projects are difficult to talk about not only in complexity but in the feelings we ascribe to them, feelings we create in lieu of having a strong enough factual basis; that is, out of ignorance of the total scope of the project (and, again, ignorance isn't a bad thing, it's literally not knowing, if you knew the complete solution you wouldn't be asking me to engineer it for you, right?!), we fall back on statistics, feelings of likelihood, and of optimism or apathy towards one approach or another.  And above some level of complexity, ignorance is a necessity, many systems are simply too complex for any one person to fully understand, and so we will always apply heuristics and emotions while operating on them.

And it's not to say we should avoid feelings.  It's unavoidable, of course.  At least at scale -- whether in terms of a given sufficiently-complex problem, or in terms of the number of people that can maintain a complete understanding within their heads (of any given topic).  Just, to avoid making rash decisions, with little buy-in from stakeholders, and avoid committing sunk-cost and gamblers' fallacies (among many others) along the way.

Back on topic, for my part -- I gave this a try,
http://www.roland-riegel.de/sd-reader/
which seemed to go just fine.  I haven't used it for writing, I haven't tested it in any meaningful way at all, but it seems capable of reading a file from root in SPI mode, and that's all that I cared about at the time.

Tim

[PlainName]

"people want to use a Windows Laptop to copy GPS flight maps from a folder on the Desktop to the SD card/USB-stick that goes into the flight module on the airplane or helicopter"

Do they, or would they?

[DiTBho]

"people want to use a Windows Laptop to copy GPS flight maps from a folder on the Desktop to the SD card/USB-stick that goes into the flight module on the airplane or helicopter"

Do they, or would they?

I don't know, last time I was in Japan, I spoken to a couple of managers of CASIO, and asked why they removed the physical SD-card support from their graphing calculators (e.g. FX-9860GIISD), and replaced it with FAT32 internal flash support in the last line of products (e.g. FX-9860GIII).

Why not an other prettier filesystem? Why still FAT32?

The answer was something like because most people, especially in the educational field, college and university, use Windows, so they are used to have FAT32 on their SD cards.

I don't know 

[PlainName]

Well perhaps the management had a point, then. It's all very well have a super-FS which is ultra-robust, fast and uses 2K of memory, but give your average user the choice of an SD card to shove in their PC/phone/whatever and have it Just Work, or minimally having to install a special driver (where's the Mac version? What about XP?) or, worse, a custom dongle and I know why they will choose every time.

Without knowing details, it seems to me that the problem wasn't the management choosing FAT32 but the shitty implementation of it.