Windows app to "erase" SD cards?

[Peabody]

I have the idea, from somewhere, that the SD card controller keeps track of which sectors are already erased, and therefore don't have to be erased before writing data to them.  Erasing a card might make data logging easier because writes would be faster.  This idea is supported by my understanding that in testing an SD card's performance, testers use a card which is fresh from the factory, and not previously used.  Presumably such a card is in the erased state other than the areas used to set up FAT32 or other file system.

Looking through the standard SD card commands, I see CMD38, which supposedly erases a sector or group of sectors.  So it appears to be possible to erase a card back to its factory state.  In fact, in the SDFAT library for Arduino, there is an example called SDFormatter.ino which purports to do exactly that, and the source shows that CMD38 is indeed called.

But everything I see for "erasing" an SD card in Windows uses the word in a different way.  They talk about "secure erasing" or "wiping", and what they really mean is overwriting the entire card with ones, zeroes or random data, even multiple times.  That's not at all what I want.  I want the flash memory to be literally erased, and I don't care if the previous data might somehow be recoverable by the NSA.  I just want the card's controller to recognize the memory as being in the erased state.

I could build an Arduino SD card eraser, but would rather find a Windows app that does the same thing - mainly because I think the SDFAT example doesn't go beyond SDHC.  Does anybody know of a Windows app that allows erasing as opposed to overwriting or wiping?

[rstofer]

Does this do what you want

https://www.sdcard.org/downloads/formatter/

[Peabody]

Does this do what you want

https://www.sdcard.org/downloads/formatter/

I don't think so.  It gives two formatting options - quick and overwrite.  I don't think overwrite is the same as erase.

[oPossum]

A block has to be erased before anything can be written to it, so writing anything will by necessity erase all the blocks used. What it won't do is erase what is held in the blocks used for overprovisioning. That requires a "sercure erase" command to be sent to the storage device. I have done this with SATA SSDs, but I don't know if it is possible with SD cards.

[oPossum]

If you are looking for maximum possible performance to be restored, that is a matter of allocated blocks rather than erased blocks. There is a difference. Secure erase can both erase and deallocate all blocks. Trim can be use to partially or fully deallocate a block.

[edavid]

The SD Association has a formatter:  https://www.sdcard.org/downloads/formatter/

You would think that if anyone could write a tool that would do the right thing, it would be them, but the documentation is so vague, it's impossible to tell 

[Kilrah]

The SD formatter does just that, it formats the card. I don't know of a way to bulk erase an SD card. For SSDs yes there is the secure erase function that does this, and TRIM commands set by the OS during use to allow the drive to reorganise data to free and pre-erase blocks.

[MosherIV]

I do not think there any PC utilities that erase a SD card and leave it blank.
All the ones I have seen and used will erase and then put some kind  of FAT filing system on the SD card.
Technically then the SD card is no longer blank, multiple blocks are used to hold the FAT table entries and directory entries.

The only PC utilites that I know of that allow direct manipulation of SD cards are Linux based.
 dd allows direct copying of blocks, ie it can ignore the FAT filing system.
The SD card can be directly accessed as a file device in the Linux filing system somewhere.

As far as being able to control which blocks are in use and need erasing, that is the point of the FAT filing system. It tracks what is in use and which blocks they are in. Hence it also knows which blocks are not in use. Whether they are actually blank or not depends on the particular implementaion of the FAT filing system.
Most in the embedded world use 'FatFS'. Source code is freely available. Im not sure what it does under the hood.

Edit: most commercial data loggers will use the FAT filing system, otherwise you would not be able to access the logs on a PC as a file.
If you bypass the FAT filing system by writing to empty blocks, then the filing system has no idea that blocks have been used and no idea where you logged data is.
Most will cache the writes and be in control of power down/sleep so that data writes to SD will complete.

[Kilrah]

The point is not about FAT, that the FAT and other formatting-related things are written is not a problem. What he wants is to make sure that the free space is pre-erased.

[Jeroen3]

I just ran a test on a Kinston 8GB Micro SDHC.
Factory card write (4GiB) 5.24 MB/s. Second try 4.61 MB/s. Third try: 5.24 MB/s.
Formatted quick: 4.82 MB/s. Formatted full: 5.24 MB/s.
However, random 4k does show a significant change. From 0.81, 0.20 to 0.53 MB/s and 0.11 / 0.72 for the quick/full format. (with the SD tool)

I didn't think there would be, but there is a difference.
But SD cards were never developed to have 4k random writes anyway. They're for pictures and video.

Read is constant.

[magic]

blkdiscard

Or plug it into any digital camera and do a "full" / "low level" format.

SD cards have TRIM functionality and that's what those cameras use to clear whole card in a few seconds.

BTW, in order to erase an SD card with software of any sort you need a card reader which presents it to the software at the level of bare SD protocol, rather than handling the SD protocol itself and pretending to be a USB pendrive, SCSI disk or something like that. So not all readers are suitable. You should be able to find it out in Device Manager, I think. Still no idea if any software for that is available on Windows.

Alternatively, perhaps there are "pretending" types of SD readers which also support translating TRIM commands from pendrive-speak to SD-speak. Then any TRIM utility will do.

But everything I see for "erasing" an SD card in Windows uses the word in a different way.  They talk about "secure erasing" or "wiping", and what they really mean is overwriting the entire card with ones, zeroes or random data, even multiple times.

Those people probably haven't heard of flash overprovisioning, or maybe they really overwrite it many times and still don't have 100% guarantee

[SoundTech-LG]

Hi,

You may be able to accomplish this with HDD low level formatter, or COPY a fresh new SD of same vintage to your problem child using HDD Raw Copy Tool...  http://hddguru.com/

[Peabody]

Thanks very much for the replies.

I think Magic has made a very important point about this issue.  And that is that the typical SD card reader which is plugged into USB, or even the typical SD card slot on a computer, which is probably also actually USB, is not going to give computer software access to the commands needed (i.e. CMD38) to erase blocks on the card.  But I do indeed have two Canon cameras which offer the option of "low-level" formatting, and it's entirely possible that they erase the card like I want before writing the FAT system files to it.  The burst rates they advertise might well depend on having pre-erased sectors.

And of course an Arduino could be programmed to do that, and it appears the example I referred to does it.  So I could just dedicate an Arduino as my SD card eraser/formatter.

The only issue would be whether either the Arduino or the Canon cameras will handle HDXC cards.  I'll have to check on that.

Anyway, thanks again for all the comments.  I think the bottom line is that I have to be able to speak directly to the card's controller to get this done.  But I will also have to confirm that pre-erasing makes a difference.  I strongly believe it does, but that might be brand-dependent.

[magic]

Yes, cameras do a low level erase. That's the whole point. You can verify the job with some byte level disk viewer/editor. (hexdump anyone?)

They also optimize filesystem layout to match alignment prescribed by the SD spec. Some cameras even complain if you insert a card with misaligned filesystem. They are nuts about performance

[Peabody]

I wanted to follow up on this with further testing results, and also report some bad news about the SD Formatter utility offered by the SD Association.

With some time on my hands, I finally got around to installing the SDFat.h library in my Arduino IDE, and testing the SdFormatter.ino example that comes with that library.  I used a Nano clone and an 8GB Transcend microSD card.  Looking at the sketch code, it's clear that the card is erased using the CMD38 command as part of the formatting.  An 8GB card takes about 10 seconds to erase.  Moreover, the sketch fetches a register flag from the card that indicates the erased state of a byte - whether it reads as zero or FF (this varies with vendor/brand).  The flag for my Transcend card says it erases to FF, and looking at the card with my hex editor on Windows, it is indeed all FFs except for the FAT, root directory, etc.  The SDFat author says all the geometry used by his formatter (cluster size, alignment) is the same as the SD Association's formatter.

So then I formatted the same card with the Association's formatter in Windows using the "Overwrite" option (there is no Erase option).  It got about half way through in 10 minutes, at which point I gave up.  Looking at the card in the hex editor again, I see all zeroes in the first half, and all FF's in the last half.  So "Overwrite" should be taken literally.  It is writing zeroes to the entire card, leaving all the sectors in the written state, with no sectors left in the erased state.  As Magic said in an earlier reply, I think it does this because it simply doesn't have access to the CMD38 command via the Windows driver, so the only way to remove all data from the card is to overwrite it.  In other words, it is unable to erase the card.  But that's just a guess.

Then I did a "low level" format of the same card in my Canon DSLR camera.  This took about 10 seconds, and returned the card to all FFs.  This is clearly a full erasure.

I haven't really done a test, but I just assume that if the card is erased, it will not be necessary to erase a sector before writing to it, and therefore tasks like high-speed data logging, or shooting stills in burst mode in a camera, will do better than if every sector is flagged as already containing data, which will require erasing before writing, and possibly even moving data around.  While a card may work ok with no unerased sectors, on the margin as things speed up, an erased card will write faster.  For data logging, that means sensors can be sampled more frequently, or be less subject to dropouts while waiting for the card to write, with an erased card.

I think the Association's formatter is fine for a Quick format, but I won't be using the Overwrite option.  In fact, I think I'm going to dedicate a Nano and an SD card module to the formatting of SD cards.  Once flashed to the Nano, the sketch will work with any terminal program (I use Termite).  It could even be modified to operate stand-alone.

The SDFat formatter only does SD and SDHC, but there is a beta update in the works that adds support for SDXC cards.  Just search Github for SDFat.

There's another example called AnalogBinLogger which logs 5000 ADC readings to the card per second.  More about that if anyone is interested.

Here's the log of the SDFat SdFormatter session:

Code: [Select]

Type any character to start

This program can erase and/or format SD/SDHC cards.

Erase uses the card's fast flash erase command.
Flash erase sets all data to 0X00 for most cards
and 0XFF for a few vendor's cards.

Cards larger than 2 GB will be formatted FAT32 and
smaller cards will be formatted FAT16.

Warning, all data on the card will be erased.
Enter 'Y' to continue: Y

Options are:
E - erase the card and skip formatting.
F - erase and then format the card. (recommended)
Q - quick format the card without erase.

Enter option: F
Card Size: 8166 MB, (MB = 1,000,000 bytes)

Erasing
................................
.............................
All data set to 0xff
Erase done

Formatting
Blocks/Cluster: 64
FAT32
................
Format done


[magic]

So then I formatted the same card with the Association's formatter in Windows using the "Overwrite" option (there is no Erase option).

What a joke

I haven't really done a test, but I just assume that if the card is erased, it will not be necessary to erase a sector before writing to it, and therefore tasks like high-speed data logging, or shooting stills in burst mode in a camera, will do better

Of course, this only applies if the writes are some minimum size (a kB or a few?) and aligned properly. I'm not familiar with the exact details for SD cards, though. If you simply append a few bytes, chances are your library/OS or the card(?) will do a read-modify-write.

[Kilrah]

Given you've verified "overwrite" writes 0s it should be easy to prepare 2 identical cards, one with that and one erased to FFs and do speed/latency comparisons to know whether the sector being already erased or not has a significant impact.

[Peabody]

Of course, this only applies if the writes are some minimum size (a kB or a few?) and aligned properly. I'm not familiar with the exact details for SD cards, though. If you simply append a few bytes, chances are your library/OS or the card(?) will do a read-modify-write.

It seems virtually all SD and SDHC cards deal in 512-byte sectors.  There's nothing in the SD commands that lets you append a byte or two.  So any attempt to do that is going to result in that read-modify-write for not only the file data area but also its directory entry (to update the file size) and both copies of the FAT when the file crosses into a new cluster.  So it's no wonder that things can slow down when appending data to a file.

But the AnalogBinLogger I referred to gets around pretty much all of the file system overhead during the logging process.  It creates a file which occupies whatever is left of the card that is consecutive sectors, then pre-erases those sectors.  Then when logging starts, it just writes successive 512-byte sectors to the card without updating the directory or the FAT.  It's essentially a continuous multi-block write, and since everything is pre-erased, the card's controller shouldn't have to do anything but carry out the writes.  Then when logging is halted, it redefines the file length and corrects the FAT entries to reflect what was actually written.

A timer is set up to interrupt at a fixed interval at which point the sensor is read and the reading added to a 512-byte buffer.  Then in the Main loop, the code notices when a buffer has been filled, and it then sends that buffer contents to the card after switching to another collection buffer for the interrrupts to use.  So there will be no lost data so long as the time required to write a block to the card doesn't exceed 512 interrupts.  And it turns out that none of the SD writing activity is bothered by being interrupted from time to time.  So something like a Nano's Atmega328P, with all of 2K of ram, is big enough to do this properly at pretty good speed.  The example defaults to 5000 samples per second, but it can probably do a good bit more than that.

So when speed is really important, you have to deal with the card at a low level, 512 bytes at a time, and stay away from all the file system overhead.

[Peabody]

Given you've verified "overwrite" writes 0s it should be easy to prepare 2 identical cards, one with that and one erased to FFs and do speed/latency comparisons to know whether the sector being already erased or not has a significant impact.

Yes, I need to think about the best way to do that.  It could be something as simple as copying a 1GB file to the cards and seeing how long that takes.  But I wonder whether Windows will taint the results in some way.  Perhaps there are SD benchmarking programs that would give valid results.  Or even Arduino benchmarking sketches.

[Peabody]

I did a comparison test writing a 1.8GB file to the SD card - with the card erased, then with the card overwritten.  There was no difference in the total write time.  I don't know what this means.  Maybe it means there's no advantage to pre-erasing the card, although I'm at a loss to understand how that could be true.   Or maybe it means that Transcend doesn't keep track of which sectors are erased.  Or something Windows 10 is doing to limit the write speed.  I don't know.

[Jeroen3]

Pre-erasing maybe improved random writes. But that's not really what you use an SD for anyway.