SD card emulator

[encore2097]

This might sound crazy but there is good reason for this and carefully considered for over a year from all angles, so if you're comment is "why not try X" without having understood the problem, please do not reply.

With that said, I need to emulate an SD card via USB.

The problem: creating an automated test and benchmark platform for embedded devices, specifically single board computers (SBCs).

During development and testing we flash a bootloader + os image to an SD card on a host computer and then move the card manually to a target device (RPi or similar) and power it on to see if it works. If it works great, if not then we have to re-flash the SD card on the host computer and retry it. This is not feasible for an automated system and solution like SD-MUX (multiplexers) still require an SD card and can wear out and require switching which increases complexity.

The proposed solution is an sd card emulator.

It would have an SD/microSD form factor <-> micro controller + buffer <-> USB 3.0 <-> host computer with modified driver that uses a file as storage (emulated NAND)

So how can we do this? The fastest, cheapest, most effective way to do this (theoretically) is to use a hacked USB 3.0 SD card reader.

Let me explain, a USB 3.0 card reader uses a micro controller to read and write to an SD card. One example is the Realtek RTS5301 used on many USB 3.0 card readers and inexpensive.

If the card reader using the SD pins can communicated with an SD card which stores and retrieves data on NAND storage, we should be able to mock out the NAND storage for storage over another protocol (in this case a filesystem accessed over USB).

The card reader micro controller shows a SDXC/MMC 4-bit I/F on the block diagram and supported features include:

   • Support SD card clock 208MHz (max.) for SDR104 mode.

   • Support SD card clock 100MHz (max.) for SDR50 mode and 50MHz (max.) for DDR50 mode.

If it is capable of R/W to a SD card at those rates, surely it must be able to emulate the opposite signaling to a host to mimic an SD card. Or at the very least drop down to a lower standard which is still faster than SD-MUX and more robust (wear and tear, switching USB, etc..)

What do you think?

[andersm]

At work we're looking into using QEMU and QTest for running automated tests against generated SD card images. For running tests on the actual hardware, I would mount the filesystem over the network, if your OS and hardware allows it.

[Rasz]

or you could buy off the shelf $20 WiFi enabled SD card and upload files over the air.
btw rasPee can boot from USB and ethernet now

[abyrvalg]

I'd follow Rasz's SD-WiFi advice, but if you still need to go long way:

There are several FPGA-based SD emulator projects by the same guy on hackaday.io (https://hackaday.io/project/6946-sd-card-emulator-lattice-xp, check "similar projects" section also), but no sources published.

RTS5301 will not work: it's a master device that clocks the interface, but you need a slave clocked by interface.

Some full size SD cards have separate SD-NAND controller talking to big TSOP NAND chips (if you can emulate NAND bus. Cypress FX3?)

There is a weird SD device with SPI flash instead of NAND (should be much easier to emulate - no scrambling, no ECC, no block mapping, I can see a plain FAT image in flash dump) - xtc2clip.org

[mikeselectricstuff]

Why do you have to reflash the SD card, as opposed to using another programmed one ?

[encore2097]

or you could buy off the shelf $20 WiFi enabled SD card and upload files over the air.
btw rasPee can boot from USB and ethernet now

This is an interesting idea.. I'm not aware that you can write to the SD card over Wi-Fi, but it should be possible since the card appears to be running embedded linux. However, there are no microSD cards with Wi-Fi capability. They do make an SD to microSD adapter/cable - if it works I have no idea.. I believe electrical pinout is the same..

I'm also concerned with performance and the issue of wear and tear on NAND. Might be possible as a quick hack.

I'd follow Rasz's SD-WiFi advice, but if you still need to go long way:

There are several FPGA-based SD emulator projects by the same guy on hackaday.io (https://hackaday.io/project/6946-sd-card-emulator-lattice-xp, check "similar projects" section also), but no sources published.

RTS5301 will not work: it's a master device that clocks the interface, but you need a slave clocked by interface.

Some full size SD cards have separate SD-NAND controller talking to big TSOP NAND chips (if you can emulate NAND bus. Cypress FX3?)

There is a weird SD device with SPI flash instead of NAND (should be much easier to emulate - no scrambling, no ECC, no block mapping, I can see a plain FAT image in flash dump) - xtc2clip.org

I'm still new to EE so this might be a dumb question: whats unique to the slave device that the master cannot emulate? They communicate via the same electrical signals over the set of wires at the same clock speed. From my research, host initializes (powers) SD card and negotiates transfer speed, then it sends read and write commands which the SD card microprocessor interprets and answers by manipulating flash. Should be possible with a firmware hack to make the host act as a slave since it has all the capabilities to communicate with a slave or am I mistaken here?

Re: NAND emulation, this would be very difficult and incredibly complex and I dont see the point to doing this.. NAND is used because of its form factor and cost, those are not constraints -- any data store will do. Besdies the host has no conception/idea of NAND it only sends read/write commands after initialization. So the emulation is the SD card interface and protocol. Taking the Wi-fi SD card idea its very similar to what I'm proposing.

[encore2097]

At work we're looking into using QEMU and QTest for running automated tests against generated SD card images. For running tests on the actual hardware, I would mount the filesystem over the network, if your OS and hardware allows it.

Works if you dont plan to modify bootloader / uboot / kernel. If you do plan to modify those ( say because your are adding in netboot or NFS ) and it doesn't boot during development, get ready for a frustratingly long development cycle with lots of SD card swapping.

Why do you have to reflash the SD card, as opposed to using another programmed one ?

Hope you are not trolling..  you are still swapping SD cards - which would not make it an automated test platform.
The SD-MUX v2 uses this idea, it has two SD cards and two interfaces and electrically switches the connections between the target and the host. This still has the wear and tear issue and switching electrical connections has reported issues; intermittently not working one day and then working the next (maybe bug in HW or SW)

[abyrvalg]

I'm still new to EE so this might be a dumb question: whats unique to the slave device that the master cannot emulate? They communicate via the same electrical signals over the set of wires at the same clock speed. From my research, host initializes (powers) SD card and negotiates transfer speed, then it sends read and write commands which the SD card microprocessor interprets and answers by manipulating flash. Should be possible with a firmware hack to make the host act as a slave since it has all the capabilities to communicate with a slave or am I mistaken here?

Unidirectional CLK signal (from master to slave, this is "hardwired" in silicon, like having a mouth to talk or having an ear to listen).

[mikeselectricstuff]

Hope you are not trolling..  you are still swapping SD cards - which would not make it an automated test platform.
The SD-MUX v2 uses this idea, it has two SD cards and two interfaces and electrically switches the connections between the target and the host. This still has the wear and tear issue and switching electrical connections has reported issues; intermittently not working one day and then working the next (maybe bug in HW or SW)

Not trolling - just trying to understand the actual problem, and why a big pile of ready-programmed and verified SD cards wouldn't solve the problem.

[NottheDan]

Try this? http://www.linuxinternals.org/blog/2014/06/04/a-microsd-card-remote-switcher/

As for the  suggestion of using multiple SD cards, that should be able to be automated fairly easily too with with an extender cable running to a switchable dock for multiple SD cards. A more elaborate version of this.

Also, for the Raspi exists Berryboot, which allows you to use multiple image from the same SD card.

[Jeroen3]

Technically, you could emulate an SD card using ram. Then press reset and it performs fast flash->ram copy to restore it to supposed pre-test state.
If you're really into it you could do something like shadow copying to restore even faster. But that obviously is more complicated.

However, It all depends if you're able to find code/hdl to function as an SD card memory interface.
And, if the costs of developing such automated test equipment is more affordable than a human plugging cards. Which it most likely is not, unless you're going above 4 zero's.

[encore2097]

Unidirectional CLK signal (from master to slave, this is "hardwired" in silicon, like having a mouth to talk or having an ear to listen).

Gotcha, signal source and sink. Thanks.

From the spec, the SD Host powers on the SD card, the SD card generates the clock, so I'd need a clock generator.. sounds like there might not be an easy solution other than making an SD card that uses RAM / NFS as storage instead of NAND.

[encore2097]

Not trolling - just trying to understand the actual problem, and why a big pile of ready-programmed and verified SD cards wouldn't solve the problem.

There are no pre-verified images, this is during development hence the testing. How this process works: a new build of the firmware is generated after making changes, now we need to see if it works and the performance.

1) Write new firmware to SD card
2) Attempt to boot device (check bootloader)
3) Attempt to boot kernel (check kernel)
4) If successful boot, run series of tests/benchmarks and record results (check drivers, application, etc)

If the firmware fails at any step prior to 4, there is no way to access the SD card through the device itself. Its like bricking a phone. This is why many embedded devices/SBCs support SD card or similar, which allows you to remove the storage and write to it from a working host.

Now to speed up development, its necessary to be able to test multiple builds, on multiple boards simultaneously. It becomes a logistically nightmare to swap 5-10 SD cards every 5-20 minutes for various boards and configurations, while keeping track of what build ran where. Computer vs abacus.

Try this? http://www.linuxinternals.org/blog/2014/06/04/a-microsd-card-remote-switcher/

As for the  suggestion of using multiple SD cards, that should be able to be automated fairly easily too with with an extender cable running to a switchable dock for multiple SD cards. A more elaborate version of this.

Also, for the Raspi exists Berryboot, which allows you to use multiple image from the same SD card.

Thanks for the link. This is another variant of the SD-MUX v1, and has the same issues. This isn't only for RaspberryPi's , I have many other SBCs from different vendors for differing projects. Having to develop/maintain a multi-boot bootloader for each one is not practical and introduces another layer of complexity which gets farther away for testing actual usage.

Technically, you could emulate an SD card using ram. Then press reset and it performs fast flash->ram copy to restore it to supposed pre-test state.
If you're really into it you could do something like shadow copying to restore even faster. But that obviously is more complicated.

However, It all depends if you're able to find code/hdl to function as an SD card memory interface.
And, if the costs of developing such automated test equipment is more affordable than a human plugging cards. Which it most likely is not, unless you're going above 4 zero's.

You would need an RTOS, as the SD interface requires signal responses within a certain time frame and the clock for UHS-I (65+ MB/s) is 100MHz or 208MHz for high speeds. Even then, with such a high clock speed, I don't think this is possible via pure software emulation.. are you aware of something that could enable RAM emulation with these constraints without a micro controller?

[nctnico]

Not trolling - just trying to understand the actual problem, and why a big pile of ready-programmed and verified SD cards wouldn't solve the problem.

There are no pre-verified images, this is during development hence the testing. How this process works: a new build of the firmware is generated after making changes, now we need to see if it works and the performance.

1) Write new firmware to SD card
2) Attempt to boot device (check bootloader)
3) Attempt to boot kernel (check kernel)
4) If successful boot, run series of tests/benchmarks and record results (check drivers, application, etc)

You are doing this wrong! When developing you should be able to access the SD card through the embedded device via the network (sshfs / fuse if it is Linux) in order to keep your sanity. After all the kernel and even the bootloader have to live somewhere on the SD card and should be easy to access. Only program SD cards for production.

[mikeselectricstuff]

OK I mis-read this as a production-line situation, hence the confusion.

As the above post says, seems like you're doing it wrong - I appreciate there may be a very few things that need testing on real cards, but surely once you've loaded some minimum boot-up code from the card, you should be able to handle anything else over a network connection. If you want to make it look as much as possible like an SD card, you write a bootloader that emulates low-level SD commands over the network.

Failing that, it ought to be reasonably simple to make a fairly dumb multiplexer between the devices SD port and a USB SD-card reader that means at least you don't need to physically plug & unplug cards.

Of course it would be possible to make what is effectively a dual-port, RAM based MicroSD card emulator, it would be a lot of work for what can probably be more easily done in other ways.

An FPGA would be able to emulate the SD interface at full speed without too much trouble- you'd probably want a small package located very close to the SD pins for signal integrity  - not sure if you'd be able to get full speed extending the pins outside the footprint of the microsd card.

The SD interface is command-based, so you could parse the commands into reads and writes to RAM, and you could have a second interface to the RAM - as you already need to do the work to emulate the SD interface, the second interface might as well also be SD as it would simply be a second instantiation of the same HDL component, hooked up to a standard USB SD card-reader IC to give an interface that can go down a cable to the other host.
 
For speed you'd want to emulate using RAM rather than flash, and you wouldn't get enough on an FPGA so would need external DRAM - not a big deal as most FPGAs can talk to DDR2/3

If you did decide to go this way, you'd probably end up with something that would be sellable as a product

[mikeselectricstuff]

Of course an FPGA based hardware emulator like this could also do interesting stuff like logging of SD transactions for debugging and reverse-engineering.

[andersm]

Works if you dont plan to modify bootloader / uboot / kernel. If you do plan to modify those ( say because your are adding in netboot or NFS ) and it doesn't boot during development, get ready for a frustratingly long development cycle with lots of SD card swapping.

If that happens, you've got problems in your development process. And in any case, realistically how often will that happen? (The same question applies to your concern about SD card wear - if you have to change cards every couple years, is it a real issue?)

[tszaboo]

Still dont get, why you would need to have to emulate the interface. A normal setup would look like this:
You place the board on a "bed of pogopins" test jig. The SD card testpoints make a contact. On the base board you have 2-3-4 whatever number of SD card sockets and a digital multiplexer, controlled by the test jig. You select the SD image you want.

If you really want to plug in the SD card jig device, you can still use the multiplexer method.

[Rasz]

Still dont get, why you would need to have to emulate the interface.

I think the end goal was remote debugging without monkeys swapping cards. Both Wifi SD card and multiple cards + mux + sd reader + some host with ethernet will allow that. Except mux setup will be >$200+time to put it together while Wifi SD is $27 with free shipping on amazon right now

https://www.amazon.com/gp/product/B00GEBTFNM
https://www.amazon.com/gp/product/B01D9JIUU0
probably $50 to have it at your doorstep tomorrow.

yes, you can upload contents over the wireless interface, here is a dude using wifi SD card in his 3D printer in order to upload prints remotely without touching the card http://www.extrud3d.com/flashair

> concerned with performance and the issue of wear and tear on NAND

it would take over a year to kill $20 SD card in this scenario (~10MB every 3 minutes for bootloader+kernel+reboot?)
instead you are planning to spend couple of hundred on elaborate contraption that will take half a year to develop and debug?

[mac.6]

For kernel and boot test you can switch to tftp boot instead of using sdcard (if your SBC has native ethernet).
You will only need to update sdcard for bootloader and rootfs, and even for this you can do a two step boot (use tftp boot for kernel and rootfs, update SDcard and reboot).

[CJay]

Can you write the SD over JTAG via the host CPU?

Would save all the effort of having to swap cards in and out, just attach your JTAG host to the board under test, either via pogo pins in a test jig or 'standard' header and blast away, if you've concerns over the longevity of the SD cards then you need to be considering the quality of the SD cards.

[technix]

If your processor support it, you can even leave the card out and boot over JTAG entirely, by holding the processor in RESET, write your bootloader into RAM directly and throw PC there boot straight from RAM. After you get a working bootloader this way (I doubt if a bootloader will get too large to fit in the RAM, and if you need a multi-stage bootloader you can load each stages using JTAG when the card is missing) you can put the bootloader on SD card and start netbooting.

[vpeter]

Browsing with Google and found this thread...

I did similar circuit like this already mentioned a-microsd-card-remote-switcher.

But as core IC I took TS3A27518E which is 6-Bit, 1-of-2 Mux/Demux. Same IC is used in linaro's and tizen's design for automatic testing. Whole circuit works pretty well as prototype. To switch targets I'm using small DPDT switch (not ideal but do the job for manual work). On both sides I'm using flat cable with micro sd card at the end like this.

I would like to make a proper PCB for it but I'm no electronic guy  It should be very small and looking like SD card which would be put in usb sd card writer. On it there is mentioned IC + micro sd card. At the other end there would be flat cable mentioned. If there is someone who is interested in this let me know and I will provide all the details. I'm sure this could be used in a different ways.

Peter

[amyk]

You would need an RTOS, as the SD interface requires signal responses within a certain time frame and the clock for UHS-I (65+ MB/s) is 100MHz or 208MHz for high speeds. Even then, with such a high clock speed, I don't think this is possible via pure software emulation.. are you aware of something that could enable RAM emulation with these constraints without a micro controller?

An SD card basically consists of a microcontroller and NAND flash with firmware. Some are only an 8051, but running at ~100MHz. There's another clump of 8051s and ARMs whose max clock frequency is specified at 208MHz, likely not a coincidence. It's only the communication that happens at such a speed, so they probably have something like a DMA controller and everything else is handled by the firmware.

https://www.bunniestudios.com/blog/?p=3554

[Bruce Abbott]

It becomes a logistically nightmare to swap 5-10 SD cards every 5-20 minutes for various boards and configurations, while keeping track of what build ran where.

That's not the only logistical nightmare.  Why would you be creating 5-10 different builds every 5-20 minutes?