SD card emulator

[encore2097]

This might sound crazy but there is good reason for this and carefully considered for over a year from all angles, so if you're comment is "why not try X" without having understood the problem, please do not reply.

With that said, I need to emulate an SD card via USB.

The problem: creating an automated test and benchmark platform for embedded devices, specifically single board computers (SBCs).

During development and testing we flash a bootloader + os image to an SD card on a host computer and then move the card manually to a target device (RPi or similar) and power it on to see if it works. If it works great, if not then we have to re-flash the SD card on the host computer and retry it. This is not feasible for an automated system and solution like SD-MUX (multiplexers) still require an SD card and can wear out and require switching which increases complexity.

The proposed solution is an sd card emulator.

It would have an SD/microSD form factor <-> micro controller + buffer <-> USB 3.0 <-> host computer with modified driver that uses a file as storage (emulated NAND)

So how can we do this? The fastest, cheapest, most effective way to do this (theoretically) is to use a hacked USB 3.0 SD card reader.

Let me explain, a USB 3.0 card reader uses a micro controller to read and write to an SD card. One example is the Realtek RTS5301 used on many USB 3.0 card readers and inexpensive.

If the card reader using the SD pins can communicated with an SD card which stores and retrieves data on NAND storage, we should be able to mock out the NAND storage for storage over another protocol (in this case a filesystem accessed over USB).

The card reader micro controller shows a SDXC/MMC 4-bit I/F on the block diagram and supported features include:

   • Support SD card clock 208MHz (max.) for SDR104 mode.

   • Support SD card clock 100MHz (max.) for SDR50 mode and 50MHz (max.) for DDR50 mode.

If it is capable of R/W to a SD card at those rates, surely it must be able to emulate the opposite signaling to a host to mimic an SD card. Or at the very least drop down to a lower standard which is still faster than SD-MUX and more robust (wear and tear, switching USB, etc..)

What do you think?

[andersm]

At work we're looking into using QEMU and QTest for running automated tests against generated SD card images. For running tests on the actual hardware, I would mount the filesystem over the network, if your OS and hardware allows it.

[Rasz]

or you could buy off the shelf $20 WiFi enabled SD card and upload files over the air.
btw rasPee can boot from USB and ethernet now

[abyrvalg]

I'd follow Rasz's SD-WiFi advice, but if you still need to go long way:

There are several FPGA-based SD emulator projects by the same guy on hackaday.io (https://hackaday.io/project/6946-sd-card-emulator-lattice-xp, check "similar projects" section also), but no sources published.

RTS5301 will not work: it's a master device that clocks the interface, but you need a slave clocked by interface.

Some full size SD cards have separate SD-NAND controller talking to big TSOP NAND chips (if you can emulate NAND bus. Cypress FX3?)

There is a weird SD device with SPI flash instead of NAND (should be much easier to emulate - no scrambling, no ECC, no block mapping, I can see a plain FAT image in flash dump) - xtc2clip.org

[mikeselectricstuff]

Why do you have to reflash the SD card, as opposed to using another programmed one ?

[encore2097]

or you could buy off the shelf $20 WiFi enabled SD card and upload files over the air.
btw rasPee can boot from USB and ethernet now

This is an interesting idea.. I'm not aware that you can write to the SD card over Wi-Fi, but it should be possible since the card appears to be running embedded linux. However, there are no microSD cards with Wi-Fi capability. They do make an SD to microSD adapter/cable - if it works I have no idea.. I believe electrical pinout is the same..

I'm also concerned with performance and the issue of wear and tear on NAND. Might be possible as a quick hack.

I'd follow Rasz's SD-WiFi advice, but if you still need to go long way:

There are several FPGA-based SD emulator projects by the same guy on hackaday.io (https://hackaday.io/project/6946-sd-card-emulator-lattice-xp, check "similar projects" section also), but no sources published.

RTS5301 will not work: it's a master device that clocks the interface, but you need a slave clocked by interface.

Some full size SD cards have separate SD-NAND controller talking to big TSOP NAND chips (if you can emulate NAND bus. Cypress FX3?)

There is a weird SD device with SPI flash instead of NAND (should be much easier to emulate - no scrambling, no ECC, no block mapping, I can see a plain FAT image in flash dump) - xtc2clip.org

I'm still new to EE so this might be a dumb question: whats unique to the slave device that the master cannot emulate? They communicate via the same electrical signals over the set of wires at the same clock speed. From my research, host initializes (powers) SD card and negotiates transfer speed, then it sends read and write commands which the SD card microprocessor interprets and answers by manipulating flash. Should be possible with a firmware hack to make the host act as a slave since it has all the capabilities to communicate with a slave or am I mistaken here?

Re: NAND emulation, this would be very difficult and incredibly complex and I dont see the point to doing this.. NAND is used because of its form factor and cost, those are not constraints -- any data store will do. Besdies the host has no conception/idea of NAND it only sends read/write commands after initialization. So the emulation is the SD card interface and protocol. Taking the Wi-fi SD card idea its very similar to what I'm proposing.

[encore2097]

At work we're looking into using QEMU and QTest for running automated tests against generated SD card images. For running tests on the actual hardware, I would mount the filesystem over the network, if your OS and hardware allows it.

Works if you dont plan to modify bootloader / uboot / kernel. If you do plan to modify those ( say because your are adding in netboot or NFS ) and it doesn't boot during development, get ready for a frustratingly long development cycle with lots of SD card swapping.

Why do you have to reflash the SD card, as opposed to using another programmed one ?

Hope you are not trolling..  you are still swapping SD cards - which would not make it an automated test platform.
The SD-MUX v2 uses this idea, it has two SD cards and two interfaces and electrically switches the connections between the target and the host. This still has the wear and tear issue and switching electrical connections has reported issues; intermittently not working one day and then working the next (maybe bug in HW or SW)

[abyrvalg]

I'm still new to EE so this might be a dumb question: whats unique to the slave device that the master cannot emulate? They communicate via the same electrical signals over the set of wires at the same clock speed. From my research, host initializes (powers) SD card and negotiates transfer speed, then it sends read and write commands which the SD card microprocessor interprets and answers by manipulating flash. Should be possible with a firmware hack to make the host act as a slave since it has all the capabilities to communicate with a slave or am I mistaken here?

Unidirectional CLK signal (from master to slave, this is "hardwired" in silicon, like having a mouth to talk or having an ear to listen).

[mikeselectricstuff]

Hope you are not trolling..  you are still swapping SD cards - which would not make it an automated test platform.
The SD-MUX v2 uses this idea, it has two SD cards and two interfaces and electrically switches the connections between the target and the host. This still has the wear and tear issue and switching electrical connections has reported issues; intermittently not working one day and then working the next (maybe bug in HW or SW)

Not trolling - just trying to understand the actual problem, and why a big pile of ready-programmed and verified SD cards wouldn't solve the problem.

[NottheDan]

Try this? http://www.linuxinternals.org/blog/2014/06/04/a-microsd-card-remote-switcher/

As for the  suggestion of using multiple SD cards, that should be able to be automated fairly easily too with with an extender cable running to a switchable dock for multiple SD cards. A more elaborate version of this.

Also, for the Raspi exists Berryboot, which allows you to use multiple image from the same SD card.

[Jeroen3]

Technically, you could emulate an SD card using ram. Then press reset and it performs fast flash->ram copy to restore it to supposed pre-test state.
If you're really into it you could do something like shadow copying to restore even faster. But that obviously is more complicated.

However, It all depends if you're able to find code/hdl to function as an SD card memory interface.
And, if the costs of developing such automated test equipment is more affordable than a human plugging cards. Which it most likely is not, unless you're going above 4 zero's.

[encore2097]

Unidirectional CLK signal (from master to slave, this is "hardwired" in silicon, like having a mouth to talk or having an ear to listen).

Gotcha, signal source and sink. Thanks.

From the spec, the SD Host powers on the SD card, the SD card generates the clock, so I'd need a clock generator.. sounds like there might not be an easy solution other than making an SD card that uses RAM / NFS as storage instead of NAND.

[encore2097]

Not trolling - just trying to understand the actual problem, and why a big pile of ready-programmed and verified SD cards wouldn't solve the problem.

There are no pre-verified images, this is during development hence the testing. How this process works: a new build of the firmware is generated after making changes, now we need to see if it works and the performance.

1) Write new firmware to SD card
2) Attempt to boot device (check bootloader)
3) Attempt to boot kernel (check kernel)
4) If successful boot, run series of tests/benchmarks and record results (check drivers, application, etc)

If the firmware fails at any step prior to 4, there is no way to access the SD card through the device itself. Its like bricking a phone. This is why many embedded devices/SBCs support SD card or similar, which allows you to remove the storage and write to it from a working host.

Now to speed up development, its necessary to be able to test multiple builds, on multiple boards simultaneously. It becomes a logistically nightmare to swap 5-10 SD cards every 5-20 minutes for various boards and configurations, while keeping track of what build ran where. Computer vs abacus.

Try this? http://www.linuxinternals.org/blog/2014/06/04/a-microsd-card-remote-switcher/

As for the  suggestion of using multiple SD cards, that should be able to be automated fairly easily too with with an extender cable running to a switchable dock for multiple SD cards. A more elaborate version of this.

Also, for the Raspi exists Berryboot, which allows you to use multiple image from the same SD card.

Thanks for the link. This is another variant of the SD-MUX v1, and has the same issues. This isn't only for RaspberryPi's , I have many other SBCs from different vendors for differing projects. Having to develop/maintain a multi-boot bootloader for each one is not practical and introduces another layer of complexity which gets farther away for testing actual usage.

Technically, you could emulate an SD card using ram. Then press reset and it performs fast flash->ram copy to restore it to supposed pre-test state.
If you're really into it you could do something like shadow copying to restore even faster. But that obviously is more complicated.

However, It all depends if you're able to find code/hdl to function as an SD card memory interface.
And, if the costs of developing such automated test equipment is more affordable than a human plugging cards. Which it most likely is not, unless you're going above 4 zero's.

You would need an RTOS, as the SD interface requires signal responses within a certain time frame and the clock for UHS-I (65+ MB/s) is 100MHz or 208MHz for high speeds. Even then, with such a high clock speed, I don't think this is possible via pure software emulation.. are you aware of something that could enable RAM emulation with these constraints without a micro controller?

[nctnico]

Not trolling - just trying to understand the actual problem, and why a big pile of ready-programmed and verified SD cards wouldn't solve the problem.

There are no pre-verified images, this is during development hence the testing. How this process works: a new build of the firmware is generated after making changes, now we need to see if it works and the performance.

1) Write new firmware to SD card
2) Attempt to boot device (check bootloader)
3) Attempt to boot kernel (check kernel)
4) If successful boot, run series of tests/benchmarks and record results (check drivers, application, etc)

You are doing this wrong! When developing you should be able to access the SD card through the embedded device via the network (sshfs / fuse if it is Linux) in order to keep your sanity. After all the kernel and even the bootloader have to live somewhere on the SD card and should be easy to access. Only program SD cards for production.

[mikeselectricstuff]

OK I mis-read this as a production-line situation, hence the confusion.

As the above post says, seems like you're doing it wrong - I appreciate there may be a very few things that need testing on real cards, but surely once you've loaded some minimum boot-up code from the card, you should be able to handle anything else over a network connection. If you want to make it look as much as possible like an SD card, you write a bootloader that emulates low-level SD commands over the network.

Failing that, it ought to be reasonably simple to make a fairly dumb multiplexer between the devices SD port and a USB SD-card reader that means at least you don't need to physically plug & unplug cards.

Of course it would be possible to make what is effectively a dual-port, RAM based MicroSD card emulator, it would be a lot of work for what can probably be more easily done in other ways.

An FPGA would be able to emulate the SD interface at full speed without too much trouble- you'd probably want a small package located very close to the SD pins for signal integrity  - not sure if you'd be able to get full speed extending the pins outside the footprint of the microsd card.

The SD interface is command-based, so you could parse the commands into reads and writes to RAM, and you could have a second interface to the RAM - as you already need to do the work to emulate the SD interface, the second interface might as well also be SD as it would simply be a second instantiation of the same HDL component, hooked up to a standard USB SD card-reader IC to give an interface that can go down a cable to the other host.
 
For speed you'd want to emulate using RAM rather than flash, and you wouldn't get enough on an FPGA so would need external DRAM - not a big deal as most FPGAs can talk to DDR2/3

If you did decide to go this way, you'd probably end up with something that would be sellable as a product

[mikeselectricstuff]

Of course an FPGA based hardware emulator like this could also do interesting stuff like logging of SD transactions for debugging and reverse-engineering.

[andersm]

Works if you dont plan to modify bootloader / uboot / kernel. If you do plan to modify those ( say because your are adding in netboot or NFS ) and it doesn't boot during development, get ready for a frustratingly long development cycle with lots of SD card swapping.

If that happens, you've got problems in your development process. And in any case, realistically how often will that happen? (The same question applies to your concern about SD card wear - if you have to change cards every couple years, is it a real issue?)

[tszaboo]

Still dont get, why you would need to have to emulate the interface. A normal setup would look like this:
You place the board on a "bed of pogopins" test jig. The SD card testpoints make a contact. On the base board you have 2-3-4 whatever number of SD card sockets and a digital multiplexer, controlled by the test jig. You select the SD image you want.

If you really want to plug in the SD card jig device, you can still use the multiplexer method.

[Rasz]

Still dont get, why you would need to have to emulate the interface.

I think the end goal was remote debugging without monkeys swapping cards. Both Wifi SD card and multiple cards + mux + sd reader + some host with ethernet will allow that. Except mux setup will be >$200+time to put it together while Wifi SD is $27 with free shipping on amazon right now

https://www.amazon.com/gp/product/B00GEBTFNM
https://www.amazon.com/gp/product/B01D9JIUU0
probably $50 to have it at your doorstep tomorrow.

yes, you can upload contents over the wireless interface, here is a dude using wifi SD card in his 3D printer in order to upload prints remotely without touching the card http://www.extrud3d.com/flashair

> concerned with performance and the issue of wear and tear on NAND

it would take over a year to kill $20 SD card in this scenario (~10MB every 3 minutes for bootloader+kernel+reboot?)
instead you are planning to spend couple of hundred on elaborate contraption that will take half a year to develop and debug?

[mac.6]

For kernel and boot test you can switch to tftp boot instead of using sdcard (if your SBC has native ethernet).
You will only need to update sdcard for bootloader and rootfs, and even for this you can do a two step boot (use tftp boot for kernel and rootfs, update SDcard and reboot).

[CJay]

Can you write the SD over JTAG via the host CPU?

Would save all the effort of having to swap cards in and out, just attach your JTAG host to the board under test, either via pogo pins in a test jig or 'standard' header and blast away, if you've concerns over the longevity of the SD cards then you need to be considering the quality of the SD cards.

[technix]

If your processor support it, you can even leave the card out and boot over JTAG entirely, by holding the processor in RESET, write your bootloader into RAM directly and throw PC there boot straight from RAM. After you get a working bootloader this way (I doubt if a bootloader will get too large to fit in the RAM, and if you need a multi-stage bootloader you can load each stages using JTAG when the card is missing) you can put the bootloader on SD card and start netbooting.

[vpeter]

Browsing with Google and found this thread...

I did similar circuit like this already mentioned a-microsd-card-remote-switcher.

But as core IC I took TS3A27518E which is 6-Bit, 1-of-2 Mux/Demux. Same IC is used in linaro's and tizen's design for automatic testing. Whole circuit works pretty well as prototype. To switch targets I'm using small DPDT switch (not ideal but do the job for manual work). On both sides I'm using flat cable with micro sd card at the end like this.

I would like to make a proper PCB for it but I'm no electronic guy  It should be very small and looking like SD card which would be put in usb sd card writer. On it there is mentioned IC + micro sd card. At the other end there would be flat cable mentioned. If there is someone who is interested in this let me know and I will provide all the details. I'm sure this could be used in a different ways.

Peter

[amyk]

You would need an RTOS, as the SD interface requires signal responses within a certain time frame and the clock for UHS-I (65+ MB/s) is 100MHz or 208MHz for high speeds. Even then, with such a high clock speed, I don't think this is possible via pure software emulation.. are you aware of something that could enable RAM emulation with these constraints without a micro controller?

An SD card basically consists of a microcontroller and NAND flash with firmware. Some are only an 8051, but running at ~100MHz. There's another clump of 8051s and ARMs whose max clock frequency is specified at 208MHz, likely not a coincidence. It's only the communication that happens at such a speed, so they probably have something like a DMA controller and everything else is handled by the firmware.

https://www.bunniestudios.com/blog/?p=3554

[Bruce Abbott]

It becomes a logistically nightmare to swap 5-10 SD cards every 5-20 minutes for various boards and configurations, while keeping track of what build ran where.

That's not the only logistical nightmare.  Why would you be creating 5-10 different builds every 5-20 minutes?
 

[mac.6]

This smells like continuous integration server (jenkins/bamboo) with automated builds...

[rob77]

There are no pre-verified images, this is during development hence the testing. How this process works: a new build of the firmware is generated after making changes, now we need to see if it works and the performance.

1) Write new firmware to SD card
2) Attempt to boot device (check bootloader)
3) Attempt to boot kernel (check kernel)
4) If successful boot, run series of tests/benchmarks and record results (check drivers, application, etc)

i did a boatload of embedded stuff and even custom linux distros (mainly for x86 based network gear) and i can tell you you're not doing it the most efficient way.

you have to have a "soft" test-bed for testing builds and once it passes all the regression tests in the soft test-bed then you can go for hardware.... a virtualized test enviroment will boot and run the tests in seconds, compared to minutes when testing on hardware (flash SD, boot HW) .. you can use QEMU for arm on x86 host, and for x86 on x86 host you can use any virtualization software.

in general you always have to test in software first, otherwise you'll waste you time... for example back in time when i was writing a network driver kernel module - i was using UML (user mode linux) for the early stage of development.. UML was running an another instance of linux kernel as process on the host linux OS and provided the ideal soft test-bed.. once the HW independent part was OK and stable i moved to the hardware part on real HW. doing that all on hardware from beginning, i would be still doing it (that was approx 10-11 years ago)

[vpeter]

So they and they are doing it also wrong?  I really doubt that.

[rob77]

So they and they are doing it also wrong?  I really doubt that.

you should distinguish between different use cases.
1. testing software during development - this is much faster in a virtualized environment (soft test-bed)
2. testing boards during manufacturing - this can be done with a bunch of SD cards

of course if you have a hardware without a possibility of virtualization then you have to go for testing on hardware where such a SD card switch or even SD emulator has it's benefits.

and as always - there are many ways to achieve the same goal.

so feel free to doubt anything you wish... but i've been there... just sharing my experience

[encore2097]

Thanks for all the answers. I ended up with a nifty solution.

This smells like continuous integration server (jenkins/bamboo) with automated builds...

Spot on. Though entirely custom, fast, feature-ful and the next step in how hardware and software is built and tested.

So they and they are doing it also wrong?  I really doubt that.

The folks who worked on this are amazing and great inspiration! I've spoken to them deeply about this problem and while this was the a fast solution at the time, there is much room for improvement.

So they and they are doing it also wrong?  I really doubt that.

you should distinguish between different use cases.
1. testing software during development - this is much faster in a virtualized environment (soft test-bed)
2. testing boards during manufacturing - this can be done with a bunch of SD cards

of course if you have a hardware without a possibility of virtualization then you have to go for testing on hardware where such a SD card switch or even SD emulator has it's benefits.

and as always - there are many ways to achieve the same goal.

so feel free to doubt anything you wish... but i've been there... just sharing my experience

Its quite amazing how many sharp people tell someone they are wrong or doing things "inefficiently" without even understanding the problem. You have made so many assumptions without even attempting to understand my use case.

I'm not using x86, but custom RISC hardware. I've already tested in software and trying to emulate in virtualization would be 100x slower than hardware and require FPGAs. e.g. How do you "test" HDMI/LVDS bring up or hardware accelerated network features of your NIC in software?

Whats the difference between theory and practice? In theory, there is no difference. If you are not testing on actual hardware, you are shipping your customers lots of bugs.

[mikeselectricstuff]

Its quite amazing how many sharp people tell someone they are wrong or doing things "inefficiently" without even understanding the problem.

But it can equally be the case that you are utterly convinced you have thought through all the possibilities, but someone else comes at it from a different angle to solve a problem in a different way
you'd not considered

You have made so many assumptions without even attempting to understand my use case.

Maybe if you'd given more details of your requirements, you'd have got more appropriate suggestions.

[Rasz]

How do you "test" HDMI/LVDS bring up or hardware accelerated network features of your NIC in software?

what does that have to do with testing bootstrapping?
hardware emulation of boot medium gives you precisely ONE advantage - ability to test bootloader. With bootloader tested and working everything else is just network/serial connection transfer away.

You dont need to swap(virtually) SD cards to test hdmi. You sound very young and enthusiastic, and quick to dismiss old farts who have been there and done that several times over.

[3db]

How do you "test" HDMI/LVDS bring up or hardware accelerated network features of your NIC in software?

what does that have to do with testing bootstrapping?
hardware emulation of boot medium gives you precisely ONE advantage - ability to test bootloader. With bootloader tested and working everything else is just network/serial connection transfer away.

You dont need to swap(virtually) SD cards to test hdmi. You sound very young and enthusiastic, and quick to dismiss old farts who have been there and done that several times over.

Are you an old fart too ?   
I'm an old fart have you graduated to grumpy old fart yet ? 

[il__ya]

Thanks for all the answers. I ended up with a nifty solution.

encore2097, sorry for bumping this 3 years old thread, but may I ask what the eventual solution was? I am facing a similar problem at the moment, and would be interested to know how you solved it.

[encore2097]

Thanks for all the answers. I ended up with a nifty solution.

encore2097, sorry for bumping this 3 years old thread, but may I ask what the eventual solution was? I am facing a similar problem at the moment, and would be interested to know how you solved it.

Its unfortunate that I missed this.

Here's what I came up with: https://hackaday.io/project/19783-sd-card-emulation

[coromonadalix]

i recently saw an sd card emulator on Ali for Dwin Hmi displays

https://www.aliexpress.com/item/32917802827.html?spm=a2g0o.cart.0.0.2c523c00ps3UBV&mp=1

10,80$ usd

and this one too

https://www.aliexpress.com/item/32830983955.html?spm=a2g0o.cart.0.0.2c523c00ps3UBV&mp=1

9.90$ usd ??

[donotdespisethesnake]

Those don't emulate an SD card, they are adapters which provide a comms channel from SPI to the USB. The SD card socket is just a way to get to the SPI signals.

[aldolo]

Any news on this topic? So far I'm considering using 2 real sd card, one on-line and one off-line. the off-line one could be written to using an arduino while the on-line one is used by the final device. then it is easy to swap the 2 with few gates.

[SimonR]

Somewhere around 2007/8 I did exactly this more or less.
My company developed an ASIC that had an SDIO interface which is effectively an SD card with a peripheral on it instead of flash memory.

During development we used an off the shelf SDIO chip until the ASIC (Test FPGA) was ready. Our application was communications device that transferred a lot of data. You could easily have used the chip to implement a memory card, which we were also considering at one point for test purposes.

The chip in question was from a company called Arasan and just a demonstration chip for their IP which I think we used. Its not available any more but Arasan seem to be going still

https://www.arasan.com/products/

Alternatively while I was searching for the Arasan chip I also found this from Lattice

http://www.latticesemi.com/en/Products/DesignSoftwareAndIP/IntellectualProperty/IPCore/EurekaCores/EP560SDSDIOMMCSlaveController

This looks like it might be what everyone is looking for.

Just a suggestion

[ethaniel]

Sorry for bumping up an old thread. I'm willing to set a $2K bounty on this if anyone is interested in exploring the idea.

I'm trying to find a way how to capture data that's being written to a microsd card and receive it live on linux.

I've got a GoPro MAX which can record 360 degrees video. It doesn't have an HDMI output.
I want to extract the h264/h265 video data as it's being written into files on the "virtual" microsd card.

I assume there is alot of wizardry involved - letting the gopro know that this it's a working sd card with an actual filesystem and free space available. Perhaps somehow patch it through to a virtual block device in linux? There is a lot of data transfer involved (100+ megabytes per second).

My goal is to use feed the raw h264 into a streaming program, so I could use the GoPro as a 24/7 live camera.

Anyways, the bounty is $2K. Looking forward to hearing back from you.