FPGA/uC high speed flash memory interface

[AlexK]

Hello,

I've been researching this for a while now and I keep hitting a dead end, so any ideas and suggestions will be appreciated.

I'm trying to design a high speed data logger whose function is to sample some analog sensor output and store the information on a flash memory. My basic requirements are:
1) The sampled data is coming at a rate of 30MBytes/sec.
2) The data must be stored to the flash as "real time" as possible (i.e. no large ram buffers).
     Basically this means: Write speed to flash memory: 30MBytes/sec.
3) The data logger must be portable: small size, battery powered, stand alone (i.e. not connected to a pc)
4) The write operation to the flash must start immidiatly at power up (i.e. no long boot times of processors running linux operating systems)

First thing I did was to search for a (small form factor) flash memory. The obvious choices for flash memory card that supports 30MBytes/sec are:
1) SD in UHS-I mode (SD 3.0 compatible)
2) Compact Flash with Ultra DMA - 4 mode.

Next step was to find a host that can interface with the above cards(and this is why this post is in the FPGA/uC section of the forum):

• All the microControllers I've looked at (STM32, Atmel UC3 for example) only support the SD 2.0 specification
• An FPGA IP core that supports SD/CompactFlash is too expensive (over 10K$).
• OpenCores doesn't have any cores that support SD 3.0 or CF UDMA-4.
• I also looked at USB storage devices. It seem's that the required write is speed only possible with usb 3.0 interface, but I couldn't find any uC that supports usb 3.0 either.

I've also looked at MemoryStick,MMCplus,NAND flash memories and even SATA storage, but the problem remains: either the memory card doesn't support the 30MB/s or I can't find a host that does.

Any ideas on how I might tackle this problem ?

Thank you in advance,
Alex

[Christe4nM]

Forgive me for asking, but the 2nd requirement: is that really what the requirement is, or how the requirement was understood?

What I mean to say is that, I've seen to many designers delving right into finding a solution for a certain requirement while the requirement was not actually what was needed, but how it was understood. So maybe, just maybe, it isn't necessary to write to memory at that speed. Still if so, there might be clever design methods so you can still use "slower" memory. Unfortunately I'm not able to help you there since my knowledge is limited in that regard.

[nctnico]

How about using a small ARM board with Linux?

[mikeselectricstuff]

A significant issue is erase time - if you can 'pre-format' the flash beforehand so it never needs erasing, this will make the task easier.
If not you'll need to include enough buffer RAM to cover both erase and write times

For that bandwidth you should take a serious look at whether some compression is possible. 
NXP LPC18xx supports a fairly fast SD flash interface - not sure offhand how fast, however for max speed, an FPGA will probably give you the best performance.

If you can't get the  throughput, the only solution is to go parallel - use multiple devices simultaneously. This is very scalable, and an FPGA would be the way to do it as you can instantiate as many interfaces as you need, as well as interface to any buffer RAM you may need to deal with lumpiness of flash transfers. In terms of pin count, SD would probably be the most practical.
If you want to ensure multiple cards don't get mixed up , you could even glue them together to make a 'wide card'...!

Another option, if your capacity requirement is well defined, is to use discrete NAND flash chips - this will give you better low-level control over things like erase time and bad-block handling, but you will need to implement ECC. It also give you some options to do clever stuff like erasing one chip (or bank within a chip) while writing to another chip or bank. Similarly you can also use this method to cover any program times. From memory, program time for nand is fairly quick (few 100 uS per page), but erase is tens to hundreds of mS, and the erase page is generally quite a lot bigger than the program page.



 
 

[marshallh]

You will need one or two SD interfaces.You can write a CF controller in a couple days. If you really get with it, you can have SD 4bit mode written in several days. Done both.
Block NAND devices like this will spontaneously require breathing time while writing for block erases and other things. Up to 100-200ms latency sometimes. So, you will need 32mb SDRAM or somewhere around there. On top of the blockram fifo buffers inside your design.

I just looked at the UHS-I spec. It would be a minor amount of work to modify the sd core I have for that using DDR output buffers.

These things aren't that bad, but if you have not everything all together before it will be a huge challenge.

[fpga]

Hi Alex,

First, you need to clarify several requirements.

1. How much data do you need to store?
2. Is the 30MBytes/sec sustained or is it just peak throughput with a lower average throughput?
3. Why does the data storage need to be "real time" with no large buffers?
4. Can the device start from a low power sleep state?
5. What is more critical, development cost or production cost?

These clarifications will help clear up which architecture choices need to be made.

Factors that need to be considered are:
1. None of the small ARM Cortex M3/M4 micros will be able to sustain that type of data flow.
2. SD card high speed interfaces, i.e., UHS-I, SDXC, are expensive to get the full SD Association specs, and even more expensive to license the IP cores.
3. Discrete NAND flash devices require implementation of error correction and wear leveling.
4. Writing to any type of Flash memory is done sector by sector, so somewhere the buffering needs to be done.

If development cost is a bigger issue than production cost, then an ARM A8/A9 micro with a PCIe or SATA-II interface and a SSD drive, or a SOM board with these interfaces running embedded Linux is not a bad choice. There are a number of older ARM 9 micros that have an onboard IDE interface that could read/write to compact flash, but I doubt that they could handle that type of throughput.

To do all this in an FPGA from scratch is doable, but it is a significant project.

[mikeselectricstuff]

There may be another approach which could be substantially less development effort.
If you can find a microcontroller based hardware+software solution that is in a similar order of magnitude as your transfer rate, you could set something up where the incoming data is divided into multiple interleaved streams (simple CPLD,FPGA, or maybe creative use of internal counters to catch every n'th  sample), each feeding its own MCU+storage module.

Even if you needed as many as, say, 16 of these modules, each module only needs to handle less than 2MBytes/sec, with maybe 100-200K buffer to cover erase times. This is getting close to what devices like the LPC1800 can handle using readily available free libraries. 
OK it's not pretty, and readout might get interesting, but it could do the job at a not-ridiculous hardware cost, and fairly low and predictable development effort.

[AlexK]

Thank you all for your ideas and suggestions.

If you can find a microcontroller based hardware+software solution that is in a similar order of magnitude as your transfer rate

Actually we did manage to reach a writing speed of almost 15MBytes/sec using an ATMEL uC and a class 10 SD card. So I guess a configuration with an FPGA+2 micros+2 SD cards is possible but I'm still hoping there is a more elegant solution.

1. How much data do you need to store?
2. Is the 30MBytes/sec sustained or is it just peak throughput with a lower average throughput?
3. Why does the data storage need to be "real time" with no large buffers?
4. Can the device start from a low power sleep state?
5. What is more critical, development cost or production cost?

1. 2GByte should be enough.
2. The 30MBytes/sec is (unfortunately) sustained.
3. The device will basically serve as a "black box" for a high impact platform. Since I'm not sure whether the circuitry will survive the impact, I prefer that the data will already be stored on the flash memory at the moment of impact (I'm assuming of course that the flash will survive).
4. No,I'm afraid it can't. It should start from complete power down.
5. At the moment only a few prototypes are planned so development cost is more critical.

You will need one or two SD interfaces.You can write a CF controller in a couple days. If you really get with it, you can have SD 4bit mode written in several days. Done both.

You took me a bit by surprise there. Since I'm not an experienced HDL programmer I wasn't too eager to build the flash memory controller from scratch, but if you say that it can be done in a short time maybe I should reconsider. If I'm not mistaken the SD specification costs around 1000$ while the CF spec costs only 100$ (and is probably easier to implement). Do you think this is the way to go? Write my own CF controller (with UDMA-4) for an FPGA? (BTW, no file system is required so maybe this simplifies things a little more)

Block NAND devices like this will spontaneously require breathing time while writing for block erases and other things. Up to 100-200ms latency sometimes. So, you will need 32mb SDRAM or somewhere around there. On top of the blockram fifo buffers inside your design

I was completely unaware of that. Are those "breathing times" required even if no block erase is performed (i.e. I write to the memory card until it is full and then stop)?

Thanks again.

[nctnico]

IMHO you are way better off using a Beagle board or something similar. The development cost will be close to zero compared to building something.

[marshallh]

Simplified spec here
https://www.sdcard.org/downloads/pls/simplified_specs/part1_410.pdf
It is completely possible to implment 4bit mode off this spec. And from the looks of it, UHS-I as well. The AC timing and voltage info is broken off into the paid material ($2k'year) but you can most likely forgo that.

Skimming through it seems most prudent to use DDR50 mode. This lets you keep the main controller logic in a 50mhz domain which will be easy to close timing on. You'll have to use the device specific DDR outputs, in altera's case that would be ALTDDIO. Here it would be convenient because it'd let you pump 8 bits at a time through the interface at 50mhz, even though it will be 100mhz effectively.
This is better than using the SDR100 mode because it doesn't seem to require phase tuning. (When using old SDRAM at >= 100mhz since you have no data strobes, it's common to provide the chip with a phase shifted clock to compensate for pcb delays). This will give you 50mbyte/sec theoretical throughput

So the FPGA option will get you extremely precise and completely deterministic timing. Buffering the data locally in some sdram will complicate things but it's still doable. The downside is the dev time if you haven't done much FPGA work. I'm not an HDL expert but I'm also far from a noob anymore. If you want PM me and I can send you the SD controller I wrote and you can get an idea of what your are getting yourself into.

I looked at some of the bigger ARM SoCs and they seem to support UHS-I but it's hard finding exact details on this. And then if you go this route you have to design for the device which is more complex than plopping an FPGA on a pcb will be. Take a look at Freescale's i.MX6 series, it is the most likely to support UHS-I and you don't need NDAs or anything to buy them in singles off digikey (Looking at you, Broadcom)

[marshallh]

Also regarding SD "breathing times" these vary depending on the manufacturer, production date, revision, etc. There is basically a 32bit ARM in every SD card with some custom LSI for interfacing on both ends. The controller ALWAYS erases before it writes, this is how flash rom works. Erasure sets the cells to 0xFF and "writing" just flips the necessary bits to 0.

For best throughput your SD contorller will do a specific command (WRITE MULTIPLE) that tells the SD card "I want to erase X contiguous blocks in preparation for writing." The specifics are in the spec. It is critical that you do this. Find the typical NAND block size of the SD card (maybe 16/64kbyte) and use at least that for your minimum write size.

The reason for this is that the NAND can ONLY be erased in these large block sizes. So if you say "I want to write 2 blocks and that's it" a block according to the SD spec is 512 bytes, so the card has to read off the entier blocks' contents, ERASE the entire block, write your 1024 bytes of data, then copy back all the remainder of the block that was there before.

CF has the same deal, historically it has been much faster thanks to its wide physical data bus, but the new UHS additions are closing that gap anymore.

[mikeselectricstuff]

Thank you all for your ideas and suggestions.

If you can find a microcontroller based hardware+software solution that is in a similar order of magnitude as your transfer rate

Actually we did manage to reach a writing speed of almost 15MBytes/sec using an ATMEL uC and a class 10 SD card. So I guess a configuration with an FPGA+2 micros+2 SD cards is possible but I'm still hoping there is a more elegant solution.

How is your data arriving ? you might be able to do it with no FPGA at the front end, or a simple CPLD. If, for example it's a parallel data stream, you can probably use timer/capture peripherals to grab every n'th sample. However routing a parallel 30mbyte/sec bus round several MCUs could be fun, so a CPLD/FPA may make things easier, with minimal HDL effor

You will need one or two SD interfaces.You can write a CF controller in a couple days. If you really get with it, you can have SD 4bit mode written in several days. Done both.

You took me a bit by surprise there. Since I'm not an experienced HDL programmer I wasn't too eager to build the flash memory controller from scratch,

This might be an option http://www.toshiba-components.com/memory/benand.html
SLC NAND flash with inbuilt ECC. You still need to manage bad blocks, but that can be pretty easy.
ECC during write isn't all that hard to do - it's just generating parity bits. read is a little more fiddly but you're not so  time constrained & you could do that on a host PC after uploading

 - I found that bad blocks can be identified by longer erase times, so a simple pre-formatting operation can find all the bad blocks. You can then deal with them by building a simple lookup table of logical address to physical address, e.g. in a seperate SPI flash. As you are streaming data out, you will be needing to do the translation in a very predicatable fashion so this should be easy.
A big advantage of going with discrete NAND flash is you have full control over erase & program times, and when you do them - the problem with a generic disc type device is it doesn't knwo what you are about to do next, so for example it will always erase a block before writing. In your application you will be able to pre-format, and so have zero erase time issues. (Assuming you fill the memory then stop, and not a rolling FIFO buffer)
Your page program times will also be very predictable, so you will be able to have minimal RAM buffering (easily low enough to use FPGA's internal RAM) by interleaving writes between 2 chips, or 2 banks within one chip (i.e. write to one chip during page program time of the other).
I just looked at a random NAND datasheet, which shows typical page write times of 200uS (2K page) and block erase (128) of 500uS.
 
In terms of ruggedness, not having a device in a socket could be a big improvement.
Raw NAND chips are very easy to interface - the interface is standard - I recommend Microns datasheets as being particularly well written. Certainly easier than any CF or SD type device for your 'write a big single lump of data' type application.

As you only need 2gbytes, I'd say the simplest solution by quite a long way is FPGA and NAND flash chip(s), probably a microcontroller to oversee things like start/stop control & generating the badblock map etc., but wouldn't need to ever see the actual data, so no major speed or memory requirements.

One disadvantage of this is you can't just plug it into a PC to get the data out, but getting fast data out with  FTDI's high-speed USB devices is pretty easy if you don't need ultimate performance.
 

[KedasProbe]

Simplified spec here
https://www.sdcard.org/downloads/pls/simplified_specs/part1_410.pdf
It is completely possible to implment 4bit mode off this spec. And from the looks of it, UHS-I as well. The AC timing and voltage info is broken off into the paid material ($2k'year) but you can most likely forgo that.

An older version but with all the info:
http://forums.parallax.com/attachment.php?attachmentid=100220&d

[amyk]

There is basically a 32bit ARM in every SD card with some custom LSI for interfacing on both ends.

More likely 8051.

[andersm]

There is basically a 32bit ARM in every SD card with some custom LSI for interfacing on both ends.

More likely 8051.

Samsung have used ARM7TDMI at least at some point (see near bottom of post).

[Jeroen3]

Can you use some kind of data compression to reduce the throughput?
Does quad SPI flash write fast enough?

[mikeselectricstuff]

Can you use some kind of data compression to reduce the throughput?
Does quad SPI flash write fast enough?

QSPI flash has much lower capacity than NAND (but without the headache of ECC and bad-block stuff), but is mainly optimised for very fast reads and random access.
Write time is much longer than NAND (2ms for 256 byte page) , and erase time is  horrific (about 1 second per MByte).
 

[Jeroen3]

You can use multiple SD cards on one bus. The only limit is the SD card interface speed.