Generic deep memory data acquisition system

[allanw]

I was thinking up of something ridiculously ambitious to attempt to do for next year. I'd like to make a data acquisition system that takes standard computer DDR2 RAM modules and buffers data in there before transferring it out with USB. Theoretically it's possible with dual channels to get to 1600Mbit/s (which is only 200M/s at 8-bits)

Since this would require an FPGA anyway, I thought it'd be a good idea to just make it a modular DAQ: have an FPGA board with generic inputs and build separate input boards that have specific ADC's. For instance, one input could be a 14-bit ADC at 100MS/s and a different one could be an 8-bit ADC at 200 Msamples/s. This also cleanly separates the analog and digital portions of the system.

The Spartan-6's are low-cost and already have two memory controllers on-board. There's a relatively cheap $200 dev board with 128Mbytes of DDR2 RAM on there already. The only issue is the cost of the software and the additional cost of using the memory controller, which I haven't really researched yet.

But is this something that will have a use? The benefit is that with 4 gigs of ram, you can acquire a full 2.5 seconds of data at 200M/s. I know 200M/s isn't all that impressive but it seems like it's impossible to get any faster storage with using off-the-shelf technology. If the waveform is repetitive and I can get sampling clock jitter low enough, then the equivalent sampling rate would be ridiculously high, which could be useful, right?

Or maybe I should try DDR3, which is twice as fast as DDR2. But then there aren't cheap dev boards with that and I'd have to lay out a board myself which is definitely a daunting 6-layer task.

A company called Ultraview sells DAQ's that also stream to PCIe (2Gbit/s (but where does it go?)) which seems to be the next plateau in sampling speed. Picoscope also offers similar ones. They are both above $4000 which seems expensive to me.

I thought it'd be interesting to do it to learn about high speed design and could be useful to others if well documented. Any thoughts?

[slburris]

The Spartan 6 has only 1 non-BGA package, the 128 pin quad flat pack, and that package
doesn't have memory controllers.  Availability of any Spartan 6 chip seems to be pretty
poor, except at Avnet.

That being said, are you planning on using a dev board and building daughterboards for it
that take the RAM modules?  Or do you have the capability to mount BGAs?

You could get your speed up by using RAM modules in parallel, although the details
get trickier as you go wider.

Whose dev board do you like?  I only know of the Digilent Atlys which is
$199 for students, but $349 for me.

I've tried hard to convince Digilent that hobbyists are "students of life" but
they aren't buying it :-(


Scott

[allanw]

Going to have to use BGA's for this. I can probably hand solder them with a reflow oven.

Yep I got the $200 Digilent board (student here ). Initially I'll see if this is feasible with the on-board 128MB DDR, and then develop a prototype with DDR2 slots. Unfortunately it seems like 6 layers is the minimum here, with 4/4 spacing and tiny holes. Each prototype board revision is going to cost at least $500 which sucks since I've never actually done any high speed design before. Hopefully I can get it right after reading the tons of app notes available and finding some good simulation software. I'll start off small with the relatively high speed ADC boards for this dev board first though.

I'm splitting this up with a partner - he will do the Verilog and digital work and I'll figure out the analog and board-level stuff. We actually don't really know how DDR2 works, so it's going to take some research into seeing what kind of continuous bandwidth is achievable for parallel DDR sticks. (is one stick more parallel than one DDR chip or what?) Luckily the hardware memory controller handles much of the details.

If this works out then the next step is to build the ultimate DAQ with DDR3 and PCI-Express.

[slburris]

For the DDR memory design, you'll need to make sure your PCB traces to memory are approximately equal
length if you want to run the memory at full speed, I think.

But if you are taking on BGAs, you probably have PCB software that helps out with that :-)

Does it make any sense to capture to static RAM and then use the SDRAM as a
deep store?  You might be able to burst capture at higher speeds for a bit
given the simpler timing.  I don't know -- it's an interesting problem.

Scott

[allanw]

Yes, SRAM might be necessary, to buffer up enough data to write to DDR at once. The memory on the FPGA might not be fast enough to do it. However it seems like it'd be a very complex scheme of writing to the SRAM as fast as possible while reading from another one into the FPGA into DDR.

[allanw]

Looks like my idea for modular FPGA DAQ inputs has already been done:

http://www.xilinx.com/products/boards_kits/fmc.htm

But I bet these are ridiculously expensive.

[TechGuy]

A few  comments

1. DDR2 DIMMS are 64-bit (8 byte) devices. at 200 MS/s you would only need to write at 25 Mhz, You would need to store the ADC data into a set of eight 8-bit registers and then write them to DDR2 every eight clock cycle. This is of course is assuming your using an 8 bit flash ADC. for 14/16-bit resolution every fourth cycle will be commited to DDR2. DDR2-1066 Maximum I/O is 8533 Megabytes per second, not megabits.  You should have no problem storing 200 M/s using even DDR2-400 (3200 MB/s with a Memory clock speed of 100 Mhz).
http://en.wikipedia.org/wiki/DDR2_SDRAM

2. To get even higher sampling frequency you can divide the data into multiple ADCs\DDR2 channels. Many years ago I used four 7-bit flash ADCs to quadruple the samping rate. For a pair of ADCs you just a use an inverter. so that the one ADC samples on the high side of the clock, and the other on the low side of the clock cycle. This can be increases to four using a frequency divider so that each ADC samples at 1/4 or the clock speed. This can also be applied to RAM I/O. Back then FPGA where really just getting started. I used a standard logic devices using  a bunch of 8 bit counters to provide the address bus I/O and a set of 4 Static RAM chips to store it in. I used a PC Parallel port to down the the data to a PC and a few of the parallel port I/O pins to toggle between data capture and data dumps.

3. I would avoid buffering to static RAM, since it will greatly increase the design complexity.

4. You might want to consider using an external DD2 memory controller instead of an integrated controller in a FPGA.

For instance, one input could be a 14-bit ADC at 100MS/s and a different one could be an 8-bit ADC at 200 Msamples/s.

I would pick one ADC resolution instead of trying to support two different resolutions. You can use two 14-bit ADCs to get 200MS/S or four 14-bit ADCs to get 400MS/s. Although I don't really see the need for 14-bits at such sampling rates. usually 8 Bits is plenty of resolution for high speed acquisition systems. Virtually all digitial oscilloscopes use 8 or 9 bit ADCs.

[allanw]

Thanks for the help!

DDR2-1066 Maximum I/O is 8533 Megabytes per second, not megabits.  You should have no problem storing 200 M/s using even DDR2-400 (3200 MB/s with a Memory clock speed of 100 Mhz).[/url]

Ah yes, I didn't account for how wide the memory is. But that's only if you don't need to change banks right? Or is this time almost negligible?

4. You might want to consider using an external DD2 memory controller instead of an integrated controller in a FPGA.

I couldn't find any. Unless I put a motherboard northbridge in there, but that'd make things complicated. The Spartan-6 has two memory controllers in the smaller packages, and 4 in the bigger ones. I think that should be enough.

I would pick one ADC resolution instead of trying to support two different resolutions. You can use two 14-bit ADCs to get 200MS/S or four 14-bit ADCs to get 400MS/s. Although I don't really see the need for 14-bits at such sampling rates. usually 8 Bits is plenty of resolution for high speed acquisition systems. Virtually all digitial oscilloscopes use 8 or 9 bit ADCs.

Since it's an FPGA it shouldn't really matter what I do right? Since the FPGA will take in a bunch of general IO from the input board. I decided to start with a 14-bit 100Msample/s board because the ADC is really cheap at only $30 and I wanted to try designing a board with that high of a resolution (low jitter clock, etc)

edit: oops, post got screwed up

[TechGuy]

I couldn't find any. Unless I put a motherboard northbridge in there, but that'd make things complicated. The Spartan-6 has two memory controllers in the smaller packages, and 4 in the bigger ones. I think that should be enough.

Yup, You're probably right. There seems to be slim pickens for DDR memory controllers. I had assumed that there were standalone DDR2 controllers for MCU's that didn't have DDR memory support built in. Looks like FPGA is probably the only reasonable way to go.

Ah yes, I didn't account for how wide the memory is. But that's only if you don't need to change banks right? Or is this time almost negligible?

I am not sure if bank changes would impact I/O. I know that DRAM refresh may require a delay. I am not sure if SDRAM now has built in refresh logic or if the memory controller still need to run refresh manually. I really wish DRAM manufactures made DRAM so it works just like SRAM, so its simple to interface.

[slburris]

I still think it would be easy to read from the ADC's into Spartan block RAM.
Since the BRAM is dual-port, *and* you can do format conversion, i.e x8 on
input, x32 on output for example, you could then burst out to the DDR RAM
and hide little things like pre-charge, bank switching, etc.

Mind you I haven't done such a design, just thinking off the top of my head.

Scott

[jahonen]

Timing constraints should be considered quite early on design, although it seems that 100 MHz should be easy, there can easily be quite slim window of success, when you add all the timing budget things together. For example, bad selection of FPGA pins may lead to non-achievable timing target, be careful with that.

Signal integrity simulations are also useful. I can only give advice that use point-to-point wiring wherever possible, multidrop bus is considerably more difficult to design, and may be even impossible if IO-pin drivers are not hefty enough. You can easily get very strange and extremely difficult to find faults if your signal edges are not monotonic enough (been there, done that!). Again, the signal integrity simulator will show a good estimate if your driver is up to task.

Dynamic RAM needs refresh cycles now and then (I believe that in intervals of ~8 µs), furthermore there is a maximum time that a page can be open at one time. Result is that data rate is somewhat lower than one would expect. Synchronous static RAM is able to sustain new data on each clock period, but that is several orders of magnitude more expensive. For best latency and difficulties, the acquisition clock frequency should be designed in such way that there is no need for clock domain crossing logic, that can be a pain sometimes. Although FPGAs usually feature a dual-clock FIFO, which solves the problem.

Regards,
Janne

[allanw]

I've decided to go with this ADC for now: http://www.national.com/ds/DC/ADC11DV200.pdf

It seems like the timing is very tight for the DDR LVDS output at 200MHz. The datasheet says hold and setup times are 1.2ns typical, 0.7ns minimum, with a max data-data skew of 0.47ns, typical 0.02ns.

If I put the data outputs through an LVDS repeater, it adds a channel-channel skew typ 0.02ns, max 0.15ns. I'd need two repeaters so then there's another 0.5ns max part-part skew.

In the worst case accounting for all the skews, it's not even possible. WTF?

[joelby]

You could have a look at Linear's PStache board - it's a data acquisition thing that you can plug a bunch of different Linear demo boards for ADCs into. I don't think the schematics are available, but from the photo it looks like it has two FPGAs and some external memory.

http://cds.linear.com/docs/Reference%20Design/DC1371A2-QS.pdf