EEVblog Electronics Community Forum

Electronics => Microcontrollers => Topic started by: Mechatrommer on April 21, 2018, 10:21:37 pm

Title: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 21, 2018, 10:21:37 pm
as someone said it, let me consider a theoretical idea of a spherical dead cow in a vacuum (whatever that means is)... this is just out of my imagination...

lets say i have 4X 8 bit ADC sampling at 1.5GSps each, and then all of the data are sent in staggered manner like Rigol DSO did, or just simultaneously into a FPGA to get effective 6GSps acquisition. and then the data are sent to memory RAM bank (1 bank is enough? or 4 banks is ok?). if 1 memory bank/chip per adc architecture is used (ie 4 banks total), at 1.5GSps data rate, i'm guessing for parallel interface, the clock is the same as that or twice. if using serial interface, then the clock at least 8X of that ie 12GHz, i dont see anything closer to that in digikey (2GHz is the RAM max clock in stock/list), am i delusional here? so question...

1) is there a FPGA that can handle this magnitude?
2) same as RAM? from digikey parallel interface @ > 1.5GSps, there are few candidates, but mostly no picture, not in single quantity purchase, or obsolete model.

Waiting your reply thanks.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NiHaoMike on April 21, 2018, 10:42:30 pm
6GS/s at 8 bits is 6GB/s, which is a lot but common DDR3 easily does more than that.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 21, 2018, 11:11:27 pm
6GS/s at 8 bits is 6GB/s, which is a lot but common DDR3 easily does more than that.
so i guess 1 RAM per ADC is a wise choice. 1.5GB/s should give more lee-way to the requirement and broader option in RAM IC... i suspect a FPGA should be powerfull enough to send 32bits data to RAMs @ 1.5GHz clock rate, no? i'll search for it later, i just dont know which specification is 6GHz and how many cells and internal memory required.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NiHaoMike on April 21, 2018, 11:14:46 pm
The tricky part is going to be the PCB design. Just look at some PC motherboards to get an idea of how intricate high speed wiring is.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 21, 2018, 11:32:46 pm
There are few issues here.

First - data acquisition interface. A lot of really high-speed ADCs have JESD204B interface, and for that you will need to have FPGA with MGTs to interface with them as regular IO pins are not fast enough. If you use Artix-7, there are one or two MGT quads (4 tx and 4 rx each, the only device with two quads is the largest Atrix-200). If the ADC of your choice has LVDS interface, this will need quite a bit of pins and quite fast SERDES. There is a couple of "traps for young players" though - MGT can only do up to 3Gbps at speed grade -1, you will need -2 to get full 6G, and the same goes for SERDES - at grade 1 they can only go up to 950 Mbps in DDR mode, while they will go up to 1250 Mbps at speed grade 2.

Second, memory. Same Artix-7 can do 800 MT/s (400 MHz x 2 transfers per clock), so you can get 800 MB/s for a byte-wide interface (x8). The widest you can do is x64 (you will need to use FBGA-676 package for that), that will give you 8 x 800 = 6400 MB/s = 6.25 GB/s. But this is the theoretical maximum, real bandwidth is going to be less than that because DDR3 needs to be refreshed every once in a while. If you want faster, you will either have to run the memory at 533 MHz (that will yield 1066 MT/s, but that is only possible for speed grade 3 devices), of use Kintex-7 devices which can run the memory at up to 1600 MT/s, which is essentially a double of what Artix can do.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 21, 2018, 11:35:32 pm
i suspect a FPGA should be powerfull enough to send 32bits data to RAMs @ 1.5GHz clock rate, no?
No way that's happening. Nothing can move inside FPGA at that kind of rate (excluding MGTs, but they work as SERDES where their parallel interface works relatively slowly).
Title: Re: 6GSps FPGA to Memory Handling...
Post by: BrianHG on April 22, 2018, 12:09:50 am
i suspect a FPGA should be powerfull enough to send 32bits data to RAMs @ 1.5GHz clock rate, no?
No way that's happening. Nothing can move inside FPGA at that kind of rate (excluding MGTs, but they work as SERDES where their parallel interface works relatively slowly).
800Mhz clock on DDR3 (This means 1.6 gigawords per pec), Altera Stratix V and Arria V GV can handle that and then some.  (Some of these need QDR ram to get this speed, others can do it with off the shelf DDR3/DDR4) Read here: (I would still use 64bit ram, most likely a SODIMM ram module, at this rate, you can use Stratix IV, Arria V)
https://www.altera.com/support/support-resources/external-memory.html (https://www.altera.com/support/support-resources/external-memory.html)

Use a smart ram controller with a smart crossed interleaved upper/low\er address page banking, when sampling, your ram will never need a refresh cycle.  (Done this trick in my video scan rate converter, however, I wrote my own 8 port ram controller and the smart refresh generator 'ticked' off memory pages being access, so, when rendering video continuously, the memory would almost never need a refresh cycle.  I would have to turn off all the video access for the refresh cycles needing to begin.)

If you want some good clearance with only a single 32bit port, you will have to use the Intel Arria 10, which can run DDR4 at 1.2Ghz, or 2.4 gigawords/sec, or, DDR3 at 1.066Ghz.  (You are cutting things really close with DDR3, don't forget there is overhead and you can only effectively get this speed with nice clean sequential page bursts.  Trying to address and write single bytes wont cut it.  Also, don't forget that in the FPGA, your cache buffers are running in a double or quadruple sized buss at maybe 1/4 your sample clock rate.  IE, 8 bits at 1.5Ghz are cached as 32 bits at 375Mhz.)

This is big $$$.  When Stratix 10 becomes available, it's internal cache ram can only operate at 1GHz, though you get that beautiful 2.6GHz DDR4 support.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 22, 2018, 12:47:42 am
800Mhz clock on DDR3 (This means 1.6 gigawords per pec), Altera Stratix V and Arria V GV can handle that and then some.  (Some of these need QDR ram to get this speed, others can do it with off the shelf DDR3/DDR4) Read here: (I would still use 64bit ram, most likely a SODIMM ram module, at this rate, you can use Stratix IV, Arria V)
https://www.altera.com/support/support-resources/external-memory.html (https://www.altera.com/support/support-resources/external-memory.html)
That is Kintex-7 territory. They can also run DDR3 at 800 MHz (1600 MT/s), but unlike greedy Antel Xilinx allows you to do that using K70/K160 devices without paying a single cent for the software or IPs. For me personally that's a big advantage. Moreover, even some cutting edge UltraScale/UltraScale+ devices are included in free WebPack edition - but here it's getting to the territory where the chip's price starting to approach a ballpark of what license would cost. Still - I'd rather buy 6 Zynq UltraScale+ MPSocSs (~$500 each) than spend that money on a license ;)

If you want some good clearance with only a single 32bit port, you will have to use the Intel Arria 10, which can run DDR4 at 1.2Ghz, or 2.4 gigawords/sec, or, DDR3 at 1.066Ghz.
That is multi-thousand $$ affair, Xilinx UltraScale/US+ can do that too (DDR4 up to 1333 MHz IIRC). But that doesn't mean that an internal bus can run that fast. If my memory serves me, even on super hi-end FPGAs you can only run logic at like 600-700-800 MHz. That's what I meant when I said "No way to run anything inside at 1.5 GHz".
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NiHaoMike on April 22, 2018, 01:40:14 am
A long time ago, ATI (before they merged with AMD) did make GPUs with video input for connecting cameras. I think the real problem is lack of demand.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: ejeffrey on April 22, 2018, 03:12:51 am
I've got a design that does that with DACs.  8x AD9736 connected to an Arria V FPGA.  Those are 14 bit DACs operating at 1 GS/s.  They are kind of old school and use a parallel LVDS interface, so that is 120 LVDS pairs on the FPGA.  You can do the same with ADCs, and up to 1-2 GS/s there are still available with LVDS.  Faster chips typically use JESD204b, which requires an FPGA with lots of 10G+ transceivers.

We only use SRAM.  DDR memory has plenty of bandwidth, but to have both DDR memory interface *and* 120 LVDS transmitters would require a much larger FPGA.

The big gotcha with DRAM is the refresh.  You don't want the ADC to stall out because of a memory refresh.  That means you either have to have a big SRAM buffer or find a way to avoid hitting the refresh.  The way to make that work is application dependent.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 22, 2018, 07:01:49 am
sorry i wasnt clear enough about this, i'm so naive in this thing... i dont even know how to program in FPGA :palm: i wish there is MCU to do this kind of thing, but at GSps multibus rate, FPGA seems inevitable. i was looking at https://www.digikey.my/product-detail/en/texas-instruments/ADC081000CIYB-NOPB/ADC081000CIYB-NOPB-ND/1302378 (https://www.digikey.my/product-detail/en/texas-instruments/ADC081000CIYB-NOPB/ADC081000CIYB-NOPB-ND/1302378) ADC so the data interface is 16 bits LVDS (i'll need to study more on this, your advice will always be more than welcomed), so its effectively 800MSps @ 16 bits X 4 ADC.

If you want some good clearance with only a single 32bit port, you will have to use the Intel Arria 10, which can run DDR4 at 1.2Ghz, or 2.4 gigawords/sec, or, DDR3 at 1.066Ghz.
That is multi-thousand $$ affair, Xilinx UltraScale/US+ can do that too (DDR4 up to 1333 MHz IIRC).
this one of the main concern, i dont want to be left with $500 useless part that cant do the job just as good as wall decoration in frame. now i got few names to start with.

I've got a design that does that with DACs.  8x AD9736 connected to an Arria V FPGA.  Those are 14 bit DACs operating at 1 GS/s.  They are kind of old school and use a parallel LVDS interface, so that is 120 LVDS pairs on the FPGA.  You can do the same with ADCs, and up to 1-2 GS/s there are still available with LVDS.  Faster chips typically use JESD204b, which requires an FPGA with lots of 10G+ transceivers.

We only use SRAM.  DDR memory has plenty of bandwidth, but to have both DDR memory interface *and* 120 LVDS transmitters would require a much larger FPGA.

The big gotcha with DRAM is the refresh.  You don't want the ADC to stall out because of a memory refresh.  That means you either have to have a big SRAM buffer or find a way to avoid hitting the refresh.  The way to make that work is application dependent.
hey! i didnt know there is 14 bits ADC @ > 1GSps that is half the price of the 8 bits ADC081000 ADC. thanks! i maybe consider AD9736 in the design plan (that is lingering in the mind) but BGA footprint is no fun for hobbiest one-off project.

to be more specific, there are 4 "paths" for the design (maybe 5 or 6 paths @ 1GSps depending on part used), lets just take one, the other 3-5 will just duplicates. an ADC (8 - 14 bits @ 500 - 800MHz data clock) -> FPGA -> RAM. the total RAM is about 50 - 100MBytes. so its 12.5 - 25MB per RAM (4 paths) or 8 - 16MB per RAM (6 paths). all of this 100MB must be filled at effective rate of 6GSps (total 4x ADC), and then later there will be blind time (no data capture) where another MCU, or the same FPGA will process this data, FFT to be exact, before sending it to PC (USB) or screen for human viewing. anyway i think if i go RAM -> CPU path for FFT processing, this is the easiest part (so i also need a RAM that can be clocked a lot slower to interfacce to MCU), but if i go FFT in FPGA route, this may open another can of worm for a discussion, short FFT up to 1024 pts is common, but MBytes length of FFT is another thing... https://www.ll.mit.edu/HPEC/agendas/proc04/powerpoints/Posters/Poster%20B/dillon_posters.ppt (https://www.ll.mit.edu/HPEC/agendas/proc04/powerpoints/Posters/Poster%20B/dillon_posters.ppt) let alone staggered clock to get 6GSps from 4 x 1.5GSps ADC parts.

so the total FPGA pins required is 4 (ADC) X 16 x 2 (LVDS) = 128 pins, thats for input only (ADC), i guess for RAM requires the same pins amount, so total 256 pins, excluding other pins for comm/command/synch etc... man!

this discussion has lead me to few app notes... google "lvds to fpga" (for my reference later)...
https://www.altera.com/en_US/pdfs/literature/an/an479.pdf (https://www.altera.com/en_US/pdfs/literature/an/an479.pdf)
https://www.xilinx.com/support/documentation/application_notes/xapp1071_V6_ADC_DAC_LVDS.pdf (https://www.xilinx.com/support/documentation/application_notes/xapp1071_V6_ADC_DAC_LVDS.pdf)
but not quite ADC->FPGA->RAM thing. anyway this is good when we have somebodies helping shoulder to shoulder, thanks guys.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: ali_asadzadeh on April 22, 2018, 12:56:05 pm
Following ;)
Title: Re: 6GSps FPGA to Memory Handling...
Post by: C on April 22, 2018, 02:07:03 pm

Instead of 4 ADC's into one fpga, you might think of 4 ADC's synced into 4 fpga's.
With good design this could expand by ADC.

Worked with a CAD system back in 80's that had each thing as a separate bit plane. Made actual processing easy. A simple logic board then turned many planes into video.

Todays 1.5GSps could become tomorrows 3GSps with a 3GSps input to two 1.5GSps output block.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 22, 2018, 02:58:52 pm
FPGA is not MCU - you decide on the bus width and you can go as wide as you want. Thus, this is mostly a question of available pins.

On the ADC side it'll be expensive. For example, 2 of ADC08D1520 or one EV8AQ160 - approaching $1K. That'll take around 100 pins.

On the memory side, this is a matter of bus width. 1.5GS/s x 8 x 4 is 48 Gb/s. If you use a 64-bit DDR3 (standard DIMM), then you need 48/64 = 667 Mb/s. If you don't get much overhead then 800 Mb/s (400 MHz clock) will be enough. That's around 120 pins.

So, you'll need 100 + 120 = 220 pins - quite feasible in one FPGA, such as Spatan-7, or Artix-7 if you decide that 484-pin package is too tough. FPGA will cost much less than what you pay for ADCs.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: ejeffrey on April 22, 2018, 03:03:59 pm
hey! i didnt know there is 14 bits ADC @ > 1GSps that is half the price of the 8 bits ADC081000 ADC. thanks! i maybe consider AD9736 in the design plan (that is lingering in the mind) but BGA footprint is no fun for hobbiest one-off project.

DACs are generally cheaper and higher resolution than ADCs.

The ADC081000 is not recommended for new designs.  We are also using it in one of our designs (only 2 channel) When I did a survey, the best replacement I found was the ADC8D1020.  Unlike the ADC081000, it runs the LVDS pairs at 1 Gb/s instead of 500 MBit/s.  This cuts your needed LVDS pairs in half, and also makes synchronizing easier.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 22, 2018, 06:58:35 pm
hey! i didnt know there is 14 bits ADC @ > 1GSps that is half the price of the 8 bits ADC081000 ADC. thanks! i maybe consider AD9736 in the design plan (that is lingering in the mind) but BGA footprint is no fun for hobbiest one-off project.
DACs are generally cheaper and higher resolution than ADCs.
epic fail :palm: i tought its a ADC...

The ADC081000 is not recommended for new designs.  We are also using it in one of our designs (only 2 channel) When I did a survey, the best replacement I found was the ADC8D1020.  Unlike the ADC081000, it runs the LVDS pairs at 1 Gb/s instead of 500 MBit/s.  This cuts your needed LVDS pairs in half, and also makes synchronizing easier.
but ADC8D1020 is not available in digikey nor ebay... sad.

On the ADC side it'll be expensive. For example, 2 of ADC08D1520 or one EV8AQ160 - approaching $1K. That'll take around 100 pins.
i thought FPGA cost (SW license and the chip) will be few magnitude more expensive. on the adc side, thats why we go Rigol's style. 4 X ADC081000 (6GSps staggered) is the cost of 1 X EV8AQ160 (3GSps), i cant find ADC08D1520 price in digikey...
Title: Re: 6GSps FPGA to Memory Handling...
Post by: nctnico on April 22, 2018, 07:04:55 pm
It depends on what the ADC interface is but if you have a wide memory bus (like 128 or 256 bit) then you can run the memory at much lower speeds. You'll need to implement your own DDR controller though.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 22, 2018, 08:10:25 pm
i thought FPGA cost (SW license and the chip) will be few magnitude more expensive. on the adc side, thats why we go Rigol's style. 4 X ADC081000 (6GSps staggered) is the cost of 1 X EV8AQ160 (3GSps), i cant find ADC08D1520 price in digikey...

https://www.digikey.com/product-detail/en/texas-instruments/ADC08D1520CIYB-NOPB/296-41497-ND/1870663 (https://www.digikey.com/product-detail/en/texas-instruments/ADC08D1520CIYB-NOPB/296-41497-ND/1870663)

Also:

https://www.digikey.com/product-detail/en/texas-instruments/ADC08D1020CIYB-NOPB/ADC08D1020CIYB-NOPB-ND/1762580 (https://www.digikey.com/product-detail/en/texas-instruments/ADC08D1020CIYB-NOPB/ADC08D1020CIYB-NOPB-ND/1762580)

Xilinx's Vivado is free for smaller chips. DDR3 IP is free.

Entry level FPGAs which may fit your needs:

https://www.digikey.com/product-detail/en/xilinx-inc/XC7S50-2FGGA484I/122-2046-ND/7389946 (https://www.digikey.com/product-detail/en/xilinx-inc/XC7S50-2FGGA484I/122-2046-ND/7389946)

https://www.digikey.com/product-detail/en/xilinx-inc/XC7A15T-2FGG484I/122-1931-ND/5248118 (https://www.digikey.com/product-detail/en/xilinx-inc/XC7A15T-2FGG484I/122-1931-ND/5248118)

If you want to go ballistic (but still free Vivado):

https://www.digikey.com/product-detail/en/xilinx-inc/XC7A200T-3FFG1156E/122-1858-ND/3925786 (https://www.digikey.com/product-detail/en/xilinx-inc/XC7A200T-3FFG1156E/122-1858-ND/3925786)

Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 22, 2018, 10:18:38 pm
If you want to go ballistic (but still free Vivado):

https://www.digikey.com/product-detail/en/xilinx-inc/XC7A200T-3FFG1156E/122-1858-ND/3925786 (https://www.digikey.com/product-detail/en/xilinx-inc/XC7A200T-3FFG1156E/122-1858-ND/3925786)
There is no point going with this device, as Kintex-160 is about the same price, while it will give you double memory bandwidth and double transceivers speed.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: BrianHG on April 22, 2018, 11:00:37 pm
Cyclone V supports binding two 32-bit, DDR3/3L 800MT/s controllers to form a single 64-bit, 800MT/s controller, which gives you a theoretical BW of 6.4GB/s.
You need to keep the flow of data continuous, otherwise random addressing (CL time) will eat away your actual BW quickly.

Xilinx entry level Artix supports 1066MT/s, so you have a bit more margin.
I would be worried about both being so close to the nose and the core clock speed for the internal dual port memory cache ram.
ArriaV would bring him from CycloneV 800MT/s to ArriaV 1336MT/s in an approximate 350$ IC, and internal core memory speeds from CycloneV 300-420Mhz to ArriaV 400-500Mhz.  The down side is you need to buy Quartus to work with the ArriaV where it is free for the simpler variants of CycloneV.

I would still go 64 bit ram on the ArriaV and give a wide margin for play.  You never know...  Slightly off PCBs, bad impedance controls, shady DDR3 memory, poor assembly, all which might lead you to lowering your DDR3 clock rate.  With memory running at max 10GB/sec, you have a clear 1.5x headroom which will allow row precharge & refresh cycles and you can safely slow-down your ram clock to around 1Ghz from 1.336Ghz top speed.  800Mhz is just too close and would fail without ultra firmware tweaking.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 22, 2018, 11:05:28 pm
If you want to go ballistic (but still free Vivado):

https://www.digikey.com/product-detail/en/xilinx-inc/XC7A200T-3FFG1156E/122-1858-ND/3925786 (https://www.digikey.com/product-detail/en/xilinx-inc/XC7A200T-3FFG1156E/122-1858-ND/3925786)
There is no point going with this device, as Kintex-160 is about the same price, while it will give you double memory bandwidth and double transceivers speed.

Not really. Here's a comparably priced Kintex:

https://www.digikey.ca/product-detail/en/xilinx-inc/XC7K160T-1FBG676C/122-1961-ND/3911247 (https://www.digikey.ca/product-detail/en/xilinx-inc/XC7K160T-1FBG676C/122-1961-ND/3911247)

It has less pins, less logic. Since it's grade -1, while Artix is -3, the speed is roughly the same. DDR3 is rated at 800MT/s for Kintex while the rating for Artix is 1066MT/s, so Artix is actually a little bit faster.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 22, 2018, 11:55:19 pm
Not really. Here's a comparably priced Kintex:

https://www.digikey.ca/product-detail/en/xilinx-inc/XC7K160T-1FBG676C/122-1961-ND/3911247 (https://www.digikey.ca/product-detail/en/xilinx-inc/XC7K160T-1FBG676C/122-1961-ND/3911247)

It has less pins, less logic. Since it's grade -1, while Artix is -3, the speed is roughly the same. DDR3 is rated at 800MT/s for Kintex while the rating for Artix is 1066MT/s, so Artix is actually a little bit faster.
This one is even cheaper: https://www.digikey.com/products/en?keywords=XC7K160T-1FFG676C (https://www.digikey.com/products/en?keywords=XC7K160T-1FFG676C) but it supports 1600 MT/s for DDR3. It's got a bit less logic and 100 less IOs, but 400 IOs is still more than enough for the project at hand, while faster memory and MGTs are definitely useful in case you want to interface via JESD204B. Also there is a lot of headroom for growth in case you will need more logic (but you'll have to buy a license), while the Atrix-200 is a dead end.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: BrianHG on April 23, 2018, 01:34:16 am
Not really. Here's a comparably priced Kintex:

https://www.digikey.ca/product-detail/en/xilinx-inc/XC7K160T-1FBG676C/122-1961-ND/3911247 (https://www.digikey.ca/product-detail/en/xilinx-inc/XC7K160T-1FBG676C/122-1961-ND/3911247)

It has less pins, less logic. Since it's grade -1, while Artix is -3, the speed is roughly the same. DDR3 is rated at 800MT/s for Kintex while the rating for Artix is 1066MT/s, so Artix is actually a little bit faster.
This one is even cheaper: https://www.digikey.com/products/en?keywords=XC7K160T-1FFG676C (https://www.digikey.com/products/en?keywords=XC7K160T-1FFG676C) but it supports 1600 MT/s for DDR3. It's got a bit less logic and 100 less IOs, but 400 IOs is still more than enough for the project at hand, while faster memory and MGTs are definitely useful in case you want to interface via JESD204B. Also there is a lot of headroom for growth in case you will need more logic (but you'll have to buy a license), while the Atrix-200 is a dead end.
+1 for the 1600MT/sec.  Like I said, double your bus size on the ram and get that 2x clearance, which realistically will only be around 1.5x after overhead.  1000MT/sec as a limit is just cutting things way too close.  You have near 0 wiggle room.

My choice would be go for the Arria 10GX '10AX022C4U19E3SG' for 410$, https://www.digikey.com/products/en?keywords=10AX022C4U19E3SG (https://www.digikey.com/products/en?keywords=10AX022C4U19E3SG)
You get 13 megabit onchip cache at 730 MHz.
191 hardware floating point multipliers/adders for FFT with a peak 172 GFLOPS.
You should be able to feed the 1.6Ghz from your ADC directly into this Arria as it has way more than 32 1.6GHz LVDS channels, though, you will need to read through using a dedicated IO bank and dedicated PLL & differential clock channel as well, but the IOs and internal deserializes are designed for that sort of thing.
2400MT/sec DDR4 memory interface, run it at 64 bit.  Then you have enough ram speed to sample, process, and run onboard software all from 1 ram bank.  Or, even add a second smaller 32bit ram controller section.

NOTE:  Never buy a singe Altera/Xilinx IC before building the timing critical guts of your project in their dev suites & doing a full timing simulation.  Otherwise, you may be throwing out money on something which wont be fast enough.

You can learn how to do things on Altera's free web version of Quartus with a slower FPGA.  Altera also allows 1 free month with the full version once you are ready to simulate a full speed Arria 10 version, but, to program the Arria 10, you will need a license.

Others here can offer you better advice on using Xilinx, but, I still say do a full timing simulation with real-world simulated inputs, ram controller and additional processing.  Otherwise, you might design and build and buy something which wont be fast enough.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 23, 2018, 12:30:36 pm
it supports 1600 MT/s for DDR3.

1600 MT/s DDR3 is a plus because it lets you use 32-bit wide memory, while with 800 MT/s you'd need at least 64-bit wide. This saves 40 pins and some board space. This puts more strain on board design in terms of length matching and SI.

If you decide on 1600 MT/s, then Kintex will be your starting point. I would rather start from small Spartans and Artixes, which are only $50 (or less) and might be quite sufficient for the task.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 23, 2018, 01:20:42 pm
1600 MT/s DDR3 is a plus because it lets you use 32-bit wide memory, while with 800 MT/s you'd need at least 64-bit wide. This saves 40 pins and some board space. This puts more strain on board design in terms of length matching and SI.
I'd use 64bit SODIMM module as the socket tends to make layout easier. And at 1600 x 8 = 12.5 GB/s of bandwidth there is a huge margin.

If you decide on 1600 MT/s, then Kintex will be your starting point. I would rather start from small Spartans and Artixes, which are only $50 (or less) and might be quite sufficient for the task.
I don't think they will be sufficient - certainly not $50 ones. Also in some ways using Kintex is easier because they don't require small decoupling caps under BGA as they are already on a package. So you only need to fit a small amount of bulk caps (4.7 uF and up). This essentially frees up the bottom layer for additional routing space.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 23, 2018, 01:58:20 pm
I don't think they will be sufficient - certainly not $50 ones.

Why not? Acquiring 4 ADC channels to DDR3 memory doesn't require a lot of logic. The limiting factor is the number of pins. What Artix have you used on your last board? What was resource utilization for your MIG?

Of course, if there's some sort of data processing besides acquisition, this should be accounted for.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: SiliconWizard on April 23, 2018, 02:10:07 pm
The trick that has been widely in use in digital oscilloscopes is to run many memory chips in parallel so as to increase throughput without needing insanely high-speed memory.
For ADCs, the trick is to have several ADCs that sample data in an interleaved way, so that each one can sample at lower effective sample rate.

We already had digital scopes with 8 or 16 gsa/s almost 20 years ago (see the DDA series of Lecroy scopes for instance) and this is how that was done.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 23, 2018, 02:53:40 pm
I don't think they will be sufficient - certainly not $50 ones.
Why not? Acquiring 4 ADC channels to DDR3 memory doesn't require a lot of logic. The limiting factor is the number of pins. What Artix have you used on your last board? What was resource utilization for your MIG?
Of course, if there's some sort of data processing besides acquisition, this should be accounted for.
for now, the paid $4000 quartus SW is not justifiable for a one-off that probably never see the light. so using an FPGA that doesnt require software license is much preferable. earlier i said the FPGA probably will perform FFT as well, but if this means requiring larger FPGA with multi thousand dollar license, then i think it should be beneficial in term of cost to off load the FFT to MCU processing, Arm V7 1GHz quad core or the latest 64bits Arm A53 can be hard at cheap, or maybe even adding 2nd FPGA specialized in FFT alone, this... mmmv (my milage may vary). now i'm not sure on licensing scheme of other brand like the mentioned Artix KIntex etc, but if the cost of the license is near 4 digits cost then i think they have to be discarded from the list, for now while the money is still peanut small...

...have several ADCs that sample data in an interleaved way, so that each one can sample at lower effective sample rate.
yes i should have said interleaved, instead of "staggered" :palm:

We already had digital scopes with 8 or 16 gsa/s almost 20 years ago (see the DDA series of Lecroy scopes for instance) and this is how that was done.
exactly except they are proprietary, and the components cost back then i believe is unachieveably expensive by hobbiests during the time...

today, what is unachieveable is the sw license cost :palm: people just know how to get money. its understandable for FPGA development tools because they have to cover R&D and production factory, but look at things like EM Solver, the cost is just simply nuts... offtopic.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 23, 2018, 03:12:03 pm
Why not? Acquiring 4 ADC channels to DDR3 memory doesn't require a lot of logic. The limiting factor is the number of pins. What Artix have you used on your last board? What was resource utilization for your MIG?
Actually it does. I don't have exact numbers right now, but I remember it took over 10k LUTs and 10k+ FFs just for x16 controller. Another thing is since you can only run the memory at 400 MHz (speed grade -3 devices are hard to find in stock, so I tend to avoid them), you will need to have a wide memory bus to get the kind of BW you need (x16 only gives 1600 MB/s), but for that you need to use at least 484 package - and that package you will have to use logic level translators to get SPI flash to work for booting up FPGA. Unfortunately due to the way IO banks are bonded out on Atrix the only way to make x64 DDR3 using pins on a right column is to use A200 device in 676 package, but there are some gotchas with using banks 16 and 35 if you want to use MGTs at full 6G speed as well.

Of course, if there's some sort of data processing besides acquisition, this should be accounted for.
That's another Terra incognita that tends to bite you in the behind if you're not careful. It's better to have oversized device during development because debugging aids like ILA also consumes some FPGA resources.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 23, 2018, 03:16:56 pm
today, what is unachieveable is the sw license cost :palm: people just know how to get money. its understandable for FPGA development tools because they have to cover R&D and production factory, but look at things like EM Solver, the cost is just simply nuts... offtopic.
Xilinx free license gives you access to a lot of FPGAs - entire Spartan-7 and Artix-7 lines, some Kintex (K70 and K160), some Zynqs(030 and below) and also some UltraScale/UltraScale+ devices (including about half of Zynq US+ MPSoCs lineup). On top of that there are about 300 IPs available with that free license.
So you can do quite a bit with all that. This is why I've chosen to use Xilinx devices - they are much more friendly to hobbyists.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: SiliconWizard on April 23, 2018, 03:18:10 pm
We already had digital scopes with 8 or 16 gsa/s almost 20 years ago (see the DDA series of Lecroy scopes for instance) and this is how that was done.
exactly except they are proprietary, and the components cost back then i believe is unachieveably expensive by hobbiests during the time...

Well, actually, back then, even developing on microcontrollers was expensive. This is how companies like Microchip changed it all for hobbyists.

Old Lecroy scopes (80's-90's) have 68k CPUs. Very slow by today's standards, but at the time, a 68030 dev board was very expensive, so kinda outside the league of most hobbyists. I'm not even talking about fast ADCs.

today, what is unachieveable is the sw license cost :palm: people just know how to get money. its understandable for FPGA development tools because they have to cover R&D and production factory, but look at things like EM Solver, the cost is just simply nuts... offtopic.

Yes. Costs have shifted towards software, but a lot of recent digital scopes (except the low-end ones) still use some proprietary ASICs.

You want crazy costs? Look at the costs for microelectronics development (Cadence, Synopsys, Mentor...) Often over $100 000 with additional recurrent maintenance costs to top it off.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 23, 2018, 03:28:59 pm
for now, the paid $4000 quartus SW is not justifiable for a one-off that probably never see the light. so using an FPGA that doesnt require software license is much preferable. earlier i said the FPGA probably will perform FFT as well, but if this means requiring larger FPGA with multi thousand dollar license, then i think it should be beneficial in term of cost to off load the FFT to MCU processing, Arm V7 1GHz quad core or the latest 64bits Arm A53 can be hard at cheap, or maybe even adding 2nd FPGA specialized in FFT alone, this... mmmv (my milage may vary). now i'm not sure on licensing scheme of other brand like the mentioned Artix KIntex etc, but if the cost of the license is near 4 digits cost then i think they have to be discarded from the list, for now while the money is still peanut small...

Xilinx's Vivado is free for all Artixes and for smaller (70 and 160) Kintexes. The free package is called WebPack. You do need to register and apply for the free license online. Also, the free version collect usage data (called WebTack) which you cannot turn off.

Depending on the size and speed of your FFT it may take more resources or less resources. You can download Vivado from Xilinx's Web site, install it, create a project for a big FPGA (I usually use A200T frown upon here), create your design, simulate, look at the resource utilization. This will give you the basis to decide how big a chip you need.

Using external MCU will require passing data to MCU, which may be difficult. Xilinx has combined chips - ARM + FPGA. They're called Zynq. They facilitate communications between ARM and FPGA. Another possibility is to build soft CPU (Xilinx's has a core called MicroBlaze which is decent).

However, FFT is better done in FPGA fabric - they have DSP blocks which can do MADD at roughly 500 MHz (if properly pipelined). Even the smallest Artix has 40 DSP blocks. So, imagine all of them working in parallel at 500 MHz. This easily beats any MCU.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 23, 2018, 04:59:35 pm
Xilinx's Vivado is free for all Artixes and for smaller (70 and 160) Kintexes...
i just registered in Xilinx website. when i tried to download Vivado, Xilinx required me to fill my details (Malaysia) due to "U.S. Government Export Approval" and i got this... "We cannot fulfill your request as your account has failed export compliance verification." great!! we are a big threat to the USA :palm:
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 23, 2018, 05:34:09 pm
Xilinx's Vivado is free for all Artixes and for smaller (70 and 160) Kintexes...
i just registered in Xilinx website. when i tried to download Vivado, Xilinx required me to fill my details (Malaysia) due to "U.S. Government Export Approval" and i got this... "We cannot fulfill your request as your account has failed export compliance verification." great!! we are a big threat to the USA :palm:

I certainly don't know US export laws, but this might have been a bug or some sort of spelling problem as well.

See https://www.xilinx.com/support/answers/44043.html (https://www.xilinx.com/support/answers/44043.html)

Title: Re: 6GSps FPGA to Memory Handling...
Post by: Berni on April 23, 2018, 06:42:12 pm
As far as i see this project is not going to be cheap in any way you look at it.

The way to go certainly is parallel. You can only increase the clock speed up to a point before the chips become rather exotic and ridiculously expensive. DDR3 memory certainly is very quick when accessed in large chunks. I would think it makes sense to split up the load between multiple FPGAs so that you can make use of cheap small versions of them and since you are starting off with multiple interleaved ADCs its easy to split them up between multiple FPGAs to take the data.

Wide memory buses are the answer. Modern PCs use this a lot as dual channel memory is pretty much standard to get a pair of 64bit DIMMs to be 128bit wide. Some big CPUs even have quad channel memory to get you a 256bit bus to RAM. Once you get to graphics cards that tend to have even more hunger for bandwidth things can get to 384bit memory buses. Recently GPU manufacturers started putting RAM chips in the same package as the GPU and wire bonding the ram to it, this got them a truly ridiculous 3072 bit bus to the RAM.

You don't necessarily have to buy the dev tools off the bat tho. All the major manufacturers let you program the smaller chips with the free version. But they will offten want you to buy the software to get a lot of the nicer soft IP such as soft DDR memory controllers, FFT blocks etc. However as far as i know Altera gives you all of this for evaluation where using this IP will make your binary file time limited. You can program your FPGA with it just fine but after about 1 to 2 hours the FPGA will deliberately crash itself when the timer runs out.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 24, 2018, 08:53:55 am
i just realized this discussion only lingers among 2 names, Altera and Xilinx. why these are the only 2 names that produce FPGA? isnt there anyone else thats probably cheaper and SW IDE can be easily installed at cheap or free? :(
Title: Re: 6GSps FPGA to Memory Handling...
Post by: daveshah on April 24, 2018, 10:14:55 am
Lattice ECP5 is cheaper, but not realistically going to handle 6Gsps data flow to DDR3. Also you have to pay for the software if you need one with the transceivers.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: ali_asadzadeh on April 24, 2018, 12:31:07 pm
God has invented torrents ages ago! ;) Just saying

If this is a hobby project and a one off, maybe that would be ok to use torrents, but if it's a commercial project then the license cost would be nothing :)
Title: Re: 6GSps FPGA to Memory Handling...
Post by: SiliconWizard on April 24, 2018, 01:02:05 pm
Lattice ECP5 is cheaper, but not realistically going to handle 6Gsps data flow to DDR3.

Probably not. The  ECP5-5G has 5 Gbps SERDES, which is not bad.
There is a dev kit (ECP5-5G Versa), with a free 1-year licence. The board has DDR3 RAM, and apparently the max freq you can get out of it is 800 MHz, which gets you 1.6 GBytes/s since  its data bus is 16-bit.
I'm not sure what max data bus width their DDR3 controller IP supports.

Also you have to pay for the software if you need one with the transceivers.

Yes, but last I checked, the licence cost was $99/yr. It's definitely affordable.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 24, 2018, 03:00:19 pm
God has invented torrents ages ago! ;) Just saying
If this is a hobby project and a one off, maybe that would be ok to use torrents, but if it's a commercial project then the license cost would be nothing :)
torrent is invented by man, based on Laws created by God. anyway, torrent is of no good without a license provided.. ;) i tried to download from another site https://www.cmc.ca (https://www.cmc.ca) same thing happened and then i emailed them to beg, the reply is, "its only for canadian academy" gosh thats racist.

Lattice ECP5 is cheaper, but not realistically going to handle 6Gsps data flow to DDR3.
Probably not. The  ECP5-5G has 5 Gbps SERDES, which is not bad.
i'm having trouble searching for their SW name from the website. what is it again? please.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: daveshah on April 24, 2018, 03:52:53 pm
The software is Lattice Diamond.

MicroSemi PolarFire might also be worth a look.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: daveshah on April 24, 2018, 03:55:28 pm
It looks like the ECP5 DDR3 controller can do up to a 64-bus at 800Mbps. Which is 6.4Byte/s.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Berni on April 24, 2018, 04:12:03 pm
The FPGA market is in the hands of only a few vendors and they all seam to have the same sort of business model. They all offer a large range of chips from tiny cheap ones to ridiculously expensive but also incredibly powerful chips. They all give you the software for free with the cheap chips but will attempt to sell it to you for >1000$ otherwise. They all offer you some expensive but very useful IP blocks (A lot of them you get for free if you buy that expensive software)

Altera and Xilinx are simply the two biggest players in the market. They make the highest performance chips and they have the best software tools (Tho Altera IDE does some things better while Xilinx does other things better, overall they are both good, but not perfect)

Then people like Lattice or Microsemi are sort of the cheap brand ones. The chips cost less but the software is not nearly as good (Tho the software can still be just as expensive as Altera or Xilinx).

In general its a good idea to stick to LVDS output ADCs since you can run receive 800Mbit/s per lane on 20 or more lanes on most cheap FPGAs out there. The chips with transceivers get expensive really quick. You can avoid paying for the DDR3 controller license if you get a chip with a hardIP DDR3 controller(At least thats how it works at Altera for Cyclone V), but as said before you might need 2 to 4 FPGAs to get enough memory bandwidth.

If this is a one off project tho it might make sense to look into any existing hardware for this such a PXI mainframe ADC digitizer cards that can do such speeds.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: SiliconWizard on April 24, 2018, 04:18:42 pm
i'm having trouble searching for their SW name from the website. what is it again? please.


* Lattice Diamond:
http://www.latticesemi.com/en/Products/DesignSoftwareAndIP/FPGAandLDS/LatticeDiamond (http://www.latticesemi.com/en/Products/DesignSoftwareAndIP/FPGAandLDS/LatticeDiamond)

* ECP5-5G Versa Development Kit:
http://www.latticesemi.com/en/Products/DevelopmentBoardsAndKits/ECP55GVersaDevKit (http://www.latticesemi.com/en/Products/DevelopmentBoardsAndKits/ECP55GVersaDevKit)
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 24, 2018, 05:11:20 pm
... but for that you need to use at least 484 package - and that package you will have to use logic level translators to get SPI flash to work for booting up FPGA.

Silly, isn't it? It's even funnier if you want to configure the FPGA from MCU as SPI slave. This uses one pin (one pin!) in bank 14, and it mandates that all the configuration and system pins must be at 1.5V. What could they possibly think when they designed it this way? I use level shifters - $5 is nothing compared to upgrading FPGA.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 24, 2018, 11:48:55 pm
Silly, isn't it? It's even funnier if you want to configure the FPGA from MCU as SPI slave. This uses one pin (one pin!) in bank 14, and it mandates that all the configuration and system pins must be at 1.5V. What could they possibly think when they designed it this way? I use level shifters - $5 is nothing compared to upgrading FPGA.
For QSPI there are 4 data pins (D00-D03) and chip select. Using single-pin SPI is incredibly slow, boot time is going to be seconds even for small devices.
The pinout design implies that banks 14 and 15 are used for configuration, and in case of parallel flash you will need a lot of these pins. Remember it was designed before QSPI flash has become a commodity, if you take a look at configuration-related pins in newer UltraScale/UltraScale+ devices, you will see that all QSPI-related pins are a part of config bank 0, so it doesn't force any particular voltage levels for any of user I/O banks (see here (https://www.xilinx.com/support/documentation/user_guides/ug570-ultrascale-configuration.pdf), page 27). So it's hard to blame them for failing to predict where market will go almost a decade later.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 25, 2018, 12:17:19 am
For QSPI there are 4 data pins (D00-D03) and chip select. Using single-pin SPI is incredibly slow, boot time is going to be seconds even for small devices.

SPI flash can typically run at 50 MHz which is 0.3 sec for full bitstream for A50T. Good enough for me. Using SPI from MCU at 25 MHz is still less than a second - faster than JTAG. Even at 10 MHz it wouldn't be that bad.

We're hijacking a thread again ... :)
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 25, 2018, 01:44:54 am
SPI flash can typically run at 50 MHz which is 0.3 sec for full bitstream for A50T. Good enough for me. Using SPI from MCU at 25 MHz is still less than a second - faster than JTAG. Even at 10 MHz it wouldn't be that bad.
There are scenarios where this is unacceptable. For example PCI Express mandates 100 ms power-good-to-ready time limit. This is not an easy requirement to fulfill - especially as you will need to use larger device for PCI-E endpoint because it's quite logic-heavy ("AXI Memory Mapped to PCI Express" IP on one of my test designs is ~15k LUTs, 12k FF  and 19 BRAMs).

We're hijacking a thread again ... :)
Disagree. Since the project at hand practically mandates x64 memory bus, figuring out if 484 package can be made to work is important, especially since pinout layout of smaller 676 devices (A75 and A100) still force using configuration banks as only two banks are bonded out on a left column. So this is important consideration.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NiHaoMike on April 25, 2018, 05:29:25 am
SPI flash can typically run at 50 MHz which is 0.3 sec for full bitstream for A50T. Good enough for me. Using SPI from MCU at 25 MHz is still less than a second - faster than JTAG. Even at 10 MHz it wouldn't be that bad.
There are scenarios where this is unacceptable. For example PCI Express mandates 100 ms power-good-to-ready time limit. This is not an easy requirement to fulfill - especially as you will need to use larger device for PCI-E endpoint because it's quite logic-heavy ("AXI Memory Mapped to PCI Express" IP on one of my test designs is ~15k LUTs, 12k FF  and 19 BRAMs).
Many years ago, I dealt with a FPGA PCIe card at work where the instructions included holding down the reset button for a second or so when powering up the PC to allow the FPGA to boot. I suggested adding a capacitor across the reset button to provide the needed delay.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 25, 2018, 05:48:34 am
We're hijacking a thread again ... :)
hijack granted. you may proceed as you like ;D

We're hijacking a thread again ... :)
Disagree. Since the project at hand practically mandates x64 memory bus, figuring out if 484 package can be made to work is important, especially since pinout layout of smaller 676 devices (A75 and A100) still force using configuration banks as only two banks are bonded out on a left column. So this is important consideration.
agree, 32bits @ 1.6GHz, 64bits @ 800MHz and so on. both number of pins and clock speed need to be considered from every aspects. i may not fully understand what you people discussing here, but who knows this may be valuable reference later. the golden advice so far thats applicable to me is, download the IDE and simulate the program for the FPGA.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: ali_asadzadeh on April 25, 2018, 06:26:30 am
Quote
torrent is invented by man, based on Laws created by God. anyway, torrent is of no good without a license provided.. ;)
If you search a bit more, under 30 seconds, you could get the torrents with the license too, with the latest version! ;)

almost about 70% of all internet traffic is torrents

https://en.wikipedia.org/wiki/BitTorrent

But if the project is commercial, I would suggest to buy the tools and help them, create better products for us :)
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 25, 2018, 08:40:25 am
If you search a bit more, under 30 seconds, you could get...
(http://4.bp.blogspot.com/-3XjINic4TjM/UyxrlnnFtlI/AAAAAAAAAQE/XVN6M3dNPA4/s1600/Uncle+Sam+I+Want+You.jpg)
if anything you want to say, say it in.... PM!
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Scrts on April 25, 2018, 05:54:15 pm
It looks like the ECP5 DDR3 controller can do up to a 64-bus at 800Mbps. Which is 6.4Byte/s.

I don't understand this discussion... The idea is push 6Gbps of data into the memory and then what? Let's say you can achieve that speed, but you have absolute zero time then to read and process that data. The bus will always be busy writing.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 25, 2018, 06:01:04 pm
I don't understand this discussion... The idea is push 6Gbps of data into the memory and then what? Let's say you can achieve that speed, but you have absolute zero time then to read and process that data. The bus will always be busy writing.

Not necessarily. You may acquire event(s) of various duration and read them back later. It all depends on the application.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 26, 2018, 12:07:51 am
I don't understand this discussion... The idea is push 6Gbps of data into the memory and then what? Let's say you can achieve that speed, but you have absolute zero time then to read and process that data. The bus will always be busy writing.
since you asked, lets open the worm can a little bit more. as i said earlier, during capture, the circuit will get input data to adc->fpga->ram at 6GSps until 50-100MB of RAM is filled up. after that, its called blind time, no capture but processing the data RAM->processor->display. how long is this blind time? its unspecified and mainly will depends on how fast post-processing is made. but i have the feeling that it is 99.999% certain that post processing will take alot slower to complete than capture rate, so any funny idea such as double buffering will be inapplicable here, except if i have to tolerate super expensive solution such as massive parallel processing involving super expensive fpga or massive amount of them marched/arranged on the pcb like brain cells. there are few possibilities during this "blind time":

a) another mcu or fpga is connected to the same bus where writing to the RAM from FPGA is made during capture, but this time fetching the data out of RAM from the same bus for FFT processing (at slower speed if cannot be fast). or if write and read can be done concurrently for the RAM from different bus, then so be it. i'm not sure if this even possible, i've never done this.

b) the same fpga which wrote to the RAM earlier will fetch back the RAM content for FFT processing, or maybe process the captured data concurrently (while capture in progress) but this will depend on algorithm used.

c) same as (a) except reading bus is from another pin of the 1st (capture) fpga. ie for reading RAM, mcu has to command the capture FPGA to read the RAM and send the content to the another output bus to mcu or another fpga.

i should sketch something but i'm in the office right now, MechCAD is a bit difficult from here here we go (see attached). anyway this is what i can think of from spherical dead cow point of view. as for RAM, what i prefer is, it should not have the "refresh cycle" or internal housekeeping job that blocked writing to RAM. if capture fpga has to wait for the RAM to do something then its not good (unless the condition can outrun the capture rate). i prefer a RAM that we simply set the address either to serial or parallel pin, and incremented by shift register or something, if RAM can automatically increment the address internally without fpga doing anything, then this is the ideal RAM i'm looking for. another thing is how to fetch data from ram, is from the same bus? or is ram have another pin for output, this i need to study later and will impact design decision, this is another deeper into the worm can or the spherical dead cow gray matter stuffs.

feel free to share thoughts, my intention for this thread is already answered, ie the question of "whether a fpga can do 6GSps". My best regards to you all.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: C on April 26, 2018, 01:07:12 am

You have many options if you break project apart some.

First as stated you could use four sync's fpga's as input.

Then you could have many outputs on each input fpga. With a small buffer in input fpga you could output packets. A packet would be a chunk of time.
With two ports, your rate drops to 3GSps, with four ports to 1.5GSps.
This has many benefits. First the reduced data rate helps a lot. If the input fpga can do packets you can start processing each packet when received.
When you want to goto 12GSps, you just need to get the input split into two 6GSps to have a working system.

If you can not do packets, then you most likely still have to stop the capture.

When you split the four ADC to separate fpga's, it becomes a small step to do more ADC's

One FPGA puts some hard limits in place. The many fpgs's way is something that can scale. likely with lower cost fpga's.

C

Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 26, 2018, 02:16:06 am
The problem with 4 fpga is how to synch them in 6GSps (interleaving) resolution with acceptable jitter figure? That can be a whole lot chapter for that, i will need the clean 6GHz runs on the pcb traces. If we do it in 1 fpga, we can say use 50 or 60 MHz TCXO connected close to a fpga, and do the 100x multiplication in the chip, can?  If not, i may learn from Rigol circuit because that what they did. Rigol is my living proof that this concept can be done.

Edit:  or wait, we can use adf4355 synthesizer for this, yes 4 fpga each running at 1.5GHz can be another option... :-+
Title: Re: 6GSps FPGA to Memory Handling...
Post by: C on April 26, 2018, 03:02:12 am
Now packets

When you have sync, you have many streams of data.
If you have 8-bit data then to get transparent data you need an extra bit.
The 9-th bit could mark the start of a packet.
The other 8-bits could still contain sample data.
The 9-th would give you a time sync marker.

A different way would be the 9-th bit marking the difference between command/status or data.

When you look at current bit streams you find 8b-10b or something like this and you can get the sync from this.
with fpga you might convert to equal of one of these streams and still use a parallel connection. You can gain speed by increasing the 8b to larger number with smaller scale of the 10b. You might think of this as shifting from byte data to 64-bit data or larger.

The parallel connection could make displaying data easer. Think of many bit planes making up actual color display data. X bits input gets mapped to a color pixel. You can process each bit plane in parallel. Your menu or other display data could be additional planes.

Again you get to scale.

C
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Berni on April 26, 2018, 06:09:51 am
You likely want to go with the "Option C" where the MCU talks to the FPGA

The DDR3 memory controller will usually have the capability to have multiple buses accessing it (If not you can mux the bus to the controller inside the FPGA very easily). This extra bus can then be used to allow the MCU access to all of the DDR3 memory. A lot of MCUs have external memory interfaces that are mostly meant to give the MCU capability to expand its memory using a external RAM chip. What you can do is hook up a FPGA to the same pins and have it act like its a SRAM chip. That way the MCU will send all read/write operations on that area of memory out into the FPGA. There you can then interpret the simple SRAM memory bus as read/write commands and translate them into read/write commands to the DDR3 controller. As a result you have memory mapped the entire DDR3 memory into the MCU where its processor can access it in the same was as it would access its main internal RAM.

As far as FFT goes if you need it calculated fast then FPGA is the place to do it. A decent sized FPGA will have about 10 to 1000 times faster integer math performance in FFT compared to ARM microcontrollers. As for doing FFT on the fly as its coming out of the ADC that's only possible on short chunks like for example 1024 samples at a time. The reason is that the way FFT works means it needs the first and last data sample in the  beginning of the calculation so you need to hold the entire FFT length in memory before you can start calculating it. So you will need to shuffle things in and out of main memory to calculate things like 100Mpt long FFTs

If you also want to draw to the display very fast too (like 60fps) then it likely also makes sense to move the LCD and connect it directly to the FPGA too while just having the MCU continue to sit on that SRAM emulated memory bus. That way you can built a sort of custom graphics card in the FPGA that takes the waveform data and turns it into a bitmap picture. The MCU in the meantime can also fill up a framebuffer with the image of the menus that the FPGA merges on the fly with the waveform bitmap while sending it to the LCD.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 26, 2018, 11:21:12 am
If you also want to draw to the display very fast too (like 60fps) then it likely also makes sense to move the LCD and connect it directly to the FPGA too while just having the MCU continue to sit on that SRAM emulated memory bus. That way you can built a sort of custom graphics card in the FPGA that takes the waveform data and turns it into a bitmap picture. The MCU in the meantime can also fill up a framebuffer with the image of the menus that the FPGA merges on the fly with the waveform bitmap while sending it to the LCD.
This begs the question - what's the reason to have external MCU as opposed to using a softcore like Microblaze (which is free BTW for 7 series devices)? That softcore natively supports AXI4 bus which is also supported by memory controller, and you can implement input datapath as a AXI Master and connect it to the same memory controller interface (via interconnect), or you can implement it as a AXI-Stream master and use one of DMA controllers (also free IPs) to send the data into memory.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Berni on April 26, 2018, 01:24:00 pm
This begs the question - what's the reason to have external MCU as opposed to using a softcore like Microblaze (which is free BTW for 7 series devices)? That softcore natively supports AXI4 bus which is also supported by memory controller, and you can implement input datapath as a AXI Master and connect it to the same memory controller interface (via interconnect), or you can implement it as a AXI-Stream master and use one of DMA controllers (also free IPs) to send the data into memory.

That is also a perfectly valid way of doing it.

The reasons for using an external MCU might be:
- Cheap
- Higher performance than a softcore (You can get 400MHz ARM MCUs with tons of peripherals)
- Perhaps great OS support on that particular microprocessor
- Familiar development environment if you used one for a long time.
- Low power modes made easier
- Saving on development time if you haven't worked with softcores before.
- Saving resources inside the expensive FPGA if things get tight.

What you end up choosing depends entirely on your particular situation at hand.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: C on April 26, 2018, 01:33:23 pm
Mechatrommer

You might want to keep differential in mind.

High speed digital logic, differential.

The high speed ADC I have looked at, differential

The high speed analog, differential.

You should think of not crippling input an do full differential from input.

C
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 26, 2018, 01:59:31 pm
- Cheap
Softcore is $0 so MCU is definitely more expensive. Not to mention additional board space which also costs $$.
- Higher performance than a softcore (You can get 400MHz ARM MCUs with tons of peripherals)
If you need an MPU, you can use Zynq. As for peripherals - Xilinx provides free IPs for most peripherals you might find inside typical MCUs. Or you can implement several softcores if you need more horsepower.
- Perhaps great OS support on that particular microprocessor
Softcore can be configured to have MMU and there is a Linux build for that core. If you want to use something like Windows CE (which is quite popular as creating GUI is easier with it), you will need a full-blown MPU like ARM Cortex-AXX. Again, Zynq will be very fitting here with it's A9 hard MPU(s) and it will eliminate all layout-related hassle of making MCU-to-FPGA interface.
- Low power modes made easier
Low power and FPGAs don't tend to coexist peacefully as latter usually aren't that low power. For example Artix-100T device can consume up to 6 Amps just on it's Vccint rail. Same goes for DDR3 - it consumes quite a bit of power because of termination. My board with x16 DDR3L (low power version!) consumes up to 0.5 Amp from Vccddr rail (1.35V).
- Saving on development time if you haven't worked with softcores before.
If you choose to use OS, then it doesn't really matter if it's running on a softcore or on some other platform.
- Saving resources inside the expensive FPGA if things get tight.
Microblaze in MCU configuration takes less than 1k LUTs, so even the smallest Artix-A12 can fit a dozen of them. Implementing a SRAM-like interface is also going to take some resources, so it might end up being a wash.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 26, 2018, 02:14:44 pm
You should think of not crippling input an do full differential from input.
:-\ :-\ :-\

Microblaze in MCU configuration takes less than 1k LUTs, so even the smallest Artix-A12 can fit a dozen of them. Implementing a SRAM-like interface is also going to take some resources, so it might end up being a wash.
a dozen? well thats interesting, its like having duodeca-core cpu running FFT thats fantastic. but well, here is the first time i heard the name Microblaze, i cant say negative or positive about it, the only think in mind is how am i going to program a GUI for it, what IDE etc.. there are a lot to catch up about this stuffs.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: C on April 26, 2018, 02:54:11 pm
When you start looking at differential analog, one set of circuits you often find in examples is going from or to single ended, often just a resistor.
If you think, a circuit driven from single ended input has no common mode at this point. The big advantage here for differential is getting the return signal path isolated from other signals like power supply noise.

Keep thinking on what could be big picture.
When you have ADC input, could be small change to logic input. It might also be a small change to get DAC or logic output.

For ADC you have collect data, display data some way or both.

When you have input & output, then you have network analyzer.

With the chips that lets you adjust timing, you can have many things working in sync. Even programming can get easer when working on separate but parallel.
 
Have you looked at Current mode analog. Cures some things that Voltage mode has problems with.

C
Title: Re: 6GSps FPGA to Memory Handling...
Post by: asmi on April 26, 2018, 03:09:34 pm
a dozen? well thats interesting, its like having duodeca-core cpu running FFT thats fantastic. but well, here is the first time i heard the name Microblaze, i cant say negative or positive about it, the only think in mind is how am i going to program a GUI for it, what IDE etc.. there are a lot to catch up about this stuffs.
That's the name of the softcore developed by Xilinx and offered for it's FPGAs. They are often used to orchestrate hardware when doing everything in hardware is not feasible, as well as some advanced scenarios like partial reconfiguration, where some of the hardware is swapped in and out as required. It's also usually used to react to user input as usually you don't need fast clock speed for that because input is going to be very slow anyway.
Xilins also provides free IPs for Mutex and Mailbox to facilitate inter-processor communication if you want to implement a multi-core SMP system, as well as some IPs to implement triple modular redundancy if that is your thing (of course you'd better match this will memory bus with ECC). Like I said above, there is a lot of things you can do using IPs that are provided for free, and that is a distinct advantage Xilinx have over competitors, which all only offer paid IPs.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 26, 2018, 04:05:16 pm
Like I said above, there is a lot of things you can do using IPs that are provided for free, and that is a distinct advantage Xilinx have over competitors, which all only offer paid IPs.

Xilinx promotes so called BD design flow where you design is graphically represented as interconnected blocks. Most blocks are IPs. If you need something customized, you create your own block. This looks like some sort of virtual Arduino on steroids - you can assemble practically anything in a matter of few hours (excluding compile time). So, if you like such style, it'll be very easy for you.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: C on April 26, 2018, 05:47:45 pm
Xilinx promotes so called BD design flow where you design is graphically represented as interconnected blocks. Most blocks are IPs. If you need something customized, you create your own block. This looks like some sort of virtual Arduino on steroids - you can assemble practically anything in a matter of few hours (excluding compile time). So, if you like such style, it'll be very easy for you.
I would think this would be a great time saver if you can work with it.

Is coping a Rigol DSO a good idea?
What you get from Rigol is down to a cost often for a savings of cents.

Think GW Instek have shifted their foundation to Xilinx Zynq.

One scope I have read about is using two ADC's for 1 G. The ADC that supports 4 channels at 250 M so if not down to cost could be an 8 channel input.

One of the tear-downs had a separate analog board.

The more modules you create, the more options for the future. Can become easy if each channel is a clone.

Data packets with a time stamp lets you put packets in order.
Time stamps can be a simple binary counter that rolls over to 0.
Your mouse is based on a small counter for direction. With PC reading often, distance is unlimited.

I keep thinking if you keep it simple so that it can scale, you will save time building and have easer job in future to scale. The many modules makes it easer to test and for future changes or a different use.

Look at micro controller and how things have changed.
Was SPI or Jtag now often it looks like a disk drive.
A good CAN buss controller these days can get first firmware over CAN bus. Keep in mind that this could be the best/easiest bus for operator interface. You can have many and it is still just CAN packets if you do not let yourself get barrier in someone's idea of a higher level.

My look at Xilinx Zynq being inside the FPGA could make a lot of things simpler/easer then a CPU outside FPGA, and you don't lose capability for a CPU outside.


C
Title: Re: 6GSps FPGA to Memory Handling...
Post by: Mechatrommer on April 26, 2018, 06:03:35 pm
Data packets with a time stamp lets you put packets in order.
Time stamps can be a simple binary counter that rolls over to 0.
i dont think timestamp or extra bit is necessary, esp in high speed rate it can degrade performance quite significantly, in the design we usually set the ordering fixed, ie this is fpga1 that goes 1st, and then this is the second fpga and so on. when fetching the data, we do by the same ordering. whats troubling about timestamping is the jitter generated by the (interleaved) clock since nyquist theory requiring capture interval to be the same from point to point, otherwise the true signal cannot be reconstructed faithfully. but in order to handle for jitter, other mechanism or external (maybe analog) monitor/calibration circuit should be made since fpga cant detect this, it will be happy to do the job when the clock goes high or low without any sense of temporal axis well, this can be for another discussion in another thread.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: C on April 26, 2018, 07:16:20 pm

sorry did not state that very well.

Say you have 4 ADC's at X sample rate.

You get better results if all take sample at exactly the same time.
For example one is volts, one current, if not same time watts has an error.

With a mux input first step is de-mux data to process.
With parallel data input you can assume time sync.

If you break each stream into blocks, then you can parallel process. This is again assuming that each takes the same time to process.
Adding a time stamp & channel at start of each block does not change sample rate, just output data rate for short time. You now have needed information to rebuild original streams from original or computed data from the blocks of data.
This overhead of this decreases the larger the data block.

With time stamps in stream, many options become possible as you have time to link data and not just position.
With processing you often get many streams that need to be in time sink with original.
Your source is very high speed data, to decrease time not sampled you need more parallel processing cheap.

Important thing is get the best data you can at start of chain and not be so crippled in hardware.

While you are thinking of getting time sync of 4 ADC, you might think of time sync of two or more of what you are building.

With this in mind, easy be for another discussion in another thread.

C

Title: Re: 6GSps FPGA to Memory Handling...
Post by: Berni on April 26, 2018, 07:43:08 pm
I would think its the easiest to simply phase shift the clock signal to the ADC and the FPGA processing it as a whole while also having a sync pulse that each FPGA samples along with the ADC data.

At these sort of speeds you can already generate the delay by simply making the clock take long squiggly route to that last ADC. Additionally digital clock delay chips can be put in the path or a few pF of capacitance placed on the clock line in order to fine tune the delays to be exactly 1/4 the clock period.

Additionally if the ADCs don't have the required analog bandwidth you might need to have a separate sample and hold in front that can take bites at full bandwidth and hold them steady for the ADC to convert correctly. Having 10GSPS is not very useful if your bandwidth is only 100MHz for example.
Title: Re: 6GSps FPGA to Memory Handling...
Post by: NorthGuy on April 26, 2018, 08:28:07 pm
At these sort of speeds you can already generate the delay by simply making the clock take long squiggly route to that last ADC.

Not that long - slightly less than an inch (22-25 mm) for 6Gs/s :)

Many ADCs will have an ability to adjust phase through SPI or alike. Thus, the delay can be inserted and calibrated at run time with great precision (such as 10 ps). You can use a fast comparator to produce calibration edges.

Title: Re: 6GSps FPGA to Memory Handling...
Post by: BrianHG on April 27, 2018, 01:42:06 am
Make sure you get top notch, impedance controlled PCBs with this project.  Plus, PCB cad software which will calculate and manage delay routing.  Even with this, you will most likely need to have calibrated delay timing settings for each PCB and have a test jig with reference signals to perform calibration.  Also, don't forget about temperature drift.

I'm not saying it's impossible, it's just going to be difficult to make many of these with such tight tolerances across 4 FPGA to weave those ADCs the way you want.