DSP 101: Developing FOSS FW for Zynq and Cyclone V based DSOs

[rhb]

This is a continuation of a thread with the same title in Test and Measurement. Please read it if this interests you.

I am embarking on a project to write a general open source firmware framework for instruments based on the Zynq 7010/20 devices.

I have no prior experience with FPGAs, but lots of DSP and hardware experience.

For development platforms  I have a Zynq 7010 based Instek GDS-2072E and a Zybo Z7-20 with an Instek MSO-2024EA to probe the targets.

My first objective is to implement sampling.  Read one or more 8 bit values and store them to DRAM at maximum speed.  For a 4 channel DSO this implies 2 GB/S of write to the DRAM.  I do not expect that to be easy to do, but obviously it can be done.

The initial sampling IP will do the following:

regularly sample 1-2 ADC channels at 1 GS/S
regularly sample 3-4 ADC channels at 500 MS/S
sample up to 32 bits of logic state and run length compress on the fly
randomly sample 1-4 ADC channels and resample to uniform sampling on the ARM cores

I am particularly interested in finding examples of reading GPIO and writing to DRAM on Zynq boards.

[edit] changed title to highlight this is Zynq oriented

[nctnico]

IMHO you should stay closer at home and try to add features to the existing firmware of the GDS-2072E.
Creating an oscilloscope from scratch without FPGA experience won't be finished before the Zync is an obsolete piece of technology.

[NorthGuy]

Here's an example of DDR3 driver for Zynq. It overcomes some of the MIG shortcomings, so you can read/write DDR3 faster. They did it for camera, but the usage for scope is not much different. I didn't look at the details, but it looks very reasonable. I think it's open source, but I don't think it would work on 7010 because it uses HP pins.

https://blog.elphel.com/2014/06/ddr3-memory-interface-on-xilinx-zynq-soc-free-software-compatible/

[rhb]

Xilinx has a DMA core and a "Central DMA"  core.  I've not been able to sort out the difference, but I'm in acronym overload at the moment.  There is also a video oriented DMA core.

From what I *think* I understand so far, to transfer 2 GS/S requires multiplexing  multiple AXI  DMA streams.

My first priority is gaining some understanding of FPGAs and the DSO implementation.  Once I know enough to modify the existing DSO FW I shall certainly do so, but I've got a lot to learn first.  Good Will  doesn't provide a lot of support information with regard to hacking their product.  So the Xilinx documentation is pretty important.  I've already spent a good bit of time on the boot process which has helped a great deal.  I still have not figured out how to read an installed  image over the JTAG interface though.

[asmi]

Here's an example of DDR3 driver for Zynq. It overcomes some of the MIG shortcomings, so you can read/write DDR3 faster. They did it for camera, but the usage for scope is not much different. I didn't look at the details, but it looks very reasonable. I think it's open source, but I don't think it would work on 7010 because it uses HP pins.

https://blog.elphel.com/2014/06/ddr3-memory-interface-on-xilinx-zynq-soc-free-software-compatible/

This controller is infested with GPL virus. For me that's reason enough to avoid it like a plague And the only "shortcoming" the author seem to have "fixed" is he avoided using hard blocks (phasers) which are built into FPGA exactly for use in memory controllers (as far as I understand these blocks are used for write levelling or DDR3/3L). To me that's not that much of a shortcoming.
You don't need MIG to access DDR3 that is connected to Zynq's PS side (which is how vast majority of Zynq-based systems built). Hard DDR controller that is built into PS supports up to 1066 MT/s via 16 or 32bit bus in all speed grades, and if you want to achieve that kind of performance using Artix fabric, you'll need very expensive -3 speed grade. The only Kintex-based Zynq (which have HP pins and as such can go beyond 1066 MT/s in fast speed grades) supported by free Vivado edition is Z030.

The real problem of all Zynq SoCs (except UltraScale/US+) is that PS supports only 1Gb of RAM max, so if your design requires more - you're forced to implement a soft controller *and* some way to tunnel data from PL-side DDR3 into PS-side DDR3.

But as far as I understand the reason Zynqs are used in DSO/MSO is that ARM cores are used as application processors to run GUI and other system management-related tasks, while all of the actual work is done in fabric.

[nctnico]

But as far as I understand the reason Zynqs are used in DSO/MSO is that ARM cores are used as application processors to run GUI and other system management-related tasks, while all of the actual work is done in fabric.

It is the other way around. Having a powerful ARM CPU on board allows to do lots of things like decoding, FFT, math in software which shortens the development time and makes the system more flexible.

[NorthGuy]

This controller is infested with GPL virus. For me that's reason enough to avoid it like a plague

You don't necessarily need to use the code. I don't know much about the code and I didn't look at it. It only runs on HP pins. HR pins don't have FINEDELAY (not even any output delay), so it's pretty much useless on 7010 or Artix. But reading the discussion is interesting by itself.

And the only "shortcoming" the author seem to have "fixed" is he avoided using hard blocks (phasers) which are built into FPGA exactly for use in memory controllers (as far as I understand these blocks are used for write levelling or DDR3/3L). To me that's not that much of a shortcoming.

The shortcomings are basically two:

First, MIG uses very slow IO FIFOs which actually work as a limiting factor (limits the memory speed to 800Mb/s for -1 and -2 grades and 1066Mb/s for -3, which is 4x speed of the FIFO).

Second is the PHASER design. Reading from the memory is synchronized by strobes transmitted from the DDR chip on DQS lines. The strobes only persist while the data is transmitted. Then the Xilinx's ISERDERS needs few more clocks to put the data through. For this purpose, PHASER maintains a free-running clock which is synchronized with DQS strobes. In the absence of reads for some span of time (1 us precisely) they desynchronize. Therefore, MIG has to switch to reading at least once every 1 us. Not a big deal if you constantly read and write, but if you want to write in big blocks (such as when acquiring data from an ADC), these mandatory reads force the controller to switch the direction twice every microsecond, which will take at least 10% off of your write speed.

You don't need MIG to access DDR3 that is connected to Zynq's PS side (which is how vast majority of Zynq-based systems built). Hard DDR controller that is built into PS supports up to 1066 MT/s via 16 or 32bit bus in all speed grades, and if you want to achieve that kind of performance using Artix fabric, you'll need very expensive -3 speed grade.

I don't know the details of Zynq construction, but PS side needs memory for its own purposes. If you want to use substantially all memory bandwidth, I would guess you have to have DDR3 memory on the PL side. But I may be wrong about this.

[rhb]

Why wouldn't one want to use one of the AXI DMA IP cores provided Xilinx?  I've not sorted out the difference among  "DMA" , "Central DMA" and "video DMA" but the general verbiage suggests one of these is what one wants to use to write the ADC output to DRAM.

The AXI DMA controller will support widths up  to 128 bytes though there may be a lower limit imposed at the DRAM end.  Of course, wider transfers imply feeding the ADC to a FIFO to buffer the DMA.

[AndreZheng]

I'm also interested in this project. I can do zynq-soc hardware design, QT GUI design.

[rhb]

It's a very long way off, but I'd like to see instrument FW become predominately an off the shelf, open source component that instrument OEMs use and modify to suit their hardware. That would let them focus their efforts on HW design and manufacture and give users the ability to add features and fix bugs.

Some features  I'd like:

moving window mean and standard deviation traces in addition to the most recent trace
ability to perform arbitrary processing on a trace before feeding it to an XY display (motivated by plotting core hysteresis)
color coded probability density function plots.
spectrum analyzer quality FFT displays

The general outline of what I have in mind at the moment is as follows:

DMA from ADC to DRAM
DRAM to PL to DRAM for processing
DRAM to PS for display

The Instek has a Spartan 6 that appears to be dedicated to handling the UI and LCD, so the display part may be a bit more involved.  But a multiprocess, interrupt driven architecture seems to me the easiest to implement successfully.

[NorthGuy]

The AXI DMA controller will support widths up  to 128 bytes though there may be a lower limit imposed at the DRAM end.  Of course, wider transfers imply feeding the ADC to a FIFO to buffer the DMA.

Looking at the pinouts, depending on the package, you get either 16 or 32 DQ pins on the PS side. Which means that there's a limit of 32*1066 = 34Gb/s = 4GB/s you absolutely cannot jump over. You don't have much control of the PS side DDR3 controller (as you would if you had DDR3 on the PL side). To figure out how close you can get to this theoretical maximum, you need to try.

It is entirely possible that someone has already tried. Quick googling brought these results:

https://forums.xilinx.com/t5/Embedded-Processor-System-Design/Debug-DDR3-memory-controller-for-Zynq-7000/td-p/395503

https://www.reddit.com/r/FPGA/comments/5h2lbt/xilinx_zynq_ddr_performance/

I wouldn't keep the hopes high for the PS side.

[andrey_filippov]

This controller is infested with GPL virus.

This is funny - even Microsoft was for one year a top contributor to the "infested with GPL" code :-) .  And you may compare Apple (with their OS started from the non-viral BSD) and Android that was built on top of "infested" GNU/Linux code. It is not that Google is "less evil" they just could not kill the "virus". And so my kids can use smartphones thanks to Cyanogen/Lineage.

Phaser and other Xilinx proprietary modules are bad not only because they are not moral (as RMS would probably say), they are encrypted and so I can not use the simulation tools I prefer - Icarus with Cocotb. And I can not troubleshoot my code because of this "black box".  I expect it to have some bugs that you will hit immediately when you'll try to do something differently from what they had in mind. And then what? Pray to get support? I tried, not once - this is the latest: https://forums.xilinx.com/t5/Synthesis/After-upgrading-from-15-3-to-17-4-I-cannot-produce-working/td-p/827195 . So I would never spend my time on dealing with "black boxes" until there is any possibility to go around. I described my experience with MIG in an old blog post: https://www.elphel.com/blog/2013/10/fpga-is-for-freedom/

Using just the system memory over AXI-4 is not always an option when you need bandwidth, sometimes you have to use a dedicated controller. Additionally our controller benefits from having very predictable memory access patterns for image processing - mostly in line-scan order and rectangular tiles. This and the fact that latency can be tolerated, controller makes possible accesses from 16 concurrent channels providing over 95% memory bandwidth utilization.

[asmi]

This is funny - even Microsoft was for one year a top contributor to the "infested with GPL" code :-) .  And you may compare Apple (with their OS started from the non-viral BSD) and Android that was built on top of "infested" GNU/Linux code. It is not that Google is "less evil" they just could not kill the "virus". And so my kids can use smartphones thanks to Cyanogen/Lineage.

Phaser and other Xilinx proprietary modules are bad not only because they are not moral (as RMS would probably say), they are encrypted and so I can not use the simulation tools I prefer - Icarus with Cocotb. And I can not troubleshoot my code because of this "black box".  I expect it to have some bugs that you will hit immediately when you'll try to do something differently from what they had in mind. And then what? Pray to get support? I tried, not once - this is the latest: https://forums.xilinx.com/t5/Synthesis/After-upgrading-from-15-3-to-17-4-I-cannot-produce-working/td-p/827195 . So I would never spend my time on dealing with "black boxes" until there is any possibility to go around. I described my experience with MIG in an old blog post: https://www.elphel.com/blog/2013/10/fpga-is-for-freedom/

GPL is not free, nor was it ever designed to be. It's the biggest cancer of entire OS. So spare that moral BS pls to someone who doesn't know better. Also I find it funny when somebody's talking about openness and then just proceed to use PROPRIETARY ASICs which is what FPGAs are
Now, whether PHASERs have bugs or not is absolutely irrelevant because they are fricken HARD blocks, so you will have to just deal with the way they work, or don't use them at all. Same goes for any other hard IPs that are inside FPGAs.

Using just the system memory over AXI-4 is not always an option when you need bandwidth, sometimes you have to use a dedicated controller. Additionally our controller benefits from having very predictable memory access patterns for image processing - mostly in line-scan order and rectangular tiles. This and the fact that latency can be tolerated, controller makes possible accesses from 16 concurrent channels providing over 95% memory bandwidth utilization.

Yes that's true, which is why Xilinx documented two other ways to interface memory using MIG - either using "User Interface Block", or just using PHY and implementing the actual controller yourself.
But that is only possible when memory is connected to fabric, as in case of Zynq the only way to connect that soft controller to the PS is to use AXI bus, which defeats the whole point, and might actually make matters worse in some cases.

[jonnyb]

At the risk of hijacking the thread...

GPL is not free, nor was it ever designed to be. It's the biggest cancer of entire OS

This "cancer" theme was pushed by outspoken Steve Ballmer more than 15 years ago https://www.theregister.co.uk/2001/06/02/ballmer_linux_is_a_cancer/

Yet Microsoft (including Ballmer himself http://www.zdnet.com/article/ballmer-i-may-have-called-linux-a-cancer-but-now-i-love-it/) have now embraced it https://www.theverge.com/2016/11/16/13651940/microsoft-linux-foundation-membership and were actually the top contributor at one point.

Microsoft are in it because it makes money, I don't think Microsoft does stuff for moral (BS or otherwise) reasons

[rhb]

The Xilinx documentation makes the claim that AXI supports power of 2 widths from 32 to 1024 bits.  I did not attempt to dig into the fine print, but as noted, I'm rather skeptical of that.

The larger problem is I can't get Vivado to talk to the Zybo Z7.  It crashed attempting to install the USB drivers and crashed again when I tried to do it manually.  The instructions I was directed to by Xilinx on their forum asked me to go to <install_path>/bin/nt[64].  There are 30 directories in the installation that meet that criterion. Then I was instructed to run "wdreg" which Windows says it does not recognize and which does not exist on my system.  I was dead in the water all day reading documentation as a result.  So far as I can tell, I'm the only one who has viewed my followups on the Xilinx forum.  Hopefully that will change.

Vivado consumes ~45 GB of disk space.  It has 68 bin directories scattered around this mess and multiple copies of the same files.  I did check the driver install files to make sure the duplicates were identical, but I decline to fix Vivado at any price.  My biggest concern at this point is that the Xilinx staff appear unable to communicate clearly.  Telling me to navigate to one of 30 directories does not inspire confidence. I'm not all that surprised as Vivado has over 163,000 files and over 3000 DLLs.  It gives new meaning to "spaghetti code". 

Sadly, I have seen this sort of mess before.  I was once asked to help get a software package working for which my employer paid $80,000 US each year for support.  After a week of beating on it, the support guy said, "If you get it working please send me the fixes so I can give them to the other customers."  They had to scarpe me off the ceiling with a putty knife.

As for the GPL, could we please leave out politics and religion?  I just want instruments that work correctly instead of doing stupid things some fool wrote while thinking about their weekend plans.

[asmi]

The Xilinx documentation makes the claim that AXI supports power of 2 widths from 32 to 1024 bits.  I did not attempt to dig into the fine print, but as noted, I'm rather skeptical of that.

There is no need to be skeptical. The bus itself will transmit as much as sender and receiver will be able to push, and it intrinsically doesn't add any overhead (or can be made not to).
MIG documentation recommends AXI data bus width of 8x memory data width for 4:1 controller and 4x for 2:1 controller - this way entire DDR3 burst lasts exactly one UI clock cycle.

The larger problem is I can't get Vivado to talk to the Zybo Z7.  It crashed attempting to install the USB drivers and crashed again when I tried to do it manually.  The instructions I was directed to by Xilinx on their forum asked me to go to <install_path>/bin/nt[64].  There are 30 directories in the installation that meet that criterion. Then I was instructed to run "wdreg" which Windows says it does not recognize and which does not exist on my system.  I was dead in the water all day reading documentation as a result.  So far as I can tell, I'm the only one who has viewed my followups on the Xilinx forum.  Hopefully that will change.

This is odd. I've been using Vivado since pretty much first release and never had any issues.

Vivado consumes ~45 GB of disk space.  It has 68 bin directories scattered around this mess and multiple copies of the same files.  I did check the driver install files to make sure the duplicates were identical, but I decline to fix Vivado at any price.  My biggest concern at this point is that the Xilinx staff appear unable to communicate clearly.  Telling me to navigate to one of 30 directories does not inspire confidence. I'm not all that surprised as Vivado has over 163,000 files and over 3000 DLLs.  It gives new meaning to "spaghetti code". 

My Vivado folder takes ~25 GBs, and that includes DovNav, SDK and HLS. I'd suggest you to do a complete reinstall.
And no need to insult their developers without having any facts to back it up.

As for the GPL, could we please leave out politics and religion?

There is no politics nor religion. MIT is free license, and so is Apache and some others. GPL is not.

[andrey_filippov]

Now, whether PHASERs have bugs or not is absolutely irrelevant because they are fricken HARD blocks, so you will have to just deal with the way they work, or don't use them at all. Same goes for any other hard IPs that are inside FPGAs.

I do not think so. PHASER exists in two ways - as a hard macro in the silicon, and as a simulation module. Yes, I know that hard macro somehow works, at least in certain designs. But I need it to work in my design, and I need the simulation to perfectly match the hardware. And the quality of the silicon is usually better than that of the simulation modules , because nobody really cares. And I do not want to spend any time on guessing what is going inside the PHASER. It is not just a "black box", it is not even documented (as some other similar modules). So yes, you are right - "don't use them at all". And these modules anyway do not agree with my simulation software.

[andrey_filippov]

As for the GPL, could we please leave out politics and religion?  I just want instruments that work correctly...

I'm an atheist. And as "access to the source code is a precondition" for this software to be free, it is also a precondition to make sure that it is in your own hands to make "instruments to work correctly", I do not want to use black magic to tame the black boxes, and spells to satisfy the software.

And the right license just makes sure it will stay that way.

[NorthGuy]

Vivado certainly is the champion of the bloat, with all the usual consequences - low usability, slowness, bugs. There's absolutely no doubts about this.

About GPL. If someone publishes his code, it is up to him what kind of license he uses. However, open source code is open, so anyone has a right to look at it regardless of the license. If, on the other hand, you want to use GPL-ed code in your project, it strips off your divine right to protect the code you have written, thus using such code looks like pure madness to me.

[nctnico]

About GPL. If someone publishes his code, it is up to him what kind of license he uses. However, open source code is open, so anyone has a right to look at it regardless of the license. If, on the other hand, you want to use GPL-ed code in your project, it strips off your divine right to protect the code you have written, thus using such code looks like pure madness to me.

That depends entirely on the project. Using GPL is an easy way to share resources between companies without needing complicated legal structures. There are many examples of companies contributing to GPL software which they need but it is not their core business (the Linux kernel for example but I've also spend time on improving things in OpenOCD while getting paid).

[NorthGuy]

That depends entirely on the project. Using GPL is an easy way to share resources between companies without needing complicated legal structures. There are many examples of companies contributing to GPL software which they need but it is not their core business (the Linux kernel for example but I've also spend time on improving things in OpenOCD while getting paid).

Contributing to a running open source is one thing. Doing your own development is another.

At any rate, Andrey Filippov made a brilliant controller and clearly demonstrated that Xilinx's FPGA is capable of much higher speeds than Xilinx's MIG. His work only characterizes HP pins (and thus is not directly applicable to this project as 7010 doesn't have any HP pins) but a similar approach may be developed for HR pins too. Since this doesn't involve using the source code, it doesn't matter whether it is GPL-ed or not. Whether HR pins can yield similar performance remains to be seen.

[rhb]

Could we *please* drop GPL and such? 

 I started this thread to discuss the details of configuring a Zynq to function as a DSO.  But I'm beginning to think that open source firmware for COTS DSOs is impossible because any discussion immediately leads to irrelevant arguments.

I am *not* at all interested in whether anyone  does or does not like the GPL, BSD, MIT or any other license.

[andrey_filippov]

"Open Source" is in the title of the thread you started. If you do not care about FLOSS licenses - why do you care about "Open Source" ? There are plenty of the proprietary DSOs on the market.
Trying to just use some unlicensed freeware will lead you to create some poor man's DSO, not a state-of-the-art device.

[rhb]

 

"Open Source" is in the title of the thread you started. If you do not care about FLOSS licenses - why do you care about "Open Source" ? There are plenty of the proprietary DSOs on the market.
Trying to just use some unlicensed freeware will lead you to create some poor man's DSO, not a state-of-the-art device.

There are plenty of  DSOs with *buggy* proprietary firmware.  What would you call a project to write portable FW that the user can modify or fix?

I was hoping to get some input on how to implement the sampling task as it appears to be the most difficult.  It's also a key part of implementing  compressive sensing which is certainly state of the art.  So far as I know that's not actually been implemented by any vendor. I've got 30 years of experience writing and modifying large, complex DSP codes.  Scopes have a lot of knobs and buttons to modify parameters, but they don't do anything very complex. At least not relative to what I'm used to doing which typically takes many  pages of advanced math just to describe.  Things like Fourier transforms are so routine they hardly require any thought.

[NorthGuy]

I was hoping to get some input on how to implement the sampling task as it appears to be the most difficult.

Not that difficult. The ADC most likely provides a parallel LVDS interface (probably DDR), possibly de-serialized by 1:2 or 1:4. FPGA has built-in ISERDES objects to de-serialize it further. You may need to calibrate input delays, but once it's done you simply get a stream of numbers coming out of the set of registers. At 100-200 MHz you simply write it to FIFO and then, on the other end of the FIFO (which is another hardware object inside FPHA), you store the data into the memory, possibly forking out some of the frames for immediate processing. You only need to get memory which is fast/wide enough to accept all the data.

If you want to use an existing scope, you probably should start from reverse-engineering the schematics and getting the datasheet for the ADC. This seems like a lot of work to me. Once you have the datasheet, you either need to find a way to connect to FPGA's JTAG to do scans and detect which pins are connected to the ADC and in what order. Or you can unsolder the FPGA and detect the connections with DMM. You also need to figure out how the memory is connected and possibly few other things. Once you know the hardware, you can start the design. Nothing too difficult, but takes lots of time, and if you encounter some "oops" in the hardware which prevent you from getting the desired speed, the project is doomed.

I think it's easier to build your own hardware, but ADCs are expensive, so you're probably looking at $20k+ until you get a working board. You probably won't be able to recover this by selling boards - who's going to pay $1-2k for a dubious board? Instead, for that money, you may be able to buy off-the-shelf PCIe acquisition card for the PC and then write custom PC software which will be completely customizable and will do any mathematics you want.