DSP 101: Developing FOSS FW for Zynq and Cyclone V based DSOs

[rhb]

Most of the possibilities are nailed down pretty tightly by the data sheets.  A rather modest amount of testing should find the correct FPGA pins without doing anything as drastic as removing the ADC.  Unless I missed something, the Zybq 7010 used in the Instek only has 32 GPIOs.  The whole point of getting the Zybo was to be able to test FW before loading it to the DSO.

The 7010-1 part has a maximum clock of 120 MHz IIRC.  After several thousand pages of documentation I'm not entirely sure how much I remember correctly.  So it appears to my ignorant eye that the ADC has to be buffered with a FIFO to handle 1-2 GS/S using 16 byte transfers.

I have no interest in designing a DSO board and flatly reject as pointless doing that.  There is no way to produce it at a competitive price.  And it's a lot of work.  At the end of which you still have to write software for it.

The schematics are not obtainable except by reverse engineering the board.  I'd like to do that and may attempt it at some point.  SMD parts make it look reasonably tractable with suitable software and a USB microscope.  But that is a project for another day and my main goal there is the ability to make device level repairs.

Can anyone point me to an example of using DMA to transfer 8 bit values read from 8 GPIO pins at 400 MS/S?  That's the start of a good LA. I know it can be done, but I've not yet found out how.  Just people telling me it's all impossible which casual inspection shows is a specious argument.

[nctnico]

The easiest is to have a clock of 200MHz and use a IDDR (input DDR flipflop) to sample the data. Put that into a dual buffer (dual port) memory and have a DMA engine write it to the memory. I don't think there are exact examples for this because it is quite straightforward to make once you get the hang of how the internal bus works.

[asmi]

Most of the possibilities are nailed down pretty tightly by the data sheets.  A rather modest amount of testing should find the correct FPGA pins without doing anything as drastic as removing the ADC.  Unless I missed something, the Zybq 7010 used in the Instek only has 32 GPIOs.  The whole point of getting the Zybo was to be able to test FW before loading it to the DSO.

What makes you think that ADC is connected to PS GPIO? I would bet they are connected to PL as PL pins can work up to 950 Mbps in DDR mode using SERDES for speed grade 1. Depending on package, Zynq Z010 has 54 or 100 pins on PL side, and 32 or 54 on PS side (if we exclude dedicated DDR3 pins).

[NorthGuy]

What makes you think that ADC is connected to PS GPIO? I would bet they are connected to PL ...

That's true. What's the point in using Zynq if you only use PS?

[rhb]

What makes you think that ADC is connected to PS GPIO? I would bet they are connected to PL ...

That's true. What's the point in using Zynq if you only use PS?

I never thought the DSO  ADC was connected to the PS.  Why on earth would anyone do that or think I suggested that? 

Please forgive my having conflated the specs for the Zybo  Z7-10 and Z7-20 with the chips themselves.  I've been reading a lot of documentation the last few days.  But testing  54 PL GPIOs (7010) is not a lot worse than testing 32.   Dump all the pins into memory via DMA with a square wave on the DSO inputs and then have the PS look for the pins that change at that rate.  I'm sure there are annoying little gotchas, but hardly rocket science. And much easier than removing and reinstalling a BGA.  The UI is likely to be a bit more difficult as the front panel and LCD are connected to a Spartan 6.  But the ADC traces are easy to identify going to the Zynq. Avoiding stomping on the Spartan bitstream is probably going to take some study.  But it might be static.

In any case, the devices in the DSO place rather tight bounds on how the DSO is wired.  And I'm sure the ARM cores wouldn't mind testing a few possibilities if one asked them politely.

I found reference to examples for DMA, CDMA and VDMA in Xilinx documentation about the DMA engines which also made clear the distinction among the three.   I haven't looked at them yet because I'm still battling Vivado. It should be pretty straightforward once I get Vivado to talk to the Zybo. My first objective is to get the PL to scribble into DRAM as fast as possible.   But once I can read and write the Zybo memory I should be able to hook a JTAG to the DSO and read out the FW to a file for safe keeping.  I don't want to stomp on the DSO FSBL or SSBL, but it may be necessary, so I want to be sure I can recover if I have to do that.

PS Aside from being able to fix bugs and bad UIs, I want to add features such as computing mean and standard deviation of the trace relative to the trigger and stop the sweep if the trace goes outside of a user specified deviation.  Essentially a statistical  automask function.  To do that one has to be able to change the bitstream itself.

[langwadt]

What makes you think that ADC is connected to PS GPIO? I would bet they are connected to PL ...

That's true. What's the point in using Zynq if you only use PS?

I never thought the DSO  ADC was connected to the PS.  Why on earth would anyone do that or think I suggested that? 

Please forgive my having conflated the specs for the Zybo  Z7-10 and Z7-20 with the chips themselves.  I've been reading a lot of documentation the last few days.  But testing  54 PL GPIOs (7010) is not a lot worse than testing 32.   Dump all the pins into memory via DMA with a square wave on the DSO inputs and then have the PS look for the pins that change at that rate.  I'm sure there are annoying little gotchas, but hardly rocket science. And much easier than removing and reinstalling a BGA.  The UI is likely to be a bit more difficult as the front panel and LCD are connected to a Spartan 6.  But the ADC traces are easy to identify going to the Zynq. Avoiding stomping on the Spartan bitstream is probably going to take some study.  But it might be static.

In any case, the devices in the DSO place rather tight bounds on how the DSO is wired.  And I'm sure the ARM cores wouldn't mind testing a few possibilities if one asked them politely.

I found reference to examples for DMA, CDMA and VDMA in Xilinx documentation about the DMA engines which also made clear the distinction among the three.   I haven't looked at them yet because I'm still battling Vivado. It should be pretty straightforward once I get Vivado to talk to the Zybo. My first objective is to get the PL to scribble into DRAM as fast as possible.   But once I can read and write the Zybo memory I should be able to hook a JTAG to the DSO and read out the FW to a file for safe keeping.  I don't want to stomp on the DSO FSBL or SSBL, but it may be necessary, so I want to be sure I can recover if I have to do that.

PS Aside from being able to fix bugs and bad UIs, I want to add features such as computing mean and standard deviation of the trace relative to the trigger and stop the sweep if the trace goes outside of a user specified deviation.  Essentially a statistical  automask function.  To do that one has to be able to change the bitstream itself.

does the DSO run linux?

[nctnico]

Yes.
BTW there is also a 'Red Pitaya' which IIRC is an open source oscilloscope/data aqcuisition board. That may be a good start as well to look at working examples. IMHO for this project to succeed it shouldn't be started from scratch.

[langwadt]

Yes.
BTW there is also a 'Red Pitaya' which IIRC is an open source oscilloscope/data aqcuisition board. That may be a good start as well to look at working examples. IMHO for this project to succeed it shouldn't be started from scratch.

has anyone checked if the PL is secured on the DSO? afair there is no way to reconfigure a PL that has been loaded with encryption except a hardware reset, so you'll have to rebuild the bootloader. It is also possible to require encrypted configuration then the chip is useless without the key

[rhb]

Yes.
BTW there is also a 'Red Pitaya' which IIRC is an open source oscilloscope/data aqcuisition board. That may be a good start as well to look at working examples. IMHO for this project to succeed it shouldn't be started from scratch.

has anyone checked if the PL is secured on the DSO? afair there is no way to reconfigure a PL that has been loaded with encryption except a hardware reset, so you'll have to rebuild the bootloader. It is also possible to require encrypted configuration then the chip is useless without the key

It seems unlikely that it is currently, but I would not be surprised if  GW did that.  They removed dropbear once the root password was leaked.  At present that is easily restored, but it could be made much more difficult.  The FW on my 2072 still has dropbear running which is one of the reasons I want to make sure I save a copy of the FW.

I may get a Red Pitaya also.  Certainly I'm going to try to use the Red Pitaya code.  For some reason I was not aware of how advanced that was.  I picked the Zybo for the 1 GB of DRAM.  There are a lot of cheap Zynq boards and I was getting weary of reading spec sheets.

[NorthGuy]

I don't think 50Ms/s scopes have much in common with 2Gs/s scopes, so I doubt the Red Pitaya is of any help.

[nctnico]

I don't think 50Ms/s scopes have much in common with 2Gs/s scopes, so I doubt the Red Pitaya is of any help.

The basics are the same: shuffling data around. At higher sampling rates there is just more data to deal with which means higher speeds and wider paths. AFAIK the Zync has several GByte/s of memory bandwith to play with anyway.

[rhb]

I've been browsing github and other related sites.  I found this comment disturbing.

"API1+SCPI: this code is now proprietary, will not be part of this rel… "

That's fine if the the author did all the work, but not so nice if people were conned into working for free.

The 14 bit ADC/DAC is *very* attractive.  So I'm likely to get one of these once I make more headway. But the price tag gets pretty high quickly. 

The 7010 part can move at least 2 GB/S as that is what the 4 channel scopes based on it are doing.  There's a lot of unused real estate in the 2072E PL, so it may be possible to wedge in an internal LA function to help probe the HW.  Moving 2 GB/S is probably not easy, but clearly possible.  If I can get 32 bits of GPIO at 500 MS/S with a 200 MS buffer using the Zybo  I'll be very pleased.  Put a proper front end on it and I'll have a very good LA.

Hopefully I will get Vivado working later today.  I've got to sort out USB access on a VBox VM which does not look as if it will be fun.  I've also got to figure out why the UBIFS and MDT-Tools don't work on the Debian 9.3 VM.  I had some, but not all of it working on a physical system before I decided to try using VMs.  I have plenty of hardware, but the heat buildup gets to be a problem.

But for now I have to go finish a wall :-(

[rhb]

Well, I'm finally getting moving on this again.  I've almost finished the development system.  It has a Zybo Z7-20 and a BeagleBoard X15, dedicated gigabit switch, PSU, HDMI to VGA convertors, two port KVM switch, SATA drive for the X15.  Still to install are a pair of switched USB 3.0 hubs and a USB 2.0 hub for initial board programming.  And I still don't have the KVM USB ports connected to the boards.

I've started analyzing the signal path.  Coming out of the ADC the first thing is the filter to correct for AFE and ADC errors.  From there it needs to go to the BW filters.  The next stage is distribution to the math and trigger sections before going into main memory.

Basic plan is to create an AWG waveform generator IP block.  Once I can feed that to memory, I'll create the calibration filter block to correct for part variations in the AFE and ADC.  The BW filters are pretty trivial.  I plan to implement 20, 50, 100, 200 and user specified with a choice of best rise time with <3% overshoot and a Gaussian step with no overshoot.  Once that is writing to memory on the Zybo then comes the task of writing AWG waveforms for all the trigger modes and implementing the trigger IP blocks.  In addition to the normal start trigger, I plan to implement a stop trigger so that the scope will stop data acquisition so one can look at the trace before a fault condition was encountered. This is particularly aimed at embedded debugging work.

After that comes the display and measurement IP blocks.  Visual persistence is the only thing that looks difficult.  The display will be able to operate in triggered YT, roll and XY with math functions as input to XY mode so that one can measure hysteresis loops without needing an analog integrator.  I've only just started on the display requirements, so those are still pretty vague.

The DAQ chain is very standard stuff in seismic processing, so the only mysteries are the particulars of Verilog and the Zynq.  I'm giving very serious thought to building a 2nd development system with one or two FPGAs from other vendors for testing the Verilog code for portability.

The availability of high performance ADC eval boards at nominal prices makes a bespoke USB DSO practical if there is a portable set of FPGA IP blocks for the FW framework.

The SATA drive for the X15 is in the lower left under the ethernet switch. 

[wpwrak]

Well, I'm finally getting moving on this again.

Excellent ! The world desperately needs state of the art Open Source scopes. Most just don't know it yet ;-)

So your plan is to build a scope "from scratch", instead of repurposing a mass-market scope, like the many Zynq7000 series devices that are on the market now ? Or is this just a first step to familiarize yourself with the issues ?

I like the use of a PC case to keep things clean and tidy. My setups tend to end up looking like a Cthulhu family reunion ...

- Werner

[rhb]

The original aim was simply to write new software for mass market scopes like the Instek and Siglent.  For most purposes that is the best option.  I have an Instek  GDS-2072E I purchased expressly for testing.  It's much more practical than designing a scope and you could never meet that price point.

However, after playing with $20k instruments from a couple of the top tier OEMs I was quite appalled at how buggy they were. Then I got an AD email promoting a new ADC.  The eval board has two SMA inputs and will sample at 2 GSa/S dual channel and 4 GSa/S single channel at 14 bits for $1200.  So if you figure around $2k for a high end FPGA eval board with HDMI output, $5-6K for a bespoke instrument is plausible.  The main concern is the complex protocol used by the ADCs in that class.

Previously I had rejected designing a scope, but assembling one from COTS parts is a different matter.

I've ordered an Altera DE10-Neo so I'll be testing the Verilog code on both. I'd like to get a 3rd FPGA w/ embedded ARM, but haven't been able to find one.  With only two, each vendor will claim theirs is correct and the other wrong.  But if it works on devices from two competitors it gets their attention.

I once had a problem with some code on a Sun.  I got the standard line, "It's probably a bug in your code."  Until I said, "Well, it works on the IBM and HP workstations."  Very quick, "Oh, I see what you mean."  It was resolved the next day as I recall.  I needed some obscure compiler flag which was poorly documented.

The important thing that the AD email did was remind me that I should generalize the library so that it is not device specific except at key points such as the ADC driver.  Testing simultaneously on multiple systems is the easiest way to achieve that.  You get immediate feedback which avoids writing more code which has some  unknown dependency.  Resolving a platform dependency in a hundred lines of code is *much* easier than doing it in 100,000 lines of code.

I know how to develop bug free code.  I wrote two 15,000 line libraries for a project that *never* had a bug found.  After 6 years of operation, a call to getcwd(3c) started failing on Sun systems.  On investigation it turned out that Sun had broken their C standard library.  I was testing for errno == 0 as per the standard.  The fix was an #ifdef SUN cwdstr == 0 instead.

The project as a whole had fewer than a dozen bugs reported the first year.  It went down from there and the code continued in service for 12-16 years until there was no longer any need for the functionality.  It was a 500,000 line port of mostly scientist written VAX FORTRAN code from VMS to Unix.  We found and fixed a very large number of bugs during the process of porting the code. This was very heavily used.  Every execution sent an 80 byte UDP datagram back to a company server.  Once a month I generated plots and tables of usage at all the business locations. There were about 250 users and about 3 million program runs per year.

[wpwrak]

Great ! I'm hoping for something that will run on mass-market scopes, but building your own system has obvious advantages, too. Probably not so much in terms of bugs, but in terms of not needing to reverse-engineer parts of the design.

In my experience, one of the most important things for an Open Source project to "catch on" is to create a low barrier of entry. Once you have something someone can just install on their off-the-shelf scope, provided that this scope isn't too exotic, people will begin to trickle in, even if the project is still very rough around the edges. Since there's now a number of architecturally similar scopes (i.e., centered around a Zynq), it shouldn't be too long before ports will happen.

For ports, reverse-engineered schematics or at least block diagrams, BOMs (part number of the major chips), and pin assignment tables would be very useful.

For ease of installation, it would be best if one could run the experimental Open version without having to replace the original firmware. If the scope lets you boot from external media, that would be great (1). If it lets you log in over USB or the network and download your own application, that would be just as good. (I'd expect that you can do almost everything you need from user space, and it's usually possible to disable kernel drivers that may get in the way.) This will need a bit of investigation. Another issue is to load the FPGA part (PL). With a little luck it should be possible to treat it separately from the ARM, but that depends on a number of things.

(1) The "Copying Linux from USB to RAM..." in the boot log for the 1104X-E (https://www.eevblog.com/forum/blog/eevblog-1044-siglent_s-$499-sds1104x-e-4ch-oscilloscope-teardown/msg1358858/#msg1358858) looks promising.

I have a Siglent 1104X-E coming and am getting started on the Zynq-7000 series for a work project, so I hope I'll be able to figure out some things there.

Have you created a project/repo on github, gitlab, or similar yet ?

- Werner

[rhb]

I'm primarily interested in running on the Instek, Siglent and similar.  Other stuff is just because I know from experience that portability improves code quality a lot.  And there will be new devices.

I plan to use the existing FW update process if at all possible.  So that it's just like doing an OEM update and can be replaced by the OEM FW.

My hope is to make it good enough that the OEMs decide to use it.

Except for the protocol for the front panel, there's really not a lot of reverse engineering required that I can see.  The choice of ADC nails that down and I don't see much need to modify the front panel operation.  At least on the Instek, that appears to be done by a separate FPGA.


While I've not *done* anything with the Zynq yet, I've read quite a lot of documentation about the factory bootloader, so writing a substitute for the Instek bootloader shouldn't be all that hard if it becomes necessary.

[rhb]

I've modified the title as I discovered that Intel/Altera have an approximate equivalent to the Zynq.  I should have a Cyclone V based DE10-Nano in the next couple of days.  I shall be testing the Verilog code on both.  If anyone knows of another FPGA with an ARM integrated on chip please let me know so I can get one.

I've been working on this a good bit the last few days.  The recommended approach to developing FW/IP blocks for FPGAs is to use the FPGA to generate a test signal.  It's a common theme through the tutorials I've looked at. That has the additional virtue of providing the rudiments of an arbitrary waveform generator in the process.

So I've developed a general outline of how to proceed.  This will be done first with Octave scripts, then I'll write Verilog versions if appropriate.  I'm going to start at the basics of DSP.   That should make it easy for anyone who is interested to play along.

Topics to follow:

Digital sampling and aliasing

Impulse response of a minimum phase boxcar or brickwall filter

Impulse response of minimum phase Butterworth filters

Impulse response of minimum phase Gaussian taper filters

Effects of component tolerances on analog filter responses

Designing digital correction filters to compensate for AFE and ADC errors

Time synchronous splitting into math, trigger and data buffer streams

Math functions

Trigger functions

Persistence and gradient displays

Those topics cover the first part of the DSP chain for a DSO.  They are critically important and I have observed that most DSO OEMs are deficient in various ways including the top tier.

The initial FPGA development will be focused on getting data from the ADC into DRAM via DMA at the necessary data rates.  Once the data are in memory, then things become more leisurely as LCDs can't  update faster than 60-120 Hz. 

I'm going to move this to the Projects section if I can figure out how.   Working alone this will take 12-18 months to complete.

[Marco]

(1) The "Copying Linux from USB to RAM..." in the boot log for the 1104X-E (https://www.eevblog.com/forum/blog/eevblog-1044-siglent_s-$499-sds1104x-e-4ch-oscilloscope-teardown/msg1358858/#msg1358858) looks promising.

Getting into Linux should really be first order of business. With any luck they have the whole UI+configuration logic in user space and only handle the ADC high speed data through the FPGA. Then you can just shim their software, only doing something special on a specific key-combo captured by the shim, and reduce the amount of software you have to create to get started. Doesn't solve the reverse engineering of the FPGA part of course, but every little bit helps.

If you can't get into linux and they do decryption in the first stage boot loader ... then it all turns into a clusterfuck, the zynq inside the oscilloscope would be dead weight for a start.

[rhb]

(1) The "Copying Linux from USB to RAM..." in the boot log for the 1104X-E (https://www.eevblog.com/forum/blog/eevblog-1044-siglent_s-$499-sds1104x-e-4ch-oscilloscope-teardown/msg1358858/#msg1358858) looks promising.

Getting into Linux should really be first order of business. With any luck they have the whole UI+configuration logic in user space and only handle the ADC high speed data through the FPGA. Then you can just shim their software, only doing something special on a specific key-combo captured by the shim, and reduce the amount of software you have to create to get started. Doesn't solve the reverse engineering of the FPGA part of course, but every little bit helps.

The UI is just a program running in user space on the Instek and I'd be really surprised if that's not the case for the Siglent.  The root passwords are known.  So by loading older FW one has full root access.

You've brought up an important point about the UI.  I'll need to check if it is using dynamic linking or not.  I don't know of a way to "shim" a staticly linked executable.  If you do, please enlighten me.  That would make it much easier to reverse engineer the front panel interface. I'd assumed putting an LA on the lines from the Zynq to the front panel FPGA  which is a Spartan 6 in the Instek.

I have no intention of trying to reverse engineer the FPGA.  I'm just going to start from scratch as that is *much* easier.  It will also result in a much better UI.  So initially this will be an AWG + DSO + LA + SA simulator displaying to a screen under keyboard control.  Actually loading it on the scope will be the last step.

I think you have to use the JTAG interface to change the FSBL on the Zynq.  At present the FSBL is very simple and doesn't use encryption, so if it is necessary to use the JTAG interface to change the FSBL then my biggest concern is moot.

The Altera may be more challenging as I could not download their software for the DE10-Nano.  Everything produced 404s ;-(

My next post will be on the subject of aliasing, impulse responses and minimum phase so that we have a well defined model and terminology.  While DSP is done in a lot of fields, the jargon tends to vary a lot.

[Marco]

It's not my expertise, but I can google. They don't need to link anything if they use the GPIO from the Zynq ARM core, they can use sysfs or /dev/input. If they use /dev/input with the gpio drivers I think you can use udev rules to rename the nodes and then uinput to resend them from your own program on the nodes their software expects. Not sure how to do the same thing with sysfs.

[langwadt]

(1) The "Copying Linux from USB to RAM..." in the boot log for the 1104X-E (https://www.eevblog.com/forum/blog/eevblog-1044-siglent_s-$499-sds1104x-e-4ch-oscilloscope-teardown/msg1358858/#msg1358858) looks promising.

Getting into Linux should really be first order of business. With any luck they have the whole UI+configuration logic in user space and only handle the ADC high speed data through the FPGA. Then you can just shim their software, only doing something special on a specific key-combo captured by the shim, and reduce the amount of software you have to create to get started. Doesn't solve the reverse engineering of the FPGA part of course, but every little bit helps.

The UI is just a program running in user space on the Instek and I'd be really surprised if that's not the case for the Siglent.  The root passwords are known.  So by loading older FW one has full root access.

You've brought up an important point about the UI.  I'll need to check if it is using dynamic linking or not.  I don't know of a way to "shim" a staticly linked executable.  If you do, please enlighten me.  That would make it much easier to reverse engineer the front panel interface. I'd assumed putting an LA on the lines from the Zynq to the front panel FPGA  which is a Spartan 6 in the Instek.

I have no intention of trying to reverse engineer the FPGA.  I'm just going to start from scratch as that is *much* easier.  It will also result in a much better UI.  So initially this will be an AWG + DSO + LA + SA simulator displaying to a screen under keyboard control.  Actually loading it on the scope will be the last step.

I think you have to use the JTAG interface to change the FSBL on the Zynq.  At present the FSBL is very simple and doesn't use encryption, so if it is necessary to use the JTAG interface to change the FSBL then my biggest concern is moot.

The Altera may be more challenging as I could not download their software for the DE10-Nano.  Everything produced 404s ;-(

My next post will be on the subject of aliasing, impulse responses and minimum phase so that we have a well defined model and terminology.  While DSP is done in a lot of fields, the jargon tends to vary a lot.

does it have an sdcard slot? if you can get to the mode pins you can boot and run a zynq off a an sdcard

[rhb]

I'm pretty sure it doesn't have an SD card slot.  I studied the Zynq documentation quite a bit with particular attention to the boot process.  The Instek FW update format is not very complex.  The only obstacle at present is sorting out how the CRC is done.  I received misleading information via a series of PMs, but the update is not encrypted, so I understand the basic structure. 

But right now I want to focus on writing FPGA code that is as portable as possible.  If someone else wants to crack the Instek FW update format please do.  But I've got other tasks for now.

[rhb]

If you regularly sample at less than twice the highest frequency component of a signal you get aliasing.  The frequency above Nyquist gets folded back to a lower frequency.  I think that DSOs have made most people aware of the issue, but I wanted to be sure people understood what happens.

The point here is that aliased  frequencies that are slightly above Nyquist show up slightly below Nyquist.  So by following the ADC with a digital filter one can avoid needing to be 24 dB down at Nyquist for an 8 bit ADC.  This is pretty commonly done by the OEMs and is also how they implement the software license BW upgrades.

One can also compensate for variations in the AFE due to component tolerances in the same filter.

In addition to the plots, I've attached the Octave and Gnuplot files.

[hamster_nz]

Having worked on a Video pipeline on a Cyclone SoC, the DDR memory bandwidth is precious.

- CPU Instruction fetches
- CPU Data read & writes
- Video buffer write (for the CPU to access the video stream)
- Memory reads for screen overlay & GUI

Bang! All gone! And it isn't just the aggregated MB/s rates, you have to allow for the priority of access, and vendor spec sheets always quote max performance, not achievable-while-doing-five-things-at-once performance.

The on-chip block RAM is quite fast - why not reduce aspirations (at least for an early version) to sampling into 64k of Block RAM, and leave the DDR problem till later?