Idea for improving LTDZ Spectrum Analyzer & Tracking Generator 35MHz-4.4GHz

[Kalvin]

I am interested in the LTDZ 35MHz - 4.4GHz spectrum analyzer & tracking generator containing STM32F103 processor for a hobby use (amateur radio). After watching some videos and reading VMA's satellite blog, it is obvious that there are some problems and shortcomings with the existing design. For example, the default RBW is about 120 kHz, and it cannot be changed dynamically. For a general RF filter measurement and SSB spectrum analysis, this wide RBW makes this otherwise very nice instrument almost useless.

Looking at the schematics, the output of the RX mixer is fed through a 120 kHz low pass filter (read: RBW-filter) into input of a AD8307 LOG amplifier/detector. The output of the LOG detector is then fed to STM32F103's ADC input.

Since the STM32F103 has a 12-bit ADC which can run up to 1Msps, why not drop the AD8307 altogether, feed the 120 kHz filter directly into the ADC input, and make the STM32F103 do the required math and DSP. The ADC would give theoretically 12*6dB = 72dB of dynamic range, which is quite similar to AD8307 anyway. Then the STM32F103 could implement adjustable RBW, perform even FFT if needed. At the same time one can even obtain 1 - 3 bits of conversion gain (6dB - 18dB) with narrower RBW for improved dynamic range (if the noise floor is practically low enough). There are some alternative firmwares available which could be used as a starting point for this modification.

What do you think? Do you see any reasons why this scheme would not work?

----

Project at Github: https://github.com/kalvin2021/ltdz-dsp

----
Here are some pointers to books related this subject:

Freely available ebook "The Scientist & Engineer's Guide to Digital Signal Processing, 1999" by Analog Devices:
https://www.analog.com/en/education/education-library/scientist_engineers_guide.html

Freely available ebook "Software-Defined Radio for Engineers, 2018" by Analog Devices:
https://www.analog.com/en/education/education-library/software-defined-radio-for-engineers.html

Practical introductory book about digital signal processing without too complex mathematics (pun intended):
Lyons: Understanding Digital Signal Processing, 3rd Edition

----
PCB:


Schematics:


Edit: Added the books, PCB and schematics.

[radiolistener]

why not drop the AD8307 altogether, feed the 120 kHz filter directly into the ADC input, and make the STM32F103 do the required math and DSP

Using digital filter slows down STM32 and don't provide any benefit. There is no need for configurable bandwidth. It just make sweep very slow and low dynamic range.

And AD8307 is logarithmic amplifier with 92 dB dynamic range, so it allows to increase dynamic range by using logarithmic scale. If you remove it, you will get linear scale and bad dynamic range.

If you're not satisfied with software results, you can try my version NWTSHARP-LTDZ, it supports LTDZ and NWT7 devices and use better calibration algorithm, also you can see RAW ADC charts. Just select COM port, press Connect and then Sweep.

My LTDZ instance dynamic range is 50 dB. And it's limited by RF frontend design. In order to get better, it needs a new PCB. There is already exist exactly the same hardware with better PCB layout and shielding, but it cost much more.

[Kalvin]

why not drop the AD8307 altogether, feed the 120 kHz filter directly into the ADC input, and make the STM32F103 do the required math and DSP

Using digital filter slows down STM32 and don't provide any benefit. There is no need for configurable bandwidth. It just make sweep very slow and low dynamic range.

For your satellite spectrum observation purposes the 120 kHz RBW might be ok, but for applications that require more detailed spectral analysis, this 120 kHz RBW is way too wide, and makes the board basically useless.

If one wants to obtain narrower RBW with more spectral details, it is necessary to reduce the RBW from 120 KHz down to 1 KHz or even less, depending of the spectral details one wants to see. Yes, this will slow down the sweep rate, but in applications that requires more spectral details like tuning a narrowband filter or investigating narrow SSB spectrum, this needs to be done either in analog domain (AD8307) or in digital domain (using numerical DSP-algorithms).

Bypassing the AD8307 LOG-detector altogether, feeding the output of the 120 kHz lowpass-filter directly into STM32F103 ADC input, sampling the signal with up to 1Ms/s, and performing the final RBW-filtering and LOG-detector in MCU with some math/DSP, one can select narrow RBW-filter dynamically, and gain some extra dynamic range due to processing gain.

For more advanced spectrum analysis, one can even implement FFT for computing the spectral components, and one can get very narrow RBW and good spectral resolution without slowing down the sweep rate. That is something that really cannot be done with AD8307 if you reduce the RBW for improved spectral resolution.

And AD8307 is logarithmic amplifier with 92 dB dynamic range, so it allows to increase dynamic range by using logarithmic scale. If you remove it, you will get linear scale and bad dynamic range.
<snip>
My LTDZ instance dynamic range is 50 dB. And it's limited by RF frontend design. In order to get better, it needs a new PCB. There is already exist exactly the same hardware with better PCB layout and shielding, but it cost much more.

This is partially true. By taking advantage of the processing gain you can compensate the "seemingly lost" dynamic range of 72dB of a 12-bit ADC when using narrower RBW filters. With the original 120 kHz RBW, you can still get extra dynamic range by oversampling and averaging, but this will obviously slow down the scanning rate as more samples are needed.

As you have stated, the actual obtainable dynamic range may be only 50 dB for a given board/design. Even if AD8307 is technically able to provide 92 dB of dynamic range, in practice you will get much less than that due to the noise floor of the board. In that sense the 12-bit ADC with 72dB of dynamic range is sufficient, and the STM31F103 is able to compute the LOG-detector with 72dB dynamic range in software without any problems.
 
Some newer boards are said to be able to give up to 70 dB of dynamic range, so the 12-bit ADC is still sufficient for the LOG-detector, and AD8307 does not provide any real benefits. Of course, the signal level needs to be set correctly so that the full dynamic range of ADC can be utilized. With the processing gain obtainable using the narrower RBW and/or oversampling/averaging, one can compensate lost dynamic range to some extent, even if the signal level is not perfect.

[Bicurico]

If you want to use such small RBW you will either need a professional class spectrum analyser like the Siglent SSA3021X/X-P/-RT/SVA series or you might try your luck with SDR alike devices like the HackRF one or Adalm Pluto (using Satsagen).

I just tested the Arinst SSA-R2 TG and it is a much better performing device than the SMA/NWT/D6/LTDZ devices. However, it uses a fixed RBW of 200 kHz (https://vma-satellite.blogspot.com/2021/05/arinst-ssa-r2tg-one-of-themost-frequent.html).

The last cheap option would be a TinySA. At 50 Euro it is very cheap, but in EU you will probably end up getting a fake clone, unless you buy at AliExpress from Zeenko Store and handle customs yourself.

It seems to have a configurable RBW, but it is otherwise limited in terms of frequency range.

At the end of the day, it all depends on what exactly you want to do.

Regards,
Vitor

[Kalvin]

I agree that the phase noise may become an issue with smaller RBWs when using AD4351. Unfortunately I do not have any measurement data for the phase noise for these chinese boards. Fortunately there is some measurement data available, giving -60 ... -70 dBc@1 kHz for AD4351: https://www.eevblog.com/forum/rf-microwave/adf4351-simple-you-would-think/msg2514417/#msg2514417

I do have a genuine TinySA and Adalm Pluto, but wanted to take a look at these inexpensive AD4351-based spectrum analyzers / tracking generator boards, and see how their performance could be improved so that these simple and inexpensive boards would be better suited for a typical radio amateur-related measurements (spectrum analysis, filter measurements, signal generator) from the upper HF frequency range, up to UHF frequencies even beyond 1 GHz. Adding a cheap SWR bridge would create a decent and very inexpensive antenna analyzer for frequencies above 35 MHz, for example.

[radiolistener]

You can improve input SWR by removing 51 Ω on the input. With that mod input impedance will be more close to 50 Ω, because by default it has about 30 Ω input impedance. But it will leads to overload for LTDZ output, you're needs to use attenuator.

Regarding to the filter bandwidth, you can modify filter circuit and get more narrow bandwidth. But narrow bandwidth leads to gaps for a large frequency step. Since max point count is 9999, you will get:
- sweep from 35 MHz to 1 GHz = 96.5 kHz step
- sweep from 35 MHz to 2 GHz = 196.5 kHz step.
- sweep from 35 MHz to 4 GHz = 396.5 kHz step

So, 120 kHz bandwidth is good for a usual sweep from 35 MHz to 1.23 GHz

[Kalvin]

You can improve input SWR by removing 51 Ω on the input. With that mod input impedance will be more close to 50 Ω, because by default it has about 30 Ω input impedance. But it will leads to overload for LTDZ output, you're needs to use attenuator.

Regarding to the filter bandwidth, you can modify filter circuit and get more narrow bandwidth. But narrow bandwidth leads to gaps for a large frequency step. Since max point count is 9999, you will get:
- sweep from 35 MHz to 1 GHz = 96.5 kHz step
- sweep from 35 MHz to 2 GHz = 196.5 kHz step.
- sweep from 35 MHz to 4 GHz = 396.5 kHz step

So, 120 kHz bandwidth is good for a usual sweep from 35 MHz to 1.23 GHz

Thanks, I will investigate the input resistor/impedance as soon as I will receive my board. I will use SMA-attenuator(s) as needed, which will also help with the SWR.

I would not like to modify the board so that the filter will become fixed (again), because the whole idea is to modify the firmware so that it would be possible to select the RBW dynamically without further hardware modifications. So far the only modification in the hardware would be to bypass the AD8307 LOG-detector, and let the STM32F103 do the filtering, LOG-detector and possibly some further signal processing like FFT. It seems that there are some unused MCU ADC-input pins available, so it might be even possible to keep the AD8307 operational, and add a wire from filter output to the MCU ADC-input pin for the dynamically selectable RBW feature. Let's see what is possible/useful with this hardware.

[gf]

The 0Hz IF is not really suitable for channelizing the IF band into narrower bands (e.g. via FFT), since the input signal bands [f_LO-BW...f_LO] and [f_LO...f_LO+BW] do alias in the down-mixed (real-valued) IF signal, and the original upper and lower bands cannot be separated any more. Upper/lower band separation were not a problem if the IF signal were complex (I/Q), but I guess it is not a quadrature mixer.

For the original use case, this aliasing does not matter since only the total power of upper and lower band together -- i.e. [f_LO-BW...f_LO+BW] -- is calculated. But for further channelizing it does matter.

Still you have the option, of course, to apply a digital lowpass with an even lower cutoff frequency to the pre-filtered IF signal.

[ BTW, if the lowpass filter has (single-sided) bandwidth of BW, then the corresponding bandpass bandwidth is actually 2*BW, i.e. RBW is even wider. ]

EDIT:
If the frequency response of the analog lowpass filter (on this web page) is correct, then digitizing is challenging, too. If I extrapolate the filter's transition band to -100dB below peak, then I get a required Nyquist frequency of about 4MHz, requiring a sampling rate of 8MSa/a. If only the (say) 0...300kHz band is eventually evaluated in the digital domain, then one could accept some foldback and allow a lower Nyquist frequency of (4000+300)/2=2150kHz. Required sampling rate were still 4.3MSa/s then. Depends of course on the desired image rejection (I assumed 100dB).

[Kalvin]

The 0Hz IF is not really suitable for channelizing the IF band into narrower bands (e.g. via FFT), since the input signal bands [f_LO-BW...f_LO] and [f_LO...f_LO+BW] do alias in the down-mixed (real-valued) IF signal, and the original upper and lower bands cannot be separated any more. Upper/lower band separation were not a problem if the IF signal were complex (I/Q), but I guess it is not a quadrature mixer.

For the original use case, this aliasing does not matter since only the total power of upper and lower band together -- i.e. [f_LO-BW...f_LO+BW] -- is calculated. But for further channelizing it does matter.

Still you have the option, of course, to apply a digital lowpass with an even lower cutoff frequency to the pre-filtered IF signal.

[ BTW, if the lowpass filter has (single-sided) bandwidth of BW, then the corresponding bandpass bandwidth is actually 2*BW, i.e. RBW is even wider. ]

Good points. These devices do not have a quadrature mixer, so your observations are valid indeed. The DC-path from the mixer output to the input of the low-pass filter (120 kHz RBW-filter) is dc-blocked, so there will be a problem with spectral components near 0Hz. Since the tracking generator and the receiver mixer are fed with two separate, independent oscillators, it may be possible to offset the receiver LO a bit higher or lower frequency*, sample the output of the 120 kHz RBW-filter with the SMT32F103 ADC, and then perform a complex spectral shift down to 0Hz in order to get a proper analytical quadrature signal, which should resolve this 0Hz problem. Please correct me I have have overlooked something.

*For example, If one wants to have RBW of 1 kHz, the receiver mixer LO needs to be offset at least by this same amount ie by 1 KHz. In reality, the offset needs to be a somewhat more because the spectral components near DC will be attenuated due to the DC-blocks in the signal path.

[Kalvin]

EDIT:
If the frequency response of the analog lowpass filter (on this web page) is correct, then digitizing is challenging, too. If I extrapolate the filter's transition band to -100dB below peak, then I get a required Nyquist frequency of about 4MHz, requiring a sampling rate of 8MSa/a. If only the (say) 0...300kHz band is eventually evaluated in the digital domain, then one could accept some foldback and allow a lower Nyquist frequency of (4000+300)/2=2150kHz. Required sampling rate were still 4.3MSa/s then. Depends of course on the desired image rejection (I assumed 100dB).

The noise floor of these boards is probably somewhere -70dB, so the filter's transition band attenuation may not have to go all the way down to -100dB. One option is to reduce the filter's bandwidth until the attenuation is sufficient for the sample rate and the desired dynamic range to be used.

STM32F103 seems to have a 12-bit ADC which is able to sample at 1 MHz. Since STM32F103 has only 20KB RAM, and STM32F103 may not be able to process the samples in real-time, the available RAM space will be a challenge and probably cause some pain during implementation and cause some compromises in the final design. But let's see.

Edit: Thank you for the link! It looks that the original RBW/low-pass filter is quite poorly designed, so the response can be improved by changing the component values. If nothing else will help, it is possible to design and build a new low-pass filter of higher order on a separate PCB, but at this point I would not consider this as an option yet.

Edit2: I wonder whether the DC-blocking capacitor at the output of the mixer could be removed, because the low-pass filter will be dc-blocked by its design topology anyway.

[gf]

The maximum dynamic range for these boards is probably somewhere -70dB, so the filter's transition band attenuation may not have to go all the way down to -100dB. One option is to reduce the filter's bandwidth until the attenuation is sufficient for the sample rate and the desired dynamic range to be used.

STM32F103 seems to have a 12-bit ADC which is able to sample at 1 MHz. Since STM32F103 has only 20KB RAM, and STM32F103 may not be able to process the samples in real-time, the available RAM space will be a challenge and probably cause some pain during implementation and cause some compromises in the final design. But let's see.

Edit. It looks that the original RBW/low-pass filter is quite poorly designed, so the response can be improved by changing the component values. If nothing else will help, it is possible to design and build a new low-pass filter of higher order on a separate PCB, but at this point I would not consider this as an option yet.

My 100dB image rejection desire was based on 70dB + (say) 30dB processing gain. 16-bit integer quantization (for calculations) were ~98dB, too.
If 1MSa/s is the limit, then I'd expect to end up with ~50..55dB, when allowing significant fold-back (again, granted that the depicted frequency response is correct).
According to AN4841, STM32F103 can do a 1024-point Q31 FFT in about 3ms. That were not too bad. 20k RAM is indeed pretty meager, though.

[Kalvin]

The maximum dynamic range for these boards is probably somewhere -70dB, so the filter's transition band attenuation may not have to go all the way down to -100dB. One option is to reduce the filter's bandwidth until the attenuation is sufficient for the sample rate and the desired dynamic range to be used.

STM32F103 seems to have a 12-bit ADC which is able to sample at 1 MHz. Since STM32F103 has only 20KB RAM, and STM32F103 may not be able to process the samples in real-time, the available RAM space will be a challenge and probably cause some pain during implementation and cause some compromises in the final design. But let's see.

Edit. It looks that the original RBW/low-pass filter is quite poorly designed, so the response can be improved by changing the component values. If nothing else will help, it is possible to design and build a new low-pass filter of higher order on a separate PCB, but at this point I would not consider this as an option yet.

My 100dB image rejection desire was based on 70dB + (say) 30dB processing gain. 16-bit integer quantization (for calculations) were ~98dB, too.
If 1MSa/s is the limit, then I'd expect ~50..55dB, when allowing significant fold-back (again, granted that the depicted frequency response is correct).
According to AN4841, STM32F103 can do a 1024-point Q31 FFT in about 3ms. That were not too bad. 20k RAM is indeed pretty meager, though.

Let's think this backwards: What would be the maximum usable signal bandwidth at the output of a properly designed 4th/5th order low-pass filter when the sample rate is 1MHz, and the desired image rejection is 100dB (Zin=50ohms and Zout=1100ohms). After that we could gradually relax the filter requirements from 100dB down to 70dB, and see what bandwidth we will get using this current filter topology on the PCB.

Edit: Although the 20KB RAM is pretty small, the actual firmware is quite simple. If the algorithms do not use long sample buffers, then the RAM should not be a problem. What I have seen some alternative firmware implementations, they are quite straight forward and do not have any OS, so there is no need to allocate space for multiple stack frames, for example.

Edit2: One interesting observation from AN4841: STM32F103 can perform 1024-point Q31 FFT faster than Q15 FFT (214098 cycles/2.97ms vs 248936 cycles/3.46ms).

[Kalvin]

I was playing with this LC-filter design tool to get some idea what kind of performance is possible with the current filter topology available on the PCB: https://rf-tools.com/lc-filter/

The parameters I have used are:
- Low-pass
- Chebyshev or Elliptical
- Shunt first
- Order 5
- Cut-off 120 kHz
- Pass-band ripple 0.1 dB
- Input and output impedance 50

The Chebyshev gives 80 dB attenuation at 630 kHz, and Elliptical gives 80 dB attenuation at 492 kHz. Since the system is using 1 MHz sample rate, and we are mostly interested in the frequency components below 120kHz, we can tolerate some aliasing, and the MCU can perform additional filtering for the sampled data in order to attenuate the frequency components that are not in our interest and which are aliased.

[gf]

Let's think this backwards: What would be the maximum usable signal bandwidth at the output of a properly designed 4th/5th order low-pass filter when the sample rate is 1MHz, and the desired image rejection is 100dB (Zin=50ohms and Zout=1100ohms). After that we could gradually relax the filter requirements from 100dB down to 70dB, and see what bandwidth we will get using this current filter topology on the PCB.

Very rough estimate: 4th order is about 6*4=24dB/octave (at least asymtotically) => 100dB gives of cut-off of ~4 octaves below Nyqist then, or ~5 octaves below samling rate. When allowing significant fold-back across Nyquist, we gain almost one octave, leading to a low-pass cut-off of roughly 1M/2^5, i.e. ~30kHz (-> 60kHz RBW). In detail one would relly need to consider the actual filter design.

What is actually the oscillator frequency resolution (smallest frequency step)?

[gf]

I was playing with this LC-filter design tool to get some idea what kind of performance is possible with the current filter topology available on the PCB: https://rf-tools.com/lc-filter/

The parameters I have used are:
- Low-pass
- Chebyshev or Elliptical
- Shunt first
- Order 5
- Cut-off 120 kHz
- Pass-band ripple 0.1 dB
- Input and output impedance 50

The Chebyshev gives 80 dB attenuation at 630 kHz, and Elliptical gives 80 dB attenuation at 492 kHz. Since the system is using 1 MHz sample rate, and we are mostly interested in the frequency components below 120kHz, we can tolerate some aliasing, and the MCU can perform additional filtering for the sampled data in order to attenuate the frequency components that are not in our interest and which are aliased.

  80dB @500kHz should be well sufficient.

[Kalvin]

Let's think this backwards: What would be the maximum usable signal bandwidth at the output of a properly designed 4th/5th order low-pass filter when the sample rate is 1MHz, and the desired image rejection is 100dB (Zin=50ohms and Zout=1100ohms). After that we could gradually relax the filter requirements from 100dB down to 70dB, and see what bandwidth we will get using this current filter topology on the PCB.

Very rough estimate: 4th order is about 6*4=24dB/octave (at least asymtotically) => 100dB gives of cut-off of ~4 octaves below Nyqist then, or ~5 octaves below samling rate. When allowing significant fold-back across Nyquist, we gain almost one octave, leading to a low-pass cut-off of roughly 1M/2^5, i.e. ~30kHz (-> 60kHz RBW). In detail one would relly need to consider the actual filter design.

What is actually the oscillator frequency resolution (smallest frequency step)?

The topology on the PCB seems to be 5th order, and if we use 120kHz RBW as our goal, and modify the filter to become Elliptical (by adding capacitors in parallel with the two inductors), we might be able to get theoretically 90dB ... 100dB dynamic range (including the processing gain).

[Kalvin]

What is actually the oscillator frequency resolution (smallest frequency step)?

The LTDZ board info says that the minimum frequency step is 1kHz. I am not sure if the steps become larger at the higher frequencies, as I have not looked at the ADF4351 datasheet in detail yet. https://www.analog.com/media/en/technical-documentation/data-sheets/ADF4351.pdf

[radiolistener]

I'm impressed with your desire, because working with ADC @ 1 MS/s on STM32F103 is a real headache 

[Kalvin]

I'm impressed with your desire, because working with ADC @ 1 MS/s on STM32F103 is a real headache 

Do not scare me! The plan is *not* to do any real-time processing at 1 Ms/s rate, but to get only the ADC samples into RAM buffer using DMA at the given sample rate, and then process the samples from the RAM buffer without actual real-time requirements. After processing the buffer, the system will start capturing the next buffer etc.. Since the available RAM is less than 20KB, the sample buffer and processing buffers may not be very long, but for simple signal analysis it should be sufficient. Even 1024 point FFT should be quite possible without any problems.

[Kalvin]

It would great to have a small bootloader installed into STM32F103, so that the firmware could be updated over the USB serial interface without any special development tools after initial bootloader programming. Any suggestions?

[radiolistener]

using bootloader is very inconvenient and problematic way to deal with STM32F103. There is JTAG/SWD connector on LTDZ board, so you can upload and download firmware with no needs for any kind of booloader. Also you can do incircuit debugging with it.

All what you need is just ST-LINKv2: https://www.aliexpress.com/item/32887597480.html

Original ST-LINKv2 has SWO signal which can be useful to receive debugging messages into terminal, but unfortunately such pin is missing on Chinese ST-LINKv2 clone. It doesn't limit you with in circuit debugging features, but printing debug messages over SWD interface is too slow. It will be better to have dedicated terminal through SWO pin.

If  you can find ST-LINKv2 with SWO pin it will be better, but also you can modding Chinese ST-LINKv2 to get SWO and RST for STM32 by soldering some wires:
https://habr.com/ru/post/402927/

PS: some years ago I bought this ST-LINKv2 for 1 USD, now it cost 4-5 USD  This is probably due to logistics issues...

By the way, here is firmware from my LDTZ instance. LTDZ bought from different sellers have different firmware, and a little different results and bugs. I don't know if source code is available, I just read it from my LTDZ with ST-LINKv2. Also I found that firmware build from sources for NWT D6 device works ok with LDTZ.

[gf]

PS: some years ago I bought this ST-LINKv2 for 1 USD, now it cost 4-5 USD This is probably due to logistics issues...

This is just the indicator for the true inflation rate

[Kalvin]

Thanks for quick briefing for the tools, much appreciated! I do have bunch of those tools (probably got them also 1 USD each or something), but I have not been using them with STM32F103 yet.

The reason why I would like to have a bootloader is that after the actual firmware development is mostly completed, the enclosure will be closed, and the tools will be moved to some other project. In this situation it is much more convenient to use the bootloader to update the firmware than to open the enclosure once again, setup the development environment and close the enclosure. When I say a boot loader, I mean a custom bootloader that can receive a firmware [ASCII hex-file] over the serial port and Flash that into the memory. Thus, no need to install any special drivers other than the normal USB serial port driver, if needed.

My plan for the firmware development is as follows:

1. Create a simple STM32F103 firmware application which uses the serial port:
- command to to configure TX frequency of the ADF4351
- command to to configure RX frequency of the ADF4351
- command to start sampling the RX signal at 1Ms/s into RAM buffer
- command to dump the ADC sample buffer to PC over the serial port
- (command to update the firmware)
2. Do as much development on PC as possible, because it is much easier to do signal analysis and algorithm development on PC with proper tools.
3. After the algorithms are ready, implement the algorithms for firmware in C, and do the signal processing in STM32F103.
4. Create a simple GUI application for spectrum analysis (RX only).
5. Create a simple GUI application for filter and antenna measurement using the tracking generator.

I try to keep things as simple as possible, nothing fancy or bling-bling, only basic features. I will publish the project source code in Github after step 1, and keep it updated until step 5. After step 5 I may not be working actively on the project as it has served its purpose for me: The goal was to implement dynamically adjustable RBW for LTDZ platform, and as such make this inexpensive hardware more useful tool/instrument for amateur radio purposes. Since the source code will be freely available, someone interested in this project can fork my code, start adding the missing features and improve the GUI etc.

Edit: After step 1 and early step 2 it will be possible estimate how technically feasible this project will be. If it turns out that for some reason it is not possible to achieve the main goal of this little project (adjustable RBW with only minimum hardware changes), I will not proceed to steps 3 - 5. However, at this point everything looks pretty good, and no show stoppers are at sight.

[radiolistener]

here is some firmware source which I found. I don't remember if this source code works ok with LTDZ, but at least it compatible with LTDZ hardware and as I remember it even works. Just too lazy to flash it to test.

As I remember I found it on some site related to D6 device. This is not the same firmware as shipped with LTDZ, and it uses a little different device ID and use a little different protocol, but as I remember it works ok with LTDZ.
All things mixed together into single C file, but code is not too large so it is readable.

I didn't dig into source code deep, just tried to flash different versions and see how it works with LTDZ. As I remember, my NWTSHARP app implements both protocols, so it should works with original firmware and with this one. The difference with original is that one firmware requires to send sweep command with pause, otherwise it doesn't works properly. I don't remember exactly if it happens with original firmware or with this one.

[gf]

Good points. These devices do not have a quadrature mixer, so your observations are valid indeed. The DC-path from the mixer output to the input of the low-pass filter (120 kHz RBW-filter) is dc-blocked, so there will be a problem with spectral components near 0Hz. Since the tracking generator and the receiver mixer are fed with two separate, independent oscillators, it may be possible to offset the receiver LO a bit higher or lower frequency*, sample the output of the 120 kHz RBW-filter with the SMT32F103 ADC, and then perform a complex spectral shift down to 0Hz in order to get a proper analytical quadrature signal, which should resolve this 0Hz problem. Please correct me I have have overlooked something.

For virtually any {desired input frequency, LO frequency} pair there exist an image input frequency, so that both (desired and image) input frequencies are down-converted by the mixer to the same IF frequency. Once the IF contains contributions from both input frequencies, they can't be separated any more unambiguously. So no, a digital down-conversion step after samling the IF cannot fix it either. Traditional superheterodyne radios solve the problem by ensuring that the image input frequencies do not even enter the mixer (-> band pass filter, tuned to the desired input frequency range, excluding image frequencies). If this is not feasible, then an image reject mixer or quadrature mixer is required, in order to either reject the contribution of the image frequency, or to retain the contributions of both frequencies in a separable way.

The tinySA avoids these issues by using a full-featured tranceiver chip, which includes LNA, VCO, quadrature mixer, IF filter, 2x ADC, some DSP processing, and which outputs the down-converted and decimated digital signal to the MCU.

Are you rather after SA functionality, or Network Analyzer functionality? While a SA has to deal with arbitrary input spectra, a NA only needs to receive/detect only a single frequency at a time (namely the current frequency transmitted by its tracking generator). The latter makes life easier.

[Kalvin]

Good points. These devices do not have a quadrature mixer, so your observations are valid indeed. The DC-path from the mixer output to the input of the low-pass filter (120 kHz RBW-filter) is dc-blocked, so there will be a problem with spectral components near 0Hz. Since the tracking generator and the receiver mixer are fed with two separate, independent oscillators, it may be possible to offset the receiver LO a bit higher or lower frequency*, sample the output of the 120 kHz RBW-filter with the SMT32F103 ADC, and then perform a complex spectral shift down to 0Hz in order to get a proper analytical quadrature signal, which should resolve this 0Hz problem. Please correct me I have have overlooked something.

For virtually any {desired input frequency, LO frequency} pair there exist an image input frequency, so that both (desired and image) input frequencies are down-converted by the mixer to the same IF frequency. Once the IF contains contributions from both input frequencies, they can't be separated any more unambiguously. So no, a digital down-conversion step after samling the IF cannot fix it either. Traditional superheterodyne radios solve the problem by ensuring that the image input frequencies do not even enter the mixer (-> band pass filter, tuned to the desired input frequency range, excluding image frequencies). If this is not feasible, then an image reject mixer or quadrature mixer is required, in order to either reject the contribution of the image frequency, or to retain the contributions of both frequencies in a separable way.

The tinySA avoids these issues by using a full-featured tranceiver chip, which includes LNA, VCO, quadrature mixer, IF filter, 2x ADC, some DSP processing, and which outputs the down-converted and decimated digital signal to the MCU.

Yes, you are absolutely right that my idea of spectrum shift was mathematically flawed. That's why I was thinking would it be possible to remove the DC-blocking capacitor from the mixer output in order to improve low-frequency performance near DC. The output of the 120 kHz low-pass filter will then have a DC offset, but it can be removed after sampling the signal by computing the average of the buffered samples and then subtracting this average from the samples.

Are you rather after SA functionality, or Network Analyzer functionality? While a SA has to deal with arbitrary input spectra, a NA only needs to receive/detect only a single frequency at a time (namely the current frequency transmitted by its tracking generator). The latter makes life easier.

Scalar network analyzer with adjustable RBW is the major goal, so that it would be possible to use this simple, inexpensive and readily available hardware for measuring RF filters, use it as a simple antenna analyzer, and make it usable for other RF measurements for amateur radio purposes.

I know that there are different versions of very inexpensive nanoVNA vector network analyzers available, which are handy devices to be used for RF network measurements. The idea of this project is to find out whether the LTDZ could be improved and made more useful by only very minimal hardware modifications and some digital signal processing in the firmware.

Spectrum analyzer would a nice by-product for this project, but with the existing LTDZ's hardware design/architecture the spectrum analyzer will suffer the same problem that you addressed above ie. lack of image-rejection without proper filtering at the input. However, the device can still be used as a rudimentary spectrum analyzer as long as the user knows its limitations, and the image frequency components are well below the actual signal of interest. Of course, tinySA is definitely more suitable as a spectrum analyzer, but as far as I know, tinySA cannot be used as a network analyzer.

It would be possible to add a low-pass filter, an upconverter and a bandpass filter before the LTDZ input for a proper spectrum analyzer. The tracking generator could be used as LO for the upconverter, and the RX LO would mix the bandpass signal down to Zero-IF. I guess the blocks needed can be purchased from eBay or Aliexpress. But this would be a bit outside of the project's scope.

[Kalvin]

here is some firmware source which I found. I don't remember if this source code works ok with LTDZ, but at least it compatible with LTDZ hardware and as I remember it even works. Just too lazy to flash it to test.

As I remember I found it on some site related to D6 device. This is not the same firmware as shipped with LTDZ, and it uses a little different device ID and use a little different protocol, but as I remember it works ok with LTDZ.
All things mixed together into single C file, but code is not too large so it is readable.

I didn't dig into source code deep, just tried to flash different versions and see how it works with LTDZ. As I remember, my NWTSHARP app implements both protocols, so it should works with original firmware and with this one. The difference with original is that one firmware requires to send sweep command with pause, otherwise it doesn't works properly. I don't remember exactly if it happens with original firmware or with this one.

Thanks! I found the same firmware source code from Github. Great to hear that the code is working with the existing PC applications. The code is not too complex to be modified, which encouraged me to start thinking about modifying LTDZ if the first place.

The idea is to create a superset of commands so that the firmware will remain backwards compatible with the existing implementation as much as possible, but will add new commands for configuring the RBW, capturing the samples into sample buffer, reading the captured samples from the sample buffer etc.

[Bicurico]

That has been done before, for example by a user called "Domenico". Her provided me with a feature-increased firmware for the LTDZ, as well as, some modifications of the original hardware design (like the filter).

This is documented in my blog and the extra commands (which allow for example to control the output power of the signal generator in 4 steps) have been implemented in my "VMA Simple Spectrum Analyser" software.

However, in my opinion, it is a waste of time trying to modifiy and/or improve the LTDZ design. If you guys want to obtain a better/good design with variable RWB and what-not, I recommend to basically start from scratch and develop a proper design. You can (and probably should) use the same main components (ADF4351, M810, etc.). But the filter section needs to be redesigned and I would honestly break with backwards protocol compatibility and write a completely new and more efficient firmware.

The current one for example does not support the huge frequency range and considers frequencies to be 10x smaller (which is why you need the 10x multiplicator in WinNWT/LinNWT). Also, the protocol sends each sample twice, which is a waste of time. Also, there is no calibration and the 10 bits sent in two bytes do not represent a dBm value - the final sample power needs to be calculated from a weird formula, that I copied from the LinNWT source code. While my measurements where acceptably accurate (considering the price and quality of these devices), I do have my doubts if this couldn't be implemented in a better way.

Anyway, I am following this with interest and count on me to adjust my software for any new protocol that might be developed.

Also, as a note, I recently tested the Arinst SSA-R2 TG in my blog, which is a much nicer piece of equipment than the SMA/NWT/D6/LTDZ series of devices.

Regards,
Vitor

[radiolistener]

Also, there is no calibration and the 10 bits sent in two bytes do not represent a dBm value - the final sample power needs to be calculated from a weird formula, that I copied from the LinNWT source code.

This is very good, because device sends RAW data and allow to perform much better and more precise calibration on PC side where you can store megabytes of calibration data and perform different calibration calculations.

For example, NWTSHARP-LTDZ software that I posted above store measurement points during calibration process for every frequency point. And then, when you perform measurement it looking for nearest reference point for the measured frequency and use it to perform correction. As result I can get 0.003 dB resolution on NWT7 device with perfectly flat response (about ±0.2 dB across full frequency range).

Actually NWTSHARP-LTDZ store calibration file as an XML with raw adc codes in hex for every frequency point. There are two lists ref_zo and ref_xx for two attenuation levels used during calibration. The value of attenuation level is specified in value_zo and value_xx in dB. Calibration file name contains hardware device ID, so you can use it for two different NWT devices with different ID and calibration file will not be overwritten.

For comparison here is RAW ADC and calibration result on my NWT7 for entire frequency range from 1 kHz to 90 MHz (linear detector).
 

[Bicurico]

My software allows to generate a whole compensation map!

https://vma-satellite.blogspot.com/2019/08/error-mapping-smanwtd6ltdz-devices.html
https://vma-satellite.blogspot.com/2019/09/checking-out-performance-of-siglent.html

I understand what you mean with raw output. However, note that the same PCB could and should (and is) used by some MCU with TFT display for mobile use - for example this one: https://www.aliexpress.com/item/1005001762421373.html

It would, in my opinion, make sense that he firmware could output a sweep as a series of samples in dBm output for quick implementations.

A second, different, command should/could output the same data as raw data (without repetition).

Note that nothing prevents calibration/compensation of the streamed samples in correct dBm format.

My main observation is that in my opinion it does not make much sense trying to improve the performance of the LTDZ, as it was ENGINEERED to be AS CHEAP AS POSSIBLE. It is difficult to afterwards modify it for best performance. It is simpler and better to design a new (correct) layout. And once this is being done, one might as well improve the protocol.

Note that WinNWT/LinNWT was never meant to be open source or freeware! It was developed as a custom software for the original NWT-7, NWT-9, NWT-500 designs published by German magazin "AMATEURFUNK". When BG7TBL designed the original SMA/NWT4000 devices he just used this software, making use of the possibility to set the frequency multiplicator, thus supporting the much increased frequency range. However, the software was illegally shipped with the devices, without the consent of the author or AMATEURFUNK.

Worse than that, the author started to be contacted for support and complains of the not fully adequate software! This is why the relevant download site was taken down by the author.

If one is to develop a better than SMA/NWT/D6/LTDZ design, then one should design a brand new firmware with a new protocol.

Just my opinion, though.

Regards,
Vitor

[gf]

Yes, you are absolutely right that my idea of spectrum shift was mathematically flawed. That's why I was thinking would it be possible to remove the DC-blocking capacitor from the mixer output in order to improve low-frequency performance near DC. The output of the 120 kHz low-pass filter will then have a DC offset, but it can be removed after sampling the signal by computing the average of the buffered samples and then subtracting this average from the samples.

Spectrum analyzer would a nice by-product for this project, but with the existing LTDZ's hardware design/architecture the spectrum analyzer will suffer the same problem that you addressed above ie. lack of image-rejection without proper filtering at the input.

The existing design does actually not suffer from images. Since it is a direct conversion (zero-IF) receiver, all images which are mapped by the mixer into the RBW filters' passband happen to lie inside the passband anyway (just at a different position), therefore they are supposed to be included in the power calculation and don't need to be excluded. For measuring the power at the low-pass filter output, the design is OK in this regard.

What I wanted to say is that this IF signal is not suitable for sub-dividing the sampled RBW passband into smaller sub-bands, e.g. via FFT.
That's a different use case for which the images inside the RBW passband do matter.
[ Asking Google, I did find approaches here and here to detect potential images post fact in the spectrum and to cut them out. ]

The AC coupling unfortunately splits the frequency response of the RBW bandpass in the midde, so that it has two lobes eventually. It is however difficult to deal with DC offsets. At (say) 1V full scale, one could only tolerate 10µV offset error, if the aim is an error of -100dB. That's not trivial. Adding a capacitor is much easier.

Scalar network analyzer with adjustable RBW is the major goal, so that it would be possible to use this simple, inexpensive and readily available hardware for measuring RF filters, use it as a simple antenna analyzer, and make it usable for other RF measurements for amateur radio purposes.

Why adjustable bandwidth for NA? The basic functionality of a simple NA is that you send a single frequency (i.e. more ore less zero bandwidth) to the DUT, and you measure the signal coming out out the DUT, which is the same single frequency which you did send to the DUT (only amplitude and phase are different). Then you step to the next freyency and repeat the procedure again and again. Ideally, images should not be a problem either. Why should a potential image frequency come out from the DUT if you don't send it into the DUT? ADC aliasing is mitigated too - for the same reason - granted that the frequency plan is carefully chosen. Harmonics still need to be considered in the frequency plan, since not all DUTs are perfectly linear. And the NA's signal generator does not necessarily generate a pure sine wave either.

One thing which does indeed not fit well is the combination of the AC coupled filter with the zero-bandwidth CW signal and a zero-IF receiver. Since the filter has a null at DC it can't detect the zero bandwidth signal which is down-mixed to exactly DC. Therefore I'd rather pick a non-zero IF for the NA use case (i.e. add an offset to the receiver LO). NanoVNA V2.2 for instance uses 12kHz IF, which is sampled @300kSa/s by the MCU (it is also STM32F103 based, IIRC), and 25 50 samples (one two 12kHz periods) are correlated with a windowed complex sine wave ¹⁾ to obtain one measurement (btw, since the NanoVNA is a VNA, the ADC needs to run continuously in order not to lose phase in the pause when the receiver is switched to a different signal source).

EDIT: Regarding "it is also STM32F103 based, IIRC":
I checked again. The schematic mentions both, STM32F103C8 and GD32F303CCT6, but it is obviously the latter. Sorry for the mistake.

EDIT: ¹⁾ That's basically a Goertzel dedector, i.e. a DFT for a single frequency. The chosen window function eventually acts as bandpass filter.

EDIT: Regarding "Scalar network analyzer with adjustable RBW is the major goal":
If you intend anyway to sample the pre-filtered IF with the ADC, then why limit to "scalar", and not do "vector"?
External directional bridge and RF switches to switch the (single) receiver input between reference signal, refelected singal, DUT output,... are required for both.
As long as TX oscillator, RX oscillator and ADC clock are frequency-locked to the same xtal by rational factors, no frequency drift between them is to be expected either, and phase can be retained.
Eventually I don't see any difference in the required hardware then - it's just software. Do i miss anything?

[Kalvin]

Yes, you are absolutely right that my idea of spectrum shift was mathematically flawed. That's why I was thinking would it be possible to remove the DC-blocking capacitor from the mixer output in order to improve low-frequency performance near DC. The output of the 120 kHz low-pass filter will then have a DC offset, but it can be removed after sampling the signal by computing the average of the buffered samples and then subtracting this average from the samples.

Spectrum analyzer would a nice by-product for this project, but with the existing LTDZ's hardware design/architecture the spectrum analyzer will suffer the same problem that you addressed above ie. lack of image-rejection without proper filtering at the input.

The existing design does actually not suffer from images. Since it is a direct conversion (zero-IF) receiver, all images which are mapped by the mixer into the RBW filters' passband happen to lie inside the passband anyway (just at a different position), therefore they are supposed to be included in the power calculation and don't need to be excluded. For measuring the power at the low-pass filter output, the design is OK in this regard.

Thank you for correcting me again, very much appreciated. This is a subject that I know a little, but especially this zero-IF architecture with a scalar mixer is something that I really haven't had to think before at this detail. I will take this as an excellent learning experience/exercise. Quadrature zero-IF architecture (for example Tayloe mixer used in SDR transceivers) and quadrature mixers in general are somewhat more familiar.

What I wanted to say is that this IF signal is not suitable for sub-dividing the sampled RBW passband into smaller sub-bands, e.g. via FFT.
That's a different use case for which the images inside the RBW passband do matter.

Ok, this is something that was confusing me before. Let's see if I manage to grasp the idea better this time: Since this hardware architecture is essentially a direct conversion zero-IF receiver with a scalar mixer, the frequency components around the LO are mixed, folded and summed so that the resulting spectrum at the output of the mixer is centered symmetrically around DC so that the original "negative" frequency components and the "positive" frequency components relative to LO are now combined. If we now perform spectrum analysis for this signal at the mixer output, we really cannot tell whether the spectrum components we now see were originally below LO or above LO before mixing, thus we would see a distorted signal spectrum. However, the power of the signal can be computed correctly nevertheless. Please correct me if I have missed something.

[ Asking Google, I did find approaches here and here to detect potential images post fact in the spectrum and to cut them out. ]

Thank you for the links. Really clever ways for suppressing the images and spurs after the "spectral damage has already been done" in the mixer. I need more time to digest those papers.

The AC coupling unfortunately splits the frequency response of the RBW bandpass in the midde, so that it has two lobes eventually. It is however difficult to deal with DC offsets. At (say) 1V full scale, one could only tolerate 10µV offset error, if the aim is an error of -100dB. That's not trivial. Adding a capacitor is much easier.

The input of the ADC needs to be biased somewhere near VCC/2, so this bias voltage needs to be removed from the ADC samples anyway before actual power calculation. I was thinking that it would not really matter whether we get this bias voltage from the mixer output or from a resistor divider after a capacitor. If the ADC is coupled directly to the mixer output without a capacitor, there would not be a need for any ADC bias resistors. However, as a seconds thought, using capacitor coupling with the resistor divider, the bias voltage may remain more stable compared to mixer output, especially if the DC-voltage at the mixer output varies with the mixer input signal level.

Scalar network analyzer with adjustable RBW is the major goal, so that it would be possible to use this simple, inexpensive and readily available hardware for measuring RF filters, use it as a simple antenna analyzer, and make it usable for other RF measurements for amateur radio purposes.

Why adjustable bandwidth for NA? The basic functionality of a simple NA is that you send a single frequency (i.e. more ore less zero bandwidth) to the DUT, and you measure the signal coming out out the DUT, which is the same single frequency which you did send to the DUT (only amplitude and phase are different). Then you step to the next freyency and repeat the procedure again and again. Ideally, images should not be a problem either.

Yes, so true. The tracking generator is essentially injecting only a sinusoidal signal into DUT, so the DUT will also output a sinusoidal signal at the same frequency (with amplitude/phase change but this architecture will ignore the phase information). If the DUT is a non-linear circuit there will be other spectral components, though.

Why should a potential image frequency come out from the DUT if you don't send it into the DUT? ADC aliasing is mitigated too - for the same reason - granted that the frequency plan is carefully chosen. Harmonics still need to be considered in the frequency plan, since not all DUTs are perfectly linear. And the NA's signal generator does not necessarily generate a pure sine wave either.

Tracking generator (ADF4351) is generating output signal with some harmonics, but those harmonics should not cause problems in general.

One thing which does indeed not fit well is the combination of the AC coupled filter with the zero-bandwidth CW signal and a zero-IF receiver. Since the filter has a null at DC it can't detect the zero bandwidth signal which is down-mixed to exactly DC. Therefore I'd rather pick a non-zero IF for the NA use case (i.e. add an offset to the receiver LO). NanoVNA V2.2 for instance uses 12kHz IF, which is sampled @300kSa/s by the MCU, and 25 50 samples (one two 12kHz periods) are correlated with a windowed complex sine wave ¹⁾ to obtain one measurement (btw, since the NanoVNA is a VNA, the ADC needs to run continuously in order not to lose phase in the pause when the receiver is switched to a different signal source).

This LO frequency offset and Goertzel detector is quite simple to do, as you have described.

EDIT: Regarding "Scalar network analyzer with adjustable RBW is the major goal":
If you intend anyway to sample the pre-filtered IF with the ADC, then why limit to "scalar", and not do "vector"?
External directional bridge and RF switches to switch the (single) receiver input between reference signal, refelected singal, DUT output,... are required for both.
As long as TX oscillator, RX oscillator and ADC clock are frequency-locked to the same xtal by rational factors, no frequency drift between them is to be expected either, and phase can be retained.
Eventually I don't see any difference in the required hardware then - it's just software. Do i miss anything?

The ADC is sampling low-pass filtered zero-IF baseband (ie. sampling the signal at the output of the 120 kHz RBW filter connected to the output of the scalar mixer). Too many hardware modifications are a bit outside of the original project scope, although it is very hard to resist feature creep. However, if this hardware would be better, those extra features might be more attractive to be implemented. Now the intention is just to improve the existing design with only few hardware changes.

[gf]

Ok, this is something that was confusing me before. Let's see if I manage to grasp the idea better this time: Since this hardware architecture is essentially a direct conversion zero-IF receiver with a scalar mixer, the frequency components around the LO are mixed, folded and summed so that the resulting spectrum at the output of the mixer is centered symmetrically around DC so that the original "negative" frequency components and the "positive" frequency components relative to LO are now combined. If we now perform spectrum analysis for this signal at the mixer output, we really cannot tell whether the spectrum components we now see were originally below LO or above LO before mixing, thus we would see a distorted signal spectrum. However, the power of the signal can be computed correctly nevertheless. Please correct me if I have missed something.

Exactly
Regarding "we really cannot tell whether the spectrum components we now see were originally below LO or above LO before mixing":
You only see the sum of the original upper sideband and the mirrored lower sideband. The negative frequencies in the IF spectrum are just a conjucate complex mirror of the positive IF frequencies, which applies to the spectrum of any real-valued signal.

Yes, so true. The tracking generator is essentially injecting only a sinusoidal signal into DUT, so the DUT will also output a sinusoidal signal at the same frequency (with amplitude/phase change but this architecture will ignore the phase information). If the DUT is a non-linear circuit there will be other spectral components, though.

If you only feed only a single frequency into the DUT, then the "other spectral components" are supposed to be only harmonics (unless the DUT would contain oscillators, mixers, etc. so that IM products can arise).

The ADC is sampling low-pass filtered zero-IF baseband (ie. sampling the signal at the output of the 120 kHz RBW filter connected to the output of the scalar mixer).

Sure. But for the NA use case you are still free to choose a non-zero IF (by adding an offset to the receiver's LO frequency). You just need to ensure that the chosen IF frequency is inside the passband of the filter (zero is actually not inside the passband, due to AC coupling).

Too many hardware modifications are a bit outside of the original project scope, although it is very hard to resist feature creep. However, if this hardware would be better, those extra features might be more attractive to be implemented. Now the intention is just to improve the existing design with only few hardware changes.

Which hardware changes in particular (besides IF being sampled by the ADC) were required for a VNA, but not for a scalar NA? My understanding of a scalar NA architecture is still something like figure 1 in this document. In your case, tracking generator and receiver/detector are built-in, so you don't need them externally. But since you have only a single receiver, there is still the need to multiplex the three inputs with external RF switches. This applies to both, scalar and vector NA. Directional bridge needs to be external too, since you don't have a built-in bridge.

[ If I consider the additinally required external coponents, I wonder, whether buying a NanoVNA V2 for 50 bucks isn't the cheaper solution. ]

[Kalvin]

Which hardware changes in particular (besides IF being sampled by the ADC) were required for a VNA, but not for a scalar NA? My understanding of a scalar NA architecture is still something like figure 1 in this document. In your case, tracking generator and receiver/detector are built-in, so you don't need them externally. But since you have only a single receiver, there is still the need to multiplex the three inputs with external RF switches. This applies to both, scalar and vector NA. Directional bridge needs to be external too, since you don't have a built-in bridge.

[ If I consider the additinally required external coponents, I wonder, whether buying a NanoVNA V2 for 50 bucks isn't the cheaper solution. ]

You are right that any modern NA has an architecture that is shown in figure 1. However, this hardware with an external, inexpensive SWR/reflection bridge can be used as a simple, manually operated NA already. This requires extra work from the user as the user needs to perform required measurement steps manually. It is very trivial to add a new command that can be used to set/clear some GPIO pins which can be used for relay control, and the PC software can then perform the necessary steps required by a modern NA. I just want to avoid feature creep and want to proceed one step at a time. This is my first STM32F103 project, so this is quite challenging project already as there are lots of details to learn. Luckily the original firmware C source code is quite easy to understand, and the required modifications are straightforward to implement. In order to make this project successful, it is better to keep it simple.

I would not consider this project as a replacement for any existing instrument on the market (tinySA, nanoVNA etc.). The sole purpose of this project is just to improve the existing hardware so that it will become more usable for common amateur radio measurements (adjustable RBW for improved spectral resolution, shifted IF for NA measurements).

[Kalvin]

If you intend anyway to sample the pre-filtered IF with the ADC, then why limit to "scalar", and not do "vector"?
External directional bridge and RF switches to switch the (single) receiver input between reference signal, refelected singal, DUT output,... are required for both.
As long as TX oscillator, RX oscillator and ADC clock are frequency-locked to the same xtal by rational factors, no frequency drift between them is to be expected either, and phase can be retained.
Eventually I don't see any difference in the required hardware then - it's just software. Do i miss anything?

After some thinking, this is not extraordinary difficult to implement, given that there are external relays and directional coupler/reflection bridge available, and the two ADF4351s can maintain sufficiently stable frequency. Since the tracking generator is sending a sinusoidal signal, the RX can use whatever shifted IF and ADC sample clock that is suitable. For example the RX could use 12kHz IF offset, thus the ADC sample-clock needs to be at least 24kHz, preferably more like 4x, which makes signal detector quite simple. Since the ADC sample-rate can now be reduced from 1Ms/s down to 48Ks/s, ADC can be kept running continuously between different measurements, STM32F103 can process the samples in real-time, and the phase information between tracking generator and the LO can be preserved -> it is quite possible to implement a VNA with this hardware, external directional coupler/reflection bridge and external signal relays.

On the other hand, using this hardware as a spectrum analyzer with digital, adjustable RBW, the ADC sample rate needs to be high enough so that the complete 120kHz band of the analog RBW/low-pass filter can be sampled without aliasing, probably up to 1Ms/s. In this mode the signal can be sampled into a RAM buffer using ADC+DMA, and then processes the RAM buffer without actual real-time requirements.

[Kalvin]

About an firmware bootloader: It is most useful if the onboard application firmware could be updated over the USB port without opening the enclosure. The device firmware contains two parts: 1) the firmware bootloader and 2) actual application part of the firmware. The firmware bootloader needs to be installed only once using the JTAG/SWD interface/tools, and after that the application part of the firmware can be updated over the USB port without any tools.

The firmware bootloader resides in the Flash memory page(s) that won't be erased during firmware update process. After each device reset the program starts executing from the firmware bootloader, which will compute and validate the checksum of application part of the Flash memory. If the checksum is valid, the execution continues from the application run-time initialization which eventually calls the application's main() function (this needs some linker magic). If the firmware application checksum is invalid, the bootloader will take over the USB-serial port and open a command interface for updating the application firmware.

The application firmware needs to have a command for erasing/invalidating the application part of the firmware itself. After erasing/invalidating the application firmware, the device will reset itself, and the firmware bootloader will notice that the application part of the firmware needs to be updated, and will open the command interface for updating the application firmware.

Pretty standard stuff.

Does someone know a pointer for a small & simple STM32F103 bootloader which could be used as a basis for this kind of bootloader?

[radiolistener]

I was thinking would it be possible to remove the DC-blocking capacitor from the mixer output in order to improve low-frequency performance near DC. The output of the 120 kHz low-pass filter will then have a DC offset, but it can be removed after sampling the signal by computing the average of the buffered samples and then subtracting this average from the samples.

The problem of DC bias is that it add offset for the ADC input. ADC max/min limits are fixed and adding offset on the input leads to remove dynamic range. If ADC is overloaded you can't fix it by remove DC bias from ADC output because it don't allows you to restore dynamic range.

Using IIR filter on DSP side to remove DC offset is useful for analog quadrature detector, which may have a slightly phase shift difference between I and Q branches. LTDZ don't use quadrature detector, it use usual mixer which folds negative and positive frequencies.

So there is no advantage of removing DC block capacitor, much better solution is to use some IF (measure magnitude at some frequency shift) to keep DC frequency in outside of your digital filter.

About an firmware bootloader: It is most useful if the onboard application firmware could be updated over the USB port without opening the enclosure.

The bootloader is useful for mass production, it allows to deliver firmware to millions users who don't have knowledge about MCU and don't have JTAG/SWD adapter.

If you're want to develop and debug your own firmware for your own usage, then bootloader is complete useless for you. You're needs JTAG/SWD which allows you to upload firmware directly and use in circuit debugging (you can run your firmware step-by-step in the source code on the live hardware).

[Kalvin]

About an firmware bootloader: It is most useful if the onboard application firmware could be updated over the USB port without opening the enclosure.

The bootloader is useful for mass production, it allows to deliver firmware to millions users who don't have knowledge about MCU and don't have JTAG/SWD adapter.

If you're want to develop and debug your own firmware for your own usage, then bootloader is complete useless for you. You're needs JTAG/SWD which allows you to upload firmware directly and use in circuit debugging (you can run your firmware step-by-step in the source code on the live hardware).

Like I wrote above, I do not want to open the enclosure if I just want to update the firmware. It has been quite possible to develop the firmware without JTAG/SWD-tools and the ability to single-step the code. Those tools are handy at the initial phase of the project, but after a while those tools are not that important anymore.

[radiolistener]

Like I wrote above, I do not want to open the enclosure if I just want to update the firmware.

LTDZ don't have enclosure. Without JTAG/SWD you will be unable to debug your code, so it is must have if you want to develop your own firmware.

The bootloader is only useful to upload already existing firmware (designed, compiled and tested) for these users who don't have JTAG/SWD adapter. It is not intended for firmware development.

The bootloader is the same as installer engine on windows. It allows to install already existing and prepared software for end-users. But it is complete useless if you develop and debug your own software.

[radiolistener]

It has been quite possible to develop the firmware without JTAG/SWD-tools and the ability to single-step the code. Those tools are handy at the initial phase of the project, but after a while those tools are not that important anymore.

No. This is impossible. You can compile code and upload it to device through the bootloader, but you won't be able to debug your code on the live hardware.

Another issue is that upload firmware through the bootloader is more complicated than through JTAG/SWD. The bootloader needs to get your hands on device to enable bootloader mode with solder some jumper on PCB or by selecting appropriate menu on the device firmware (if it has such ability). You will need to damage your LTDZ PCB (cut some wires and solder jumpers) in order to enter into bootloader mode. Sometimes the bootloader mode is just unavailable. JTAG/SWD don't needs any action before upload firmware, it just uploaded by your IDE directly just after compilation and with no needs to do something.

STM32F103 has the bootloader in OTP memory, in order to enter into bootloader mode you're needs to solder jumpers for specific pins. As I know these jumpers don't available on LTDZ PCB. And I'm not sure if serial port for the bootloader is the same as used for PC communication on LTDZ device.

Just believe me, you will get a lot of trouble in order to use bootloader for firmware development. It will be much cheaper to buy JTAG/SWD adapter like ST-LINKv2 than trying to use the bootloader.

If you're not experienced with STM32 programming and debugging, I think using the bootloader from OTP memory for STM32F103C8T6 will be almost impossible for you with LTDZ, which PCB don't have switch to enter into the bootloader.

[Kalvin]

We are probably talking about two different bootloaders here. Yes, there is an on-chip bootloader in STM32F103, created by the chip manufacturer, which can be activated by a certain GPIO-pin and device reset. There is even an application note for that: AN3155 USART protocol used in the STM32 bootloader. It seems that you are referring to this factory programmed bootloader here.

What I am talking about is a bootloader which is a part of the firmware and resides in the on-chip Flash memory (the user erasable Flash memory). This bootloader is customized for updating the application firmware in the Flash memory using the standard USB-UART-serial port. The on-chip Flash memory is partitioned into two parts: Few Flash pages are reserved for the bootloader itself, and the rest of the Flash memory pages are used for the application part of the firmware. No need to use jumpers any more for updating the firmware.

[Kalvin]

It has been quite possible to develop the firmware without JTAG/SWD-tools and the ability to single-step the code. Those tools are handy at the initial phase of the project, but after a while those tools are not that important anymore.

No. This is impossible. You can compile code and upload it to device through the bootloader, but you won't be able to debug your code on the live hardware.

It has been possible to develop firmware successfully with only a serial port available for printf(), and LTDZ is no exception. Only very seldom, as the last resort, there has been a real need to use JTAG/SWD-tools for debugging. Single-stepping a real-time system is quite often useless or even impossible, as the word "real-time" will suggest.

[radiolistener]

What I am talking about is a bootloader which is a part of the firmware and resides in the on-chip Flash memory (the user erasable Flash memory).

What is the reason for such bootloader? It still needs JTAG/SWD to program it and for debugging. It consume a lot of memory. It requeres some jumper or communication protocol command to enter into bootloader, it needs to develop firmware uploader tool etc. Only headache and no any advantage.

This bootloader is customized for updating the application firmware in the Flash memory using the standard USB-UART-serial port.

You can't use existing USB-UART for printf, because it already used for communication protocol.

This bootloader is customized for updating  The on-chip Flash memory is partitioned into two parts: Few Flash pages are reserved for the bootloader itself, and the rest of the Flash memory pages are used for the application part of the firmware. No need to use jumpers any more for updating the firmware.

And it still needs JTAG/SWD to upload such bootloader and if something going wrong and bootloader page will be erased, you will need JTAG/SWD to upload it again.

When you try to use JTAG/SWD, you will realize that bootloaders is a nightmare, which consume MCU memory, needs special software and additional actions and  just provide you with a very basic functionality for upload firmware and nothing else.

It's complete useless, I'm implemented user bootloader and know what I'm talking about. 

Using the bootloader for firmware development is like using Update Service in the Windows for software development. It's complete useless for such purposes.

[radiolistener]

It has been possible to develop firmware successfully with only a serial port available for printf(), and LTDZ is no exception.

Yes, it is possible to develop firmware without debugging... if you're a guru in STM32 development and have a long experience in using it with ADF4351 and other peripherals.

But if you're a guru, you're already understand why JTAG/SWD is must have why it simplify your life.

The same as a guru can write software with no using IDE. He can, but he don't want do it in such way. If you're novice it will be just crazy to start programming in such way.

Only very seldom, as the last resort, there has been a real need to use JTAG/SWD-tools for debugging. Single-stepping a real-time system is quite often useless or even impossible, as the word "real-time" will suggest.

No. If you're working on firmware for MCU, you will needs JTAG/SWD for incircuit debugging. Otherwise you will just don't know what is going on in the MCU if it don't works as expected. For example, you upload firmware but it don't works and don't printf debug message on serial port for unknown reason. Without JTAG it's just a lot of headache to understand why it don't works as expected and what is happened.

JTAG/SWT is used 100% time for firmware development and debugging. The bootloader is useless for such purpose. For example, I never used bootloader for debugging and don't know who is using it for that.

Anyway, in order to develop and debug your bootloader you will needs JTAG/SWD.

The bootloader for firmware development is just like a fifth leg for a dog.

I'm sure for almost 100% that you will not implement firmware if you start to fight with bootloader problems instead of firmware debugging with JTAG/SWD adapter.

[Kalvin]

Let's say I have a friend who also has this LTDZ hardware. So, my friend brings it to me, I open the enclosure, flash the firmware once using the JTAG/SWD-tools, and close the enclosure. Now, when there is a new version of firmware available, I can email the firmware to him (or my he can download it from Github), and he is able to update his LTDZ without opening the enclosure or without any special development tools - just like magic. All he needs is a PC software supporting the new protocol and a USB-cable.

Same thing even if I do it for me: If I just make a simple fix in the firmware, I prefer to do it using this bootloader instead of doing the whole chore of opening the enclosure, flashing the firmware using the tools, and finally closing the enclosure again.

[radiolistener]

Let's say I have a friend who also has this LTDZ hardware. So, my friend brings it to me, I open the enclosure, flash the firmware once using the JTAG/SWD-tools, and close the enclosure. Now, when there is a new version of firmware available, I can email the firmware to him

I suggest to write firmware first and then thinking on how to delivery it to the end-user.

PS: Self updating firmware is dangerous for user who don't have JTAG/SWD, because it may fail and brick the device. And he will be unable to restore it, because he don't have JTAG/SWD.

The user with bootloader can brick his device and it will be like magic. No needs to open enclosure, no needs for jumpers, just brick device through USB. I had such experience when debug bootloader.

Another issue is that you will need to edit memory map in the linker file. It will be complicated for novice.

[Kalvin]

PS: Self updating firmware is dangerous for user who don't have JTAG/SWD, because it may fail and brick the device. And he will be unable to restore it, because he don't have JTAG/SWD.

What is so special about this that makes it dangerous? What are the places in the update process where the system can brick itself during the firmware update? Is there a way that can prevent this from happening, I mean is there a way that the firmware could be updated in a safe manner without bricking the device?

[radiolistener]

What are the places in the update process where the system can brick itself during the firmware update?

if you write your own bootloader, it is placed in erasable memory page, so it can be easily erased due to some random error or mistake in the firmware. Or just random power off during update. If bootloader is erased, you will not be able to use it anymore. You will needs JTAG/SWD to upload it again.

The bootloader placed in OTP memory cannot be erased, because it is placed in one time programmable memory. But it requires special jumpers configuration at MCU reset in order to enter into bootloader mode. As I mentioned before, you will need to desolder/solder components, cut wires or add jumpers on LTDZ PCB in order to enter into bootloader mode. And after update it will needs to restore original configuration in order to allow to run firmware instead of bootloader.

Is there a way that can prevent this from happening, I mean is there a way that the firmware could be updated in a safe manner without bricking the device?

Yes, do it through JTAG/SWD adapter. If something bad happens you always can upload firmware again. 

[Kalvin]

Another issue is that you will need to edit memory map in the linker file. It will be complicated for novice.

True. Although I have not worked with STM32F103 before, it should not be too different from other Cortex-M3 processors.

[radiolistener]

NanoVNA uses a hack to use OTP bootloader from user code. It jumps into OTP firmware code from user menu on device.

Since bootloader needs the same MCU state as after hardware reset, it use a trick. It write special value into RAM variable with fixed address and then perform MCU reset. After reset it check RAM variable and if there is request for bootloader it jumps into OTP bootloader. It allows to use DfuSeDemo tool to upload firmware with no needs to switch BOOT jumper.

But I'm not sure that this way will works for STM32F103C8T6. As I know some part of memory are write protected from bootloader in STM32F103C8T6, so it needs very special configuration for the code to compile into specific pages.

Using STM32F103C8T6 OTP bootloader may needs to buy external USB-serial and solder wires directly to the chip pins. But I'm not sure, because I never used bootloader on STM32F103C8T6.

For JTAG/SWD no needs to change something, JTAG/SWD connector is already present on LTDZ PCB.

I think it will be more easy for your friend to buy ST-LINKv2 than modify LTDZ PCB by adding hardware switch for bootloader mode and by adding external USB-serial adapter, which may cost more than ST-LINKv2.

[gf]

After some thinking, this is not extraordinary difficult to implement, given that there are external relays and directional coupler/reflection bridge available, and the two ADF4351s can maintain sufficiently stable frequency.

It's a PLL, so once it is locked, the frequency ratio with regard to the xtal will be perfectly stabe. However, you need to give it enough time to settle after changing the frequency. The remaning "instabilities" are rather phase noise, eventually. The ADF4351 isn't that bad, though.

Since the tracking generator is sending a sinusoidal signal,

According to the datasheet (see attachment) It is not. So be prepared for mixing products having frequencies n * f_IF (where n is an integer). I do not see a problem, though, to get rid of the IF harmonics by filtering. For instance, the frequency response of a one-period rectangular window (simple moving average filter) has zeros at all harmonics. If multiple periods are collect, then even more sophisticated window/filter can be used.

the RX can use whatever shifted IF and ADC sample clock that is suitable. For example the RX could use 12kHz IF offset, thus the ADC sample-clock needs to be at least 24kHz, preferably more like 4x, which makes signal detector quite simple. Since the ADC sample-rate can now be reduced from 1Ms/s down to 48Ks/s, ADC can be kept running continuously between different measurements, STM32F103 can process the samples in real-time, and the phase information between tracking generator and the LO can be preserved -> it is quite possible to implement a VNA with this hardware, external directional coupler/reflection bridge and external signal relays.

Likely you still want to run the ADC as fast as possible, in order that you can collect enough samples for suffient processing gain, w/o slowing down the measurements too much.

I also tried to find ADC characteristics for the STM32, like SINAD, SFDR, INL, DNL, etc. But I did not find any numbers
Strange. For other serious ADCs I'm used to find these numbers and corresponding figures in the datasheets.

Btw, I also wonder, which of the STM32F103 models is it? And what's the clock speed?
is there a schematic of LTDZ available somewhere for download? My search was not successful.

I'm impressed with your desire, because working with ADC @ 1 MS/s on STM32F103 is a real headache 

Can you elaborate where exactly you did experience problems? One interrupt per sample is of course a no-go. But isn't it possible to setup free-running ADC with DMA (possibly with double-buffering if limited real-time processing were desired), steaming the converted data at the maximum sampling rate to memory w/o CPU intervention?

[gf]

After some thinking, this is not extraordinary difficult to implement, given that there are external relays and directional coupler/reflection bridge available, and the two ADF4351s can maintain sufficiently stable frequency.

It's a PLL, so once it is locked, the frequency ratio with regard to the xtal will be perfectly stabe. However, you need to give it enough time to settle after changing the frequency. The remaning "instabilities" are rather phase noise, eventually. The ADF4351 isn't that bad, though.

I just did take a closer look at one of the photos of the PCB. Are the ADF4350 clocked from a different xtal than the CPU/ADC? This would complicate things, since the frequency lock is not granted.

[radiolistener]

Since the tracking generator is sending a sinusoidal signal,

According to the datasheet (see attachment) It is not. So be prepared for mixing products having frequencies n * f_IF (where n is an integer).

yes, ADF4351 has nice square wave with rise time about 70 ps.
See picture with 40 MHz LTDZ output on 100 MHz oscilloscope.

Btw, I also wonder, which of the STM32F103 models is it? And what's the clock speed?
is there a schematic of LTDZ available somewhere for download? My search was not successful.

my LTDZ has STM32F103C8T6, it is clocked from 8 MHz oscillator. ADF4351 is clocked from separate 25 MHz oscillator.

I also didn't find schematic, but it looks that schematic is the same as D6 device, see attachment. At least LTDZ use the same components and firmware from D6 is compatible with LTDZ.

There is difference between this schematic and LTDZ on RF input, on LTDZ R16 = 51 Ω, but on schematic it R16 = 510 Ω. Looks just like mistake of manufacturer, because this resistor is marked as 510R on schematic, and they soldered SMD resistor with marking 510 

After some thinking, this is not extraordinary difficult to implement, given that there are external relays and directional coupler/reflection bridge available, and the two ADF4351s can maintain sufficiently stable frequency.

My LTDZ output frequency is not sufficiently stable. It has frequency drift, probably due to cheap 25 MHz oscillator. See picture with 1.5 GHz output. But I'm not sure, about the source of this drift, may be part of this drift is related with RTLSDRv3 oscillator. I'm using original RTLSDRv3 bought in official store.

I'm not sure if such drift is normal at GHz frequency, because I don't have experience with GHz band. I bought LTDZ as a startup generator for GHz band frequency. For short wave point of view such drift is awful.

[gf]

What units is the horizontal axis of the waterfall? Hz, kHZ, MHz? (I mean the zoomed one)
RTL SDR V3 is supposed to have a 1ppm TCXO (max. 1kHz drift @1GHz)

EDIT: If the scale is Hz, i.e. drift by only ~100Hz, then I'd say it is well within specs.

but on schematic it R16 = 510 Ω.

I guess the designation "510R" is possibly supposed to mean 51*10⁰ 

[radiolistener]

What units is the horizontal axis of the waterfall? Hz, kHZ, MHz? (I mean the zoomed one)

Bottom waterfall units is Hz.
Top waterfall units is kHz.

When device is cold started (first 15 minutes), the drift is much worse. Even automatic frequency control cannot keep carrier within bandwidth, because carrier moving too fast. After 20 minutes board temperature is stabilized, but the drift still noticeable, when using SSB demodulation the tone shift is very significant.

For example here is 1.5 GHz drift after 10 minutes. May be it worth to replace 25 MHz oscillator with better one.

[gf]

If you need significantly better than 1ppm, then TCXO won't suffice any more and you rather need an OXCO.
Is the oscillator of the LTDZ actually a TCXO, or just a normal quartz oscillator w/o temperature compensation?

[radiolistener]

I guess the designation "510R" is possibly supposed to mean 51*10⁰ 

I don't think so. Because according to the datasheet for IAM81008 mixer it's RF input VSWR about 1.4.

I suppose that IAM81008 input impedance is about 70 Ω. With 51 Ω resistor in parallel it give about 29 Ω impedance.

And this well corresponds with actual LTDZ input impedance measured as 30 Ω, see picture.

I also read some blog about LTDZ modification which recommends to remove 51 Ω and as result it leads to input impedance more close to 50 Ω. With such mod the input is more sensitive, which is good for spectrum analyzer mode. But also LTDZ output will be too strong and direct connection RF-IN to RF-OUT will leads to overload, so there is needs to use 10 dB attenuator.

If you need significantly better than 1ppm, then TCXO won't suffice any more and you rather need an OXCO.
Is the oscillator of the LTDZ actually a TCXO, or just a normal quartz oscillator w/o temperature compensation?

According to the marking this is VCC1 (Ultra Low Jitter, Fundamental or 3rd OT Crystal Design with 20ppm temperature stability). But who knows what exactly is installed on Chinese factory...

[gf]

20ppm is not very good, though. As already mentioned, RTL SDR V3 is supposed to have a 1ppm TCXO (i.e. 20x better), and an OCXO (crystal oven) can have a stability of even 10ppb. But I guess price of a good OCXO alone is likely higher than the price of the whole LTDZ board.

[Kalvin]

Here are some pictures I found.

PCB:


Schematics:


They are from two different sites, but I believe they are both for the same LTDZ hardware.

The two ADF4351s are driven from the same 25 MHz oscillator, which is good news as they will be synced together and they will drift alike. It should not be too difficult to replace the existing oscillator with a better one.The MCU is driven by a 12 MHz its internal 8MHz oscillator. The MCU is either genuine STM32F103 or a clone. I do not have any detailed information about the ADC, but let's not hope too much from it - It is whatever it is.

Since the MCU is not using USB for the serial port (there is a separate USB-to-SERIAL-controller CH340G), it might even be possible to clock the STM32F103 from the same 25 MHz oscillator. That would make things a bit easier when implementing the PLL as all blocks are driven from the same clock. Just checked: The STM32F103 datasheet specifies that the maximum external clock frequency is 25MHz. Adding any extra wiring carelessly to the existing clock network may cause serious problems to the clock signal quality, however.

Edit: Corrected the MCU oscillator.

[radiolistener]

20ppm is not very good, though.

yeah, but at least it is declared as ultra low jitter.

The MCU is driven by a 12 MHz oscillator.

no, MCU is clocked from it's own 8 MHz oscillator (see below the STM32 chip).

12 MHz oscillator is used for CH340 USB-USART converter chip.

Since the MCU is not using USB for the serial port (there is a separate USB-to-SERIAL-controller CH340G), it might even be possible to clock the STM32F103 from the same 25 MHz oscillator.

This is a bad idea, the clock ADF4351 will be more noisy due to interference from MCU. There is no sense to clock MCU from the same clock as ADF4351.

[gf]

There is no sense to clock MCU from the same clock as ADF4351.

For non-simultaneous phase measurements, it does make sense to have ADC clock, transmitter clock and receiver LO clock frequency locked to each other. NanoVNA uses a single clock source either, obviously for the same reason.

If this is not realizable, then a kind of software PLL were required to compensate the drift between ADC clock and ADF4351 clock.

[Kalvin]

The MCU is driven by a 12 MHz oscillator.

no, MCU is clocked from it's own 8 MHz oscillator (see below the STM32 chip).

Thanks for spotting my error.  I fixed that in my post.

[Kalvin]

After some thinking, this is not extraordinary difficult to implement, given that there are external relays and directional coupler/reflection bridge available, and the two ADF4351s can maintain sufficiently stable frequency.

It's a PLL, so once it is locked, the frequency ratio with regard to the xtal will be perfectly stabe. However, you need to give it enough time to settle after changing the frequency. The remaning "instabilities" are rather phase noise, eventually. The ADF4351 isn't that bad, though.

Since the tracking generator is sending a sinusoidal signal,

According to the datasheet (see attachment) It is not. So be prepared for mixing products having frequencies n * f_IF (where n is an integer). I do not see a problem, though, to get rid of the IF harmonics by filtering. For instance, the frequency response of a one-period rectangular window (simple moving average filter) has zeros at all harmonics. If multiple periods are collect, then even more sophisticated window/filter can be used.

the RX can use whatever shifted IF and ADC sample clock that is suitable. For example the RX could use 12kHz IF offset, thus the ADC sample-clock needs to be at least 24kHz, preferably more like 4x, which makes signal detector quite simple. Since the ADC sample-rate can now be reduced from 1Ms/s down to 48Ks/s, ADC can be kept running continuously between different measurements, STM32F103 can process the samples in real-time, and the phase information between tracking generator and the LO can be preserved -> it is quite possible to implement a VNA with this hardware, external directional coupler/reflection bridge and external signal relays.

Likely you still want to run the ADC as fast as possible, in order that you can collect enough samples for suffient processing gain, w/o slowing down the measurements too much.

Instead of implementing a digital PLL, I was thinking a bit simpler approach: If the system is using external relays, and the relays are controlled by MCU, the MCU could compute the phase difference as follows:

1. The relays are set so that the signal from the tracking generator is fed into RX, and the MCU is sampling the signal from the tracking generator. This signal from the tracking generator is the reference signal with the reference phase.
2. The relays are then switched to another position (while the ADC is still continuously sampling the RX signal), so that the ADC will now sample the signal we want to measure.

If the sample rate is low enough so that it can contain these two signals in one buffer, it is possible to compute the phase difference between these two signals. Granted, at very low signal levels the accuracy will suffer. It is also possible to use two separate buffers. Anyway, this was my thinking for using the slower sample rate. Frankly, I have not given too much thought to this yet, as this is kind of advanced feature.

I am waiting for my LTDZ to arrive. Meanwhile I am setting up the development environment for the project and studying the original source code. Yesterday I spent some hours banging my head to table while trying to get my st-link V2 dongles from eBay to co-operate with the bluepill boards I have. It was found out that the st-link v2 dongles work ok, but the markings on their enclosure are totally wrong. After finally opening one dongle, the PCB had the correct signal names printed on it, and voilá, it was possible to Flash the Blinky into the bluepill boards. It was also found that I have two different kind of bluepill boards: some boards with STM32F103 and some boards with something different. I could not get those other bluepill boards flashed with the tools that I had installed, but I guess would be possible. Yes, I was using Arduino IDE to test the dongles and the boards, but just wanted to see that the dongles work.

[radiolistener]

if you want vector network analyzer, it's better to buy nanoVNA or nanoVNA2. LTDZ is a scalar analyzer and it don't worth to modify it, because it will cost you more than nanoVNA and as result you will get some kind of Frankenstein instead of device.


I bought LTDZ just as a signal generator from 35 MHz to 4.4 GHz.
RF input with IAM81008 mixer, second ADF4351 and logarithmic detector are just a nice addition.

Unfortunately output level is too low for my needs, so I'm using LNA module as an amplifier.

[Bicurico]

Unfortunately output level is too low for my needs, so I'm using LNA module as an amplifier.

The FW of the LTDZ and D6 devices can be modified in order to select at least 4 different RF output levels.

This is what Domenico did to the FW, which is available on my blog. It does not provide means for higher output level, but you can attenuate it, which could be nice if you are addinf an LNA module.

Regards,
Vitor

[Kalvin]

if you want vector network analyzer, it's better to buy nanoVNA or nanoVNA2. LTDZ is a scalar analyzer and it don't worth to modify it, because it will cost you more than nanoVNA and as result you will get some kind of Frankenstein instead of device.

I have nanoVNA already. The point here is to modify this LTDZ a little (like fixing the low-pass filter's component values for better passband and stopband performance, and using the ADC as a power meter instead of the on-board AD8307). With a little help from DSP the performance of otherwise limited hardware design can be improved. This can be taken as a learning experience, too: This is my first STM32F103-based project. Although the board has its problems, the combination of the tracking generator, LO+mixer and STM32F103 provides useful platform to do some experimenting as well. It is interesting to see how far this limited hardware can be pushed.

[radiolistener]

and using the ADC as a power meter instead of the on-board AD8307

12 bit ADC has much worse dynamic range than AD8307. If you want to replace AD8307 with direct ADC, then you're needs to use at least good 16-bit ADC.

As I know STM32 ADC has ENOB about 8-9 bit = 56 dB.

For comparison, AD8307 has dynamic range 92 dB. This is 63 times better than STM32 ADC.

[Kalvin]

and using the ADC as a power meter instead of the on-board AD8307

12 bit ADC has much worse dynamic range than AD8307. If you want to replace AD8307 with direct ADC, then you're needs to use at least good 16-bit ADC.

As I know STM32 ADC has ENOB about 8-9 bit = 56 dB.

For comparison, AD8307 has dynamic range 92 dB. This is 63 times better than STM32 ADC.

So, you are suggesting that the dynamic range of the 12-bit ADC cannot be improved, and using a 12-bit ADC as a LOG-converter (ie. power meter) is doomed from the start because AD8307 has a theoretical dynamic range of 92dB as stated in its datasheet? And what is the real dynamic range achieved using AD8307 in the LTDZ and similar boards? It is not 92dB although it is stated in the AD8307 datasheet.

Edit: Think about SDR and how it works: With a receiver with a 12-bit ADC it is possible to listen to weak signals that are well below the ADC's 72dB dynamic range. Even those cheap SDR-sticks with only 8-bit ADCs can be used to receive signals that are well below the theoretical 48dB dynamic range. According to your statement above, this should not be possible. Although the STM32F103 has pretty bad ADC, the principles are still the same. At this point I do not have any clue what will be the practical noise floor limit with different digitally implemented RBWs on this hardware.

[radiolistener]

So, you are suggesting that the dynamic range of the 12-bit ADC cannot be improved, and using a 12-bit ADC as a LOG-converter (ie. power meter) is doomed from the start because AD8307 has a theoretical dynamic range of 92dB as stated in its datasheet?

yes, I'm sure that you will get much worse dynamic range if you exclude AD8307 from measurement.

And what is the real dynamic range achieved using AD8307 in the LTDZ and similar boards? It is not 92dB although it is stated in the AD8307 datasheet.

that's interesting question. I didn't tested LTDZ RF input dynamic range separately. I will try to perform some tests.

When using RF-OUT to RF-IN, the dynamic range is about 50 dB.

With a receiver with a 12-bit ADC it is possible to listen to weak signals that are well below the ADC's 72dB dynamic range. Even those cheap SDR-sticks with only 8-bit ADCs can be used to receive signals that are well below the theoretical 48dB dynamic range.

ADC dynamic range is specified for bandwidth = sample rate / 2. When ADC works at 28 MHz sample rate, it's bandwidth is 14 MHz and if you filter it to 500 Hz bandwidth you will get processing gain 14 MHz / 500 Hz = 28000 = 44.47 dB.

There is no any improvement due to DSP processing. It's just balance between bandwidth and noise level, nothing else. If apply analog filter 500 Hz to original signal, you will get the same result.

[radiolistener]

According to your statement above, this should not be possible. Although the STM32F103 has pretty bad ADC, the principles are still the same. At this point I do not have any clue what will be the practical noise floor limit with different digitally implemented RBWs on this hardware.

Processing gain depends on bandwidth reduction. If you use 1 MHz ADC, it has 500 kHz bandwidth and if you apply 120 kHz filter, you will get processing gain = 500/120 = 4.16 = 6 dB. ADC has about 56 dB, so in total you will get 56+6=62 dB. It cannot beat 92 dB of AD8307.

In order to get the same dynamic range as AD8307 on STM32 ADC with help of processing gain, you're needs processing gain = 92-56 = 36 dB = 3981. It means you're needs ADC with bandwidth 120 kHz * 3981 = 477.72 MHz. So, you're needs ADC running from 955.44 MHz clock.

STM32 ADC cannot run at 955 MHz and STM32F103 cannot process output from so high speed ADC. So, you will be unable to get the same dynamic range as AD8307 with using STM32 ADC directly.


By the way, don't forgot that dynamic range is limited not only with AD8307, but also with IAM81008 mixer and ADF4351 output phase noise. And it looks that the main bottleneck here is not AD8307, but IAM81008, ADF4351 and phase noise of 25 MHz oscillator.

By replacing AD8307 with direct ADC sampling you cannot improve dynamic range, because it is already limited on IAM81008 mixer output. And AD8307 already has much better dynamic range than you can achieve by using direct ADC sampling.

[gf]

Not only bandwidth reduction, but averaging of multiple measurements also results in processing gain - no need for 1GSa/s. It is just a matter of patience...

[radiolistener]

Not only bandwidth reduction, but averaging of multiple measurements also results in processing gain.

averaging is a kind of low pass filter, so averaging is just a special case of bandwidth reduction.

[gf]

averaging is a kind of low pass filter, so averaging is just a special case of bandwidth reduction.

Sure, somehow it is. But it can be applied in a different domain, without reducing the (digital) RBW. Due to the DC-blocking analog IF filter, a too small RBW becomes conterproductive at some point, since it suffers from the analog filter's low frequecy attenuation then (so that even more processing gain were required to compensate that).

[Kalvin]

Here are some pointers to books related this subject:

Freely available ebook "The Scientist & Engineer's Guide to Digital Signal Processing, 1999" by Analog Devices:
https://www.analog.com/en/education/education-library/scientist_engineers_guide.html

Freely available ebook "Software-Defined Radio for Engineers, 2018" by Analog Devices:
https://www.analog.com/en/education/education-library/software-defined-radio-for-engineers.html

Practical introductory book about digital signal processing without too complex mathematics (pun intended):
Lyons: Understanding Digital Signal Processing, 3rd Edition

[gf]

So, you are suggesting that the dynamic range of the 12-bit ADC cannot be improved, and using a 12-bit ADC as a LOG-converter (ie. power meter) is doomed from the start because AD8307 has a theoretical dynamic range of 92dB as stated in its datasheet? And what is the real dynamic range achieved using AD8307 in the LTDZ and similar boards? It is not 92dB although it is stated in the AD8307 datasheet.

Edit: Think about SDR and how it works: With a receiver with a 12-bit ADC it is possible to listen to weak signals that are well below the ADC's 72dB dynamic range. Even those cheap SDR-sticks with only 8-bit ADCs can be used to receive signals that are well below the theoretical 48dB dynamic range. According to your statement above, this should not be possible. Although the STM32F103 has pretty bad ADC, the principles are still the same. At this point I do not have any clue what will be the practical noise floor limit with different digitally implemented RBWs on this hardware.

Let's play with some numbers and visualize:

Low end of the AD8307's input range is specified with -75dBm. So let's set our aim at -75dBm, too. That's only about 40µV_RMS. Let's try to capture that with the ADC. 40µV_RMS is well below 1LSB. If it can be captured at all with the ADC, then only in the presence of sufficient dithering noise. Likely this is even granted per se, w/o explicitly adding additional noise.

I don't know the exact specs and operating conditions of the ADC, but let's make the following assumptions:
12 bits, full-scale input voltage = 2V_pp, ADC noise = 707µV_RMS Gaussian noise (-30dB_FS), wanted signal is sine wave with 40µV_RMS.

Attached is a 10000 and 100000 point FFT of the simulated signal (with added noise and 12-bit quantization). Y-axis isnormallized to dBm input level (i.e. 0 is not full-scale)
With 10000 samples the noise floor can be lowered to a level such that the wanted signal peeks out a little bit. With 100000 samples the signal becomes clearly visible then.

So I guess it is not doomed from the beginning. But capturing that many samples takes some time. For input levels below say -50dBm, a LNA in front of the ADC were IMO helpful, in order that the ADC's dynamic range can be utilized more efficiently. Also unclear to me is the suitability of the existing noise for dithering such low signal levels (distribution unknown, while the simulation was done with perfect Gussian random numbers).

EDIT: I worry about potential unpredictable non-random noise/errors, though, like power supply ripple on the reference voltage (it seems not to be separately filtered?) and the bias for the input, unwanted coupling of stray signals, etc. Such things can still defeat feasibility.

[Kalvin]

So, you are suggesting that the dynamic range of the 12-bit ADC cannot be improved, and using a 12-bit ADC as a LOG-converter (ie. power meter) is doomed from the start because AD8307 has a theoretical dynamic range of 92dB as stated in its datasheet? And what is the real dynamic range achieved using AD8307 in the LTDZ and similar boards? It is not 92dB although it is stated in the AD8307 datasheet.

Edit: Think about SDR and how it works: With a receiver with a 12-bit ADC it is possible to listen to weak signals that are well below the ADC's 72dB dynamic range. Even those cheap SDR-sticks with only 8-bit ADCs can be used to receive signals that are well below the theoretical 48dB dynamic range. According to your statement above, this should not be possible. Although the STM32F103 has pretty bad ADC, the principles are still the same. At this point I do not have any clue what will be the practical noise floor limit with different digitally implemented RBWs on this hardware.

Let's play with some numbers and visualize:

Low end of the AD8307's input range is specified with -75dBm. So let's set our aim at -75dBm, too. That's only about 40µV_RMS. Let's try to capture that with the ADC. 40µV_RMS is well below 1LSB. If it can be captured at all with the ADC, then only in the presence of sufficient dithering noise. Likely this is even granted per se, w/o explicitly adding additional noise.

I don't know the exact specs and operating conditions of the ADC, but let's make the following assumptions:
12 bits, full-scale input voltage = 2V_pp, ADC noise = 707µV_RMS Gaussian noise (-30dB_FS), wanted signal is sine wave with 40µV_RMS.

Attached is a 10000 and 100000 point FFT of the simulated signal (with added noise and 12-bit quantization). Y-axis isnormallized to dBm input level (i.e. 0 is not full-scale)
With 10000 samples the noise floor can be lowered to a level such that the wanted signal peeks out a little bit. With 100000 samples the signal becomes clearly visible then.

So I guess it is not doomed from the beginning. But capturing that many samples takes some time. For input levels below say -50dBm, a LNA in front of the ADC were IMO helpful, in order that the ADC's dynamic range can be utilized more efficiently. Also unclear to me is the suitability of the existing noise for dithering such low signal levels (distribution unknown, while the simulation was done with perfect Gussian random numbers).

Yes, increasing the dynamic range and spectral resolution is a trade-off with speed. If the ADC is using sample rate of 1Ms/s, the maximum sweep-rate is 1000 frequency steps per seconds when using 1000 samples, 100 frequency steps per second when using 10000 samples, and 10 frequency steps/s when using 100000 samples. In practice the step-rate will be less because the ADF4351 PLL's  require some settling time after each step.

By using the 12-bit ADC instead of AD8307, the user can choose which is more important at any given moment: fast sweep-rate or better dynamic range / spectral resolution ie narrower RBW. With AD8307 that is not possible without changing the component values of the 120kHz RBW/low-pass filter.

EDIT: I worry about potential unpredictable non-random noise/errors, though, like power supply ripple on the reference voltage (it seems not to be separately filtered?) and the bias for the input, unwanted coupling of stray signals, etc. Such things can still defeat feasibility.

This is something that remains to be seen what kind of performance can be achieved with the actual hardware, and what are the real limiting factors in the hardware.

[Kalvin]

Got my LTDZ hardware, and I have been playing with it on my Linux PC. After some intensive google search sessions, I was able to find suitable PC software. The PC is now talking to the LTDZ, and it is possible to do some measurements. In order to protect the LTDZ from overloading, a 10dB SMA-attenuator is connected to the LO input. No hardware modifications done yet.

Quickly measuring a 50ohm through-cable and with different combination of 10dB and 20dB SMA-attenuators reveal that the LOG-detector is not particularly linear. The SMA-attenuators were checked with NanoVNA.

Measuring a FM band-stop filter from RLT-SDR.com (attachment #1) looks quite promising (blue trace), but adding an additional 10dB attenuator reveals the problem with the LOG-detector non-linearity (red trace). Probably the LOG-scale can be calibrated for improved accuracy: Need to investigate this a bit further.

Coupling/crosstalk between the tracking generator and the RX in frequency span 35Mhz - 4GHz  is shown in the attachment #2. The tg output and the rx input are terminated with 50ohm impedances. After 1.2GHz the on-board crosstalk from tg to rx increases, reducing the useful dynamic range.

As I now have LTDZ hardware available, it is possible to start experimenting with the firmware.

PC software used for testing:
WinNWT5_v5_0_2_1730 running on Wine (used for the band-stop filter measurement).
winnwt4_v4_11_09 running on Wine.
nwt4_v1_10_13 native Linux application NWT4000lin from dl4jal (Configured as NWT4000-2).

[radiolistener]

PC software used for testing:

these software all show non linear dB for LTDZ, which is actually not dB due to simplified calibration. Actually they can show just a garbage at min and max values.

You can get precise dB with my tool which I posted here: https://www.eevblog.com/forum/rf-microwave/idea-for-improving-ltdz-spectrum-analyzer-tracking-generator-35mhz-4-4ghz/msg3566887/#msg3566887

Do calibration with 10 dB and 40 dB attenuators to get the real dB scale.
My tool allows to perform calibration with any attenuator values (just put wanted values under Calibrate button before calibration). It requires two attenuators for calibration in order to achieve better precision.
Note, that values < 10 dB will be distorted on LTDZ due to non linearities of AD8307 for a close to the maximum signal level, for example will bet 5 dB instead of 0 dB.

Also you can see RAW ADC output on the chart (by selecting Magnitude or RAW), it will show you actual LTDZ performance with no distortion from a math correction.

Here is the same Chinese FM-reject filter through 10 dB attenuator and proper calibration on my LTDZ. Note -45 dB rejection here is not mistake, that's real LTDZ dynamic range 55 dB with substract of 10 dB attenuator (55-10=45 dB). WinNWT shows about -60 dB for 40 dB attenuator on LTDZ, so don't believe it

[radiolistener]

just broken my LTDZ, micro-USB connector was broken with a part of PCB wires

Be careful they are soldered weak and USB connector is keeping on PCB wires...

[Kalvin]

just broken my LTDZ, micro-USB connector was broken with a part of PCB wires

Be careful they are soldered weak and USB connector is keeping on PCB wires...

Ouch! Can you fix the PCB? Looks like it is a good practice to add some solder to the USB-connector.

[Kalvin]

Some progress: I have been able to build the original source from joseluu: https://github.com/joseluu/D6_firmware/commit/babda8b84393ddd9176de9e8841653d1d0a52468
The code works pretty well. Somehow the noise floor has jumped from -80dB to -70dB. Nice thing about this code is that the scanning doesn't produce any artifacts any more.

Next thing to do is to modify the hardware a little that the system is able to sample the output of the 120kHz RBW/low-pass filter using ADC with a sample rate of 1Ms/s, and compute the LOG/RMS of the signal. Probably I also need to fix the RBW lowpass-filter response so that the filter will perform better at the lower frequencies. Otherwise there won't be any benefit from creating dynamically adjustable RBW.

One observation: Tracking generator's output frequency is 120kHz higher than the LO by default. Probably this is common to all these devices like LTDZ and D6.

[radiolistener]

One observation: Tracking generator's output frequency is 120kHz higher than the LO by default. Probably this is common to all these devices like LTDZ and D6.

This is ok, frequencies near DC and outside LPF will be removed by coupling capacitor.
LO frequency needs to be shifted relative to tracking generator in order to put scanning frequency into LPF bandwidth. Also this difference is needed to put unwanted spurs outside input bandwidth.

[radiolistener]

Ouch! Can you fix the PCB? Looks like it is a good practice to add some solder to the USB-connector.

I'm not sure, because PCB wires was broken. May be jumpers with a small wires can help to restore it, needs to try...

[Kalvin]

Ouch! Can you fix the PCB? Looks like it is a good practice to add some solder to the USB-connector.

I'm not sure, because PCB wires was broken. May be jumpers with a small wires can help to restore it, needs to try...

If the PCB traces are badly broken, it might be possible to remove/desolder the USB-to-SERIAL converter IC from the PCB, and use an external USB-to-SERIAL converter module instead. Just wire a three-pin header to GND, RX, TX and hot-glue the header to PCB. Then connect the external USB-to-SERIAL module to the header pins.

[zelea2]

The filter has already been redesigned by F4HTQ
You can read about this here http://alloza.eu/david/WordPress3/?p=542 (in french)

[Kalvin]

The filter has already been redesigned by F4HTQ
You can read about this here http://alloza.eu/david/WordPress3/?p=542 (in french)

Yes, that filter has better performance at 120 kHz compared to the original RBW-filter. F4HTQ's improved filter is able to even increase the signal amplitude seen by the AD8307 LOG-detector, giving some extra dB for dynamic range.

My intention is to implement a digital low-pass filter with selectable cut-off frequency after this RBW-filter, so the RBW filter should have a flat pass-band response from low-frequencies up to 120 kHz, and good stop-band attenuation. Here is a new filter implemented in my LTDZ (see attachments #1 and #2).

I have had also some time to play and experiment with the firmware source code and the actual hardware. After some modifications to firmware, it is now possible to use STM32F103's 12-bit ADC to sample the output of the RBW-filter into the on-chip sample buffer, and download the sample buffer contents into PC for later analysis. The maximum sample buffer size is currently 8192 x 16-bit samples. I am using a simple Python script to download the collected ADC samples from LTDZ, and GNU Octave for signal analysis.

[Kalvin]

Here are initial measurement results showing the linearity of the 12-bit ADC with different signal attenuation levels (see attachment #1). The reference level (green graph) measurement was performed with the tracking generator set to 15 kHz offset, a 6dB attenuator connected to TG output, 20dB 10dB attenuator connected to rx input, and a coax cable between the attenuators, thus creating a loop from TG to rx with 16dB attenuation.

Three 20dB attenuators were added to the loop one at a time, and the measurement was repeated for each attenuator added. Finally a fourth, a 10dB attenuator was added, but the signal level is too low to make any difference any more. From these initial measurements it can be seen that the dynamic range is already 60dB or better. No calibration was performed nor necessary.

All these measurements were performed with a sample buffer length of 4096 samples, decimation with a factor of 16, without multiple buffer averaging. Using multiple buffers and averaging, and selecting carefully the offset frequency of the tracking generator, the noise level should decrease and thus the dynamic range can be improved.

These initial measurements showed that this cheap and simple network analyzer could have at least 60dB of dynamic range.

The attachment #2 shows the noisefloor of the rx-path seen by the ADC when TG disabled (ie. in spectrum analyzer mode). The raw dynamic range is around 45dB. Reducing the bandwidth using digital filtering and averaging, the dynamic range can be improved at the expense of the slower sweep rate. It was calculated that using decimation of 16, the RBW bandwidth becomes approx. 20kHz, and the noise floor drops to -50dB. With decimation of 256, the RBW bandwidth becomes approx. 1 kHz and the noise floor drops to -56dB. It might be possible to modify the hardware a little in order to improve the board's noise floor.

Edit: Fixed the rx input attenuation.

[gf]

Nice

The noise floor seems to be roughly in the expected region.
What was the programmed output power level of the TG's ADF4351?

What catches my eye is the large width of the two peaks of the green trace in attachment #1. As the TG transmits just unmodulated CW, I'd ideally expect to see just two narrow spikes sticking out from the noise floor. Is there such a large amount of phase noise? Or did you happen to mess up anything with the decimation and FFT? What window function did you use for the FFT? How does the green trace in look, if you take the captured buffer (w/o any pre-processing) and just do a 4k point FFT, using a say Hamming window (or even better a Kaiser window with beta=13), and zoom-in the plot to the same -30kHz...30kHz frequency range?

EDIT: Was your new IF filter already in place, or were the measurements still done with the original one?

[Kalvin]

The noise floor seems to be roughly in the expected region.

Yes, the noise spectrum does contain some spikes, but other than that the noise level is pretty mach what was expected. For the dithering purposes the noise spectrum looks pretty useful. I will try to add some extra coupling caps to the power supply in order to reduce the spikes a bit.

What was the programmed output power level of the TG's ADF4351?

TG ADF4351 output level was set to its maximum. There is a option for using either "low noise" or "low spur" mode: TG was using low spur and rx was using low noise.

What catches my eye is the large width of the two peaks of the green trace in attachment #1. As the TG transmits just unmodulated CW, I'd ideally expect to see just two narrow spikes sticking out from the noise floor. Is there such a large amount of phase noise?

I can repeat my measurements with different ADF4351 modes (low spur/low noise) and see whether that makes any difference.

EDIT: Was your new IF filter already in place, or were the measurements still done with the original one?

Yes, I had already modified my filter. The mixer's output is coupled to the filer's input using a 1uF ceramic capacitor. The filter output is biased to GND by a 12k resistor, and to +5V by a 24k resistor. The +5V is the same which is used by AD8307 LOG-detector. The biased filter output is then wired to directly to ADC input pin A2. As the ADC input has now necessary bias voltage added, the added bias voltage is removed from the buffered samples simply by subtracting the buffer's mean from each sample value. For simplicity the 12-bit ADC samples are stored as 16-bit values (ie. left aligned). The 1uF coupling capacitor's type is X7R or worse, so the capacitor's dielectric may add some distortion to the signal which may contribute to the spectrum. I will change the cap to tantalum and see whether the noise level improves.

Or did you happen to mess up anything with the decimation and FFT? What window function did you use for the FFT? How does the green trace in look, if you take the captured buffer (w/o any pre-processing) and just do a 4k point FFT, using a say Hamming window (or even better a Kaiser window with beta=13), and zoom-in the plot to the same -30kHz...30kHz frequency range?

Here is my current code for computing the FFT. The x is the buffer of 4096 samples (DC bias removed), and there is an option for decimation. If the decimation option is given, the system will perform decimation in steps of 2, and adjust the frequency data accordingly. It is possible that this iterative decimation may produce some low level noise, but I have not investigated that yet. The FFT windowing function used for the initial measurements was hann().

Edit: Attached the 4096 sample buffer data of the reference measurement signal used for the analysis, which can be read Matlab/Octave load().

Code: [Select]

# System's MCU oscillator frequency
global XTAL = 72000000;

# System's default sampling rate
global Fs_Hz = (XTAL/6/14);

function [f_Hz, bins] = spectrum(x, D)
  global Fs_Hz; # System's sampling rate

  if (nargin == 1)
    D=0;
  endif

  d=D;
  while (d > 0)
    x=decimate(x, 2);
    d=d-1;
  endwhile

  N = length(x);
  w = hann(N);
  df = Fs_Hz/N;
  min_f = -Fs_Hz/2;
  max_f = Fs_Hz/2 - df;
  f_Hz = [min_f : df : max_f] / (2**D);
  spec = (fft(x .* w));
  bins = abs(fftshift(spec));
  bins = bins/N;
endfunction


[Kalvin]

Here is the FFT spectrum of the refence signal using three different windowing functions: Hann, Hamming and Kaiser 13.

[gf]

Here is the FFT spectrum of the refence signal using three different windowing functions: Hann, Hamming and Kaiser 13.

So it's not the signal . The wide width at the bottom of the peaks is FFT leakage. Hann is insufficient here. Hamming gives already nice peaks, but still the attenuation of the side lobes does not bring them below the noise floor. Finally, Kaiser 13 has >90dB attenuation for the side lobes, so there are no visible sidelobes above the noise floor any more -- at the price of a wider main lobe.
Another way to avoid leakage for the 15kHz signal (and its harmonics) is an FFT size which contains an exact integral multiple of 15kHz periods. For your sampling rate, this applies to FFT sizes of N*400 samples, where N=1,2,3,... And indeed, if you do a FFT of buffer(1:4000) -- withoud window -- then the N*15kHz peaks do not leak any more. [The latter does not avoid leakage for other frequencies in the spectrum, though, so it is not a general solution to the leakage issue.].

Besides the leakage issue, the decimation filter also garbles the initial samples in the decimated output. At the beginning, the filter has no state from previous samples, so it needs some settling time.

I'm puzzled where the peaks in the spectrum at higher frequencies >250kZh are coming from (see attachment) 
The frequencies are obviously fs/2 - 15kHz - N*30kHz, and amplitude of the rightmost one is only ~35dB below the 15kHz carrier.
I guess they are not present at the analog ADC input (maybe you could also check the ADC input with a scope and FFT, using a higher sampling rate?)
If it were an interleaved ADC, then I would consider interleaving spurs, but it isn't interlaved, is it?

EDIT: Seems that there is indeed a time-shift between even and odd samples, see 2nd attachment.

EDIT: Reading AN3116, I noticed that the STM32 ADCs can operate in dual fast and dual slow interleaved mode. Are you using one of these operating modes?

[Kalvin]

Excellent analysis, gf! 

I found out that the TG may not been configured properly due to some timing issue in the original code, thus 15 kHz frequency. In the source code the TG offset frequency was set to 10 kHz, and I started debugging why the TG frequency was a bit off. Now it is fixed, and the TG frequency offset is correct. After quick test the spectrum looks much better now.

I recreated the original test now with 15 kHz TG frequency offset, as I wanted to keep things as identical as possible, and our scripts should work with this test signal. The captured samples can be found in the attachment #1. The FFT-spectrum without any windowing for samples y(1:4000) is shown in attachment #2. The decimated by 8 version is shown in attachment #3.

As far as I understand the ADC is not running in interleaved mode. The ADC is set to continuous mode, and MCU is polling the ADC's end-of-conversion flag before reading the ADC and  storing the samples into a RAM buffer. Since the ADC is running in a continuous mode, there should be very little jitter. I will modify the sampling process so that it will use DMA, thus the jitter should be eliminated altogether.

[Kalvin]

Before proceeding any further I need to check my hardware so that power supply ripple is minimized, ADC input is clean, and ADC sampling is using DMA for minimum jitter etc. Otherwise it will be very hard to see what is going on, and any further gains will be only marginal. Anyway, the concept is working.

Edit: ADC is now using DMA.

[Kalvin]

For a quick test for ADC dynamic range and linearity with a digital 15kHz band-pass filter added, here are results repeated with the same sample set used in my earlier post https://www.eevblog.com/forum/rf-microwave/idea-for-improving-ltdz-spectrum-analyzer-tracking-generator-35mhz-4-4ghz/msg3579913/#msg3579913

The reference level (yellow trace) measurement was performed with the tracking generator set to 15 kHz offset, a 6dB attenuator connected to TG output, no attenuator at rx input, and a coax cable between TG and rx, thus creating a loop from TG to rx with 6dB attenuation. Each 4096 sample buffer data was filtered with a 4th order bandpass filter (center frequency 15kHz, BW 400Hz) before analysis.

First the signal level without any attenuation was measured (yellow trace). A 10dB attenuator was added to the loop (blue trace), followed by three 20dB attenuators one at a time, and the measurement was repeated for each attenuator added. Finally a fifth, a 10dB attenuator was added, but ADC's linearity/dynamic range wasn't quite good enough, or the board's noisefloor was too high to get correct signal attenuation and/or the 400Hz filter bandwidth was too wide. Here are the measured and filtered signal levels using different attenuators:

No attenuator: -9.1591
10dB attenuation: -18.062
30dB attenuation: -37.563
50dB attenuation:  -57.580
70dB attenuation:  -78.101
80dB attenuation:  -81.673
no signal: -87.762

After filtering measured samples and performing FFT using Kaiser 13 window, this data shows that the usable dynamic range is at least 70dB. The loopback signal with 80dB attenuation is now visible above the noisefloor, although its value is not correct any more. Probably getting more samples and performing averaging over multiple buffers would improve the dynamic range a bit, but that remains to be seen until I have tried to reduce board's noise floor in the first place.

With the original design using AD8307 LOG-detector, the board's dynamic range was around 50dB when used as a network analyzer (tracking generator enabled). Now, using the ADC and DSP-filtering, the dynamic range of the board is at least 70dB when using the board in network analyzer-mode with the frequency resolution 1kHz (limit of the tracking generator's frequency step size). Improvement of 20dB is quite impressive as the only hardware modification so far are 1) changing the original 120kHz RBW filter component values for better low-pass performance and 2) using ADC to sample the output signal of the 120 kHz RBW filter.

In spectrum analyzer mode (ie. tracking generator disabled and using 120kHz RBW bandwidth), the dynamic range is still less than 50dB due to the poor noisefloor of the board. Adding some filtering to power supplies may improve the situation. Further, as the signal level is now computed from the sampled data, it is possible to reduce the RBW's bandwidth according to the step size using digital filtering, which will effectively improve the board's noisefloor seen by the signal level estimator. However, if the board's inherent noise floor cannot improved, the dynamic range of the spectrum analyzer-mode will remain around 50dB. Even if the dynamic range cannot be improved much, the signal levels reported by the device will be more linear compared to the original design using AD8307.

Edit: Using a 4096 sample buffer, the device is able to perform theoretically 209 sweep steps in a second. In practice the sweep step rate will be less, as the software has to wait until the PLLs are locked before starting sampling RBW signal. Currently the signal processing is performed after capturing the whole buffer of 4096 samples, which will reduce the sweep rate. However, it is possible to use two sample buffers of 4096 samples so that while the other buffer is used for collecting new data using DMA, the other buffer containing samples from the previous sweep can be processed at the same time, practically eliminating the sample buffer processing time. In spectrum analyzer mode (ie. tracking generator off), the board's noisefloor will be quite bad anyway due to wide RBW (even after applying digital RBW filtering), so it might be practical to use even shorter sample buffer length, which would increase the effective sweep step rate.

[gf]

For a quick test for testing the ADC dynamic range and linearity with a digital 15kHz band-pass filter added, [...]

IMO you are thinking too complicated. You neither need decimation nor an explicit bandpass filter. The DFT acts as filter bank, and the window function already is the bandpass filter which is applied to each frequency bin. For instance, Kaiser 13 has a 3dB bandwidth of about 2 DFT bins, a stoppband attenuation of about 100dB, and a stopband bandwidth (width of the main lobe) of about 8.5 bins. When you apply it to 4000 samples captured at a sampling rate of 12/14 MSa/s, then you get a -3dB RBW of ~430Hz and a main lobe width of about 1.8kHz. A DFT calculates all frequency bins in the range -fs/2...fs/2, but for the IF detector, we are eventually only interested in a single bin, namely the bin at 15kHz. So the idea is to calculate the DFT for this bin only, and to renounce the calculation of the other bins. This leads us to a Goertzel detector then. It does exactly that: calculating the DFT for a single frequency. The functionality of the window function, acting as detection bandpass filter, still applies to Goertzel as well, of course.

EDIT: Computational complexity is eventually only 3 multiplications and two additions per sample - and still you get a narrow RBW for the detector and a huge out-of band attenuation.

EDIT: A potential hurdle for doing it on the SMT32 in real-time might be the memory consumption for the pre-calculated window function table and sin/cos table. Depends of course on the maximum desired window size. If the supported window size(s) are fixed, a set of pre-calculated tables could be possibly reside in ROM (don't know, is ROM access slower than RAM access on the SMR32?).

[Kalvin]

For a quick test for testing the ADC dynamic range and linearity with a digital 15kHz band-pass filter added, [...]

IMO you are thinking too complicated. You neither need decimation nor an explicit bandpass filter. The DFT acts as filter bank, and the window function already is the bandpass filter which is applied to each frequency bin. For instance, Kaiser 13 has a 3dB bandwidth of about 2 DFT bins, a stoppband attenuation of about 100dB, and a stopband bandwidth (width of the main lobe) of about 8.5 bins. When you apply it to 4000 samples captured at a sampling rate of 12/14 MSa/s, then you get a -3dB RBW of ~430Hz and a main lobe width of about 1.8kHz. A DFT calculates all frequency bins in the range -fs/s...fs/s, but for an IF detector, we are eventually only interested in a single bin, namely the bin at 15kHz. So the idea is to calculate the DFT for this bin only, and to renounce the calculation of the other bins. This leads us to a Goertzel detector then. It does exactly that: calculating the DFT for a single frequency. The functionality of the window function, acting as detection bandpass filter, still applies to Goertzel as well, of course.

At this point I am just performing rough signal analysis for the captured signals in order to get an overview how the signals/spectrum look in general, are there any specific noise peaks, and what kind of dynamic range is expected ie. kind of feasibility study and proof of concept. Eventually I will implement the actual signal detector in STM32F103. ARM Cortex-M3 doesn't have FPU, so performing any complicated DSP will take a certain amount of cycles, thus the simpler the better. The effects of filter's coefficient quantization and available numeric range needs to be considered carefully as well if wanting to achieve very narrow filters. If the ratio of the filter's center frequency or filter's 3dB point relative to the sampling frequency (800+ kHz) is very small, is might be practical/necessary to perform some decimation prior filtering in order to get the filter coefficients into practical numeric range.

Edit: For the network analyzer use-case, we are free to choose the rx lo frequency so that the received CW signal will land in a location which contains little noise. For example, the noise level increases near 0Hz, so it is not practical to choose rx lo frequency so that the received CW signal is close to 0Hz. Also, due to filter coefficients it is more practical to choose rx lo frequency so that the received CW signal is a bit higher than experimented 15 kHz. For the spectrum analyzer use-case, I just may implement a simple Nth order IIR low-pass filter with a set of pre-calculated coefficients so that the filter's bandwidth will be selected according to the sweep step size.

[gf]

The reference level (yellow trace) measurement was performed with the tracking generator set to 15 kHz offset, a 6dB attenuator connected to TG output, no attenuator at rx input, and a coax cable between TG and rx, thus creating a loop from TG to rx with 6dB attenuation. Each 4096 sample buffer data was filtered with a 4th order bandpass filter (center frequency 15kHz, BW 400Hz) before analysis.

I don't understand how you can reliably apply such a bandpass filter to only ~4.78ms of sampled data (4096 samples). For instance, the impulse response of an IIR filter like

Code: [Select]

cheby1(4,0.1,[(15000-200)/(fs/2) (15000+200)/(fs/2)]) has a width of> 25ms until it fades out to a value close to zero, while the captured samples have a lenght of only ~4.78ms. To eliminate the effect of filter settling you would need to discard 25+ ms from the front of the filtered buffer, and consider only the remaining samples for further processing. But after discarding leading 25ms, there are no samples left...

[Kalvin]

The reference level (yellow trace) measurement was performed with the tracking generator set to 15 kHz offset, a 6dB attenuator connected to TG output, no attenuator at rx input, and a coax cable between TG and rx, thus creating a loop from TG to rx with 6dB attenuation. Each 4096 sample buffer data was filtered with a 4th order bandpass filter (center frequency 15kHz, BW 400Hz) before analysis.

I don't understand how you can reliably apply such a bandpass filter to only ~4.78ms of sampled data (4096 samples). For instance, the impulse response of an IIR filter like

Code: [Select]

cheby1(4,0.1,[(15000-200)/(fs/2) (15000+200)/(fs/2)]) has a width of> 25ms until it fades out to a value close to zero, while the captured samples have a lenght of only ~4.78ms. To eliminate the effect of filter settling you would need to discard 25+ ms from the front of the filtered buffer, and consider only the remaining samples for further processing. But after discarding leading 25ms, there are no samples left...

I guess the same question applies to the Goertzel-detector as well? As Goertzel has a very narrow bandwidth, its impulse response will be very long too (actually it is a resonator as the pole is on the unit circle), thus its output will rise slowly until it reaches the steady state. So far I haven't seen any information which says that Goertzel-detector cannot be used to detect signals that are shorter than the Goertzel's impulse response or rise time. For the shorter signals the Goertzel's output power will be less than the maximum available after steady state, though. Please correct me if I am wrong here.

The idea is to use a bandpass IIR-filter to process the sampled signal buffer (4096 samples), and compute the RMS (energy) of the filter output over the all samples processed: kind of RMS-detector with a filter applied to its input signal. At least this seems to work in practice, but I do not know whether this works in theory.

Edit: Changed wording.

Edit 2: I do understand that when using a shorter signal than IIR filter's impulse response, the filter will not reach the steady-state. Now, if we change filter parameters (center frequency or bandwidth), the output energy between two filters will be different due to different rise-times, and it is necessary to compute a correction/calibration factor for each filter to be used. Since we are working in a digital domain, computing these calibration factors is quite trivial, though.

[gf]

I guess the same question applies to the Goertzel-detector as well? As Goertzel has a very narrow bandwidth, its impulse response will be very long too (actually it is a resonator as the pole is on the unit circle), thus its output will rise slowly until it reaches the steady state. So far I haven't seen any information which says that Goertzel-detector cannot be used to detect signals that are shorter than the Goertzel's impulse response or rise time. For the shorter signals the Goertzel's output power will be less than the maximum available after steady state, though. Please correct me if I am wrong here.

The idea is to use a bandpass IIR-filter to process the sampled signal buffer (4096 samples), and compute the RMS (energy) of the filter output over the all samples processed: kind of RMS-detector with a filter applied to its input signal. At least this seems to work in practice, but I do not know whether this works in theory.

Edit: Changed wording.

Edit 2: I do understand that when using a shorter signal than IIR filter's impulse response, the filter will not reach the steady-state. Now, if we change filter parameters (center frequency or bandwidth), the output energy between two filters will be different due to different rise-times, and it is necessary to compute a correction/calibration factor for each filter to be used. Since we are working in a digital domain, computing these calibration factors is quite trivial, though.

A regular FIR filter is based on linear convolution, which is rather supposed to be applied to an continuous infinite stream of samples. If you instead apply linear convolution to a finite number of samples, then steady state is reached only after the the length of the filter's impulse response, and the leading filtered samples are "garbage". For IIR, baiscally the same applies, but the impulse response length is actually infinite, so an arbitrary end of the impulse needs to be defined, at a point where it returns "close enough" to zero.

Goertzel is under the hood a DFT, calculated for only a single frequency (or a snapshot of a STFT, calculated for a single chunk of samples at a particular point in time, and calculated only for a single frequency).

DFT treats the samples as if they were circular. The window function smooths the wrap-around discontinunity between end and start, reducing spectral leakage.

But a DFT can be also interpreted as filter bank. According to the filter bank interpretation, the chosen window function is the impulse response of a prototype low-pass filter, which is under the hood converted to a bandpass and applied to each DFT frequency bin (see previous link). The DFT window has always the same size as of the number of samples, it cannot be longer. While there exist various commonly used window functions (Hann, Hamming, Kaiser,...), basically any FIR lowpass with N taps (where N is the DFT size -- number of samples) could be used as window function in order to give the filter bank the desired frequency response (of course, if you need special properties like "constant overlap add", this may limit the choice of suitable window functions, but this is not an issue here).

A window function with 4000 "FIR taps" (for a 4000 point DFT) is already quite a large number, thus enabling already a pretty narrow bandwidth. But the minimum feasible bandwidth is eventuall limited by the number of samples. And my feeling is that there is a trade-off between stop-band rejection (-> power of out of band frequencies leaking to the filter output) and ENBW (equivalent noise bandwidth -> i.e. noise power picked up inside the passband).

Integrating the power of the band-pass filtered signal is certainly a valid procedure (granted that the band-pass filtering is valid in the first place). Advantage of Goertzel is the implied bandpass filter, at low computational cost, and it collects both, amplitude and phase, so that phase differences between subsequent readings can be measured. Phase measurements are more sensitive to noise than the amplitude measurements, though. For phase confidence of 1° (standard deviation), you need an effective SNR of better than 30dB.

The effects of filter's coefficient quantization and available numeric range needs to be considered carefully as well if wanting to achieve very narrow filters. If the ratio of the filter's center frequency or filter's 3dB point relative to the sampling frequency (800+ kHz) is very small, is might be practical/necessary to perform some decimation prior filtering in order to get the filter coefficients into practical numeric range.

Sure, numeric ranges need to be planned carefully. I tried to check the effect of quantizaiton. Quantizing the coefficients of a 4k point Kaiser window to 16 bits seems to reduce the window's stop band attenuation to ~120dB. The question is whether this is enough or whether more than that is required? 32-bit Q31 arithmetic should not be a problem for the cortex M3. The accumulator can also be 64 bits if necessary. For real-time processing there are less than 84 cycles per sample available, which rather rules out too complex filtering - as you said yourself. Even a decimation filter with good stopband attenuation might be already too expensive. Goertzel applied to the undecimated data is computationally not so expensive, so it seems well feasible, OTOH.

Btw, could you post the raw data from the previous test?

Edit: You may be interested in this paper, too.

[Kalvin]

Btw, could you post the raw data from the previous test?

Please find the signals in the attachment. I recreated the 15kHz test signals using ADC with DMA, and correct ADF4351 configuration. The file names contain attenuation used. File with name noisefloor is measured when tg was off giving baseline for the board's noise floor. File name noloop is measured when tg on but no loop connected (for measuring on-board signal crosstalk from tg to rx). The signal with 0dB attenuation may just start overdriving the rx, but the signal with 10dB attenuation is clean.

[gf]

Is my assumtion correct that the numbers are the (signed) 12-bit ADC readings, multiplied by 16 to bring them into a signed 16-bit range?
I notice that the numbers modulo 16 are not zero. Is there alrady an offset added/subtracted?
I guess ADC reference voltage is 3.3V, is this correct?

[Kalvin]

I guess the same question applies to the Goertzel-detector as well? As Goertzel has a very narrow bandwidth, its impulse response will be very long too (actually it is a resonator as the pole is on the unit circle), thus its output will rise slowly until it reaches the steady state. So far I haven't seen any information which says that Goertzel-detector cannot be used to detect signals that are shorter than the Goertzel's impulse response or rise time. For the shorter signals the Goertzel's output power will be less than the maximum available after steady state, though. Please correct me if I am wrong here.

The idea is to use a bandpass IIR-filter to process the sampled signal buffer (4096 samples), and compute the RMS (energy) of the filter output over the all samples processed: kind of RMS-detector with a filter applied to its input signal. At least this seems to work in practice, but I do not know whether this works in theory.

Edit: Changed wording.

Edit 2: I do understand that when using a shorter signal than IIR filter's impulse response, the filter will not reach the steady-state. Now, if we change filter parameters (center frequency or bandwidth), the output energy between two filters will be different due to different rise-times, and it is necessary to compute a correction/calibration factor for each filter to be used. Since we are working in a digital domain, computing these calibration factors is quite trivial, though.

A regular FIR filter is based on linear convolution, which is rather supposed to be applied to an continuous infinite stream of samples. If you instead apply linear convolution to a finite number of samples, then steady state is reached only after the the length of the filter's impulse response, and the leading filtered samples are "garbage". For IIR, baiscally the same applies, but the impulse response length is actually infinite, so an arbitrary end of the impulse needs to be defined, at a point where it returns "close enough" to zero.

Goertzel is under the hood a DFT, calculated for only a single frequency (or a snapshot of a STFT, calculated for a single chunk of samples at a particular point in time, and calculated only for a single frequency).

DFT treats the samples as if they were circular. The window function smooths the wrap-around discontinunity between end and start, reducing spectral leakage.

But a DFT can be also interpreted as filter bank. According to the filter bank interpretation, the chosen window function is the impulse response of a prototype low-pass filter, which is under the hood converted to a bandpass and applied to each DFT frequency bin (see previous link). The DFT window has always the same size as of the number of samples, it cannot be longer. While there exist various commonly used window functions (Hann, Hamming, Kaiser,...), basically any FIR lowpass with N taps (where N is the DFT size -- number of samples) could be used as window function in order to give the filter bank the desired frequency response (of course, if you need special properties like "constant overlap add", this may limit the choice of suitable window functions, but this is not an issue here).

A window function with 4000 "FIR taps" (for a 4000 point DFT) is already quite a large number, thus enabling already a pretty narrow bandwidth. But the minimum feasible bandwidth is eventuall limited by the number of samples. And my feeling is that there is a trade-off between stop-band rejection (-> power of out of band frequencies leaking to the filter output) and ENBW (equivalent noise bandwidth -> i.e. noise power picked up inside the passband).

Integrating the power of the band-pass filtered signal is certainly a valid procedure (granted that the band-pass filtering is valid in the first place). Advantage of Goertzel is the implied bandpass filter, at low computational cost, and it collects both, amplitude and phase, so that phase differences between subsequent readings can be measured. Phase measurements are more sensitive to noise than the amplitude measurements, though. For phase confidence of 1° (standard deviation), you need an effective SNR of better than 30dB.

The effects of filter's coefficient quantization and available numeric range needs to be considered carefully as well if wanting to achieve very narrow filters. If the ratio of the filter's center frequency or filter's 3dB point relative to the sampling frequency (800+ kHz) is very small, is might be practical/necessary to perform some decimation prior filtering in order to get the filter coefficients into practical numeric range.

Sure, numeric ranges need to be planned carefully. I tried to check the effect of quantizaiton. Quantizing the coefficients of a 4k point Kaiser window to 16 bits seems to reduce the window's stop band attenuation to ~120dB. The question is whether this is enough or whether more than that is required? 32-bit Q31 arithmetic should not be a problem for the cortex M3. The accumulator can also be 64 bits if necessary. For real-time processing there are less than 84 cycles per sample available, which rather rules out too complex filtering - as you said yourself. Even a decimation filter with good stopband attenuation might be already too expensive. Goertzel applied to the undecimated data is computationally not so expensive, so it seems well feasible, OTOH.

Btw, could you post the raw data from the previous test?

Edit: You may be interested in this paper, too.

I would like to thank you very much once again for your excellent comments and insight for practical digital signal processing, signal analysis and filter design. 

I wasn't really aware that in order to apply Goertzel in properly, it is necessary to apply windowing function to the samples before computing Goertzel. Now, thinking in terms of DFT, it is more obvious that it is important to apply windowing function in order to get good results. If it could be guaranteed that the received rx signal is always integer multiple of the block size to be analyzed, this would be easier as no windowing would be required. Since the MCU's ADC clock and the ADF4351's tg/rx may be a little out of sync, windowing function is required.

Applying a windowing function to the samples before computing a Goertzel begins to feel quite expensive operation in terms of Flash memory and computation cycles. If the block size is 4096 samples, the length of the windowing lookup-table needs to be also 4096 samples. In practice we can reduce the size of the windowing table into half, if we just make sure that the windowing function is symmetrical. Now, for each sample there will be a need to compute one multiply + one Goertzel-loop.

On the other hand, we could probably keep things very simple and just create a FIR filter with 4096 taps instead, so that each sample would require one multiply + one addition. This would be less expensive than windowing + Goertzel-loop for each sample. Making sure that the coefficients of this FIR filter would be symmetrical, the size of the lookup-table can be reduced to 2048. The FIR filter needs to be computed only once for each sample buffer of 4096 samples, so there won't be any problems with reaching the steady state. The only real concern is how many bits are required for the filter coefficients to achieve  90dB-100dB dynamic range for the FIR filter of 4096 taps.  Selecting the rx lo frequency wisely will reduce the precision requirements for the coefficients in order to get the wanted dynamic range.

I have playing with a fixed-point implementation in C of the Goertzel. So far I have not been able to achieve full dynamic range of 96dB using 16 bits for the feed-back coefficients, and it seems that more bits are required for the coefficients. Implementing a Goertzel, or more general IIR bandpass filter using cascaded 2nd order sections, the precision of the coefficients is going to be critical if the ratio of the filter's center frequency and/or bandwidth to the Nyquist freqeuency is very small. Life would be too easy if the MCU would have a FPU even with single-precision floats (with some 24 bits for the mantissa).

[Kalvin]

Is my assumtion correct that the numbers are the (signed) 12-bit ADC readings, multiplied by 16 to bring them into a signed 16-bit range?
I notice that the numbers modulo 16 are not zero. Is there alrady an offset added/subtracted?
I guess ADC reference voltage is 3.3V, is this correct?

Yes, the 12-bit ADC samples are left-aligned to 16 bits, and then the computed mean of the sample buffer is subtracted from each sample, effectively removing the ADC offset voltage (approx. 1.5V - 1.6V).

The MCU is running from 3.3V, so the ADC offset voltage needs to be somewhere near the midpoint of this 3.3V. The output amplitude of the rx mixer is quite small, so the current hardware design cannot fully utilize the full dynamic range of the 12-bit ADC. It could be possible to better match the dynamic range of the 12-bit ADC to the mixer output signal amplitude if the ADC's reference voltage would be reduced.

[gf]

On the other hand, we could probably keep things very simple and just create a FIR filter with 4096 taps instead, so that each sample would require one multiply + one addition.

I'm afraid you misunderstand the computational complexity of a FIR filter. I costs one multiply and one add per sample and per tap! I.e. computational complexity is O(N*M), where N is the number of samples, and M is the number of taps. Goertzel, OTOH, is only O(N) where N is the number of samples; it costs only 3 multiply and 2 add per sample (granted that sin/cos and window come from pre-calculated tables). If window and sin/cos tables are combined, then one multiply per sample can be saved.

Edit:

Since the MCU's ADC clock and the ADF4351's tg/rx may be a little out of sync, windowing function is required.

Frequency offset (in your sample data) seems to be roughly 0.22Hz. Likely still close enough for a rectangular window (i.e no window function) if the noise were white. But it isn't, and there are also a couple of spurs in the spectrum which need to be eliminated. So I tend to use a Hamming window whose ENBW is still as low as 1.36 bins, while out of band rejection is much better. That's based on the given sample data, frequency plan, and a block size of 4000 samples. For a different configuration or different noise floor spectrum this may no longer apply.

I also wonder whether the noise floor spectrum looks always the same, or whether it changes, for instance when you tune the ADF4351s to different frequencies. In particular I don't know what's the origin of the spurs. Furthermore I wonder what's the apparent elevated noise near DC. Is it 1/f noise, or does the spectrum contain any deterministic components in this region, too? At least it is not helpful for SA use case if a low bandwidth is desired

Edit:

Applying a windowing function to the samples before computing a Goertzel begins to feel quite expensive operation in terms of Flash memory and computation cycles. If the block size is 4096 samples, the length of the windowing lookup-table needs to be also 4096 samples.

20k RAM is indeed pretty limited. But isn't quite some amount of flash memory available for a couple of tables?

Edit:

The output amplitude of the rx mixer is quite small, so the current hardware design cannot fully utilize the full dynamic range of the 12-bit ADC. It could be possible to better match the dynamic range of the 12-bit ADC to the mixer output signal amplitude if the ADC's reference voltage would be reduced.

If I understand correctly, the 48 pin package variant of the STM32F103 unfortunately does not expose the reference voltage, so it cannot be changed. Alternative would be a (variable gain?) IF amplifier. But what I wonder in the first place is the origin of the noise. Is it already present in the IF coming out from the mixer, or is it ADC noise? If it is already present in the IF, then amplification won't help much.

[Kalvin]

On the other hand, we could probably keep things very simple and just create a FIR filter with 4096 taps instead, so that each sample would require one multiply + one addition.

I'm afraid you misunderstand the computational complexity of a FIR filter. I costs one multiply and one add per sample and per tap! I.e. computational complexity is O(N*M), where N is the number of samples, and M is the number of taps. Goertzel, OTOH, is only O(N) where N is the number of samples; it costs only 3 multiply and 2 add per sample (granted that sin/cos and window come from pre-calculated tables). If window and sin/cos tables are combined, then one multiply per sample can be saved.

I was thinking that since we have 4096 samples of data available (and there won't be any more data available for the given sweep), computing only one output sample for the band-pass FIR (ie. 4096 MAC-operations) should be enough to give the power estimate of the signal at the specific center frequency of this band-pass FIR. As far as understand, that operation would be similar to correlation between two signals.

However this concept is doomed to fail because the phases of two signals do not match, thus there would be a need to slide the buffer one sample at a time, compute the correlation again for each step performed, until the buffer has rotated for a half cycle of the signal to be detected, and get the maximum of the absolute values from the individual correlations. As the unknown phase difference between the signals is a problem, it should be possible to perform kind of quadrature correlation of the signals.

And yes, the concept would be similar to this one you described earlier:

One thing which does indeed not fit well is the combination of the AC coupled filter with the zero-bandwidth CW signal and a zero-IF receiver. Since the filter has a null at DC it can't detect the zero bandwidth signal which is down-mixed to exactly DC. Therefore I'd rather pick a non-zero IF for the NA use case (i.e. add an offset to the receiver LO). NanoVNA V2.2 for instance uses 12kHz IF, which is sampled @300kSa/s by the MCU , and 50 samples (two 12kHz periods) are correlated with a windowed complex sine wave to obtain one measurement (btw, since the NanoVNA is a VNA, the ADC needs to run continuously in order not to lose phase in the pause when the receiver is switched to a different signal source).

So, I am not sure any more whether is there a real need for a FIR filter or sine/cosine look-up table any more, because we could use a quadrature oscillator for the correlation. Then again, without proper frequency lock between ADF4351 and MCU's ADC clock, there will be a slight frequency error, and the correlation would not be be identical from one measurement to another due to real-world frequency drift between the clocks. That's why I was thinking that a simple band-pass filter would be nice because it would tolerate a slight frequency error. But since we have only 4096 samples available, and the IIR/FIR will reach the steady-state only after .... 

[Kalvin]

Applying a windowing function to the samples before computing a Goertzel begins to feel quite expensive operation in terms of Flash memory and computation cycles. If the block size is 4096 samples, the length of the windowing lookup-table needs to be also 4096 samples.

20k RAM is indeed pretty limited. But isn't quite some amount of flash memory available for a couple of tables?

There is 64KB of Flash available, and currently the size of the firmware is quite small, so Flash won't be a problem at this point. Also many operations like windowing can be performed in-place, so no extra RAM is required. For 4096 samples the required RAM size is 8KB.

The output amplitude of the rx mixer is quite small, so the current hardware design cannot fully utilize the full dynamic range of the 12-bit ADC. It could be possible to better match the dynamic range of the 12-bit ADC to the mixer output signal amplitude if the ADC's reference voltage would be reduced.

If I understand correctly, the 48 pin package variant of the STM32F103 unfortunately does not expose the reference voltage, so it cannot be changed. Alternative would be a (variable gain?) IF amplifier. But what I wonder in the first place is the origin of the noise. Is it already present in the IF coming out from the mixer, or is it ADC noise? If it is already present in the IF, then amplification won't help much.

Yes, you are right that 48 pin package doesn't allow changing the ADC reference voltage. In the spirit of minimal hardware modifications, it could be possible to add an op amp [for example in a 5-pin SOT-23 package] so that the mixer/ RBW-filter's output amplitude would better match with ADC's input signal range. There are some op amps in SOT-23-5 package with fixed gain, so they would not even require external resistors for gain setting. At this point I am not too eager to do any other hardware modifications other than trying to adding extra bypass capacitors to the power supply rails. Like you have written above, there are some noise spurs that needs some investigation.

[Kalvin]

When using the device in network analyzer mode, we can pretty much select any convenient rx frequency offset we like. I have used 15 kHz rx offset in my measurements, although this choice has been more or less random (actually the original intention was to use 10kHz frequency offset, but due to error in the original ADF4351 configuration code, the offset became 15kHz instead), and it was based on assumption that there would be more noise closer to 100 kHz. As it has now turned out, the noise level will rise near 0Hz, so the frequency offset may not be too low either.

I will experiment adding extra bypass capacitors into power supply rails tomorrow, and see how they will affect the spurs/noise.

[gf]

So, I am not sure any more whether is there a real need for a FIR filter or sine/cosine look-up table any more, because we could use a quadrature oscillator for the correlation.

But it is exactly the (digital) quadrature oscillator, which requires the sin/cos lookup table (note, DFT contains under the hood a quadrature oscillator for each frequency bin, and Goertzel contains a quadrature oscillator for the frequency to be detected, too).

Then again, without proper frequency lock between ADF4351 and MCU's ADC clock, there will be a slight frequency error, and the correlation would not be be identical from one measurement to another due to real-world frequency drift between the clocks.

As long it is less than 1Hz it is negligible, at least for amplitude estimation. Phase measurements may need to be adjusted.

That's why I was thinking that a simple band-pass filter would be nice because it would tolerate a slight frequency error. But since we have only 4096 samples available, and the IIR/FIR will reach the steady-state only after ...

You are too much fixated on a conventional filter which is supposed to be applied to a stream . A Goertzel detector acts as bandpass, too, and while its bandwidth is indeed relatively narrow, it is not zero either. So it does tolerate a small frequency error, too. What's the difference at the end?

Also many operations like windowing can be performed in-place, so no extra RAM is required. For 4096 samples the required RAM size is 8KB.

Goertzel processing needs to retain only a complex-valued accumulator from sample to sample (i.e. just 8 or 16 bytes), and the index of the current sample in the current chunk. Buffers are only required for double-buffered DMA. Granted that the real-time processing is fast enough, the two buffers do not even need not to be very big, but just large enough to avoid the interrupt overhead for each single sample. Even (say) 100 samples per buffer may be enough to hide 90% or more of the interrupt overhead.

Edit: For 15kHz IF and current sampling rate, sin/cos table for the digital LO needs only 400 entries.

I have playing with a fixed-point implementation in C of the Goertzel. So far I have not been able to achieve full dynamic range of 96dB using 16 bits for the feed-back coefficients, and it seems that more bits are required for the coefficients.

I do not understand. Goertzel is not IIR and has no feedback A window function table scaled to 0...65535 range (16 bits integer) seems to suffice for the noise floor of the provided sample data. I'd need to verify whether this applies to the sin/cos table, too.

I was basically thinking of something like this: https://godbolt.org/z/he6Kr5e73 (not tested). Gcc generates 9 instructions for the hot loop.
Code assumes that DMA collects double-buffered chunks of 400 samples each, which are processed while the other buffer fills-up in the background.

Edit: updated code

[Kalvin]

I have playing with a fixed-point implementation in C of the Goertzel. So far I have not been able to achieve full dynamic range of 96dB using 16 bits for the feed-back coefficients, and it seems that more bits are required for the coefficients.

I do not understand. Goertzel is not IIR and has no feedback A window function table scaled to 0...65535 range (16 bits integer) seems to suffice for the noise floor of the provided sample data. I'd need to verify whether this applies to the sin/cos table, too.

You are pulling my leg here I was referring to this generally known IIR-implementation of the Goertzel algorithm which will generate the required sinusoidal wave without lookup-table (taken from http://www.mstarlabs.com/dsp/goertzel/goertzel.html which explains also why windowing is important and required in practice):

Code: [Select]

realW = 2.0*cos(2.0*pi*k/N);
imagW = sin(2.0*pi*k/N);

d1 = 0.0;
d2 = 0.0;
for (n=0; n<N; ++n)
  {
    y  = x(n) + realW*d1 - d2;
    d2 = d1;
    d1 = y;
  }
resultr = 0.5*realW*d1 - d2;
resulti = imagW*d1;

Of course you can generate a lookup-table for the sine and cosine for the given frequency, and use a quadratic correlator for signal detection. Your suggested implementation will produce numerically stable implementation, but requires lookup-tables for sine and cosine. Since the Flash memory is not an issue at the moment, your implementation is very favorable indeed.

I was basically thinking of something like this: https://godbolt.org/z/he6Kr5e73 (not tested). Gcc generates 9 instructions for the hot loop.
Code assumes that DMA collects double-buffered chunks of 400 samples each, which are processed while the other buffer fills-up in the background.

Your implementation is very clean and straightforward.  As your initial analysis suggests, your power estimator algorithm should be able to run in real-time, thus only short DMA buffers required.

At this point I am ready to sacrifice more RAM in order to be able to capture all samples for one sweep step into a buffer (4096 samples, 8KB per buffer) before running the power estimator. In this way it is also possible to download the captured raw data to PC, which will help analyzing the signals using Matlab/GNU Octave, and provide means for developing the signal processing algorithms on PC. Using two 4096 sample buffers (one buffer for capturing new data, the other for holding the samples from previous sweep step and used for computing the power estimation) would provide almost real-time performance as well, at the expense of requiring 16KB of RAM for the buffers. Because the firmware is using currently very little RAM (the implementation is just a simple bare-metal super-loop without any tasks or OS), this extra RAM requirement will not be a problem at the moment.

[gf]

... but requires lookup-tables for sine and cosine. Since the Flash memory is not an issue at the moment, your implementation is very favorable indeed.

The big table is rather the tabulated window function (8000 bytes).
The sin/cos tables are only 1600 bytes (together). Unfortunately sin/cos table can't be shared, due to the involved prime factor 7 (which comes from the "odd" samling rate of 12/14 MSa/s).

Code: [Select]

costab[i] = floor(32767 * cos(i / 400.0 * 2 * pi * 7) + 0.5)
sintab[i] = floor(-32767 * sin(i / 400.0 * 2 * pi * 7) + 0.5)


At this point I am ready to sacrifice more RAM in order to be able to capture all samples for one sweep step into a buffer (4096 samples, 8KB per buffer) before running the power estimator. In this way it is also possible to download the captured raw data to PC, which will help analyzing the signals using Matlab/GNU Octave, and provide means for developing the signal processing algorithms on PC. Using two 4096 sample buffers (one buffer for capturing new data, the other for holding the samples from previous sweep step and used for computing the power estimation) would provide almost real-time performance as well, at the expense of requiring 16KB of RAM for the buffers. Because the firmware is using currently very little RAM (the implementation is just a simple bare-metal super-loop without any tasks or OS), this extra RAM requirement will not be a problem at the moment.

Sure, for offline analysis it is desired to have the complete (long) buffer available. A streaming detector OTOH allows to collect even more samples than available RAM (-> say 40000, instead of only 4000, or maybe even 400000, in order to dig even deeper into the noise floor). Btw, for the given frequency plan, block size should not be a power of two, but be rather an integral multiple of 400 (so that a block contains an integral number of 15kHz cycles).

Edit: Sorry, mistake. Number of collected samples needs to fit with the length of the window function. So collecting more samples than available memory were only possible if the tabulated window function would be interpoloated, or with a rectangular window.

[Kalvin]

... but requires lookup-tables for sine and cosine. Since the Flash memory is not an issue at the moment, your implementation is very favorable indeed.

The big table is rather the tabulated window function (8000 bytes).
The sin/cos tables are only 1600 bytes (together). Unfortunately sin/cos table can't be shared, due to the involved prime factor 7 (which comes from the "odd" samling rate of 12/14 MSa/s).

It is stated in STM32F103 datasheet that the ADC clock may not exceed 14 MHz. Currently the MCU's PLL is configured so that the MCU's clock is 72MHz (9*8MHz), and the ADC clock is 72MHz/6 = 12 MHz. As each ADC conversion requires 14 clock cycles, the Fs = 12 MHz / 14 = 857142.857142...Hz.

But we do not have to use 72 MHz clock. We can choose another MCU clock, such as 56MHz (7*8MHz). In this case the ADC clock can be set to 56MHz/4 = 14MHz, thus the ADC sample rate would become 1MHz exactly. The penalty in this case is that the MCU is running a bit slower.

In the original firmware source code it is said that the minimum ADF4351 frequency step size varies from 125Hz to 8kHz, depending on the output frequency: Above 2200MHz the minimum step size is 8kHz. The two ADF4351s are using the same 25MHz oscillator for their reference timebase, so their frequencies and phases are locked. I have not checked the ADF4351 datasheets what is the actual minimum step size at 2200MHz and above, though. Let's take for granted that the common minimum step size is 8kHz for all allowed frequencies between 35 MHz - 4400 MHz.

That means that the rx frequency offset needs to be N*8kHz. As the noise level increases close to 0Hz, selecting N=1 is not terribly good idea. On the other hand, the upper limit for N due to 120kHz RBW lowpass filter is 120kHz / 8kHz = 15. Basically we are free to choose whatever N between 2 and 15, but it is important to check that the rx offset won't end up in the spectrum which contains spurs or high noise level.

Edit: ADF4351 is quite flexible fractional-PLL, so it should be possible to use smaller step sizes than 8kHz even above 2200MHz. However, I would not like to open a can of worms at this point, so let's assume that 8kHz frequency step size is the common minimum steps size for frequency range 35MHz - 4400MHz.

[gf]

The IF frequency must be an integral multiple of the step size too, in order that TG and LO can be tuned to the required frequency offset. So when 8kHz step size is chosen, the IF frequency could be 8kHz, 16kHz, 24kHz, etc. 24kHz does not look that bad; there seems to be a relatively spur-free region in the noise spectrum (unless the spurs would happen to change, when the frequency plan is changed or when the oscillators are swept -- I dont' know, this needs to be investigated). It's also not yet clear whether the observed spurs come from the ADF4351, or whether they have a different origin. ADF4351 also has a low-spur mode, which trades-off spurs agains phase noise. Is this mode enabed or disabled? [However, as long the spurs don't collide with the IF - at any TG/LO tuning frequency - they are not really a problem, and lower phase noise is rather preferred.] In relation to the sampling rate, seven 24kHz periods correspond to 250 samples, so granularity of buffer sizes (and thus sin/cos table size) were even lower than now, enabling also smaller tables.

Lowering the MCU clock speed would impact the CPU power available for real-time processing, unfortunately. And reducing the sampling rate slows down the measurements, and the anti-aliasing filter would require a lower cut-off as well. So better keep it as is if possible. I'm just a bit unsure whether the ADC's T&H sampling time of 1.5 ADCCLK cycles is suffucient, given that the filter output is not buffered - see AN2834, chapter 4.4. Admittedly I do not fully understand the complex interaction between the IF filter and the switched capacitor ADC input (even AN2834 says "A mathematical model is not accurate enough given the number of parameters and their non-linear characteristics. Only a complex design simulation can provide a very good estimate of the minimum TSMPL duration in various conditions.").

... due to 120kHz RBW lowpass filter ...

Btw, if the zero-IF filter's low-pass cut-off is 120 kHz, then the equivalent bandpass RBW (which applies to the received RF signal) is actually 2 * 120 = 240kHz.

[Kalvin]

The IF frequency must be an integral multiple of the step size too, in order that TG and LO can be tuned to the required frequency offset.

Sure, this is not a problem as the current minimum sweep step sizes are 8kHz, 4kHz, 2kHz, 1kHz, 500Hz, 250Hz and 125Hz, depending of the tg and rx lo frequency. 

[Kalvin]

I have now added extra 22uF ceramic decoupling capacitors to the 3V3 power supply lines (at the outputs of the two AMS1117 linear voltage regulators), and 10uF ceramic capacitor to the MCU VDDA voltage net. The device noise level seen by ADC has reduced quite a bit, and many of the extra spurs have mostly gone. There are still some spurs above 100kHz, but those can be filtered out with a digital filter.

Currently the ADC bias voltage (approx. 1.5V-1.6V) is derived from the AD8307 LOG-detector +5V power supply, which happens to be also the USB +5V with only simple LC-filter applied. I will change that in the next step. Hopefully the noise floor will improve a little, and at least some of the remaining spurs will be reduced.

I recreated the 15kHz test signals again with the same sample rate and attenuation steps as before, and the signals can be found as an attachment. Both ADF4351s were using low spur mode, as was the case with the previous signal sets. The file names contain attenuation used. File with name noisefloor is measured when tg was off giving baseline for the board's noise floor. File name noloop is measured when tg on but no loop connected (for measuring on-board signal crosstalk from tg to rx). The signal with 0dB attenuation may just start overdriving the rx, but the signal with 10dB attenuation is clean.

[Kalvin]

Changing the ADC bias voltage power supply improved the board's noise floor a little. Here is the current device noise floor when tg is off (attachment #1), and when tg is on with 16dB attenuation between tg output and rx input (attachment #2). I think the hardware modifications are now successfully completed, and the next step is to proceed with the firmware itself implementing the required power detector for the network analyzer use-case, and the digital low-pass RBW-filter with adjustable bandwidth for the spectrum analyzer use-case.

[Kalvin]

Lowering the MCU clock speed would impact the CPU power available for real-time processing, unfortunately. And reducing the sampling rate slows down the measurements, and the anti-aliasing filter would require a lower cut-off as well. So better keep it as is if possible. I'm just a bit unsure whether the ADC's T&H sampling time of 1.5 ADCCLK cycles is suffucient, given that the filter output is not buffered - see AN2834, chapter 4.4. Admittedly I do not fully understand the complex interaction between the IF filter and the switched capacitor ADC input (even AN2834 says "A mathematical model is not accurate enough given the number of parameters and their non-linear characteristics. Only a complex design simulation can provide a very good estimate of the minimum TSMPL duration in various conditions.").

(Bolding is mine) The output impedance of the new RBW-filter is designed to be 1.1Kohm, and the output capacitor of the RBW-filter is 33nF. With the current 72MHz system clock frequency, the ADC clock is 72MHz/6 = 12MHz, and the ADC sampling time time is 1.5 cycles ie. 125ns.

STM32F103 datasheet Table 47 tabulates the maximum allowed source impedance for 12-bit ADC and 1/4 LSB accuracy for different sampling times. For the 14MHz ADC clock and 1.5 cycle sampling time (107 ns) the maximum allowed source impedance is 0.4 Kohm. For 7.5 cycle sampling time the maximum allowed source impedance is 7.5 Kohm.

Currently we are using sampling time of 125ns, which will tolerate somewhat higher source impedance than 0.4 Kohm. Also, if we target only for 1/2 LSB accuracy, the source impedance can be higher.

Increasing the sampling time from the 125ns (1.5 cycles) to the next possible sampling time of 7.5 cycles would decrease the available ADC sample rate from 857Ks/s down to 600Ks/s.

The measurement data with different attenuation suggests that the current sampling time of 1.5 clock is probably pretty good compromise between ADC sampling rate and the ADC accuracy.

[Kalvin]

I have created a repository at Github for the LTDZ-firmware development https://github.com/kalvin2021/ltdz-dsp I will add documentation for building the code in near future. This code will be the baseline for the actual DSP algorithm development.

This firmware implements the LOG/RMS-detector using stm32f103's 12-bit ADC, replacing the AD8307 LOG-detector. The dynamic range is now similar to original AD8307 ie. 50dB at best, but it will improve as the work progress and new digital signal processing algorithms are added. It is anticipated that the dynamic range will increase to 70dB - 80dB in network analyzer mode. The sweep times will also improve in future.

This firmware version has a new feature which allows to sample RBW filter output into ADC buffer, and provide the sample buffer to PC, which can be then analyzed with Matlab/GNU Octave/Numpy etc..

Current firmware can also be used with existing PC applications WinNWT4, WinNWT5 and NWT4000lin.

[gf]

The noise floor in each of your "-80dB", "noisefloor" and "noloop" sample data files is approx. 0.57mV RMS, equivalent to an IF level (at the ADC input) of -52 dBm (at full IF bandwidth).
[ In your "-80dB" samples, the 15kHz signal already drowns in the noise floor -- at this frequency I cannot see a statistically significant difference between your "-80dB", "noisefloor" and "noloop" samples, which would exced the variance of the noise floor. ]

The maximum possible signal level is limited by the mixer's compression point, which is specified in the datasheet with -6dBm. Your data show that -6dBm is not yet the limit, since your "0dB" samples show a compression of only ~1.1dB, compared to the "10dB" samples (whose ADC-measured IF level is -6.3dBm for the 15kHz fundamental). So a maximum IF level of about 3dBm seems feasible (at least for the RF frequency you did use when you created the data files -- maybe it's different at other RF frequencies).

If I define "dynamic range" as quotient between maximum signal level and noise floor level, then it is ~55dB, for the ADC-based power detector at full IF bandwidth. Due to summing/averaging the power of 4k samples, the variance of the noise floor is supposed to be relatively low (approx +/- 1dB), i.e. the averaging acts as VBW filter, but it can't lower the noise floor. A lower noise floor can only be obtained by reducing the noise bandwidth, of course.

OTOH, the AD8307 datasheet claims a noise floor of -78dBm, which should give a dynamic range of 81dB then. Multiple ADC sapmles of the AD8307 output can still be averaged to improve the variance of the measured noise floor (-> VBW filter). But I'm in fact surprised that you don't get a better DR from the AD8307 either, than from the ADC-based power detector. The transfer function depicted in the datasheet starts to flatten below -70dBm, but that's already close to the noise floor, so it is not assumed to explain a DR loss of  more than a couple of dB. The AD8307 measures of course anything presented to its input. Does the IF signal coming out from the mixer possibly already contain a lot of noise (i.e. significanly more than -78dBm)? If yes, then it could explain why the relatively large intrinsic DR of the AD8307's cannot be exploited. But I have no idea how much noise is actually coming out of the mixer. The noise in the ADC samples is the sum of noise already present in the IF signal and ADC noise, and I've no feeling which component dominates.

Regarding the spurs in the spectrum: The dominant spurs have a spacing of about 143kHz, which is a little bit less than fs/6. I don't worry so much about the spurs near fs/6, near fs/3 and close to Nyqusit, but they also appear close to DC, which is nasty for SA usage, since they are still inside the passband of the RBW filter, even with significantly reduced bandwith. I still wonder whether the frequency of 143kHz is just an unfortunate coincidence, or whether it is related to the sampling? If they were exactly at DC, fs/6, fs/3 and Nyquist, then they were likely sampling-related. But the spacing is a little bit less than fs/6. Do you find any signal source on the board which has an integral multiple of 143kHz, or pulse widths with an integral fraction of ~7us, or anything which could produce IM products of the said frequencies? What's actually the PS3120 IC on the board? I don't find a datasheet, but it seems to be a step-up converter (so I guess it has an oscillator, too). +5V from USB is also a potential candidate for noise and spurs. Was the receiver LO actually turned on, when you captured the "noisefloor" and "noloop" samples? Does it make a difference (regarding spurs) whether it is on or off?

Edit:

It is anticipated that the dynamic range will increase to 70dB - 80dB in network analyzer mode. The sweep times will also improve in future.

(Average) noise floor at 321 Hz noise bandwidth (-> 4000 samples, Hamming window, one bin) is about -83dBm (which would imply a DR of 86dB when the maximum level is 3dBm). Since the noise is not white, it is also frequency-dependent. But variance of the noisefloor (when you repeat the measurement multiple times) is rather high, say +/- 10dB standard deviation (just a rough initial guess), so the desired signal still needs to have a level quite above the average noise floor in oder to stick out with statistical significance. The 15kHz signal in your "-70dB" samples (which is about -66.8dBm) can be clearly distinguished from the noise floor, with a repeatability of about +/-1dB. Still it is too weak for estimating the phase. And the 15kHz signal in your "-80dB" samples is likely above average noise floor, too, but cannot be distinguished from noise due to the variance. For 4k samples, already the sampling limits the sweep time to < 200 readings per second. Tuning the PLLs takes some additional time. [ And vector measurements require at least two (1 port) or 3 (1.5-port) readings per frequency point, reducing the sweep rate further. ]

Edit:

Since the noise distributions are skewed in dB space, which is not so easy to calculate analytically, I've done monte carlo / bootstrap simulation to get closer estimates for the noise floors and their variability. Below are the results for 4000 samples. dBm level are absulute IF voltage levels at the ADC input (0dBm = 0.223607 V_RMS).

Noise floor levels:

Full IF bandwidth RMS averaging (integrate power over samples):
dBm_mean = -51.446
dBm_95pct = -51.237
dBm_stdev =  0.12724

RMS averaging of filtered samples (642 Hz noise BW, NA mode):
dBm_mean = -84.318
dBm_median = -84.104
dBm_95pct = -79.171
dBm_stdev =  3.3625

Vector averaging (DFT-like) of filtered samples (642 Hz noise BW, NA mode):
dBm_mean = -85.697
dBm_median = -84.673
dBm_95pct = -78.324
dBm_stdev =  5.6498

Code: [Select]

N=10000;
NS=4000;
num_samples = NS
fs=72e6/6/14;
freq=15000;

t=[0:NS-1]/fs;

% quadrature LO
lo = exp(-1i * 2 * pi * freq * t);

% window function (lowpass)
w = hamming(NS, "periodic")';
w /= sum(w);

% corresponding bandpass fir filter centered at freq
bp = 2 * w .* cos(2 * pi * freq * t);

% bootstrap resampling of noise from given distribution
% still not perfect, since actual noise is not i.i.d.
nf1 = textread("tg-15kHz-test-signals/tg-15kHz-80dB-after-caps.dat");
nf2 = textread("tg-15kHz-test-signals/tg-15kHz-noisefloor-after-caps.dat");
nf3 = textread("tg-15kHz-test-signals/tg-15kHz-noloop-after-caps.dat");
nf = [ nf1 nf2 nf3 ];                     % combine all three
nf = nf(1:NS)' / 65536 * 3.3 / 0.223607;  % normalize to dBm
x = zeros(N,NS);
for i = 1:N
  x(i,:) = nf(randi(length(nf),1,NS));
end

% alternatively, Gaussian noise with same power
% normalized to dBm
% x = std(nf) * randn(N,NS);

% sine wave for checking correct scaling of calculation
% x = repmat(sqrt(2) * cos(2 * pi * flo * t), N, 1);

y = zeros(N,NS);    % bandpass-filtered samples
V = zeros(N,1);     % filtered vector average
for i = 1:N
  xi = x(i,:);
  y(i,:) = real(ifft(fft(xi) .* fft(bp)));
  V(i) = sum(xi .* lo .* w);
end

disp("")
disp("Noise floor levels:")

disp("")
disp("Full IF bandwidth RMS averaging (integrate power over samples):")
dBm_mean = mean(20*log10(std(x')))
dBm_95pct = sort(20*log10(std(x')))(floor(0.95*N)+1)
dBm_stdev = std(20*log10(std(x')))

disp("")
disp("RMS averaging of filtered samples (642 Hz noise BW, NA mode):")
dBm_mean = mean(20*log10(std(y')))
dBm_median = median(20*log10(std(y')))
dBm_95pct = sort(20*log10(std(y')))(floor(0.95*N)+1)
dBm_stdev = std(20*log10(std(y')))

disp("")
disp("Vector averaging (DFT-like) of filtered samples (642 Hz noise BW, NA mode):")
dBm_mean = mean(20*log10(abs(V) * sqrt(2)))
dBm_median = median(20*log10(abs(V) * sqrt(2)))
dBm_95pct = sort(20*log10(abs(V) * sqrt(2)))(floor(0.95*N)+1)
dBm_stdev = std(20*log10(abs(V) * sqrt(2)))


[Kalvin]

gf, thank you for your excellent posting once again!  Your input to this project has been most valuable, and has given great insight for both theoretical and practical aspects of signal processing and signal analysis.

The noise floor in each of your "-80dB", "noisefloor" and "noloop" sample data files is approx. 0.57mV RMS, equivalent to an IF level (at the ADC input) of -52 dBm (at full IF bandwidth).
[ In your "-80dB" samples, the 15kHz signal already drowns in the noise floor -- at this frequency I cannot see a statistically significant difference between your "-80dB", "noisefloor" and "noloop" samples, which would exced the variance of the noise floor. ]

I made the measurements at approx 60MHz tracking generator carrier frequency and 15kHz LO offset frequency. The signal  "noisefloor" was measured with the tracking generator output disabled. The signal "noloop" was measured with tracking generator output enabled, but the RX was not connected to the tracking generator output. The -80dB signal was measured with tracking generator output enabled with 80dB attenuation from tracking generator output to RX input.

The maximum possible signal level is limited by the mixer's compression point, which is specified in the datasheet with -6dBm. Your data show that -6dBm is not yet the limit, since your "0dB" samples show a compression of only ~1.1dB, compared to the "10dB" samples (whose ADC-measured IF level is -6.3dBm for the 15kHz fundamental). So a maximum IF level of about 3dBm seems feasible (at least for the RF frequency you did use when you created the data files -- maybe it's different at other RF frequencies).

ADF4351 tracking generator produces output signal that is not sinusoidal, but more or less square-wave. The mixer datasheet states that the "RF Feedthrough at IF Port @RF=2 GHz, LO=1.75 GHz" is dBc –25, also figure 7. This may limit the dynamic range of the network analyzer even at the lower frequencies if the device/filter-under-test has high attenuation at the frequencies of interest, but contain only very little attenuation at harmonics of the tracking generator signal. At least this may be a problem in the original design with AD8307 and the new design where ADC is sampling the 120KHz (240kHz) IF signal bandwidth. However, this problem may be solved in future when applying the narrow band spectrum analysis.

If I define "dynamic range" as quotient between maximum signal level and noise floor level, then it is ~55dB, for the ADC-based power detector at full IF bandwidth. Due to summing/averaging the power of 4k samples, the variance of the noise floor is supposed to be relatively low (approx +/- 1dB), i.e. the averaging acts as VBW filter, but it can't lower the noise floor. A lower noise floor can only be obtained by reducing the noise bandwidth, of course.

OTOH, the AD8307 datasheet claims a noise floor of -78dBm, which should give a dynamic range of 81dB then. Multiple ADC sapmles of the AD8307 output can still be averaged to improve the variance of the measured noise floor (-> VBW filter). But I'm in fact surprised that you don't get a better DR from the AD8307 either, than from the ADC-based power detector. The transfer function depicted in the datasheet starts to flatten below -70dBm, but that's already close to the noise floor, so it is not assumed to explain a DR loss of  more than a couple of dB. The AD8307 measures of course anything presented to its input. Does the IF signal coming out from the mixer possibly already contain a lot of noise (i.e. significanly more than -78dBm)? If yes, then it could explain why the relatively large intrinsic DR of the AD8307's cannot be exploited. But I have no idea how much noise is actually coming out of the mixer. The noise in the ADC samples is the sum of noise already present in the IF signal and ADC noise, and I've no feeling which component dominates.

The problem with the AD8307 is that it is measuring signal power over the full 120kHz bandwidth. As there are spurs above the 120kHz frequency, yet the RBW-filter doesn't provide much attenuation at these higher frequencies, the noise floor will remain relatively high, resulting the maximum available dynamic range of 50dB or so. There is very little one can do about this, without redesigning the hardware. Averaging AD8307 output will smooth the signal alright, but it will not improve the noise floor nor improve the dynamic range.

Regarding the spurs in the spectrum: The dominant spurs have a spacing of about 143kHz, which is a little bit less than fs/6. I don't worry so much about the spurs near fs/6, near fs/3 and close to Nyqusit, but they also appear close to DC, which is nasty for SA usage, since they are still inside the passband of the RBW filter, even with significantly reduced bandwith. I still wonder whether the frequency of 143kHz is just an unfortunate coincidence, or whether it is related to the sampling? If they were exactly at DC, fs/6, fs/3 and Nyquist, then they were likely sampling-related. But the spacing is a little bit less than fs/6. Do you find any signal source on the board which has an integral multiple of 143kHz, or pulse widths with an integral fraction of ~7us, or anything which could produce IM products of the said frequencies? What's actually the PS3120 IC on the board? I don't find a datasheet, but it seems to be a step-up converter (so I guess it has an oscillator, too). +5V from USB is also a potential candidate for noise and spurs. Was the receiver LO actually turned on, when you captured the "noisefloor" and "noloop" samples? Does it make a difference (regarding spurs) whether it is on or off?

I can investigate this. It is quite easy to change the sampling frequency and see whether those 143kHz spurs remain in place or move along with different sampling frequencies. PS3120 is a switching-capacitor dc-dc converter / voltage doubler (see attachment). Its typical switching frequency is [low noise constant] 360 kHz, but I have not measured its actual frequency.

It is anticipated that the dynamic range will increase to 70dB - 80dB in network analyzer mode. The sweep times will also improve in future.

(Average) noise floor at 321 Hz noise bandwidth (-> 4000 samples, Hamming window, one bin) is about -83dBm (which would imply a DR of 86dB when the maximum level is 3dBm). Since the noise is not white, it is also frequency-dependent. But variance of the noisefloor (when you repeat the measurement multiple times) is rather high, say +/- 10dB standard deviation (just a rough initial guess), so the desired signal still needs to have a level quite above the average noise floor in oder to stick out with statistical significance. The 15kHz signal in your "-70dB" samples (which is about -66.8dBm) can be clearly distinguished from the noise floor, with a repeatability of about +/-1dB. Still it is too weak for estimating the phase. And the 15kHz signal in your "-80dB" samples is likely above average noise floor, too, but cannot be distinguished from noise due to the variance. For 4k samples, already the sampling limits the sweep time to < 200 readings per second. Tuning the PLLs takes some additional time. [ And vector measurements require at least two (1 port) or 3 (1.5-port) readings per frequency point, reducing the sweep rate further. ]

Very useful analysis, gf! As your analysis shows, the dynamic range of 70dB - 80dB is quite possible. Sweep rate and the dynamic range are trade-offs the user need make. If the user wants to go for faster sweep rate, the available dynamic range will be reduced. If the user wants to have better dynamic range, it is necessary to sweep slower. This is the main goal of this little project.

Since the noise distributions are skewed in dB space, which is not so easy to calculate analytically, I've done monte carlo / bootstrap simulation to get closer estimates for the noise floors and their variability. Below are the results for 4000 samples. dBm level are absulute IF voltage levels at the ADC input (0dBm = 0.223607 V_RMS).

Noise floor levels:

Full IF bandwidth RMS averaging (integrate power over samples):
dBm_mean = -51.446
dBm_95pct = -51.237
dBm_stdev =  0.12724

RMS averaging of filtered samples (642 Hz noise BW, NA mode):
dBm_mean = -84.318
dBm_median = -84.104
dBm_95pct = -79.171
dBm_stdev =  3.3625

Vector averaging (DFT-like) of filtered samples (642 Hz noise BW, NA mode):
dBm_mean = -85.697
dBm_median = -84.673
dBm_95pct = -78.324
dBm_stdev =  5.6498

It is easy to see from your analysis that the available dynamic range will be approx. 50dB when using this device as a spectrum analyzer. When using this device as a [scalar] network analyzer, the dynamic range can be close to 80dB, giving about 30dB improvement over the original design.

[Kalvin]

I have implemented 4th order IIR lowpass-filters for different bandwidths (3.3kHz - 100kHz) to be used in spectrum analyzer operating mode. Currently the firmware selects the most suitable filter bandwidth based on the sweep step size in use. After a quick test, the available dynamic range is approx 50dB, which matches your analysis very well.

The ADC-based LOG-detector is very linear down few dB away from the noise floor.

It was necessary to use 32-bit Q31 data format for the filter coefficients as the ratio of the filter cut-off frequencies to Nyquist frequency became very small. I could not get working filters below 3300Hz due to coefficient quantization effects, but that is not a major problem at the moment. There are ways to go around these filter construction issues.

I will start experimenting with the network analyzer band-pass filter design, based on your ideas and suggestions. First, I will test it on PC with GNU Octave and C, and then port the implementation to the actual firmware when the design is working on a PC.

[gf]

It was necessary to use 32-bit Q31 data format for the filter coefficients as the ratio of the filter cut-off frequencies to Nyquist frequency became very small. I could not get working filters below 3300Hz due to coefficient quantization effects, but that is not a major problem at the moment. There are ways to go around these filter construction issues.

A very low cut-off relaxes the requirement for a preceding decimator, so that even a 2-tap moving average filter would likely suffice for a decimation by two, i.e. average samples #1 and #2 to produce the first decimated sample, average samples #3 and #4 to produce the 2nd decimated sample, and so on. With 4 cascaded stages, you get a decimation by 16. Nyqist is still 26.7kHz then, i.e. still significantly higher than the final cut-off of say 1kHz. All frequencies which are folded back to near DC by the downsamling are located near the zeros of the moving average filters, and therefore significantly attenuated. See attachment for the frequency response of such a 4 stage decimation filter, and for comparison a Chebycheff filter with 3.3kHz cut-off. Btw, don't throw away the extra precision arising from the decimation filter (1 bit per stage).

Edit:

And don't be under the illusion that you can make the cut-off arbitrarily low if only 4k samples are available. If you want a low bandwidth, you need to capture a longer time interval. If the filter happens to be computationally too expensive to run in real-time, then I see at least the aim to do the decimation by factor R (say 16) in real-rime, so that a R times longer time interval can be captured and stored in the same amount of memory.

With decreasing cut-off, I'm also more and more worried about the estimation uncertainty at low frequencies (due to limited time interval being captured). And I'm also worried about ucertainty at DC, which suffers from offset errors and drift. While the zero IF prevents images, it introduces other problems.

Btw, how exacaly did you fit the new analog filter into the curcuit? I guess you turned the 1.1k output resister (from the rf-tools design) into a 2.2k + 2.2k divider to provide the bias for the ADC? Is the mixer output still AC coupled to the input of filter? If yes, what coupling capacitor value?

[Kalvin]

Btw, how exacaly did you fit the new analog filter into the curcuit? I guess you turned the 1.1k output resister (from the rf-tools design) into a 2.2k + 2.2k divider to provide the bias for the ADC? Is the mixer output still AC coupled to the input of filter? If yes, what coupling capacitor value?

I just replaced to original components with the new ones, so this modification is quite simple to do as the filter topology did not change. The AD8307 input impedance is 1.1Kohm, so the input is already terminated. I added 12kohm+12kohm divider between +3V3 and GND providing the required bias, which is good enough for impedance matching. Attachment #1 shows the designed filter. Attachment #2 shows simulated results of the new filter and the original filter.

The bias point is at the Vout of the new filter, which is also wired to the ADC input pin PA.2. In my LTDZ PCB, the unused MCU pins were left floating, so there weren't any problems when adding this jumper wire from Vout to the ADC input pin. Some other designs have the unused pins of the MCU are connected to GND, which requires some more effort to get this extra jumper wire connected to ADC input pin.

As the mixer output has currently a 1uF ceramic capacitor feeding the filter input, the frequencies near DC are attenuated by some dB (at 1kHz the attenuation is approx. 1.3dB). Increasing the capacitor value from 1uF to 10uF would reduce the attenuation, but the transition time after the mixer LO switches frequency would be longer. The component values of the new filter are quite convenient, and the 100uH inductors could be found from an inductor kit bought from eBay.

It might be possible to reduce the PCB's noise floor a little by placing the inductors at the soldering side of the PCB. The switched-capacitor dc-dc converter PS3120 is located close the inductors, so there is a possibility that some of the switching energy can get coupled to the inductors. I have not done that yet, but it might be worth trying. On the soldering side of the PCB there is one L-shaped trace close to one inductor, and I have seen one user adding a small copper shield above the trace in order to reduce any radiated energy from coupling to the nearby inductor.

[Kalvin]

The current hardware modifications can be found as an attachment.

[Kalvin]

Removed broken GNU Octave script. Thanks gf for pointing it out.

[gf]

   
x_re = x .* cos(2.0 * pi * Fs_Hz * t)';
x_im = x .* sin(2.0 * pi * Fs_Hz * t)';

Sorry, I don't understand what this is supposed to do
For the given t and Fs_Hz this is equivalent to x_re = x; x_im = x .* 0;
I guess you rather meant cos(2.0 * pi * Lo_Hz * t) and sin(2.0 * pi * Lo_Hz * t)?

But even then, why do you multiply with the complex sine wave at all, if you calculate a RMS sum, and not a vector sum?
Note that sum((x_re .* x_re) + (x_im .* x_im)) always gives the same result as sum(x .* x), for any frequency of the complex sine wave -- so why not directly calculate the latter?

The multiplication with the complex 15kHz sine wave makes only sense if you'd calculate the vector sum, i.e sum(x_re) .** 2 + sum(x_im)  .** 2 instead.

[Kalvin]

   
x_re = x .* cos(2.0 * pi * Fs_Hz * t)';
x_im = x .* sin(2.0 * pi * Fs_Hz * t)';

Sorry, I don't understand what this is supposed to do

  Too tired after workday. Unfortunately it seemed work, but for wrong reasons. I wrote that script from collection of the manual steps I had taken, and expanded some of the functions that I have. Obviously an error crept in, and the sin/cos generators had wrong argument (Fs_Hz vs Lo_Hz). My intention was to perform quadrature correlation between the sin/cos vectors and the signal to be detected, and finally calculate the RMS from the x_re and x_im vectors.

Here is the GNU Octave script again with the corrections:

Code: [Select]

pkg load signal;

# System's MCU oscillator frequency
global XTAL = 72000000;

# System's default sampling rate
global Fs_Hz = (XTAL/6/14);

# Local oscillator frequency
global Lo_Hz = 15000;

# 0dB level
global ref_dB  = 76.260551;

# Analyze the signal level from the given file.
# Returns signal level in dB.
function dB = level_dB(filename)
  global Fs_Hz;
  global Lo_Hz;
  global ref_dB;
  Fn_Hz = Fs_Hz/2; 
  bw_Hz = 1000;

  # Create a bandpass filter for the LO signal
  [b, a] = cheby1(2, 0.1, [(Lo_Hz-bw_Hz/2)/Fn_Hz, (Lo_Hz+bw_Hz/2)/Fn_Hz]);
 
  x = load(filename);
 
  NS = length(x); 
  ts = 1 / Fs_Hz;
  t = ts * [0 : NS-1];

  # Apply the bandpass filter to the signal
  x = filter(b, a, x);
     
  # Quadrature correlator
  x_re = x .* cos(2.0 * pi * Lo_Hz * t)'; 
  x_im = x .* sin(2.0 * pi * Lo_Hz * t)';
   
  # Compute RMS of the detected signal
 
  %rms_sum = 0;
  %for n = 1:NS
  %  rms_sum += x_re(n)**2 + x_im(n)**2;
  %endfor
 
  rms_sum = sum(x_re.**2 + x_im.**2);
  rms = sqrt(rms_sum/NS);
 
  # Convert the RMS level to dB
  dB = 20*log10(rms) - ref_dB;
 
  printf("%-30s: %f dB\n", filename, dB);
endfunction

level_dB("tg-15kHz-0dB-after-caps.dat");
level_dB("tg-15kHz-10dB-after-caps.dat");
level_dB("tg-15kHz-30dB-after-caps.dat");
level_dB("tg-15kHz-50dB-after-caps.dat");
level_dB("tg-15kHz-70dB-after-caps.dat");
level_dB("tg-15kHz-80dB-after-caps.dat");

Giving these results, suggesting that close 80dB of dynamic range could be possible in the network analyzer mode:

Code: [Select]

# With x = filter(b, a, x) applied
tg-15kHz-0dB-after-caps.dat   : -1.059714 dB
tg-15kHz-10dB-after-caps.dat  : -10.000000 dB
tg-15kHz-30dB-after-caps.dat  : -30.113738 dB
tg-15kHz-50dB-after-caps.dat  : -50.100698 dB
tg-15kHz-70dB-after-caps.dat  : -70.078363 dB
tg-15kHz-80dB-after-caps.dat  : -78.857630 dB

I was playing with the idea of quadrature correlator earlier in the evening, and it seemed to work for some extent: Signals from "0dB" to "-50dB" worked alright, but signals "-70dB" and "-80dB" did not work any more. This can be observed when running the same function without the x = filter(b, a, x) statement produces the following results:

Code: [Select]

# Without x = filter(b, a, x) applied
tg-15kHz-0dB-after-caps.dat   : -0.478256 dB
tg-15kHz-10dB-after-caps.dat  : -9.552529 dB
tg-15kHz-30dB-after-caps.dat  : -29.641191 dB
tg-15kHz-50dB-after-caps.dat  : -48.463008 dB
tg-15kHz-70dB-after-caps.dat  : -53.816233 dB
tg-15kHz-80dB-after-caps.dat  : -54.793700 dB

Then I had an idea that probably adding a band-pass filter before quadrature correlator would give some benefits, and it seemed to work. Yes, it seemed to work alright, but this idea of using the bandpass filter works even without the quadrature correlator. This was not my intention in the first place. I wanted to test whether the quadrature correlator would work as a signal detector for weak signal levels as well.

[gf]

I was playing with the idea of quadrature correlator earlier in the evening, and it seemed to work for some extent: Signals from "0dB" to "-50dB" worked alright, but signals "-70dB" and "-80dB" did not work any more. This can be observed when running the same function without the x = filter(b, a, x) statement produces the following results:

After a few correction it works fine, though, withput any pre-filtering

tg-15kHz-0dB-after-caps.dat   : 0.000000 dB
tg-15kHz-10dB-after-caps.dat  : -8.935213 dB
tg-15kHz-30dB-after-caps.dat  : -29.040368 dB
tg-15kHz-50dB-after-caps.dat  : -49.056581 dB
tg-15kHz-70dB-after-caps.dat  : -69.483305 dB
tg-15kHz-80dB-after-caps.dat  : -80.770373 dB

And it works even well with rectangular window, although a rectangular window does not suppress the spurs as well as other window functions would do. OTOH, a rectangular window has the lowest  noise bandwith of all window functions. Eventually the choice of the window functions it is a trade-off between ENBW and stopband rejection. For this particular use case I tend to use Hann or Hamming. And btw, your Chebycheff filter certainly has a lager noise bandwidth than a 4k point Hann or Hamming window (-> 5...6 dB higher average noise floor), but this happens to be mitigated by the fact that RMS averaging reduces the variance of the noise floor, while vector averaging does not.

Code: [Select]

pkg load signal;

# System's MCU oscillator frequency
global XTAL = 72000000;

# System's default sampling rate
global Fs_Hz = (XTAL/6/14);

# Local oscillator frequency
global Lo_Hz = 15000;

# 0dB level
%global ref_dB  = 76.260551;
global ref_dB = 108.561235;

# Analyze the signal level from the given file.
# Returns signal level in dB.
function dB = level_dB(filename)
  global Fs_Hz;
  global Lo_Hz;
  global ref_dB;
  Fn_Hz = Fs_Hz/2;
  bw_Hz = 1000;

  # Create a bandpass filter for the LO signal
  [b, a] = cheby1(2, 0.1, [(Lo_Hz-bw_Hz/2)/Fn_Hz, (Lo_Hz+bw_Hz/2)/Fn_Hz]);
 
  x = load(filename);

  % Choose a window size which is an integral multiple of 15kHz periods
  % in order to place zeros at the harmonics of 15kHz.
  % (particularly useful if only a rectangular window with a low stop band rejection (high side lobes) is used)
  x = x(1:4000);
 
  NS = length(x);
  ts = 1 / Fs_Hz;
  t = ts * [0 : NS-1];

  # Apply the bandpass filter to the signal
  %x = filter(b, a, x);
     
  # Quadrature correlator
  x_re = x .* cos(2.0 * pi * Lo_Hz * t)';
  x_im = x .* -sin(2.0 * pi * Lo_Hz * t)';
   
  # Compute RMS of the detected signal
 
  %rms_sum = 0;
  %for n = 1:NS
  %  rms_sum += x_re(n)**2 + x_im(n)**2;
  %endfor
 
  % calculate magnitude of vector sum
  rms_sum = sum(x_re) .** 2 + sum(x_im) .** 2;
  rms = sqrt(rms_sum/NS);
 
  # Convert the RMS level to dB
  dB = 20*log10(rms) - ref_dB;
 
  printf("%-30s: %f dB\n", filename, dB);
endfunction

level_dB("tg-15kHz-0dB-after-caps.dat");
level_dB("tg-15kHz-10dB-after-caps.dat");
level_dB("tg-15kHz-30dB-after-caps.dat");
level_dB("tg-15kHz-50dB-after-caps.dat");
level_dB("tg-15kHz-70dB-after-caps.dat");
level_dB("tg-15kHz-80dB-after-caps.dat");


Edit:

And I'm still complaining a bit about the fashion how you apply the IIR filter to the samples, which are way too short compared to the impulse response of the filter. The resulting truncation has the consequence that the effective frequency response of the filter (in conjunction with the RMS averaging detector) deviates significantly from the theoretical Chebycheff response. See attachment: Ideal response, and effective response at different phase. The effective result is still reasonable, but you need to be aware that - as a consequence of the truncation - your filter does no longer work as it was designed with cheby1().

Edit:

For comparison I've added the frequency responses you get from the coherent detector with rectangular (yellow), Hamming (orange) or Hanning window (blue). As you can see, the noise bandwith of all three ones is clearly narrower than your Cheby, and for Hanning and Hamming, the stop band rejection is better, too. In fact the filter truncation ruins the inherently large stop band rejection of the Chebycheff filter, so that at the end it is effectively not better than a rectangular window. The truncated Cheby still has retained its wider bandwidth, but that's not what we want either -- eventually we rather want a narrow (equivalent noise) bandwidth.

[Kalvin]

I was playing with the idea of quadrature correlator earlier in the evening, and it seemed to work for some extent: Signals from "0dB" to "-50dB" worked alright, but signals "-70dB" and "-80dB" did not work any more. This can be observed when running the same function without the x = filter(b, a, x) statement produces the following results:

After a few correction it works fine, though, withput any pre-filtering

Pretty stupid error from me, thanks for helping me out!  The whole idea in complex correlator/detector is to keep the orthogonal Q and I signals separate from each other until computing the RMS, so my error is really fundamental.

I have implemented bandpass-based network analyzer in the firmware, but I am not really happy with its computing time performance. The dynamic range seems to be around 80dB or so, which is not too bad. The noise level at -80dB is quite high, but that is expected.

Your suggestion of using complex correlator/detector is the way to go in terms of computing time performance and simplicity.

[Kalvin]

Here are initial results for the LTDZ dynamic range in the network analyzer operating mode. The measurements used 4096 sample data for each sweep data point, and quadrature correlator with Hanning windowing.

Attachment #1 shows the network analyzer linearity using the following attenuations: 6dB, 16dB, 26dB, 36dB, 46dB, 56dB, and no loop connected (ie. noise floor). The linearity is quite good, and the noise floor is somewhere below -80dB.

Attachment #2 shows the network analyzer linearity using the following attenuations: 46dB, 56dB, 66dB, 76dB, 86dB, and no loop connected (ie. noise floor). The linearity is still quite good, although there is some fluctuation at the weaker signal levels. The useful dynamic range is somewhere around 80dB or a bit less.

These initial results are really encouraging as the dynamic range of the network analyzer operating mode has increased from 50dB -> 80dB with only minimal hardware modifications and some simple digital signal processing.

Note: The dynamic range in the spectrum analyzer mode is still only 50dB, and it is very difficult to improve it without extensive hardware modifications.

Edit: It is possible to reduce the noise at the lower signal levels by performing some filtering across the consecutive sweep data points.

Edit: Here is the current firmware memory usage report:

Code: [Select]

   text    data     bss     dec     hex filename
  29096     100    9800   38996    9854 build/ltdz_firmware.elf

Edit: It is possible to increase the sweep rate by using two separate ADC sample buffers (4096 samples each), and utilizing the fact that the DMA can fill the other sample buffer while the MCU is processing the previous sample buffer.

[gf]

The variability of the green and blue traces at -65dB and -75dB does not look like random noise only, but obviously contains a systematic component, which is also correlated between this green and blue trace. Good question where it comes from, and whether it is rather a function of frequency, or a function of time

[Kalvin]

And I'm still complaining a bit about the fashion how you apply the IIR filter to the samples, which are way too short compared to the impulse response of the filter. The resulting truncation has the consequence that the effective frequency response of the filter (in conjunction with the RMS averaging detector) deviates significantly from the theoretical Chebycheff response. See attachment: Ideal response, and effective response at different phase. The effective result is still reasonable, but you need to be aware that - as a consequence of the truncation - your filter does no longer work as it was designed with cheby1().

Thank you for your simulations, they are most helpful to see and understand the shortcomings of applying a signal to an [IIR] filter that is shorter than the impulse response of the filter. At the start of this project I did not really care how the IIR filter would perform, other than that IIR filters can be readily designed using GNU Octave, they are quite easy to implement, and that they would outperform the original design. Your contributions helped to understand that there is a better, more robust, more elegant, more powerful and even simpler way of performing the same task of detecting signal power level at presence of a noise.

For comparison I've added the frequency responses you get from the coherent detector with rectangular (yellow), Hamming (orange) or Hanning window (blue). As you can see, the noise bandwith of all three ones is clearly narrower than your Cheby, and for Hanning and Hamming, the stop band rejection is better, too. In fact the filter truncation ruins the inherently large stop band rejection of the Chebycheff filter, so that at the end it is effectively not better than a rectangular window. The truncated Cheby still has retained its wider bandwidth, but that's not what we want either -- eventually we rather want a narrow (equivalent noise) bandwidth.

I selected Hann window for the windowing function. Probably any other window would have worked fine too. I considered using a plain rectangular window (ie. no window), but I decided to go for proper windowing. It is quite easy to change things if necessary, after all it is just a matter of generating new lookup-tables. Currently the RX LO offset is set to 80kHz instead of 15 kHz due to the fact the it looked like the noise level would be somewhat less at the higher frequencies and the LO frequency offset needs to be N*8kHz due to the possible ADF4351 frequency step sizes.

[gf]

Currently the RX LO offset is set to 80kHz instead of 15 kHz due to the fact the it looked like the noise level would be somewhat less at the higher frequencies and the LO frequency offset needs to be N*8kHz due to the possible ADF4351 frequency step sizes.

Could you please post ADC samples with 80kHz IF?
I'm interested how the (aliased) harmonics are distributed in the spectrum.

[Kalvin]

Currently the RX LO offset is set to 80kHz instead of 15 kHz due to the fact the it looked like the noise level would be somewhat less at the higher frequencies and the LO frequency offset needs to be N*8kHz due to the possible ADF4351 frequency step sizes.

Could you please post ADC samples with 80kHz IF?
I'm interested how the (aliased) harmonics are distributed in the spectrum.

Sure, please find the signals as attachment. Signal "noisefloor" is TG off and no loop from TG to RX. Signal "noloop" is TG on but no loop from TG to RX. Signal "0dB" is TG looped directly to RX without any attenuation. Signal "-10dB" is TG looped to RX through 10dB attenuator etc. TG frequency is f=60032000_Hz, LO offset is 80kHz, thus LO frequency is TG + 80kHz, Fs=857142.8571428572_Hz.

[Kalvin]

I am thinking about the strategies for how to increase the available sweep rate. Currently the firmware is capturing 4096 samples for each sweep step, and the device is able to perform approx. 1000 sweep steps in 8 seconds ie. approx 125 steps in a second. That is not too bad, considering that the dynamic range is now close to 80dB in the network analyzer mode.

I have also tried with 1024 only samples per sweep step in network analyzer mode, and the results aren't too bad either. It would be great if the user could select between three sweep rates: fast, medium, slow. Corresponding sample sizes would be 1024, 2048 and 4096 samples.

Below are listed sample buffer acquisition times @857kHz ADC sample rate for different sample buffer sizes, and corresponding theoretical maximum sweep rates, including the default PLL lock-in time of 750us:

1024 samples: 1.2ms + PLL lock wait time 750us = 1.3ms = 769 sweep steps/s.
2048 samples: 2.4ms + PLL lock wait time 750us = 2.5ms = 384 sweep steps/s.
4096 samples: 4.8ms + PLL lock wait time 750us = 4.9ms = 192 sweep steps/s.

There is still quite a lot of Flash available in the MCU for the lookup-tables, so it would be quite simple task to provide an option for three different sample buffer sizes, so the user could select between three different sweep rates.

In the current implementation the sample buffer acquisition and sample buffer processing is performed back-to-back, but that can be streamlined in the future by utilizing the ADC+DMA, which will produce a system which would be able to achieve close to theoretical sweep step rates.

[gf]

The 80kHz samples look OK, too. The alias frequencies of the harmonics are still far enough apart from the desired 80kHz signal, in order that they can be rejected, and furthermore the analog filter has attenuated them enough to drown in the noise floor.

Half number of samples is supposed to raise the noise floor by 3dB (at least theoretically, for white noise), i.e. 1040 samples leads to a 6dB higher noise foor then.
I wonder whether so many frequency points are needed in NA mode, since I don't expect the device to be suitable for measuring very narrow-band DUTs, like e.g. crystal filters.

Edit:

Separating sin/cos tables and window function tables saves memory, at the cost of only 2 or 3 more instructions per sample.
Sin and cos table need only 75 entries (7 cycles) each when the IF is 80kHz. However, size of the window function table needs to be the size of the window (i.e. number of samples).
For 4k + 2k + 1k capture sizes this makes a total of 4096 + 2048 + 1024 + 75 + 75 entries, or 14636 bytes.

[Kalvin]

The 80kHz samples look OK, too. The alias frequencies of the harmonics are still far enough apart from the desired 80kHz signal, in order that they can be rejected, and furthermore the analog filter has attenuated them enough to drown in the noise floor.

Half number of samples is supposed to raise the noise floor by 3dB (at least theoretically, for white noise), i.e. 1040 samples leads to a 6dB higher noise foor then.
I wonder whether so many frequency points are needed in NA mode, since I don't expect the device to be suitable for measuring very narrow-band DUTs, like e.g. crystal filters.

Probably the current firmware with the default ADF4351 settings may not be suitable for high-Q crystal filter measurement. In frequency range 35MHz-68MHz, the minimum frequency step size is documented to be 125Hz (I have not verified that). ADF4351 is quite versatile Fractional-PLL, so there may be some settings which could provide finer frequency step size.

/** ADF4351 frequency ranges and step sizes */
#define ADF4351_RANGE_1_MIN_DIV_10  (3500000UL)
#define ADF4351_RANGE_1_STEP_Hz     (125)
#define ADF4351_RANGE_2_MIN_DIV_10  (6875000UL)
#define ADF4351_RANGE_2_STEP_Hz     (250)
#define ADF4351_RANGE_3_MIN_DIV_10  (13750000UL)
#define ADF4351_RANGE_3_STEP_Hz     (500)
#define ADF4351_RANGE_4_MIN_DIV_10  (27500000UL)
#define ADF4351_RANGE_4_STEP_Hz     (1000)
#define ADF4351_RANGE_5_MIN_DIV_10  (55000000UL)
#define ADF4351_RANGE_5_STEP_Hz     (2000)
#define ADF4351_RANGE_6_MIN_DIV_10  (110000000UL)
#define ADF4351_RANGE_6_STEP_Hz     (4000)
#define ADF4351_RANGE_7_MIN_DIV_10  (220000000UL)
#define ADF4351_RANGE_7_STEP_Hz     (8000)
#define ADF4351_RANGE_7_MAX_DIV_10  (440000000UL)

[Kalvin]

Separating sin/cos tables and window function tables saves memory, at the cost of only 2 or 3 more instructions per sample.
Sin and cos table need only 75 entries (7 cycles) each when the IF is 80kHz. However, size of the window function table needs to be the size of the window (i.e. number of samples).
For 4k + 2k + 1k capture sizes this makes a total of 4096 + 2048 + 1024 + 75 + 75 entries, or 14636 bytes.

I was a bit lazy and Flash memory was not an issue, so I created separate tables for 80kHz sin and cos values with the Hann window-function applied. It also helped to perform simple loop unrolling as everything is dividable with 4. I know, premature optimization is the root of all evil ...

Edit: Another option is to create sin/cos tables with length of 13*75 =  975 or so with the desired window function applied, and then run the table 1x, 2x or 4x depending of the number of samples captured (1024, 2048, 4096). The sample counts not have to be 2**N any more in this case. The SNR will be a bit less with 2K and 4K sample buffer sizes due to windowing compared to full 2K and 4K tables. Since 975 is multiple of the 80kHz LO frequency, windowing may not be needed at all, so there wouldn not be any penalty with the rectangular window. However, ss you mentioned, the spur rejection won't that good when using rectangular window.

[gf]

I was also thinking whether the window function table could be possibly interpolated, in order to capture even more than 4k samples for one reading (streaming accumulation, w/o storing all captured samples in the first place). This would need to be done in real-time then, but I think it is still feasible in 84 clock cycles/sample. Would not make the sweep faster, though, but demand even more patience.

Edit: 64k samples instead of 4k were (theoretically) supposed to lower the noise floor by 12dB.

[Kalvin]

I was also thinking whether the window function table could be possibly interpolated, in order to capture even more than 4k samples for one reading (streaming accumulation, w/o storing all captured samples in the first place). This would need to be done in real-time then, but I think it is still feasible in 84 clock cycles/sample. Would not make the sweep faster, though, but demand even more patience.

With rectangular window this might be possible with some loop unrolling and hand-optimization, since all that is needed are sin/cos tables with N*75 samples.

Edit: N=1 or 2, depending whether the tables need to be even or odd.

[gf]

Another option is to create sin/cos tables with length of 13*75 =  975 or so with the desired window function applied, and then run the table 1x, 2x or 4x depending of the number of samples captured (1024, 2048, 4096).

This won't work. The window function must span the whole window size. And the frequency of the sin/cos must remain 80kHz. You can't sub-sample a combined sin/cos table with integrated window function. A "periodic" window function can't be subsampled either w/o interpolation.

[Kalvin]

Another option is to create sin/cos tables with length of 13*75 =  975 or so with the desired window function applied, and then run the table 1x, 2x or 4x depending of the number of samples captured (1024, 2048, 4096).

This won't work. The window function must span the whole window size. And the frequency of the sin/cos must remain 80kHz. You can't sub-sample a combined sin/cos table with integrated window function. A "periodic" window function can't be subsampled either w/o interpolation.

My intention was to apply the sin/cos tables twice or four times back-to-back over one sample buffer. For example, if the sin/cos-tables with windowing were 975-samples in length, the 4K sample buffer would simply run the 975-sample sin/cos table four times (kind of modulo 975).

Edit: Essentially it would be like running four correlations back-to-back applying the same sin/cos tables with windowing for each correlation at a time.

Edit2: Using rectangular window, the sin/cos tables can be used simply as sin/cos lookup-tables with length of (for example) 900 or 975, depending whether the tables lengths need to be odd or even.

[gf]

With rectangular window this might be possible with some loop unrolling and hand-optimization, since all that is needed are sin/cos tables with N*75 samples.

Sin/cos tables need only 75 samples, then they repeat circularly from the beginning.
Rectangular window would require to accumulate not 2^N samples, but N*75 samples, in order to reject harmonics.
But there is quite some "garbage" in the spectrum which should be rejected, too (particularly at low levels), so I would rather not use rectangular.

My intention was to apply the sin/cos tables twice or four times back-to-back over one sample buffer. For example, if the sin/cos-tables with windowing were 975-samples in length, the 4K sample buffer would simply run the 975-sample sin/cos table four times (kind of modulo 975).

This does not work, as it would repeat the window function as well. But the window function must not be repeated 4x, but it must span all accumulated of samples.

[Kalvin]

My intention was to apply the sin/cos tables twice or four times back-to-back over one sample buffer. For example, if the sin/cos-tables with windowing were 975-samples in length, the 4K sample buffer would simply run the 975-sample sin/cos table four times (kind of modulo 975).

This does not work, as it would repeat the window function as well. But the window function must not be repeated 4x, but it must span all accumulated of samples.

For 4K sample buffer, this would be identical to situation that the system would perform four consecutive signal power estimations each 975 samples, and then combine the four results. The penalty is that the available SNR would be less than using full the 4K tables. 

Edit: The windowed sin/cos-table sizes need to be N*75 and even, so the table sizes need to be 900 for 1K buffer. Next possible size would be 1050, but it won't fit into 1024 sample buffer. But again, the buffer size do not have to be 2**N.

[Kalvin]

Let's say that the system has sin/cos tables for 80kHz with windowing applied, and the table sizes are 900 samples each (N*75 samples, and size is even).

1. The system will sample the mixer output into a buffer of 900 samples, and the system will compute the quadrature correlation and vector sums for I and Q for the 900 samples.

2. The system will sample the mixer output into a buffer of 1800 samples. First, the system will compute the quadrature correlation and vector sum for I and Q the first 900 samples. Then, the system will  compute the quadrature correlation and vector sum for I and Q for the remaining 900 samples. The same windowed sin/cos tables are "reused" for the both steps.

Won't this work? 

[gf]

For 4K sample buffer, this would be identical to situation that the system would perform four consecutive signal power estimations each 975 samples, and then combine the four results.
The penalty is that the available SNR would be less than using full the 4K tables. 

Vector averaging of 16k samples is still different from RMS averaging 4 readings, obtained from 4k samples each.

But I calculated the frequency response of a 4x repeated Hanning window. It has indeed an "unusual" shape (see figure - orange), but if I did not make a calculation mistake, the noise bandwidth seems to be the same as for the as a non-repeated Hamming window with 4x size (blue). I need to check this again. Due to the high side lobe, this wndow won't make sense as analysis window for spectrum display, but your intuition may be right that it can nevertheless work for the use case here (despite the unusual frequency response).

[gf]

Let's say that the system has sin/cos tables for 80kHz with windowing applied, and the table sizes are 900 samples each (N*75 samples, and size is even).

1. The system will sample the mixer output into a buffer of 900 samples, and the system will compute the quadrature correlation and vector sums for I and Q for the 900 samples.

2. The system will sample the mixer output into a buffer of 1800 samples. First, the system will compute the quadrature correlation and vector sum for I and Q the first 900 samples. Then, the system will  compute the quadrature correlation and vector sum for I and Q for the remaining 900 samples. The same windowed sin/cos tables are "reused" for the both steps.

Won't this work? 

RMS averaging of N readings, of 900 samples each, does certainly work. But with RMS avaraging, you loose phase information. And it does not reduce the noise floor, but only the variance of the noise floor. So the maximum benefit is limited, even if more and more samples are captured. The noise floor can only be lowered by reducing the bandwidth (-> and a longer window function corresponds to a filter with a lower bandwidth).

[Kalvin]

Let's say that the system has sin/cos tables for 80kHz with windowing applied, and the table sizes are 900 samples each (N*75 samples, and size is even).

1. The system will sample the mixer output into a buffer of 900 samples, and the system will compute the quadrature correlation and vector sums for I and Q for the 900 samples.

2. The system will sample the mixer output into a buffer of 1800 samples. First, the system will compute the quadrature correlation and vector sum for I and Q the first 900 samples. Then, the system will  compute the quadrature correlation and vector sum for I and Q for the remaining 900 samples. The same windowed sin/cos tables are "reused" for the both steps.

Won't this work? 

RMS averaging of N readings, of 900 samples each, does certainly work. But with RMS avaraging, you loose phase information. And it does not reduce the noise floor, but only the variance of the noise floor. So the maximum benefit is limited, even if more and more samples are captured. The noise floor can only be lowered by reducing the bandwidth (-> and a longer window function corresponds to a filter with a lower bandwidth).

In this example, the vector sums for I and Q can continue accumulating from the first sample set to the second sample set. Wouldn't this mean that the phase information is still there, and there won't be any other penalty other than what is caused by windowing?

[Kalvin]

For 4K sample buffer, this would be identical to situation that the system would perform four consecutive signal power estimations each 975 samples, and then combine the four results.
The penalty is that the available SNR would be less than using full the 4K tables. 

Vector averaging of 16k samples is still different from RMS averaging 4 readings, obtained from 4k samples each.

But I calculated the frequency response of a 4x repeated Hanning window. It has indeed an "unusual" shape (see figure - orange), but if I did not make a calculation mistake, the noise bandwidth seems to be the same as for the as a non-repeated Hamming window with 4x size (blue). I need to check this again. Due to the high side lobe, this wndow won't make sense as analysis window for spectrum display, but your intuition may be right that it can nevertheless work for the use case here (despite the unusual frequency response).

The spectrum looks kind of weird. 

[gf]

In this example, the vector sums for I and Q can continue accumulating

This implies eventually that you implicitly repeat the shorter (975 samples) window function N times, in order to to form the total window function which spans all N*975 samples. And this N-fold repetition leads to the weird frequency response then. But although the frequency response of this filter looks weird, it can still work for the given purpose, if the signal spectrum contains only noise at the position of the high side lobe (which is likely granted here).

Edit: Btw, the previously plotted freqency response is the low-pass filter equivalent, of course. Mirror it across the Y axis to add the negative frequencies, and then shift it to 80kHz to get the corresponding bandpass.

Edit: Regarding 80kHz: Yet another point to consider is the frequency deviation between the PLL xtal and the MCU xtal. If the (normalized) deviation between the two oscillators is (say) 20ppm, this leads to a frequency error of 0.3Hz when the IF frequency is 15kHz, but at 80kHz IF, the error is already 1.6Hz. However, the filter bandwidth is independent of the IF frequency. This means that the frequency error to bandwidth ratio is higher at 80kHz than at 15kHz. Still no problem yet and 1.6Hz error is still negligible (at least for magnitude estimation). But when we reduce the filter bandwidth more and more (by vector accumulation of more and more samples), in order to get more and more dynamic range, then at some point the frequency error will begin to matter. I'm also unsure whether the frequency deviation of all LTDZ devices is as low as on yours.

Edit: I still wonder whether the spurs are sampling-reated. In order to see whether it makes a difference, could you please change the ADC sampling time (only for this experiment) from 1.5 to 7.5 cycles, capture 4096 "noisefloor" or "noloop" samples (i.e. w/o TG signal applied, at 600kSa/s then) and post them? I'm only interested in the raw samples, so no need to change anything else for this experiment. Btw, I've now ordered a LTDZ board, too, from China, but no idea when it arrives.

[Kalvin]

Edit: I still wonder whether the spurs are sampling-reated. In order to see whether it makes a difference, could you please change the ADC sampling time (only for this experiment) from 1.5 to 7.5 cycles, capture 4096 "noisefloor" or "noloop" samples (i.e. w/o TG signal applied, at 600kSa/s then) and post them? I'm only interested in the raw samples, so no need to change anything else for this experiment.

Please find the signal data attached. I made the measurements with two different LO offset frequencies (80kHz and 56kHz), and two different ADC sample rates (1c5: Fs=857142.8571428572_Hz and 7c5: Fs=600000.0_Hz). The signal "noisefloor" if captured when TG is off, and LO offset = 0Hz (spectrum analyzer mode). The signal "noloop" is captured when TG on, and LO offset = 80kHz/56kHz (network analyzer mode).
 

Btw, I've now ordered a LTDZ board, too, from China, but no idea when it arrives.

This is nice little gadget to play with, with the known limitations. And it seems that this piece of hardware can be improved easily as the firmware source code is available, and the firmware is quite simple and straightforward. Let's hope that your device has also decent quality. Do you also have ST-Link V2 USB-dongle for programming?

I have created two Python/Numpy programs for a) obtaining the ADC samples from the device and b) performing simple frequency sweeps for firmware testing purposes. I will publish those sometime after polishing them a bit more. Otherwise I have been using WinNWT5, WinNWT4 and NWT4000lin running on Linux Wine.

[Kalvin]

Edit: Regarding 80kHz: Yet another point to consider is the frequency deviation between the PLL xtal and the MCU xtal. If the (normalized) deviation between the two oscillators is (say) 20ppm, this leads to a frequency error of 0.3Hz when the IF frequency is 15kHz, but at 80kHz IF, the error is already 1.6Hz. However, the filter bandwidth is independent of the IF frequency. This means that the frequency error to bandwidth ratio is higher at 80kHz than at 15kHz. Still no problem yet and 1.6Hz error is still negligible (at least for magnitude estimation). But when we reduce the filter bandwidth more and more (by vector accumulation of more and more samples), in order to get more and more dynamic range, then at some point the frequency error will begin to matter. I'm also unsure whether the frequency deviation of all LTDZ devices is as low as on yours.

Changing the LO offset is quite easy in the firmware. All that is required is changing this line in the source code:

Code: [Select]

/** Default RX LO frequency offset in Hz in NA mode when using new ADC dB mode */
#define CONFIG_RX_DEFAULT_NA_MODE_FREQ_OFFSET_Hz    (10*8000L)
and regenerating the sin/cos-tables for the given LO offset frequency with the windowing applied. For this I have GNU Octave/Matlab script.

[Kalvin]

In this example, the vector sums for I and Q can continue accumulating

This implies eventually that you implicitly repeat the shorter (975 samples) window function N times, in order to to form the total window function which spans all N*975 samples. And this N-fold repetition leads to the weird frequency response then. But although the frequency response of this filter looks weird, it can still work for the given purpose, if the signal spectrum contains only noise at the position of the high side lobe (which is likely granted here).

For example I do not have strong digital signal processing background, so it has been real pleasure to have you here explaining things from the theoretical and practical point of view. 

I kind of understand what is going on here, and why my intuition may be wrong, as you explained above. What we are basically doing is performing one tap DFT for the 975 samples [using a quadrature correlator]. Since DFT assumes that the signal to be analyzed is repetitive, proper windowing needs to be applied (or the buffer length needs to be selected appropriately so that the buffer length is exact multiple of the DFT bin frequency).

On the other hand, if we repeat this same process N times for different signal sample sets (while maintaining the phase, which is true in this case), this process will become as coherent averaging (Lyons, Understanding Digital Signal Processing, 3rd. ed. Chapters 11.1 Coherent Averaging and 11.3 Averaging Multiple Fast Fourier Transforms). As I understand, this  process is similar to as if were are performing [synchronous] FFT averaging over multiple FFT transforms with zero time-domain overlap. At the end of the chapter Lyons states that " ... it [coherent FFT integration] only works for periodic time-domain signal sequences that have been obtained through careful synchronous sampling. Coherent integration of multiple FFT results is of no value in reducing spectral measurement noise for non-periodic real-world, information carrying signals". Since we are satisfying this property (we are sampling stationary sinusoidal wave from LO mixer output with synchronous sampling*), it is possible to use coherent DFT integration and obtain increase in SNR. Please correct me if I have understood this wrong.

Edit: OTOH, when I wrote "we are sampling stationary sinusoidal wave from LO mixer output with synchronous sampling", this may not be 100% true because there is a small frequency difference between the ADC sampling clock and LO mixer output frequency, although it will  remain the same through out the sampling process in practical terms. As there is a small frequency offset present, the phase will change a little between sample sets. So, I am not quite sure how much this will impact the SNR gain, as we are using qudrature correlator.

[Kalvin]

About synchronous sampling: Would it make any sense to use an ADC sample buffer of, let's say, 975 samples for 80kHz LO offset, and average N*975 samples into another "working" buffer, and only after N averaged sample sets compute the DFT? If the phase stays the same in practical terms between sample sets, this would reduce the real-time processing requirements, and allow longer sampling times without need for longer sample buffers. If the phase changes too much between sample sets, the SNR gain will be reduced (because the sampling process is no longer synchronous).

[gf]

For example I do not have strong digital signal processing background

Neither do I -- just a little bit.

Since DFT assumes that the signal to be analyzed is repetitive, proper windowing needs to be applied (or the buffer length needs to be selected appropriately so that the buffer length is exact multiple of the DFT bin frequency).

You already "apply windowing" when you pick out a finite sub-sequence of consecutive samples from the infinite stream of samples that represent the discrete signal from t=-inf...inf.
The window function just defines the relative weighting of these samples then (rectangular = equally weighted).
Stricly speaking, "no window" does actually not exist, but there is always some window involved when you truncate an infinite signal to a finite number of samples.

On the other hand, if we repeat this same process N times for different signal sample sets (while maintaining the phase, which is true in this case), this process will become as coherent averaging (Lyons, Understanding Digital Signal Processing, 3rd. ed. Chapters 11.1 Coherent Averaging and 11.3 Averaging Multiple Fast Fourier Transforms). As I understand, this  process is similar to as if were are performing [synchronous] FFT averaging over multiple FFT transforms with zero time-domain overlap. At the end of the chapter Lyons states that " ... it [coherent FFT integration] only works for periodic time-domain signal sequences that have been obtained through careful synchronous sampling. Coherent integration of multiple FFT results is of no value in reducing spectral measurement noise for non-periodic real-world, information carrying signals". Since we are satisfying this property (we are sampling stationary sinusoidal wave from LO mixer output with synchronous sampling*), it is possible to use coherent DFT integration and obtain increase in SNR. Please correct me if I have understood this wrong.

Yes, you basically integrate over a the full window length, i.e. over a contiguous sequence of equally spaced samples. This implies that the ADC must not be stopped and restarted asynchronously for each chunk if this sequence of samples is acquired in multiple chunks. If you skip M samples somewhere in the midde, and insert M zeros instead, then coherency is not lost, but it has the same effect then as if the window function were zero at these samples. But the frequency response of such a window function may become weird, as you have seen.

Edit: OTOH, when I wrote "we are sampling stationary sinusoidal wave from LO mixer output with synchronous sampling", this may not be 100% true because there is a small frequency difference between the ADC sampling clock and LO mixer output frequency, although it will  remain the same through out the sampling process in practical terms. As there is a small frequency offset present, the phase will change a little between sample sets. So, I am not quite sure how much this will impact the SNR gain, as we are using qudrature correlator.

For the amplidue estimation it means that the IF frequency does not hit exactly the center of the bandpass filter response, therefore the amplitude estimate will be a little bit lower.

If you take two readings at times t, and t+dt, and want to measure their phase difference, then the frequency error integrates over the interval dt to a phase error. If the error is larger than negligible, then the frequency offset needs to be estimated in order to compensate the phase error. Btw, the two (independent) readings at times t and t+dt need to be captured coherently, too, i.e. TG keeps running, LO keeps running, and ADC keeps running as well, only the receiver input is switched between the two readings. Otherwise phase information were lost in between. Still the interval dt should be kept short, since the frequency error can only be estimated with limited accuracy.

About synchronous sampling: Would it make any sense to use an ADC sample buffer of, let's say, 975 samples for 80kHz LO offset, and average N*975 samples into another "working" buffer, and only after N averaged sample sets compute the DFT? If the phase stays the same in practical terms between sample sets, this would reduce the real-time processing requirements, and allow longer sampling times without need for longer sample buffers. If the phase changes too much between sample sets, the SNR gain will be reduced (because the sampling process is no longer synchronous).

The following two expressions are mathematically equivalent:

Code: [Select]

sum(sum(x .* repmat(lo,N,1) .* repmat(win,N,1)))
sum(sum(x) .* lo .* win)
[ where x is a Nx975 matrix, and win and lo are 1x975 row vectors ]

So I'd say this is at the end equivalent to your previous proposal, which was in turn equivalent to an N-fold repetition of the 975-tap windows function (which lead to the weird frequency response).

[Kalvin]

Let's assume that we are sampling N samples into a buffer. We have selected N in such way that ADC sampling frequency / LO frequency over N will become integer in all practical terms (for Fs=72Mhz/6/14=857142.8571428572_Hz and  LO=80kHz: Fs/LO=10.71428571428, we could select N = 93 * 10.71428571428 = 996.42857142857 = 996, for example). Now, we will compute the DFT for the given LO frequency of 80kHz (using quadrature correlator with pre-computed sin/cos tables with Hann-window). The RMS for this DFT = sqrt(sum(Q)^2 + sum(I)^2). No weird spectrum here.

Now, let's extend this a little, and that we are sampling K*N consecutive samples into a buffer. We will then divide this buffer into K blocks, each size of N samples. Then, we will compute DFT and RMS for each individual blocks of N samples. Finally, we will combine the results from all K blocks into one RMS value. There should not be no weird spectrum here, because we are just averaging K * individual DFT results. Right?

[Kalvin]

Let's continue now from my previous post.

We have chosen N is such a clever way that the buffer of N samples will always contain integer number of LO cycles, thus the sin/cos-lookup tables will also always contain integer number of sin/cos cycles. We have also selected N so that it is even. This means that there won't be any sin/cos nor LO phase discontinuation between the K blocks, thus we are essentially performing coherent sampling for the K buffers (ie. K*N samples).

We can now do the same thing as previously "We will then divide this buffer into K blocks, each size of N samples. Then, we will compute DFT and RMS for each individual blocks of N samples. Finally, we will combine the results from all K blocks into one RMS value. There should not be no weird spectrum here, because we are just averaging K * individual DFT results."

But, we can do better than that. Since we sampling in coherent way, we do not have to compute the RMS sum for each individual block of N samples, but we can integrate the RMS sum over all K blocks, which will improve the SNR. However, this should not produce weird spectrum either.

The idea here is that we need to have only one set of sin/cos tables of N samples (with Hann-windowing), and we can select the K to be from 1 to any integer number we want to have. If we want fast sweep rate, we just select K=1. If we want better dynamic range, will select K to be 2, 3, 4 ... which will result slower sweep rate but improve the dynamic range.

Alternatively, because we are using coherent sampling, we could have a buffer of N samples in which we maintain the sum for each N samples. After be have captured all K blocks (K*N samples in total), we could compute the DFT for the averaged N samples.

[gf]

Let's assume that we are sampling N samples into a buffer. We have selected N in such way that ADC sampling frequency / LO frequency over N will become integer in all practical terms (for Fs=72Mhz/6/14=857142.8571428572_Hz and  LO=80kHz: Fs/LO=10.71428571428, we could select N = 93 * 10.71428571428 = 996.42857142857 = 996, for example). Now, we will compute the DFT for the given LO frequency of 80kHz (using quadrature correlator with pre-computed sin/cos tables with Hann-window). The RMS for this DFT = sqrt(sum(Q)^2 + sum(I)^2). No weird spectrum here.

Now, let's extend this a little, and that we are sampling K*N consecutive samples into a buffer. We will then divide this buffer into K blocks, each size of N samples. Then, we will compute DFT and RMS for each individual blocks of N samples. Finally, we will combine the results from all K blocks into one RMS value. There should not be no weird spectrum here, because we are just averaging K * individual DFT results. Right?

Vector averaging of the individual DFT results turns out to be mathematically equivalent to calculating the DFT over the whole K*N sized window, using a window function which is a K-fold repetition of the shorter N-tap window function. Just write-down the equation for sum that is calculated, for both cases, re-arrange them acordingly, and you'll see that both are the same. Consequence is that the equivalent filter frequency response is eventually that of the K-fold repeated N-tap window (i.e. the "weird" one).
Edit: I mean the DFT-term for the single frequency bin that is considered. The complete K*N sized DFT contains of course more frequency bins.

OTOH, if you calculate the RMS sum (power sum) of the individual DFT results (treating them as if the were independent and not necessarily coherent to each other), then the filter frequency response still corresponds to the (short) N-tap window function. The RMS sum does not contain phase information any more, and it reduces the variance of the noisefloor. It does not lower the noise floor. I.e. even if K approaches infinity, then N and the window function determine the lowest noise floor you can get.

Edit: Actually we'd like the get the filter response of a (non-repeated) window function, which spans the whole N*K window size, since its bandwidth is 1/K of the N-tap window's bandwidth then. But neither the vector nor the RMS averaging do that if the stored window function table has only N taps, and if the table is just indexed and not interpolated.

Edit: The ENBW of the "weird" frequency response is in fact as narrow as the ENBW of the desired frequency response. Unusual are the high side lobes. The main lobe is even narrower, conversely. If the signal spectrum happens to be "empty" (besides random noise) at the side lobe frequency (i.e. no spurs or harmonics), then this window function is still usable for the detector. It just requires a decent frequency planning to ensure that.

Btw, if you are willing to accept a gap at the center of the lobe, then I've a proposal for SA mode. Actually is is based on the above principle, with K=N=64, and doing RMS averaging of the individual sub-results. I've chosen the frequency plan to exclude the spurs and also the noise near DC. My expectation were a DR improvement of say 15+ dB, compared to simple RMS averaging over all samples. It is still rather wide-band (approx +/- 67kZh @-3dB), in order that the center gap remains small when compared to the total BW.
(Edit: The frequency plan is just an initial draft - there is still room for fine-tuning)

Edit: One more thing:

for Fs=72Mhz/6/14=857142.8571428572_Hz and  LO=80kHz: Fs/LO=10.71428571428, we could select N = 93 * 10.71428571428 = 996.42857142857 = 996, for example

In order that you can continue a vector sum coherently after accumulating N samples, by wrapping-around the sin/cos tables, an exact integral multiple of LO periods need to fint into the N samples. For f_LO = 80kHz, this applies to N=75, or any integral multiple of 75, but not to 996.

[Kalvin]

(Edit: The frequency plan is just an initial draft - there is still room for fine-tuning)

I have been thinking about this as well. Since the LO frequency needs to be multiple of 8000Hz, the current sampling frequency of 857142.857142 Hz is pretty unfriendly in this regard. The following options for the ADC sample rate are possible with different MCU core clock frequencies:

MCU core frequency 72MHz:
ADC sampling time 1.5cy: ADC sampling frequency 72MHz/6/14 = 857142.857142 Hz.
ADC sampling time 7.5cy: ADC sampling frequency 72MHz/6/20 = 600000Hz (75*8000Hz).

MCU core frequency 64MHz:
No good division ratio available which would provide advantage over 72MHz or 56MHz.

MCU core frequency 56MHz:
ADC sampling time 1.5cy: ADC sampling frequency 56MHz/4/14 = 1000000Hz (125*8000Hz).
ADC sampling time 7.5cy: ADC sampling frequency 56MHz/4/20 = 700000Hz (87.5*8000Hz).

Using the faster 72MHz core clock would provide faster computation times compared to 56MHz clock (kind of obvious). On the other hand, the 56MHz core clock would provide us faster ADC sampling rate and more samples in the given time period ie. better SNR.

[Kalvin]

Edit: One more thing:

for Fs=72Mhz/6/14=857142.8571428572_Hz and  LO=80kHz: Fs/LO=10.71428571428, we could select N = 93 * 10.71428571428 = 996.42857142857 = 996, for example

In order that you can continue a vector sum coherently after accumulating N samples, by wrapping-around the sin/cos tables, an exact integral multiple of LO periods need to fint into the N samples. For f_LO = 80kHz, this applies to N=75, or any integral multiple of 75, but not to 996.

And this applies only using 72MHz and 7.5cy ADC sampling time (see my post above). If the system would use 72MHz and 1.5cy ADC sampling time, the 996 samples would not provide exact integer multiple of LO cycles. But it may be possible to nudge the ADC sample buffer pointer for each DFT computation so that the sampling will look practically coherent for each DFT computation.

[gf]

Edit: One more thing:

for Fs=72Mhz/6/14=857142.8571428572_Hz and  LO=80kHz: Fs/LO=10.71428571428, we could select N = 93 * 10.71428571428 = 996.42857142857 = 996, for example

In order that you can continue a vector sum coherently after accumulating N samples, by wrapping-around the sin/cos tables, an exact integral multiple of LO periods need to fint into the N samples. For f_LO = 80kHz, this applies to N=75, or any integral multiple of 75, but not to 996.

And this applies only using 72MHz and 7.5cy ADC sampling time (see my post above). If the system would use 72MHz and 1.5cy ADC sampling time, the 996 samples would not provide exact integer multiple of LO cycles. But it may be possible to nudge the ADC sample buffer pointer for each DFT computation so that the sampling will look practically coherent for each DFT computation.

For NA usage, 12/14 MSa/s and 80kHz IF are fine. Seven 80kHz cyles == 75 samples then.

If a later SA goal were stiching the spectrum from multiple scans, then freqency steps which are in sync with DFT bins can be helpful.
Then 1MSa/s might be indeed be favorable, even if the CPU power is a bit lower then.

Edit: E.g. 4000 samples @1MSa/s would lead to "nice" 250Hz bins, and 8kHz were 32 bins then.

[gf]

if you are willing to accept a gap at the center of the lobe, then I've a proposal for SA mode. Actually is is based on the above principle, with K=N=64, and doing RMS averaging of the individual sub-results. I've chosen the frequency plan to exclude the spurs and also the noise near DC. My expectation were a DR improvement of say 15+ dB, compared to simple RMS averaging over all samples. It is still rather wide-band (approx +/- 67kZh @-3dB), in order that the center gap remains small when compared to the total BW.

Attached is the expected frequency response of this RBW filter. As said, unfortunately with gap at the center . But there are too much spurs, noise and DC offset error in the center. Excluding it enables a higher dynamic range. IMO the gap needs to be filled eventually by stitching multiple measurements with frequency offset.

Edit: updated Link: https://godbolt.org/z/PG461P1jW
I expect a noise floor of roughly -67dBm (IF level), or a DR of almost 70dB if the max. IF level is 3dBm.

Edit: I played a bit with the gap width vs. DR trade-off.
At the cost of ~0.85dB the gap can be reduced by 40kHz, and reduction by 48 kHz costs ~1.5dB (see attached 2nd diagram).
(In the sin/cos tables this meens 36kHz or 32kHz instead of 56kHz)

[Kalvin]

For NA usage, 12/14 MSa/s and 80kHz IF are fine. Seven 80kHz cyles == 75 samples then.

If a later SA goal were stiching the spectrum from multiple scans, then freqency steps which are in sync with DFT bins can be helpful.
Then 1MSa/s might be indeed be favorable, even if the CPU power is a bit lower then.

Edit: E.g. 4000 samples @1MSa/s would lead to "nice" 250Hz bins, and 8kHz were 32 bins then.

Good, let's keep the current sampling rate 72MHz/6/14.

[Kalvin]

if you are willing to accept a gap at the center of the lobe, then I've a proposal for SA mode. Actually is is based on the above principle, with K=N=64, and doing RMS averaging of the individual sub-results. I've chosen the frequency plan to exclude the spurs and also the noise near DC. My expectation were a DR improvement of say 15+ dB, compared to simple RMS averaging over all samples. It is still rather wide-band (approx +/- 67kZh @-3dB), in order that the center gap remains small when compared to the total BW.

Attached is the expected frequency response of this RBW filter. As said, unfortunately with gap at the center . But there are too much spurs, noise and DC offset error in the center. Excluding it enables a higher dynamic range. IMO the gap needs to be filled eventually by stitching multiple measurements with frequency offset.

Edit: updated Link: https://godbolt.org/z/PG461P1jW
I expect a noise floor of roughly -67dBm (IF level), or a DR of almost 70dB if the max. IF level is 3dBm.

Edit: I played a bit with the gap width vs. DR trade-off.
At the cost of ~0.85dB the gap can be reduced by 40kHz, and reduction by 48 kHz costs ~1.5dB (see attached 2nd diagram).
(In the sin/cos tables this meens 36kHz or 32kHz instead of 56kHz)

Thank you for investigating this. You are right that spurs and the frequency components near DC limit the dynamic range in SA mode. Filtering out the frequency components near DC means that there will be a gap in the spectrum near DC. I would see that allowing/tolerating the gap is a trade-off between better dynamic range and worse dynamic range in SA mode. It is quite possible to implement this as a feature that the user can select, because we can add new commands to serial port protocol quite easily. The compile-time configuration will then provide the default option* (either better dynamic range with a gap, or gapless spectrum with worse dynamic range).

Edit: I was playing with a simple M-tap comb-filter with null at DC, placed after IIR low-pass filter, which could be used as a DC-removal filter. The IIR low-pass filter would provide the adjustable RBW, and the comb-filter would take care of removing the low-frequency components near DC. Alternative solution would be to implement a IIR band-reject filter at DC with BW somewhere 2kHz ... 8kHz, but I have not investigated this. Lyons has some other ideas for narrow-band DC removal in his book "Understanding Digital Signal Processing", 3rd ed. section 13.40 "Linear-Phase DC Removal Filters" which shows some strategies for reducing the band-pass ripple of a typical comb-filter and making filter response narrower at DC.

Edit2: * In order to maintain compatibility with the existing PC software which do not support command protocol extansions, the compile-time configuration will be used as default.

[Kalvin]

Vector averaging of the individual DFT results turns out to be mathematically equivalent to calculating the DFT over the whole K*N sized window, using a window function which is a K-fold repetition of the shorter N-tap window function. Just write-down the equation for sum that is calculated, for both cases, re-arrange them acordingly, and you'll see that both are the same. Consequence is that the equivalent filter frequency response is eventually that of the K-fold repeated N-tap window (i.e. the "weird" one).
Edit: I mean the DFT-term for the single frequency bin that is considered. The complete K*N sized DFT contains of course more frequency bins.

Ok, here is my attempt (see attachment). Let's assume that we are sampling our LO into a buffer of K*N samples, and the want to estimate the RMS of the periodic LO signal by splitting the effort into K individual blocks (each block has size N samples). Because we have chosen our ADC sample rate so that the N is always an integer multiple LO period (75 samples), we can combine individual one-bin DFT RMS values to get total signal RMS power. Due to a fact that we are actually performing coherent sampling, we can integrate the real and the imaginary parts separately over all blocks before computing the RMS of the signal, which will increase the SNR. But this will not introduce weird spectral response.

[gf]

Due to a fact that we are actually performing coherent sampling, we can integrate the real and the imaginary parts separately over all blocks before computing the RMS of the signal, which will increase the SNR. But this will not introduce weird spectral response.

In general this does introduce the weird response. Only for a rectangular window function it does not, since its K-fold repetition is still rectangular. See below.

Code: [Select]

// complex sine wave
lo = exp(-1i*2*pi*freq*[0:74]/fs);

// 4 chunks of samples
samp_1 = rand(1,75);
samp_2 = rand(1,75);
samp_3 = rand(1,75);
samp_4 = rand(1,75);

// regular coherent summation over all samples, note: window function spans all 75*4 samples!
w_all = hanning(75*4,"periodic")';
iq_all = sum([samp_1 samp_2 samp_3 samp_4] .* [lo lo lo lo] .* w_all);

// cohrerent summation in chunks
w = hanning(75,"periodic")';
iq_1 = sum(samp_1 .* lo .* w);
iq_2 = sum(samp_2 .* lo .* w);
iq_3 = sum(samp_3 .* lo .* w);
iq_4 = sum(samp_4 .* lo .* w);
iq_sum = iq_1 + iq_2 + iq_3 + iq_4;

// this is the same as
iq_sum = sum([samp_1 samp_2 samp_3 samp_4] .* [lo lo lo lo] .* [w w w w]);


But at the end iq_sum is not the same as iq_all !!!
Because [w w w w] != w_all.
While fft(w) and fft(w_all) look as usual, fft([w w w w]) is the "weird" frequency response, with the extra side lobe.

OTOH, the following would lead to iq_sum == iq_all:

Code: [Select]

w_1 = w_all(1:75);
w_2 = w_all(76:150);
w_3 = w_all(151:225);
w_4 = w_all(226:300);

iq_1 = sum(samp_1 .* lo .* w_1);
iq_2 = sum(samp_2 .* lo .* w_2);
iq_3 = sum(samp_3 .* lo .* w_3);
iq_4 = sum(samp_4 .* lo .* w_4);
iq_sum = iq_1 + iq_2 + iq_3 + iq_4;

But this requires to store the whole N*K sized window function table, in order that w_1 ... w_k can be extracted.

[Kalvin]

I think we have had some misunderstanding, or I have not been able to explain my intention properly. However, you have captured my intention here in compact format:

Code: [Select]

// this is the same as
iq_sum = sum([samp_1 samp_2 samp_3 samp_4] .* [lo lo lo lo] .* [w w w w]);

If the system has a maximum sample buffer size of K*N samples, then the user may select to capture (1, 2, 3 or ... K) * N samples, depending how fast sweep step rate and dynamic range the user wants to have. Thus the system is required to store only on set of windowed sin and windowed cos tables of N samples for all different sweep rates.

I was originally thinking that the system would provide four different sweep rates, for example 0.5K, 1K, 2K and 4K samples, and the system would need to have four set of windowed sin/cos tables: 0.5K, 1K, 2K and 4K in size. Since this is not very flexible and would waste Flash memory because of four set of tables, I thought that the K*N sample approach would be more flexible and more economic.

Edit: This is something that I do not understand again: While fft(w) and fft(w_all) look as usual, fft([w w w w]) is the "weird" frequency response, with the extra side lobe. Why this fft([w w w w]? At least my intention is more like fft(w)+fft(w)+fft(w)+fft(w), like averaging four different FFTs/DFTs. Since we are capturing in coherent way, we can integrate over all four FFTs/DFTs in order to get better dynamic range, compared to four individual FFTs/DFTs.

[gf]

I was originally thinking that the system would provide four different sweep rates, for example 0.5K, 1K, 2K and 4K samples, and the system would need to have four set of windowed sin/cos tables: 0.5K, 1K, 2K and 4K in size. Since this is not very flexible and would waste Flash memory because of four set of tables, I thought that the K*N sample approach would be more flexible and more economic.

Yes, I understand the reason, but

the result having the intended frequency response were rather this one:

Code: [Select]

N=75; K=4;
w_all = hanning(N*K,"periodic")';
iq_all = sum([samp_1 samp_2 samp_3 samp_4] .* [lo lo lo lo] .* w_all);

while the following calculatioin leads to the "weird" frequency response with the extra side lobes:

Code: [Select]

iq_sum = sum([samp_1 samp_2 samp_3 samp_4] .* [lo lo lo lo] .* [w w w w]);

I don't say that the "weird" freqency response won't work. It still can work, too.
But the latter filter is not equivalent to the former one.

This is something that I do not understand again: While fft(w) and fft(w_all) look as usual, fft([w w w w]) is the "weird" frequency response, with the extra side lobe. Why this fft([w w w w]? At least my intention is more like fft(w)+fft(w)+fft(w)+fft(w), like averaging four different FFTs/DFTs. Since we are capturing in coherent way, we can integrate over all four FFTs/DFTs in order to get better dynamic range, compared to four individual FFTs/DFTs

General form of single DFT term (with window function) is basically

Code: [Select]

sum(SAMPLES .* LO .* WINDOW)
where SAMPLES, LO, WINDOW are row vectors of equal size (-> number of samples).
LO is the complex sine wave. The frequency response of the corresponding filter is fft(WINDOW) ¹⁾.

When you calculate

Code: [Select]

iq_1 = sum(samp_1 .* lo .* w);
iq_2 = sum(samp_2 .* lo .* w);
iq_3 = sum(samp_3 .* lo .* w);
iq_4 = sum(samp_4 .* lo .* w);
iq_sum = iq_1 + iq_2 + iq_3 + iq_4;
which is the same as

Code: [Select]

iq_sum = sum([samp_1 samp_2 samp_3 samp_4] .* [lo lo lo lo] .* [w w w w]);
then you effectively calculate a single DFT term for SAMPLES=[samp_1 samp_2 samp_3 samp_4], LO=[lo lo lo lo] and WINDOW=[w w w w]
I.e. the effecive window function for this single-term DFT calculation is [w w w w] then.
Therefore the frequency response of the corresponding filter is fft([w w w w]).

When you do the following "brute force" calculation where you sweep frequency and calculate the vector-averaged detector response for each frequency point, then you get the same frequency response as from fft([w w w w]). So I think it can't be so wrong.

Code: [Select]

fs = 12000000/14;
freqs = 30000:100:130000;
lo = exp(-1i*2*pi*80000*[0:74]/fs);
w = hanning(75,"periodic")';
w /= sum(w);

H = freqs .* 0;
for i = 1:length(freqs)
  f = freqs(i);
  samples = sin(2*pi*f*[0:299]/fs);
  for j=0:75:299
    H(i) += sum(samples(j+1:j+75) .* lo .* w) / 2;
  end
end

clf
plot(freqs,20*log10(abs(H)));
ylim([-100 0])
xlim([freqs(1) freqs(end)])
grid on


¹⁾ Plot frequency responses. Zero-fill to align different lengths, and to interpolate for a higher freqency resolution.

Code: [Select]


fs = 12000000/14;

% approx. 100Hz resolution for the plot
NFREQ=floor(fs / 100 + 0.5);            % number of frequency points
f = [0:NFREQ-1] / NFREQ * fs - fs / 2;  % frequency scale

% the window functions we want to plot
w = hanning(75, "periodic")';
w_all = hanning(75*4, "periodic")';
w4 = [w w w w];

% normalize to sum==1, in order that we get 0dB for DC
w /= sum(w);
w_all /= sum(w_all);
w4 /= sum(w4);

% zero-fill in order to interpolate and align different lengths
w = [w zeros(1,NFREQ-length(w))];
w_all = [w_all zeros(1,NFREQ-length(w_all))];
w4 = [w4 zeros(1,NFREQ-length(w4))];

clf; hold on;
plot(f,20*log10(abs(fftshift(fft(w)))),";w;")
plot(f,20*log10(abs(fftshift(fft(w_all)))),";w-all;")
plot(f,20*log10(abs(fftshift(fft(w4)))),";w4;")
xlim([-50000 50000])
ylim([-100 0])
grid on



[gf]

Kalvin, here is yet another idea to calculate the 75*K point single-frequency DFT term with Hanning window, for any K > 1, using only 900 bytes of table data.
Note that It is also possible to apply the window function in the frequency domain, after performing the DFT.
fft(a.*b) is equivalent to circular_convolution(fft(a), fft(b)), consequently fft(samples.*window) == circular_convolution(fft(samples), fft(window)).
The DFT of a periodic Hanning window fft(hanning(M,"periodic")')/M is always [ 0.5 -0.25 0 ... 0 -0.25 ], for any M >= 3.
This means the Hanning window can easily be applied in the frequency domain, by (circular) convolution with [ -0.25 0.5 -0.25 ]. Fortunately, these are only 3 non-zero taps
In order that this convolution can be calculated at the desired frequency, the DFT term needs to be calculated not only for the desired frequency, but also for the lower and upper neighbor bins of the desired frequency.
This is computationally more expensive (a couple of additional instructions per sample), but it saves a significant amount of memory.
The tables need to be calculated individually for each K, so they cannot reside in ROM, but must be in RAM. OTOH, only 75*6*2 = 900 bytes are required.

Edit:

I have to retract my statement partially. A window function can indeed be applied in the frequency domain as well. But unfortunately it does not work out with the sizes as I was thinking. Sorry for my calculation mistake. While the basic priciple is still valid, a phase correction seems unavoidable for each chunk, which comes at the cost of two additional complex multiplications per chunk. My aim is still to keep the algorithm so cheap that it can be calculated in real-time.

[Kalvin]

gf, thank you for your ideas!  Sorry that I have not been active for a week or so, but I have now time to proceed with this little project again.

I just added the documentation for the hardware modifications at Github:
https://github.com/kalvin2021/ltdz-dsp

The documentation for building the actual firmware is still in progress.

[gf]

Sorry for the mistake in my previous statement which I did retract.
I'm still waiting for my order to arrive. Likely I won't have much free time during summer either to play with it when it arrives.

[Kalvin]

gf, You asked whether the LTDZ board could be used for measuring narrow band crystal filters. I read the ADF4351 datasheet and wrote the following script for testing the register configurations. It seems that it is possible in theory to generate frequency steps in 10Hz resolution (or even better), but I have not tested this in practice. Hopefully I didn't miss something.

Code: [Select]

# Reference oscillator frequency
global REF_Hz = 25000000;

# Reference doubler: 0|1
global D = 0;

# Refecence divide by 2: 0|1
global T = 1;

# Preset divide ratio in range [1, 1023]
global R = 100;
 
# Modulus in range [2, 4095]
global MOD = 4095; 

# PFD frequency
global pfd_Hz = REF_Hz * ((1 + D)/(R * (1 + T)));

# Compute ADF4351 INT and FRAC register values for the given frequency.
#
# f       Desired frequency 35MHz <= f < 4400MHz.
# INT     INT-register value.
# FRAC    FRAC-register value.
# rf_Hz   Actual output frequency.
# err_Hz  Frequency error.
#
function [INT, FRAC, rf_Hz, err_Hz] = adf4351_reg(f)
  global REF_Hz;
  global MOD;
  global D;
  global T;
  global R;
  global pfd_Hz;
 
  if f <= 68750000
    RFDIV = 64;
  elseif f <= 137500000
    RFDIV = 32;
  elseif f <= 275000000
    RFDIV = 16;
  elseif f <= 550000000
    RFDIV = 8;
  elseif f <= 1100000000
    RFDIV = 4;
  elseif f <= 2200000000
    RFDIV = 2;
  else
    RFDIV = 1;
  endif

  VCO = f * RFDIV;
  assert(2200000000 <= VCO && VCO <= 4400000000);
 
  INT  = fix(VCO / pfd_Hz);
  assert(75<= INT && INT <= 65535);

  FRAC = round(((VCO / pfd_Hz) - INT) * MOD);
  assert(0 <= FRAC && FRAC <= (MOD-1));

  rf_Hz = round(REF_Hz * ((1 + D)/(R * (1 + T))) * (INT + FRAC / MOD) / RFDIV);

  err_Hz = f - rf_Hz;
endfunction

For example:

Code: [Select]

>> [INT, FRAC, f, err] = adf4351_reg(35000000)
INT =  17920
FRAC = 0
f =  35000000
err = 0
>> [INT, FRAC, f, err] = adf4351_reg(35000001)
INT =  17920
FRAC =  2
f =  35000001
err = 0
>> [INT, FRAC, f, err] = adf4351_reg(35000002)
INT =  17920
FRAC =  4
f =  35000002
err = 0


[gf]

R=100 is quite much. I wonder what's the phase noise then. My understanding is: Lower f_PFD -> larger INT -> more phase noise. To mitigate phase noise then, I also wonder whether the loop filter bandwidth would need to be reduced (significantly), at the expense of a larger lock time.
See also https://www.analog.com/media/en/training-seminars/tutorials/MT-086.pdf

[Kalvin]

R=100 is quite much. I wonder what's the phase noise then. My understanding is: Lower f_PFD -> larger INT -> more phase noise. To mitigate phase noise then, I also wonder whether the loop filter bandwidth would need to be reduced (significantly), at the expense of a larger lock time.
See also https://www.analog.com/media/en/training-seminars/tutorials/MT-086.pdf

Good points! I did not consider the performance at this point, so I overlooked that issue. It is possible to experiment with some other R values in range [10, 150] as well. The point here is that it is quite possible to perform sweeps with smaller steps. For limited frequency range(s), the R can be hand-selected so that the performance will be optimized in that regard.

[Kalvin]

Added a simple Python/Tkinter-utility for reading and plotting LTDZ ADC sample buffer data. This is a development tool for the project development purposes, not intended to be used as a general purpose spectrum analyzer or network analyzer tool. The sweep data can be exported to MATLAB and GNU/Octave for later analysis.

Project page:
https://github.com/kalvin2021/ltdz-dsp

Edit: Changed Matplotlib style/theme.

[Kalvin]

Added the firmware build instructions in the project page at Github:
https://github.com/kalvin2021/ltdz-dsp