Idea for improving LTDZ Spectrum Analyzer & Tracking Generator 35MHz-4.4GHz

[Kalvin]

   
x_re = x .* cos(2.0 * pi * Fs_Hz * t)';
x_im = x .* sin(2.0 * pi * Fs_Hz * t)';

Sorry, I don't understand what this is supposed to do

  Too tired after workday. Unfortunately it seemed work, but for wrong reasons. I wrote that script from collection of the manual steps I had taken, and expanded some of the functions that I have. Obviously an error crept in, and the sin/cos generators had wrong argument (Fs_Hz vs Lo_Hz). My intention was to perform quadrature correlation between the sin/cos vectors and the signal to be detected, and finally calculate the RMS from the x_re and x_im vectors.

Here is the GNU Octave script again with the corrections:

Code: [Select]

pkg load signal;

# System's MCU oscillator frequency
global XTAL = 72000000;

# System's default sampling rate
global Fs_Hz = (XTAL/6/14);

# Local oscillator frequency
global Lo_Hz = 15000;

# 0dB level
global ref_dB  = 76.260551;

# Analyze the signal level from the given file.
# Returns signal level in dB.
function dB = level_dB(filename)
  global Fs_Hz;
  global Lo_Hz;
  global ref_dB;
  Fn_Hz = Fs_Hz/2; 
  bw_Hz = 1000;

  # Create a bandpass filter for the LO signal
  [b, a] = cheby1(2, 0.1, [(Lo_Hz-bw_Hz/2)/Fn_Hz, (Lo_Hz+bw_Hz/2)/Fn_Hz]);
 
  x = load(filename);
 
  NS = length(x); 
  ts = 1 / Fs_Hz;
  t = ts * [0 : NS-1];

  # Apply the bandpass filter to the signal
  x = filter(b, a, x);
     
  # Quadrature correlator
  x_re = x .* cos(2.0 * pi * Lo_Hz * t)'; 
  x_im = x .* sin(2.0 * pi * Lo_Hz * t)';
   
  # Compute RMS of the detected signal
 
  %rms_sum = 0;
  %for n = 1:NS
  %  rms_sum += x_re(n)**2 + x_im(n)**2;
  %endfor
 
  rms_sum = sum(x_re.**2 + x_im.**2);
  rms = sqrt(rms_sum/NS);
 
  # Convert the RMS level to dB
  dB = 20*log10(rms) - ref_dB;
 
  printf("%-30s: %f dB\n", filename, dB);
endfunction

level_dB("tg-15kHz-0dB-after-caps.dat");
level_dB("tg-15kHz-10dB-after-caps.dat");
level_dB("tg-15kHz-30dB-after-caps.dat");
level_dB("tg-15kHz-50dB-after-caps.dat");
level_dB("tg-15kHz-70dB-after-caps.dat");
level_dB("tg-15kHz-80dB-after-caps.dat");

Giving these results, suggesting that close 80dB of dynamic range could be possible in the network analyzer mode:

Code: [Select]

# With x = filter(b, a, x) applied
tg-15kHz-0dB-after-caps.dat   : -1.059714 dB
tg-15kHz-10dB-after-caps.dat  : -10.000000 dB
tg-15kHz-30dB-after-caps.dat  : -30.113738 dB
tg-15kHz-50dB-after-caps.dat  : -50.100698 dB
tg-15kHz-70dB-after-caps.dat  : -70.078363 dB
tg-15kHz-80dB-after-caps.dat  : -78.857630 dB

I was playing with the idea of quadrature correlator earlier in the evening, and it seemed to work for some extent: Signals from "0dB" to "-50dB" worked alright, but signals "-70dB" and "-80dB" did not work any more. This can be observed when running the same function without the x = filter(b, a, x) statement produces the following results:

Code: [Select]

# Without x = filter(b, a, x) applied
tg-15kHz-0dB-after-caps.dat   : -0.478256 dB
tg-15kHz-10dB-after-caps.dat  : -9.552529 dB
tg-15kHz-30dB-after-caps.dat  : -29.641191 dB
tg-15kHz-50dB-after-caps.dat  : -48.463008 dB
tg-15kHz-70dB-after-caps.dat  : -53.816233 dB
tg-15kHz-80dB-after-caps.dat  : -54.793700 dB

Then I had an idea that probably adding a band-pass filter before quadrature correlator would give some benefits, and it seemed to work. Yes, it seemed to work alright, but this idea of using the bandpass filter works even without the quadrature correlator. This was not my intention in the first place. I wanted to test whether the quadrature correlator would work as a signal detector for weak signal levels as well.

[gf]

I was playing with the idea of quadrature correlator earlier in the evening, and it seemed to work for some extent: Signals from "0dB" to "-50dB" worked alright, but signals "-70dB" and "-80dB" did not work any more. This can be observed when running the same function without the x = filter(b, a, x) statement produces the following results:

After a few correction it works fine, though, withput any pre-filtering

tg-15kHz-0dB-after-caps.dat   : 0.000000 dB
tg-15kHz-10dB-after-caps.dat  : -8.935213 dB
tg-15kHz-30dB-after-caps.dat  : -29.040368 dB
tg-15kHz-50dB-after-caps.dat  : -49.056581 dB
tg-15kHz-70dB-after-caps.dat  : -69.483305 dB
tg-15kHz-80dB-after-caps.dat  : -80.770373 dB

And it works even well with rectangular window, although a rectangular window does not suppress the spurs as well as other window functions would do. OTOH, a rectangular window has the lowest  noise bandwith of all window functions. Eventually the choice of the window functions it is a trade-off between ENBW and stopband rejection. For this particular use case I tend to use Hann or Hamming. And btw, your Chebycheff filter certainly has a lager noise bandwidth than a 4k point Hann or Hamming window (-> 5...6 dB higher average noise floor), but this happens to be mitigated by the fact that RMS averaging reduces the variance of the noise floor, while vector averaging does not.

Code: [Select]

pkg load signal;

# System's MCU oscillator frequency
global XTAL = 72000000;

# System's default sampling rate
global Fs_Hz = (XTAL/6/14);

# Local oscillator frequency
global Lo_Hz = 15000;

# 0dB level
%global ref_dB  = 76.260551;
global ref_dB = 108.561235;

# Analyze the signal level from the given file.
# Returns signal level in dB.
function dB = level_dB(filename)
  global Fs_Hz;
  global Lo_Hz;
  global ref_dB;
  Fn_Hz = Fs_Hz/2;
  bw_Hz = 1000;

  # Create a bandpass filter for the LO signal
  [b, a] = cheby1(2, 0.1, [(Lo_Hz-bw_Hz/2)/Fn_Hz, (Lo_Hz+bw_Hz/2)/Fn_Hz]);
 
  x = load(filename);

  % Choose a window size which is an integral multiple of 15kHz periods
  % in order to place zeros at the harmonics of 15kHz.
  % (particularly useful if only a rectangular window with a low stop band rejection (high side lobes) is used)
  x = x(1:4000);
 
  NS = length(x);
  ts = 1 / Fs_Hz;
  t = ts * [0 : NS-1];

  # Apply the bandpass filter to the signal
  %x = filter(b, a, x);
     
  # Quadrature correlator
  x_re = x .* cos(2.0 * pi * Lo_Hz * t)';
  x_im = x .* -sin(2.0 * pi * Lo_Hz * t)';
   
  # Compute RMS of the detected signal
 
  %rms_sum = 0;
  %for n = 1:NS
  %  rms_sum += x_re(n)**2 + x_im(n)**2;
  %endfor
 
  % calculate magnitude of vector sum
  rms_sum = sum(x_re) .** 2 + sum(x_im) .** 2;
  rms = sqrt(rms_sum/NS);
 
  # Convert the RMS level to dB
  dB = 20*log10(rms) - ref_dB;
 
  printf("%-30s: %f dB\n", filename, dB);
endfunction

level_dB("tg-15kHz-0dB-after-caps.dat");
level_dB("tg-15kHz-10dB-after-caps.dat");
level_dB("tg-15kHz-30dB-after-caps.dat");
level_dB("tg-15kHz-50dB-after-caps.dat");
level_dB("tg-15kHz-70dB-after-caps.dat");
level_dB("tg-15kHz-80dB-after-caps.dat");


Edit:

And I'm still complaining a bit about the fashion how you apply the IIR filter to the samples, which are way too short compared to the impulse response of the filter. The resulting truncation has the consequence that the effective frequency response of the filter (in conjunction with the RMS averaging detector) deviates significantly from the theoretical Chebycheff response. See attachment: Ideal response, and effective response at different phase. The effective result is still reasonable, but you need to be aware that - as a consequence of the truncation - your filter does no longer work as it was designed with cheby1().

Edit:

For comparison I've added the frequency responses you get from the coherent detector with rectangular (yellow), Hamming (orange) or Hanning window (blue). As you can see, the noise bandwith of all three ones is clearly narrower than your Cheby, and for Hanning and Hamming, the stop band rejection is better, too. In fact the filter truncation ruins the inherently large stop band rejection of the Chebycheff filter, so that at the end it is effectively not better than a rectangular window. The truncated Cheby still has retained its wider bandwidth, but that's not what we want either -- eventually we rather want a narrow (equivalent noise) bandwidth.

[Kalvin]

I was playing with the idea of quadrature correlator earlier in the evening, and it seemed to work for some extent: Signals from "0dB" to "-50dB" worked alright, but signals "-70dB" and "-80dB" did not work any more. This can be observed when running the same function without the x = filter(b, a, x) statement produces the following results:

After a few correction it works fine, though, withput any pre-filtering

Pretty stupid error from me, thanks for helping me out!  The whole idea in complex correlator/detector is to keep the orthogonal Q and I signals separate from each other until computing the RMS, so my error is really fundamental.

I have implemented bandpass-based network analyzer in the firmware, but I am not really happy with its computing time performance. The dynamic range seems to be around 80dB or so, which is not too bad. The noise level at -80dB is quite high, but that is expected.

Your suggestion of using complex correlator/detector is the way to go in terms of computing time performance and simplicity.

[Kalvin]

Here are initial results for the LTDZ dynamic range in the network analyzer operating mode. The measurements used 4096 sample data for each sweep data point, and quadrature correlator with Hanning windowing.

Attachment #1 shows the network analyzer linearity using the following attenuations: 6dB, 16dB, 26dB, 36dB, 46dB, 56dB, and no loop connected (ie. noise floor). The linearity is quite good, and the noise floor is somewhere below -80dB.

Attachment #2 shows the network analyzer linearity using the following attenuations: 46dB, 56dB, 66dB, 76dB, 86dB, and no loop connected (ie. noise floor). The linearity is still quite good, although there is some fluctuation at the weaker signal levels. The useful dynamic range is somewhere around 80dB or a bit less.

These initial results are really encouraging as the dynamic range of the network analyzer operating mode has increased from 50dB -> 80dB with only minimal hardware modifications and some simple digital signal processing.

Note: The dynamic range in the spectrum analyzer mode is still only 50dB, and it is very difficult to improve it without extensive hardware modifications.

Edit: It is possible to reduce the noise at the lower signal levels by performing some filtering across the consecutive sweep data points.

Edit: Here is the current firmware memory usage report:

Code: [Select]

   text    data     bss     dec     hex filename
  29096     100    9800   38996    9854 build/ltdz_firmware.elf

Edit: It is possible to increase the sweep rate by using two separate ADC sample buffers (4096 samples each), and utilizing the fact that the DMA can fill the other sample buffer while the MCU is processing the previous sample buffer.

[gf]

The variability of the green and blue traces at -65dB and -75dB does not look like random noise only, but obviously contains a systematic component, which is also correlated between this green and blue trace. Good question where it comes from, and whether it is rather a function of frequency, or a function of time

[Kalvin]

And I'm still complaining a bit about the fashion how you apply the IIR filter to the samples, which are way too short compared to the impulse response of the filter. The resulting truncation has the consequence that the effective frequency response of the filter (in conjunction with the RMS averaging detector) deviates significantly from the theoretical Chebycheff response. See attachment: Ideal response, and effective response at different phase. The effective result is still reasonable, but you need to be aware that - as a consequence of the truncation - your filter does no longer work as it was designed with cheby1().

Thank you for your simulations, they are most helpful to see and understand the shortcomings of applying a signal to an [IIR] filter that is shorter than the impulse response of the filter. At the start of this project I did not really care how the IIR filter would perform, other than that IIR filters can be readily designed using GNU Octave, they are quite easy to implement, and that they would outperform the original design. Your contributions helped to understand that there is a better, more robust, more elegant, more powerful and even simpler way of performing the same task of detecting signal power level at presence of a noise.

For comparison I've added the frequency responses you get from the coherent detector with rectangular (yellow), Hamming (orange) or Hanning window (blue). As you can see, the noise bandwith of all three ones is clearly narrower than your Cheby, and for Hanning and Hamming, the stop band rejection is better, too. In fact the filter truncation ruins the inherently large stop band rejection of the Chebycheff filter, so that at the end it is effectively not better than a rectangular window. The truncated Cheby still has retained its wider bandwidth, but that's not what we want either -- eventually we rather want a narrow (equivalent noise) bandwidth.

I selected Hann window for the windowing function. Probably any other window would have worked fine too. I considered using a plain rectangular window (ie. no window), but I decided to go for proper windowing. It is quite easy to change things if necessary, after all it is just a matter of generating new lookup-tables. Currently the RX LO offset is set to 80kHz instead of 15 kHz due to the fact the it looked like the noise level would be somewhat less at the higher frequencies and the LO frequency offset needs to be N*8kHz due to the possible ADF4351 frequency step sizes.

[gf]

Currently the RX LO offset is set to 80kHz instead of 15 kHz due to the fact the it looked like the noise level would be somewhat less at the higher frequencies and the LO frequency offset needs to be N*8kHz due to the possible ADF4351 frequency step sizes.

Could you please post ADC samples with 80kHz IF?
I'm interested how the (aliased) harmonics are distributed in the spectrum.

[Kalvin]

Currently the RX LO offset is set to 80kHz instead of 15 kHz due to the fact the it looked like the noise level would be somewhat less at the higher frequencies and the LO frequency offset needs to be N*8kHz due to the possible ADF4351 frequency step sizes.

Could you please post ADC samples with 80kHz IF?
I'm interested how the (aliased) harmonics are distributed in the spectrum.

Sure, please find the signals as attachment. Signal "noisefloor" is TG off and no loop from TG to RX. Signal "noloop" is TG on but no loop from TG to RX. Signal "0dB" is TG looped directly to RX without any attenuation. Signal "-10dB" is TG looped to RX through 10dB attenuator etc. TG frequency is f=60032000_Hz, LO offset is 80kHz, thus LO frequency is TG + 80kHz, Fs=857142.8571428572_Hz.

[Kalvin]

I am thinking about the strategies for how to increase the available sweep rate. Currently the firmware is capturing 4096 samples for each sweep step, and the device is able to perform approx. 1000 sweep steps in 8 seconds ie. approx 125 steps in a second. That is not too bad, considering that the dynamic range is now close to 80dB in the network analyzer mode.

I have also tried with 1024 only samples per sweep step in network analyzer mode, and the results aren't too bad either. It would be great if the user could select between three sweep rates: fast, medium, slow. Corresponding sample sizes would be 1024, 2048 and 4096 samples.

Below are listed sample buffer acquisition times @857kHz ADC sample rate for different sample buffer sizes, and corresponding theoretical maximum sweep rates, including the default PLL lock-in time of 750us:

1024 samples: 1.2ms + PLL lock wait time 750us = 1.3ms = 769 sweep steps/s.
2048 samples: 2.4ms + PLL lock wait time 750us = 2.5ms = 384 sweep steps/s.
4096 samples: 4.8ms + PLL lock wait time 750us = 4.9ms = 192 sweep steps/s.

There is still quite a lot of Flash available in the MCU for the lookup-tables, so it would be quite simple task to provide an option for three different sample buffer sizes, so the user could select between three different sweep rates.

In the current implementation the sample buffer acquisition and sample buffer processing is performed back-to-back, but that can be streamlined in the future by utilizing the ADC+DMA, which will produce a system which would be able to achieve close to theoretical sweep step rates.

[gf]

The 80kHz samples look OK, too. The alias frequencies of the harmonics are still far enough apart from the desired 80kHz signal, in order that they can be rejected, and furthermore the analog filter has attenuated them enough to drown in the noise floor.

Half number of samples is supposed to raise the noise floor by 3dB (at least theoretically, for white noise), i.e. 1040 samples leads to a 6dB higher noise foor then.
I wonder whether so many frequency points are needed in NA mode, since I don't expect the device to be suitable for measuring very narrow-band DUTs, like e.g. crystal filters.

Edit:

Separating sin/cos tables and window function tables saves memory, at the cost of only 2 or 3 more instructions per sample.
Sin and cos table need only 75 entries (7 cycles) each when the IF is 80kHz. However, size of the window function table needs to be the size of the window (i.e. number of samples).
For 4k + 2k + 1k capture sizes this makes a total of 4096 + 2048 + 1024 + 75 + 75 entries, or 14636 bytes.

[Kalvin]

The 80kHz samples look OK, too. The alias frequencies of the harmonics are still far enough apart from the desired 80kHz signal, in order that they can be rejected, and furthermore the analog filter has attenuated them enough to drown in the noise floor.

Half number of samples is supposed to raise the noise floor by 3dB (at least theoretically, for white noise), i.e. 1040 samples leads to a 6dB higher noise foor then.
I wonder whether so many frequency points are needed in NA mode, since I don't expect the device to be suitable for measuring very narrow-band DUTs, like e.g. crystal filters.

Probably the current firmware with the default ADF4351 settings may not be suitable for high-Q crystal filter measurement. In frequency range 35MHz-68MHz, the minimum frequency step size is documented to be 125Hz (I have not verified that). ADF4351 is quite versatile Fractional-PLL, so there may be some settings which could provide finer frequency step size.

/** ADF4351 frequency ranges and step sizes */
#define ADF4351_RANGE_1_MIN_DIV_10  (3500000UL)
#define ADF4351_RANGE_1_STEP_Hz     (125)
#define ADF4351_RANGE_2_MIN_DIV_10  (6875000UL)
#define ADF4351_RANGE_2_STEP_Hz     (250)
#define ADF4351_RANGE_3_MIN_DIV_10  (13750000UL)
#define ADF4351_RANGE_3_STEP_Hz     (500)
#define ADF4351_RANGE_4_MIN_DIV_10  (27500000UL)
#define ADF4351_RANGE_4_STEP_Hz     (1000)
#define ADF4351_RANGE_5_MIN_DIV_10  (55000000UL)
#define ADF4351_RANGE_5_STEP_Hz     (2000)
#define ADF4351_RANGE_6_MIN_DIV_10  (110000000UL)
#define ADF4351_RANGE_6_STEP_Hz     (4000)
#define ADF4351_RANGE_7_MIN_DIV_10  (220000000UL)
#define ADF4351_RANGE_7_STEP_Hz     (8000)
#define ADF4351_RANGE_7_MAX_DIV_10  (440000000UL)

[Kalvin]

Separating sin/cos tables and window function tables saves memory, at the cost of only 2 or 3 more instructions per sample.
Sin and cos table need only 75 entries (7 cycles) each when the IF is 80kHz. However, size of the window function table needs to be the size of the window (i.e. number of samples).
For 4k + 2k + 1k capture sizes this makes a total of 4096 + 2048 + 1024 + 75 + 75 entries, or 14636 bytes.

I was a bit lazy and Flash memory was not an issue, so I created separate tables for 80kHz sin and cos values with the Hann window-function applied. It also helped to perform simple loop unrolling as everything is dividable with 4. I know, premature optimization is the root of all evil ...

Edit: Another option is to create sin/cos tables with length of 13*75 =  975 or so with the desired window function applied, and then run the table 1x, 2x or 4x depending of the number of samples captured (1024, 2048, 4096). The sample counts not have to be 2**N any more in this case. The SNR will be a bit less with 2K and 4K sample buffer sizes due to windowing compared to full 2K and 4K tables. Since 975 is multiple of the 80kHz LO frequency, windowing may not be needed at all, so there wouldn not be any penalty with the rectangular window. However, ss you mentioned, the spur rejection won't that good when using rectangular window.

[gf]

I was also thinking whether the window function table could be possibly interpolated, in order to capture even more than 4k samples for one reading (streaming accumulation, w/o storing all captured samples in the first place). This would need to be done in real-time then, but I think it is still feasible in 84 clock cycles/sample. Would not make the sweep faster, though, but demand even more patience.

Edit: 64k samples instead of 4k were (theoretically) supposed to lower the noise floor by 12dB.

[Kalvin]

I was also thinking whether the window function table could be possibly interpolated, in order to capture even more than 4k samples for one reading (streaming accumulation, w/o storing all captured samples in the first place). This would need to be done in real-time then, but I think it is still feasible in 84 clock cycles/sample. Would not make the sweep faster, though, but demand even more patience.

With rectangular window this might be possible with some loop unrolling and hand-optimization, since all that is needed are sin/cos tables with N*75 samples.

Edit: N=1 or 2, depending whether the tables need to be even or odd.

[gf]

Another option is to create sin/cos tables with length of 13*75 =  975 or so with the desired window function applied, and then run the table 1x, 2x or 4x depending of the number of samples captured (1024, 2048, 4096).

This won't work. The window function must span the whole window size. And the frequency of the sin/cos must remain 80kHz. You can't sub-sample a combined sin/cos table with integrated window function. A "periodic" window function can't be subsampled either w/o interpolation.

[Kalvin]

Another option is to create sin/cos tables with length of 13*75 =  975 or so with the desired window function applied, and then run the table 1x, 2x or 4x depending of the number of samples captured (1024, 2048, 4096).

This won't work. The window function must span the whole window size. And the frequency of the sin/cos must remain 80kHz. You can't sub-sample a combined sin/cos table with integrated window function. A "periodic" window function can't be subsampled either w/o interpolation.

My intention was to apply the sin/cos tables twice or four times back-to-back over one sample buffer. For example, if the sin/cos-tables with windowing were 975-samples in length, the 4K sample buffer would simply run the 975-sample sin/cos table four times (kind of modulo 975).

Edit: Essentially it would be like running four correlations back-to-back applying the same sin/cos tables with windowing for each correlation at a time.

Edit2: Using rectangular window, the sin/cos tables can be used simply as sin/cos lookup-tables with length of (for example) 900 or 975, depending whether the tables lengths need to be odd or even.

[gf]

With rectangular window this might be possible with some loop unrolling and hand-optimization, since all that is needed are sin/cos tables with N*75 samples.

Sin/cos tables need only 75 samples, then they repeat circularly from the beginning.
Rectangular window would require to accumulate not 2^N samples, but N*75 samples, in order to reject harmonics.
But there is quite some "garbage" in the spectrum which should be rejected, too (particularly at low levels), so I would rather not use rectangular.

My intention was to apply the sin/cos tables twice or four times back-to-back over one sample buffer. For example, if the sin/cos-tables with windowing were 975-samples in length, the 4K sample buffer would simply run the 975-sample sin/cos table four times (kind of modulo 975).

This does not work, as it would repeat the window function as well. But the window function must not be repeated 4x, but it must span all accumulated of samples.

[Kalvin]

My intention was to apply the sin/cos tables twice or four times back-to-back over one sample buffer. For example, if the sin/cos-tables with windowing were 975-samples in length, the 4K sample buffer would simply run the 975-sample sin/cos table four times (kind of modulo 975).

This does not work, as it would repeat the window function as well. But the window function must not be repeated 4x, but it must span all accumulated of samples.

For 4K sample buffer, this would be identical to situation that the system would perform four consecutive signal power estimations each 975 samples, and then combine the four results. The penalty is that the available SNR would be less than using full the 4K tables. 

Edit: The windowed sin/cos-table sizes need to be N*75 and even, so the table sizes need to be 900 for 1K buffer. Next possible size would be 1050, but it won't fit into 1024 sample buffer. But again, the buffer size do not have to be 2**N.

[Kalvin]

Let's say that the system has sin/cos tables for 80kHz with windowing applied, and the table sizes are 900 samples each (N*75 samples, and size is even).

1. The system will sample the mixer output into a buffer of 900 samples, and the system will compute the quadrature correlation and vector sums for I and Q for the 900 samples.

2. The system will sample the mixer output into a buffer of 1800 samples. First, the system will compute the quadrature correlation and vector sum for I and Q the first 900 samples. Then, the system will  compute the quadrature correlation and vector sum for I and Q for the remaining 900 samples. The same windowed sin/cos tables are "reused" for the both steps.

Won't this work? 

[gf]

For 4K sample buffer, this would be identical to situation that the system would perform four consecutive signal power estimations each 975 samples, and then combine the four results.
The penalty is that the available SNR would be less than using full the 4K tables. 

Vector averaging of 16k samples is still different from RMS averaging 4 readings, obtained from 4k samples each.

But I calculated the frequency response of a 4x repeated Hanning window. It has indeed an "unusual" shape (see figure - orange), but if I did not make a calculation mistake, the noise bandwidth seems to be the same as for the as a non-repeated Hamming window with 4x size (blue). I need to check this again. Due to the high side lobe, this wndow won't make sense as analysis window for spectrum display, but your intuition may be right that it can nevertheless work for the use case here (despite the unusual frequency response).

[gf]

Let's say that the system has sin/cos tables for 80kHz with windowing applied, and the table sizes are 900 samples each (N*75 samples, and size is even).

1. The system will sample the mixer output into a buffer of 900 samples, and the system will compute the quadrature correlation and vector sums for I and Q for the 900 samples.

2. The system will sample the mixer output into a buffer of 1800 samples. First, the system will compute the quadrature correlation and vector sum for I and Q the first 900 samples. Then, the system will  compute the quadrature correlation and vector sum for I and Q for the remaining 900 samples. The same windowed sin/cos tables are "reused" for the both steps.

Won't this work? 

RMS averaging of N readings, of 900 samples each, does certainly work. But with RMS avaraging, you loose phase information. And it does not reduce the noise floor, but only the variance of the noise floor. So the maximum benefit is limited, even if more and more samples are captured. The noise floor can only be lowered by reducing the bandwidth (-> and a longer window function corresponds to a filter with a lower bandwidth).

[Kalvin]

Let's say that the system has sin/cos tables for 80kHz with windowing applied, and the table sizes are 900 samples each (N*75 samples, and size is even).

1. The system will sample the mixer output into a buffer of 900 samples, and the system will compute the quadrature correlation and vector sums for I and Q for the 900 samples.

2. The system will sample the mixer output into a buffer of 1800 samples. First, the system will compute the quadrature correlation and vector sum for I and Q the first 900 samples. Then, the system will  compute the quadrature correlation and vector sum for I and Q for the remaining 900 samples. The same windowed sin/cos tables are "reused" for the both steps.

Won't this work? 

RMS averaging of N readings, of 900 samples each, does certainly work. But with RMS avaraging, you loose phase information. And it does not reduce the noise floor, but only the variance of the noise floor. So the maximum benefit is limited, even if more and more samples are captured. The noise floor can only be lowered by reducing the bandwidth (-> and a longer window function corresponds to a filter with a lower bandwidth).

In this example, the vector sums for I and Q can continue accumulating from the first sample set to the second sample set. Wouldn't this mean that the phase information is still there, and there won't be any other penalty other than what is caused by windowing?

[Kalvin]

For 4K sample buffer, this would be identical to situation that the system would perform four consecutive signal power estimations each 975 samples, and then combine the four results.
The penalty is that the available SNR would be less than using full the 4K tables. 

Vector averaging of 16k samples is still different from RMS averaging 4 readings, obtained from 4k samples each.

But I calculated the frequency response of a 4x repeated Hanning window. It has indeed an "unusual" shape (see figure - orange), but if I did not make a calculation mistake, the noise bandwidth seems to be the same as for the as a non-repeated Hamming window with 4x size (blue). I need to check this again. Due to the high side lobe, this wndow won't make sense as analysis window for spectrum display, but your intuition may be right that it can nevertheless work for the use case here (despite the unusual frequency response).

The spectrum looks kind of weird. 

[gf]

In this example, the vector sums for I and Q can continue accumulating

This implies eventually that you implicitly repeat the shorter (975 samples) window function N times, in order to to form the total window function which spans all N*975 samples. And this N-fold repetition leads to the weird frequency response then. But although the frequency response of this filter looks weird, it can still work for the given purpose, if the signal spectrum contains only noise at the position of the high side lobe (which is likely granted here).

Edit: Btw, the previously plotted freqency response is the low-pass filter equivalent, of course. Mirror it across the Y axis to add the negative frequencies, and then shift it to 80kHz to get the corresponding bandpass.

Edit: Regarding 80kHz: Yet another point to consider is the frequency deviation between the PLL xtal and the MCU xtal. If the (normalized) deviation between the two oscillators is (say) 20ppm, this leads to a frequency error of 0.3Hz when the IF frequency is 15kHz, but at 80kHz IF, the error is already 1.6Hz. However, the filter bandwidth is independent of the IF frequency. This means that the frequency error to bandwidth ratio is higher at 80kHz than at 15kHz. Still no problem yet and 1.6Hz error is still negligible (at least for magnitude estimation). But when we reduce the filter bandwidth more and more (by vector accumulation of more and more samples), in order to get more and more dynamic range, then at some point the frequency error will begin to matter. I'm also unsure whether the frequency deviation of all LTDZ devices is as low as on yours.

Edit: I still wonder whether the spurs are sampling-reated. In order to see whether it makes a difference, could you please change the ADC sampling time (only for this experiment) from 1.5 to 7.5 cycles, capture 4096 "noisefloor" or "noloop" samples (i.e. w/o TG signal applied, at 600kSa/s then) and post them? I'm only interested in the raw samples, so no need to change anything else for this experiment. Btw, I've now ordered a LTDZ board, too, from China, but no idea when it arrives.

[Kalvin]

Edit: I still wonder whether the spurs are sampling-reated. In order to see whether it makes a difference, could you please change the ADC sampling time (only for this experiment) from 1.5 to 7.5 cycles, capture 4096 "noisefloor" or "noloop" samples (i.e. w/o TG signal applied, at 600kSa/s then) and post them? I'm only interested in the raw samples, so no need to change anything else for this experiment.

Please find the signal data attached. I made the measurements with two different LO offset frequencies (80kHz and 56kHz), and two different ADC sample rates (1c5: Fs=857142.8571428572_Hz and 7c5: Fs=600000.0_Hz). The signal "noisefloor" if captured when TG is off, and LO offset = 0Hz (spectrum analyzer mode). The signal "noloop" is captured when TG on, and LO offset = 80kHz/56kHz (network analyzer mode).
 

Btw, I've now ordered a LTDZ board, too, from China, but no idea when it arrives.

This is nice little gadget to play with, with the known limitations. And it seems that this piece of hardware can be improved easily as the firmware source code is available, and the firmware is quite simple and straightforward. Let's hope that your device has also decent quality. Do you also have ST-Link V2 USB-dongle for programming?

I have created two Python/Numpy programs for a) obtaining the ADC samples from the device and b) performing simple frequency sweeps for firmware testing purposes. I will publish those sometime after polishing them a bit more. Otherwise I have been using WinNWT5, WinNWT4 and NWT4000lin running on Linux Wine.