Let's assume that we are sampling N samples into a buffer. We have selected N in such way that ADC sampling frequency / LO frequency over N will become integer in all practical terms (for Fs=72Mhz/6/14=857142.8571428572_Hz and LO=80kHz: Fs/LO=10.71428571428, we could select N = 93 * 10.71428571428 = 996.42857142857 = 996, for example). Now, we will compute the DFT for the given LO frequency of 80kHz (using quadrature correlator with pre-computed sin/cos tables with Hann-window). The RMS for this DFT = sqrt(sum(Q)^2 + sum(I)^2). No weird spectrum here.
Now, let's extend this a little, and that we are sampling K*N consecutive samples into a buffer. We will then divide this buffer into K blocks, each size of N samples. Then, we will compute DFT and RMS for each individual blocks of N samples. Finally, we will combine the results from all K blocks into one RMS value. There should not be no weird spectrum here, because we are just averaging K * individual DFT results. Right?
Vector averaging of the individual DFT results turns out to be mathematically equivalent to calculating the DFT over the whole K*N sized window, using a window function which is a K-fold repetition of the shorter N-tap window function. Just write-down the equation for sum that is calculated, for both cases, re-arrange them acordingly, and you'll see that both are the same. Consequence is that the equivalent filter frequency response is eventually that of the K-fold repeated N-tap window (i.e. the "weird" one).
Edit: I mean the DFT-term for the single frequency bin that is considered. The complete K*N sized DFT contains of course more frequency bins.
OTOH, if you calculate the RMS sum (power sum) of the individual DFT results (treating them as if the were independent and not necessarily coherent to each other), then the filter frequency response still corresponds to the (short) N-tap window function. The RMS sum does not contain phase information any more, and it reduces the variance of the noisefloor. It does not lower the noise floor. I.e. even if K approaches infinity, then N and the window function determine the lowest noise floor you can get.
Edit: Actually we'd like the get the filter response of a (non-repeated) window function, which spans the whole N*K window size, since its bandwidth is 1/K of the N-tap window's bandwidth then. But neither the vector nor the RMS averaging do that if the stored window function table has only N taps, and if the table is just indexed and not interpolated.
Edit: The ENBW of the "weird" frequency response is in fact as narrow as the ENBW of the desired frequency response. Unusual are the high side lobes. The main lobe is even narrower, conversely. If the signal spectrum happens to be "empty" (besides random noise) at the side lobe frequency (i.e. no spurs or harmonics), then this window function is still usable for the detector. It just requires a decent frequency planning to ensure that.
Btw, if you are willing to accept a gap at the center of the lobe, then I've a
proposal for SA mode. Actually is is based on the above principle, with K=N=64, and doing RMS averaging of the individual sub-results. I've chosen the frequency plan to exclude the spurs and also the noise near DC. My expectation were a DR improvement of say 15+ dB, compared to simple RMS averaging over all samples. It is still rather wide-band (approx +/- 67kZh @-3dB), in order that the center gap remains small when compared to the total BW.
(Edit: The frequency plan is just an initial draft - there is still room for fine-tuning)
Edit: One more thing:
for Fs=72Mhz/6/14=857142.8571428572_Hz and LO=80kHz: Fs/LO=10.71428571428, we could select N = 93 * 10.71428571428 = 996.42857142857 = 996, for example
In order that you can continue a vector sum coherently after accumulating N samples, by wrapping-around the sin/cos tables, an
exact integral multiple of LO periods need to fint into the N samples. For f
LO = 80kHz, this applies to N=75, or any integral multiple of 75, but not to 996.