why not drop the AD8307 altogether, feed the 120 kHz filter directly into the ADC input, and make the STM32F103 do the required math and DSP
Using digital filter slows down STM32 and don't provide any benefit. There is no need for configurable bandwidth. It just make sweep very slow and low dynamic range.
For your satellite spectrum observation purposes the 120 kHz RBW might be ok, but for applications that require more detailed spectral analysis, this 120 kHz RBW is way too wide, and makes the board basically useless.
If one wants to obtain narrower RBW with more spectral details, it is necessary to reduce the RBW from 120 KHz down to 1 KHz or even less, depending of the spectral details one wants to see. Yes, this will slow down the sweep rate, but in applications that requires more spectral details like tuning a narrowband filter or investigating narrow SSB spectrum, this needs to be done either in analog domain (AD8307) or in digital domain (using numerical DSP-algorithms).
Bypassing the AD8307 LOG-detector altogether, feeding the output of the 120 kHz lowpass-filter directly into STM32F103 ADC input, sampling the signal with up to 1Ms/s, and performing the final RBW-filtering and LOG-detector in MCU with some math/DSP, one can select narrow RBW-filter dynamically, and gain some extra dynamic range due to processing gain.
For more advanced spectrum analysis, one can even implement FFT for computing the spectral components, and one can get very narrow RBW and good spectral resolution without slowing down the sweep rate. That is something that really cannot be done with AD8307 if you reduce the RBW for improved spectral resolution.
And AD8307 is logarithmic amplifier with 92 dB dynamic range, so it allows to increase dynamic range by using logarithmic scale. If you remove it, you will get linear scale and bad dynamic range.
<snip>
My LTDZ instance dynamic range is 50 dB. And it's limited by RF frontend design. In order to get better, it needs a new PCB. There is already exist exactly the same hardware with better PCB layout and shielding, but it cost much more.
This is partially true. By taking advantage of the processing gain you can compensate the "seemingly lost" dynamic range of 72dB of a 12-bit ADC when using narrower RBW filters. With the original 120 kHz RBW, you can still get extra dynamic range by oversampling and averaging, but this will obviously slow down the scanning rate as more samples are needed.
As you have stated, the actual obtainable dynamic range may be only 50 dB for a given board/design. Even if AD8307 is technically able to provide 92 dB of dynamic range, in practice you will get much less than that due to the noise floor of the board. In that sense the 12-bit ADC with 72dB of dynamic range is sufficient, and the STM31F103 is able to compute the LOG-detector with 72dB dynamic range in software without any problems.
Some newer boards are said to be able to give up to 70 dB of dynamic range, so the 12-bit ADC is still sufficient for the LOG-detector, and AD8307 does not provide any real benefits. Of course, the signal level needs to be set correctly so that the full dynamic range of ADC can be utilized. With the processing gain obtainable using the narrower RBW and/or oversampling/averaging, one can compensate lost dynamic range to some extent, even if the signal level is not perfect.