You should make an effort to support the discussion
Sorry I don't have time for checking anything that is after the mod (for me does not worth the trouble since it requires getting rid of the 1.8V on pin 24 and it seems that it can become a pain; more on this later).
Before the mod case I've already made a quick check (no screenshots though), but with rather wide bandwidth (so anything below the -70dBm noise floor was not visible to me).
CH0/1 had a -40dBm 125 MHz signal (5th harmonic of the clock). Can't remember anymore which, but one of the CH2/3 channels was entirely clean and the other one had just above the noise floor 125MHz again.
Note: For this exercise I've directly connected the board output to the input of my SA that has 50 Ohm impedance, so I've followed your route of loading the output of the board with 50 Ohm although I'm still not convinced that this is the only route to follow.
In short: The designer could have done better...
I provided the S11 chart in Part 2. I measured the filter components and placed in a simulator, the simulator produced the chart. I could not measure because don't have access to my equipment until shack renovation is complete. Do you know the meaning of S11?
I think I more or less understand it, so since the filter (at least in simulation) does not have sink component (resistive that dissipate, radiating whatever) to be able to filter it has to reflect.
Thus where reflected signal is the same as the input (0dB) it filters all the signal and where reflected power is less (- whatever dB) it's less because some of the signal passed the filter. The less reflected, the more passed.
I still don't get though where the need for 50 Ohm impedance matching is coming, still don't know what you've used in the simulation what it would change if a high impedance load is used etc. (probably my fault).
Couple of notes still (somewhat loosely related).
1) Since the output of the DAC follows the sinc function (I guess you know, but for those readers who don't it's worth checking MT-085 from ADI; I remember also a longer variant but could not quickly find it) probably the filter is designed in a way it's designed to compensate somewhat the loss. So that's why the signal output bandwidth is not 200 MHz, as in sampling the rule of the thumb is that sampling should be at least 2.5 times the max. signal (exception is digital filters, next point).
2) Since good quality (minimal parasitic) and precise components are expensive the escape route is DSP where precision comes almost free (at least nowadays) but with RLC components one should rather lower than raise the bar.
So in short: Maybe the anti-aliasing filter -3dB point is around 200 MHz but signal probably drops around 3dB even below 200 MHz (due to DAC output drop) and in my view you should never try to go above 200 MHz of a 500 MHz sampling system (aim actually something like 30% of the sampling rate and that is also the case here; btw. so do you know why I assume based on your chart that this board is designed to be used till around 160-170 MHz?).
as I said the original oscillator was not a TCXO.
Quick reading/quick writing/quick checking (sorry). If just a crystal oscillator is used (which is beyond me why anyone is still using crystal probably even the price advantage is negligible nowadays) it can explain a lot of things. In case of crystal used to have reasonable rise time the signal level must be rather high and clipping (done by the 9959) is normal. Since for things I build I use TCXO only, to me this kind of setup was not known.
So it has power (probably the reason for DC separation), could do everything by its own but still somewhat relying on 9959 crystal input/mode.
This can be a real pain point with this board. To fix it the 1.8V has to be removed from pin 24 (there is a leadless packaged chip...) but it seems that the designer has not really designed for this option.(?) Normally a minor re soldering is required (sometimes not even that), looking on the components I've assumed that to use external clock source only removal of the 0 Ohm resistor (close to the crystal oscillator) is required (some other boards have solder pads, even others jumper). If this is not the case it's really a bad design. I can't tell how much I miss a schematic here...
This kind of chips are best clocked externally without any PLL usage (to keep phase noise reasonable), this is very well spelled out in the datasheet.