Thanks for all the replies!
Here is a screenshot (attached) of my spice simulation (I didn't want to make the problem description more complicated!).
In brief, the AD9838 is a DDS with high impedance current outputs that are complementary (green trace in plot). The maximum compliance voltage on the pins at 0.8V, so the current (3mA max) is developed across the reisistors, 200R. This is passed to a low pass filter to reduce signal spurs (LT1565, with 650k cutoff, blue trace). I am sure the outputs up to this stage are good because I probed the signal and only see the FFT peak corresponding to the output frequency.
DDS:
http://www.analog.com/media/en/technical-documentation/data-sheets/AD9838.pdfLP:
https://cds.linear.com/docs/en/datasheet/156531fa.pdfHowever, the problem comes in after amplification by the op amp (ideal signal is given in the red trace), that turns the complementary signal into a single ended one, I start seeing peaks at harmonics of the main frequency. It is not driving any load at the moment. Unfortunately, I don't have the scope at the moment so can't show the time domain data, but to my naked eyes, the 500kHz signal looked "good". A strange thing I was seeing when I had higher gain was that the output signal was hitting the rail cutoff much sooner than expected (4.2V vs ~4.8V from datasheet). This may not be related though.
Distortion in an opamp circuit output is actively reduced via the negative feedback, but only to the extent that the available gain allows. GBP comes into play here. As frequency increases, available open loop gain is reduced. As your closed loop gain is increased, the available gain for correcting the output is reduced. You didn't say what your circuit gain is, but with it 10 MHz GBP, at 500 kHz, you have only gain of 20 to play with. If you have gain set to 4 for example then the extra gain is 20/4=5. Then you can expect up to 1/5 or 20% distortion.
TLDR: for 500 kHz signals, you may need way more than 10 MHz GBP.
I was thinking this may be the case but was not sure how to actually relate to the GBP. Currently, the gain is at 6 to give ~3Vp-p. The problem does appear sooner at lower frequency with higher gain. What do you mean in the last sentence referring to the % distortion?