Interesting
core datasheet : many informations provided but I do not see the 'Al value' (permeance, whose usual unit is nH/t
2) (*) nor the material mix. Actually, the two informations I needed are just missing(**)

Could not find any information either on the 'Philips FX 2240' core that was mentionned by Jaycar as an equivalent. Bummer.
So, I won't dive into the theorical flux density calculation (max 500 mT) because of the air gap (plastic washer) that significantely alters the part caracteristics.
Just putting 650 VDC onto the secondary will not lead to a valuable measurement because the megohmmeter outputs a DC voltage. So you will just short the voltage supply by the DC resistance of the secondary wire.
It would need a pulsed DC high voltage source to perform the mesurement you suggest.
To check the insulation of your windings, you can try to put about 250 volts between any transformer connexion and the ferrite core wrapped in aluminium foil. If there's an isolation problem from the winding to the ferrite core, it will show up.
The isolation of an enameled wire against a conductive surface can withstand about 100 to 200 VDC.
If we really narrow down to the transformer as the culprit, in-deep mesurements of it's behaviour is not trivial at all especially if it comes to leakage and shorted turns. Maybe just rewinding it with much care and new wire could lead to a positive issue.
I would at first measure some waveforms around the switcher to see if there's not something else that creates the issue.
Core saturation can be guessed by measuring the instantaneous current through the winding.
Indeed, as we don't have the core Al value, inductance can not be calculated, so no relationship with the switcher frequency can be made.
Mesuring the core inductance (LCR meter) would not lead to a trustable result either : a potentially damaged winding + low measurement voltage of the LCR meter cannot provide reliable values for high voltage transformers.
(*) From a top tier manufacturer, one can see the
different Al values (here in mH/1000 turns) of the 'PC26/16' pot core vs 6 different magnetic materials mixes (R, P, F, ...).
(**) Very few manufacturers provide comprehensive data of the magnetic parts they sell.