After a few days of break, I come back from the battlefield. I checked that on the secondary side power supply, the one that powers the SG3525 and the operational amplifier and noticed noise and ripple. That's why I'm now powering the secondary side with a separate power supply and this problem is solved.
The next test consisted in using the common mode choke at the output of the inverter along with the 4000uF capacitor package. The voltage feedback signal has been wound on the common common mode choke core with the same as the power supply. The effect? Very bad, big oscillations.
Then, for this configuration above, I used a differential amplifier so that the coupling voltage would be the difference between the output voltage i and the ground terminal. Also the lack of a positive reaction.
Another test is also error voltage read differently but this time without common mode choke in common with power lines. Also, the effect is poor.
I added a separate common mode choke to the differential amplifier independent of this at the output of the inverter, the effect also does not cause much noise and squeaking of the transformer.
The last attempt was to replace the resistor R9 in the feedback loop with a 500k potentiometer. It turns out that there is no point for which the transformer does not write or does not have any strange waveforms at the output of the inverter.
I will still try to replace the operation amplifier with another one. Maybe it will change something.
I did some experimenting with the compensation in my bench supply to find what should be guaranteed stable albiet slow.
In your Half-Bridge_Analog_Controller-5.pdf schematic, R9,C8 would become 5.6K,1uF. R1,C1 are open circuit.
When I do a load transient test, I see a few cycles 200Hz ringing at the output. This is close to the calculated output LC filter resonance of 229Hz, normal and expected.
Passing the output voltage to the control circuit via a CMC definitely should not make things worse. Mine is 5 turns on a small torroidal core, about 50uH.
How fast are your MOSFETs switching?
I was planning to use IRFP460 MOSFETs . The heat-sink I used came from a plasma TV PSU and had a pair of SD20n60 MOSFETs bolted to it.
The plan was to use them while debugging the SMPS instead of risking my IRFP460 MOSFETs .
These MOSFETs must have very low charge causing them to switch way too fast. I had to use 220Ω and 220Ω with diode at the Gate to get the Gate plateau time up to about 500ns.
I didn't bother using the IRFP460 MOSFETs.