I have tried adding a 2200uf capacitor right at the main voltage rail next to the mosfet as shown in the image below. I also have a current transformer to measure the capacitor current.
The output waveform improves with transformer still connected when electrolytic cap is added. CH1 : Full bridge output voltage CH2: Electrolytic capacitor current.
The problem with this is: the electrolytic cap gets warm under very light loading of the full bridge. At high loads the current through the capacitor was about 30 amps at the peak. The capacitor was very hot. If adding more capacitance to the supply rail would improve the ringing, what kind of capacitor should I use? Would a larger film capacitor help the ringing? Is the ringing a layout problem? If so, should the pcb power traces be shorter?
Yes, you have found precisely the problem.
Consider this: the full AC current, from the load, must flow into and out of the supply rails (give or take phase). Any loop area, between DC+, switching nodes, and DC-, is inductance that must be accounted for. Any load current, manifest as DC supply ripple, must be accounted for.
That the electrolytic gets hot, is proof that you need more of them, not less! A 30A load draws, well, about 30A from the supply, so you need 30A worth of capacitors there. (It's actually more like 15A because of switching back and forth, so that's nice at least.)
Note that stray inductance is proportional to length (of the wires/traces used, for a given [constant] cross-section between +/- wires), or roughly to enclosed area (of a generic loop). Every mm of trace length you introduce, is another nH or so of stray inductance, and that puts V = L * dI/dt right in series with that trace and anything connecting to it.
The full bridge doesn't win you anything here. Treat it as two half-bridges sharing a supply. The supply ripple current is about the same. The dI/dt within each half bridge must be managed -- that's where your ringing is coming from. The two half-bridges can be kind of anywhere, as long as you mind that AC load current is flowing between them -- in one and out the other. I guess you aren't minding inductance in this path as much, given the long leads to the transformer, and the apparently simple windup of the transformer. (Or maybe that's the next issue to look at, I don't know.)
The root problem is also a lesson: current does not simply flow in conductors. AC current flows
between conductors, as a transmission line. There is always a nearest opposing conductor, which the "image" or return current flows in.
The impedance of that transmission line can be easily calculated for standard cross-sections, and hand-waved for intermediate cases.
The transmission line impedance and length directly gives the stray inductance of the design. (Stray inductance is usually the desired result, because the switching impedance, Vsupply / Ipeak, is a lot lower than the transmission line impedance, and so inductance dominates. But when it's the other way (in low current, high voltage applications), capacitance dominates!)
You're probably running too low of an impedance even for this to be a slam-dunk solution, but it will still help greatly: rip up your layout, and change to a ground plane based design. Easiest, add internal layers for BUS+ and BUS-, and swallow the cost of a 4-layer board (which really isn't much at all, these days). Now the current loop is between, not side by side traces (the highest impedance geometry you can have!), but between a trace flat against a ground plane (the lowest you can have!). The side-by-side case might have an impedance of 100-150 ohms; the ground-plane case, more like 20. That a five-fold (or better) reduction, which means the ringing frequency will be at least sqrt(5) times higher, and maybe its impedance (the resonant impedance is sqrt(L/C), where L = stray inductance and C = Coss of the off-side transistor) will be low enough that it becomes better damped.
If you insist on a 2-layer build, then reserve one layer for solid BUS- (say) fill, and route everything on the other layer if at all possible. You inevitably have to use vias to the opposite side to cross traces, which must be done away from the ground-return path, and with as little area cut out of the pour as possible. The geometry isn't as good here (it is flat-facing trace to ground plane, but the distance is the whole board thickness, not to an internal plane, so probably 40-100 ohms transmission line impedance), so you may not be able to see much benefit.
Much better: place the capacitors in a row, in front of the transistors. You
can't route away the inductance here, the only thing you can do is improve the placement so the routing doesn't need to try.
Again, you need about 15A worth of caps, so shop for low-ESR types that specify their ripple current, and buy enough of them to put in parallel to handle that.
One final note: TO-220 transistor leads are about 10nH alone, soldered into the board. About double that, hanging out on those terminal blocks. This is significant already! There's not much you can do about this, except changing things much more dramatically. Example: using D2PAK or DFN package SMT transistors with little or no lead length (about 5 and 2nH respectively), and soldering them to the heatsink, which is also the PCB. This takes much more layout optimization, of course, and probably can't be done without 4 layers, as much for routing as for heat-sinking purposes. That would be the preferred modern approach (not that it's necessarily the easiest or best method for your application!).
Tim