Thank you both for your replies!
The MOSFET gates are high impedance with the IC removed.
It is my first time designing such a board, so my layout has probably maaaany improvements. If you see any you'd like to point out, feel free!
The controller works at 150kHz, with a 180 degree phase shift between the 2 sides, therefore generating a 300kHz at Cin and Cout. I have measured this to be actually the case. I wouldn't know how to test the ceramic caps at their used voltage and frequency, but the voltage ripple is pretty much what I expect, so should be fine then right?
The other possibility is the layout, I must confess I cannot understand your plots but maybe thats just me, for example the top layer seems to have copper pour over everything obliterating for example the inductor pads that I cannot see.
Most of the power electronics stuff, i.e. the inductor, is on the 'bottomlayer'. Is it the convention to put large components on the 'top'-labeled layer?

I guess the picture could be way more cleared up, disabling the view of some layers/masks when taking a picture. I'll upload a better one tomorrow!
A 100nF cap might not do the trick, but my 4.7uF does?
I think if saturation was the problem it would be the mosfets that would be suffering.
Have you seen this thread - in particular reply #4 and #16.
Do I not understand how this forum works, or is there no #16? Thanks for pointing out though! I also thought the FETs should suffer, so nice to see my electronics understanding somewhat confirmed. I will try to get a good look at the gates of the MOSFETs, and check whether they for example go below 0V. I'm wondering though, is the load for the gatedriver increasing as the load increases? As far as I understand it doesn't at all. All in all doesn't seem to be my problem, but I'll certainly look into it.