Ah, the capacitors, yes, ceramic capacitors have a very low ESR, the cabling and traces to the capacitors have inductance, this forms a "tank" circuit with is an LC circuit, given a pulse an LC circuit will oscillate until the power dissipates, but the capacitors have very low ESR and usually the ESR in the capacitor uses the power in the oscillation up quickly but with no ESR it takes longer for the oscillation to subside.
try a diode into the capacitor close to it or some series resistance which will only help out your voltage regulator in dissipating the power.