Incidentally, note the one thing you are
NOT doing -- energy storage.
If we could reduce supply impedance without storing any energy at all, we would!
Indeed, we often do. Linear regulators have arbitrarily high performance, from the old LM317 and relatives (pretty slow, good enough for audio, say), all the way up to "RF" rated parts (which have especially high PSRR and low Zo up into the MHz). Linear regulators of course don't store energy at all, they merely control power. The price we pay is the dissipation of some of that power.
Fast switching regulators store a minimum of energy; with a control-loop response time in the 100s of us, their load and supply currents track pretty quickly. There's very little hold-up time, in case of power interruption, say. Even an offline supply stores only enough energy to keep alive between mains cycles.
But in the power supply, the key is this: we want a low supply impedance. That means, for a given change in load current, the supply voltage changes little. Energy stored or released in a capacitor is proportional to the voltage change (exactly, E = C Vcc ΔV). Indeed, you might even say we are actively trying to minimize energy storage; it doesn't do anything for us, just makes our supplies come up slower!
If we could get bypass capacitors that have maximum capacitance around quiescent supply voltage, and nearly zero everywhere else, we would actually prefer that. This means minimal energy storage starting from zero, and minimum impedance around the operating point.
It sounds imaginary, but such a component is actually possible -- an electret is a dielectric with a built-in electric field (in much the same way that a magnet has a built-in magnetic field). If we use a dielectric that saturates easily (i.e., as field increases past a point, capacitance decreases sharply), we can control where that saturation occurs, and in fact shift the maximum-capacitance point to the middle of our nominal range.
They are even available! TDK Cera-Link capacitors are made for industrial inverter application, so, with a built-in field corresponding to about 400V.
An interesting consequence of this -- what energy it does store, is mainly stored around some voltage, rather than around zero. Energy is voltage times charge, so if we're putting in the same amount of charge, but doing it around some nonzero voltage, suddenly all that charge stores a ton of energy!
And actually, we're putting in up to double the charge -- effectively, it's like charging a capacitor starting from a negative voltage, so we can go through negative saturation, to zero field, to positive saturation, in a single swing. Whereas with a regular capacitor, we'd most often start from zero, using only the positive charge area.
We're still not concerned about energy storage for bypassing purposes; but in essence, the capacitor is that much more effective, so we can use less of it. And if we are storing energy, they have much higher energy density than anything else in its class. Which helps, as they aren't cheap...
Tim