Alright, cool.
1. Supply ripple
Use bigger capacitors. Add moderate (LC) filtering. Add a regulator or converter stage that rides through the ripple.
Mains PFC is an extreme example of this: the input voltage is fluctuating between 0 and 100%. You literally can't draw any power at 0%, and you also can't draw the power required most of the time, because of the restriction of high power factor (current draw proportional to voltage). Rather, the power varies up and down, with the average controlled to meet demand. The power output from the PFC stage is varying 0-200%, and goes into a storage capacitor, which is usually sized for a small ripple voltage (say 10% of total). The ripple voltage is very nearly a sine wave at twice the mains frequency (i.e., the power ripple frequency). To get clean power from this, use a converter (which also gives important mains isolation..) which regulates the output. Its power input is fairly steady (thanks to the ~10% ripple) so it's not a big deal using a conventional control loop based regulator.
You can't get stable power from a single stage and finite filter capacitance, nor can you do the inverse, a DC-AC inverter stage drawing low input ripple. Conservation of energy.
2. Low frequency filtering
Use a regulator or converter. LDOs have okay PSRR below 10kHz or so, and HDOs are usually better. If you can't afford throwing away some voltage, or need some boost (Vin(min) is near or below Vout), consider a SEPIC or buck-boost ("flying inductor") stage.
If you can afford a lot of size and cost, an LC filter is still just fine, and doesn't cost the voltage drop or noise of an active solution. Iron core chokes are available from Hammond and others. It's a traditional method in tube amps, of course -- which are still being made today, if more for amusement than production.
There's also a kind of hybrid approach: you can shunt the ripple by coupling an amplifier into the output rail, feeding the amplifier with the inverse ripple so it cancels out. This doesn't lose DC voltage, but does draw some current (to power the amplifier). This is sometimes done for very quiet high voltage supplies, where the amplifier can run on a fraction of the total supply voltage (otherwise if it's running from the same supply, it would be worse than just pass-regulating the supply).
3. High frequency filtering (light)
LC filtering is efficient. Ferrite beads or chip inductors are cheap and compact.
LDOs suck at high frequency, probably don't bother. (There are a couple specialty LDOs designed for RF applications, that have good PSRR up to the low MHz. Pricey though.) C mult. is good.
4. High frequency filtering (heavy)
a. Shielding
Conceptually speaking: start with a shielded enclosure, with feedthroughs for each low-bandwidth connection (e.g., power, bias, control voltages, slow data, etc.). If you don't need enterprise-grade shielding and filtering, pare it back. Open up holes in the shield (this lets in/out more radiation), but keep the filtered connections within the perimeter of the remaining shield. Shrink shielding from whole-board to critical areas, or even critical traces. (Topologically speaking, as long as the shield surrounds the signals of interest, it doesn't much matter how it's actually shaped. That's why coax works -- and why it must be grounded on both sides!)
Presumably, you still need some high frequency connections: make sure those are well shielded or CM filtered. Minimize crosstalk between signals, and to the body of the circuit. Don't be afraid to apply the same shielding scheme recursively.
Note that a board with solid ground plane counts as this. It's not at all perfect, what with everything exposed on the board face -- but signals coming in around the edges meet the ground plane immediately, to which their shields are tied, and so common mode noise is conducted around the periphery of the board, not through it.
b. Filtering
Consider higher order filters, and common mode filters.
Signal filters: design all your signals for a characteristic / system impedance. There must always be a resistance somewhere, whether it's the source, load or both. If your system has a lot of mismatch (common in switching circuits, where the on-resistance is very much smaller than the load resistance, and the off-resistance very much larger), something like a constant-resistance filter can be used to introduce the necessary damping/matching resistance. There are tables for single and double terminated filters, and unequal resistances.
Common mode filters are constructed the same way, but usually to higher impedances (100s of ohms), with poorer control (the CM impedance of a single box on the end of a wire isn't very resistive!), and a less precise filter profile. You still want to check that the response is not peaky, given typical situations. Mostly you design it around adequate stopband attenuation, rather than a precisely flat passband or a sharp transition band. Pay attention to differential mode response, because CM chokes come in different types and you need to use the right one.
Power line filters should be low impedance, to keep ripple voltage nominal under changes in load current (Z ~= dV/dI). At least, for CV supplies. CC supplies of course must take the opposite approach (keep ripple current nominal under changes in load voltage --> higher Z).
A PDN (power distribution network) typically has an LC chain topology, with therefore a characteristic impedance and cutoff frequency, and that chain shall be terminated with resistance. It would be terrifically inefficient to terminate it with a bulk resistor that draws DC -- so we use a coupling capacitor in series with that resistor. The coupling capacitor has to be large enough that the RC pair isn't reactive at the cutoff frequency, i.e., Cbulk > 3 * Ctotal (Ctotal being the total bypass capacitance along the network).
It's a common misconception that bulk caps are for storing energy. In actuality, we only care that it acts as a coupling cap, terminating the network with just the right impedance and frequency. Indeed, the fact that it's there to reduce ripple voltage, is proof that it's not for energy storage: energy depends on voltage squared, so an energy storage capacitor is only useful if its voltage is changing a lot.
(Which brings us full circle! Remember that mains PFC stage with the bulk capacitor? The ripple on that capacitor is precisely the energy filled in between cycles. As long as the subsequent converter stage can handle the ripple, and the capacitor itself is okay, that ripple can be fairly large. The main reason we go for a relatively low ripple like 10%, is simply because electrolytic capacitors are not very good. More ripple voltage makes them boil and explode. Film capacitors would be perfectly happy, but they're bulky and expensive. Another issue is hold-up time, which again is an energy storage issue.)
As for filter design itself, that's a very complex issue of course. I would recommend starting with typical numbers and tables, then tweaking in SPICE.
Remember these relationships:
Supply filter impedance: Z = dV/dI, where dV is peak ripple voltage, resulting from peak load current change dI.
Characteristic impedance: Zo = sqrt(L/C)
Cutoff frequency: Fo = 1 / (2*pi*sqrt(L*C))
The last two are used to scale filter tables; filter calculators don't actually know anything about filtering, they just apply these to give you a filter at the Zo and Fo you requested. Ratios only. They will quite happily give you nonsense results (say, nH inductors, or mH inductors surrounded by pF capacitors, etc.), it's up to you to sanity check your inputs and your outputs.
There are also transformations that can be used to make an unreasonable filter practical. Example, using series-parallel transforms, to convert a series-parallel (ladder) bandpass into a coupled-resonators topology. And using impedance matching (of various types), so the resonators can be practical impedances (100s ohms?), while the port impedances are nominal. But this is mainly applicable to RF circuit design.
Tim