This is a tricky subject:
- You need to measure CM and diff, and match them up to observations.
- CM chokes are not simple inductors, but complex elements all their own. Here's a curve fit I did of a VAC common mode choke:


The R+L parts correspond to the flattened rising slope, characteristic of eddy currents in the laminated metallic core material. The dominant, naive model would be just R1, L1, C1, but as you can see, many more components are necessary to accurately capture the complex behavior of such an element. Even so, it breaks down above 10MHz, where I didn't bother doing it too accurately (and maybe their measurements are a bit questionable as well).
I think they measured that curve for a single winding, the other open circuit; so the other winding will act as a 1/2 wave resonant stub, and that's probably what causes the notch (at 10MHz, represented by C2, L2 I think). Noteworthy that R5 is not physically realistic (it has to be DCR only), it could stand to be another R||L element.
And ALL of this circuit, is simply the model of one winding. It's not the transfer characteristic between windings -- I don't have data for that anyway. Replace V1 with your entire circuit around L1, say, and you'd have the right impedance (if you're using this VAC part), but just for the one line.
- Similarly, the capacitors need to be modeled. This is usually pretty straightforward, as film capacitors are pretty well behaved, and only their low impedance properties are important here (the finest details of loss, of Q factor, won't affect the filter much). An RLC series equivalent is usually the way to go.
- The very circuit itself -- the layout -- should ideally be modeled as well. Traces have low-frequency-equivalent series inductance and parallel capacitance, and nearby traces couple to each other (the inductors have small k).
All of this is not to make you feel hopeless about things. There are only
finitely many things to worry about, and all those things absolutely can be measured, modeled and solved for. Such is the domain of engineering: a lot of busy work, driven by some smart theory, and often ultimately implemented by turning a bunch of knobs until it looks right.

In actual answer to one of your questions: CM chokes are usually in the k = 0.98 range. Depends on type. In my above example, if it is reasonable to assume L2, L3 and L4 are the leakage between windings, then the total of about 2.5uH, in relation to the primary inductance (lowest frequency equivalent) of 37mH, gives k ~= 0.992. This is fairly exceptional, because nanocrystalline cores have very high permeability, but still not far off (about 0.99, versus 0.98 for most ferrite cores, or maybe down to 0.90 for powder (MPP) cores, if you happen to find such a thing used for CMC).
CMCs can be placed on the DC side, yes: the difference is they don't filter FWB diode recovery noise.
The source impedance can be 50 ohms (this is how most filter manufacturers rate their products -- there's also a 1/100 ohm asymmetrical test, or something like that). The load depends on what circuit you're looking at. If you're considering the filter performance during the AC input peak, where the FWB is conducting (low impedance), then the load is a big honking electrolytic, and the common mode impedance is whatever capacitance to ground, and to secondary side, is present (often capacitance from transistor tab to heatsink, from primary to secondary inside the transformer (these two are the most common offending sources, by the way -- you can model this as well!), and any Y capacitors added in these locations).
Note that, anywhere you have L and C together, you also have an impedance, whether you meant to or not. This is Z = sqrt(L/C), the resonant impedance. Typically you'll have a large Z for common mode and a small Z for diff mode.
Remember that a filter always has a load impedance. If you do not provide one, it will find one of its own, and it probably won't be what you were expecting.

(Perhaps the response peaks at an unlucky frequency, corresponding to the filter's load being some parasitic resistance in the components.) The source being 50 ohms helps with this, and you can use that as the basis for damping the filter; you may want other resistances though, such as an R+C across the line to provide diff mode dampening (the diff mode is a rather low impedance, because leakage inductance is small -- a few uH -- and C is large, around 1uF), or a ferrite bead in series with a Y capacitor for CM damping.
Distributing losses throughout the filter helps by making the filter response less sensitive to source and load impedances. Like the VAC core example: that flattened rising slope is very lossy (lots of R+L), so a filter can be well damped if it has a resonant peak in that frequency range.
Oh, regarding measurements: you're applying a half CM, half diff signal, and measuring some mixture of the two at the load end. The definition of CM is (Vout+) + (Vout-) / 2, and diff is (Vout+) - (Vout-). Apply source voltages accordingly!
Tim