Perserverence furthers. By golly, it works! The ruminations of the last day or two helped identify three errors I was making.
One of the main problems was errors caused by the mixing of real and complex math operations in my code. The equations themselves were fine; but a couple were being executed wrongly. Some operations, like correcting for the R channel L-pad, need to be done on the magnitude vector, in polar. But others, like (especially) scaling the data to the 2.35 Ohm reference R, must be done as a complex number in rectangular coords, which leaves the imaginary term intact, or else the phase gets demolished, and rectances measure wrong.
Another issue is the fixture's sensitivity to the BNC cables' shield resistance. One of my big beefs with BNC cables is that they don't solder the ground braid to the connector shell. So with some of 'em you get variable ground impedance with each cable. Just move it a tad and it changes. Well this fixture needs short cables with as-low-as-possible shield impedance.
And lastly, when using Open-Short-Load compensation, doing a Thru sweep on the analyzer is unnecessary and will cause major errors. It turns out, the channel imbalances which the Thru quantifies are taken care of by the OSL routines.
Having corrected these things, I'm getting good results, all the way up to 10MHZ and down to about 20-ish milliOhms. Including large capacitors. See the attached plot. Three current sense R's; a 1 Ohm Caddock radial-lead, a 100mOhm axial, and a 25mOhm axial. And a 1,000uF lytic which measures 44mOhm ESR on the Wayne Kerr meter.
As you can see, the traces start getting noisy at say 50mOhm and below. This isn't caused by a measurement dynamic range problem, per se; there was plenty of dynamic range to spare on both channels, and very little difference between sweeps with a 30Hz IF and 10Hz IF. it's a numeric precision issue with the Anritsu analyzer. It rounds and stores its magnitude data with 0.01dB precision. Normally, that would be plenty good. But with this measurement, at low impedances, .01dB in the raw measurement converts to linear voltage ratio and then impedance at 2-4 milliOhms per hundredth of a dB! The lower the impedance value, the coarser the conversion, hence what looks like "noise". That coarseness is also transferred into the OSL routines. So an analyzer which stores its data with more precision will give much smoother traces (and better data) with this method.
Anyway, I'm a happy camper. Being able to measure the impedance of thru-hole parts up into the MHZ region was one of my major goals. Once I get the noise issue solved (I have a couple ideas), I can live with a 20mOhm lower limit over a 10Hz - 10MHz range.
I also have Omicron's B-SMC fixture, which uses the same network inside. So it should be plug-and-play and give the same results with SMD parts.
BTW, for anyone making their own fixture, this approach is worth considering. If anyone wants details on the corrected math routines, let me know.