Input current noise is not something I've dealt with before so thank you for reminding me about that.
If the input stage were an RC highpass with 68uF / 1kOhm, the cutoff would be about 2.4Hz. Is there something wrong with that kind of setup?
Large coupling capacitors tend to create other problems like long settling times and temperature sensitivity, and they may contribute "excess noise", but you could try it. In order of preference, wet slug tantalum (expensive), sealed dry tantalum, and high voltage low leakage aluminum electrolytic capacitors seem to be the best. I have not tried any of the modern "polymer" aluminum or tantalum capacitors though. I have done it myself for this exact application but not with a low noise preamplifier simply because I could always design the preamplifier with high impedance inputs so that large coupling capacitors are not required.
There is something to beware of when using a large AC coupling capacitor with a low impedance input; the current spike from connecting the probe without precharging the capacitor may damage the low impedance input. This is why some oscilloscope inputs have a "precharge" function to be used before AC coupling is selected.
I didn't imagine input bias current would be much of a problem here: won't it just lead to a DC offset in the mV range which I can AC-couple out at the output stage? An issue could arise if it's too high for the circuit to deal with after the gain is applied but I think we should get away with this -- 1uA input bias current becomes 1mV of offset, so even if I went nuts with gain it would still be manageable.
You can do that, but ignoring the input current induced offset does not gain much since the input current noise still sets an upper limit on the input series resistance.
Something else to consider, which helps for once, is that with a low impedance source and a large coupling capacitor, higher frequency noise from the input shunt resistor is shorted out into the source so higher frequency noise above the cutoff is actually dominated by the source. On oscilloscope high impedance inputs, this shows up as a rise in noise level above a relatively low frequency when there is no low impedance input connection but it is seldom noticeable.
double terminated means 50 ohms in series at the source and 50 ohms in parallel at the load
You've mentioned scope probes or coax a couple of times, I think in respect of the input to this amplifier stage. I hadn't thought beyond using short wires to connect directly, or a twisted pair. If using coax, what would be the connections? The device under test has two terminals (it's output and 0V) and this amplifier has I suppose three (two inputs and its circuit ground). It's making me feel stupid that I can't figure out what you have in mind here.
I have done it with twisted pair but always got better results using 1x probes or coaxial cable. Sometimes I attach short coaxial cable "pig tails" which connect coaxially into the 1x probes.
With a differential input, "chassis" ground and "power" or "signal" ground do not have to be the same location. Presumably in the DUT (device under test) there is a single point ground somewhere that the coaxial or probe shield can connect to. If the DUT and oscilloscope share power ground, then connecting the coaxial shields to either other and nothing else may be best.