I was thinking about this some more, and I believe there is a simpler way to accomplish the goal of determining the higher order transfer function coefficients. You just feed a resistor divider with a stable voltage source, say from a DAC. The divider should have a reasonably consistent ratio over the voltage range in question, at least with the math I did, but there may be a way of working it out so you can compensate for voltage coefficients. So with the resistor divider, we have three nodes, the DAC output (DAC), the center tap of the divider (CT), and ground. So you can connect these nodes to the front panel with relays as in OP's schematic. The procedure is as follows.

1. Call the ADC transfer function Y(x) = a0 + a1*x + a2*x^2 + a3*x^3 + a4 * x^4 ...

2. For each DAC voltage you measure V(CT,GND) + V(DAC,CT) - V(DAC,GND) to get your error function. For a reasonably accurate ADC, x in the transfer function can just be V(DAC,GND).

3. Run a polynomial fit of the error function to get the coefficients delta(x) = c0 + c2 * x^2 + c3 * x^3 + c4 * x^4 ... (the linear term should be insignificant)

4. Calculate the divider ratio as b = V(CT,GND)/(V(CT,GND) + V(DAC,CT)).

5. The relationships between the coefficients are

a0 = c0

a2 = c2/(-2b + 2b^2)

a3 = c3/(-3b + 3b^2)

a4 = c4/(-4b + 6b^2 -3b^3 + 2b^4)

a5 = c5/(-5b + 10b^2 - 10b^3 + 5b^4)

In practice, you can add reversal to extend the voltage range and/or check the even order coefficients. The system will be most forgiving around a 1:1 ratio, as each of the functions of the ratio b (in step 5) has a first derivative of 0 at b = 0.5. On the hardware side, I believe this will be easier to implement than a string DAC using discrete resistors because only relays are necessary for switching the voltages (minimally two, three DPDT for reversal), and temperature homogeneity of a monolithic divider is going to be much easier to achieve than it is with a handful of discretes. The primary requirement for the voltage source (i.e. the voltage reference and the DAC) is low noise and reasonable short term stability.

I will probably make one of these (with some modifications for low voltages) for testing the nanovoltmeter I am working on. I will probably just use an ADR1000 reference as I have a handful of reference modules that use these sitting around.