I just did some calculations around the DAC:

With an R_{set} of 2,2k we get a fullscale output current of around 18mA, which results in a voltage drop of about 0,9V across the 50 Ohm output load (This disregards the input impedance of the OpAmp input dividers, but as I used 51 Ohm resistors anyway, the difference should be neglectible.)

As the gain of the output driver is 3,733 (560/150 Ohm) the fullscale output voltage at the BNC (without load) is 0,9V * 3,733 = +/- 3,36V.

So the DAC range isn't fully used - or is there a mistake in my calculation?

a) The voltage gain is lower than 560/150, since you neglected the 50 Ohm resistor.

b) On the other hand, the voltage gain applies to the voltage difference between the DAC outputs, which is higher than the single-ended AC voltage between one ADC output and GND.

c) Just applying a factor of 2 - in order to account for (b) - would not be valid either, since the AC voltage is not the same at both ADC outputs, because the current sources drive into different impedances. The inverted output of the DAC (leading to IN+ of the amp) is terminated with (50 Ohm || (150+560 Ohm)) == 46.7 Ohm towards GND, but the effective termination impedance of the non-inverted DAC output towards GND is affected by the feedback from the amp output, and is lower.

EDIT:

A more detailed analysis/calculation of this kind of ADC termination circuit can be actually found e.g. here:

http://www.ti.com/lit/an/sbaa135/sbaa135.pdf. They also don't consider voltage gain, but rather transimpedance of the amp, as the inputs are current-driven. Quote from this paper:

...This relatively simple-looking circuit is, however, not giving a matched gain nor a matched input impedance for the two current source outputs...

Anyway, I share the conclusion that the DAC not fully utilized. But it may make sense to leave a couple of percent headroom for gain/offset adjustments in the digital domain.

(1) DC bias measurement:

On my board, Rset = 25B = 1.78k (not 2.2k). If the IC were a DAC902, this would correspond to I

_{OUTFS} = 22mA. However, it is not a genuine DAC902, but a chinese chip with unreadable markings and unknown specs. So we can only guess. At least it seems to be plausible that the specs are similar.

When I select the "Arb4" signal (-> eventually DC with 50% digital count of full-scale), then I measure about 500mV DC each at the inverted and non-inverted DAC output pins, and the EL5166 output is about 0V (all readings against GND). At this bias point, each DAC output is loaded with 50 Ohm || (150+560 Ohm) == 46.7 Ohm towards GND. 500mV / 46.7 Ohm = 10.7mA at each output, and this is indeed pretty close to the predicted I

_{OUTFS} / 2. Looks reasonable

(2) AC measuremt:

For simplicity it is IMO sufficient to consider only the AC current at the inverted DAC output whose termination impedance is not affected by the amp feedback (-> the AC current at the other DAC output is supposed to be the same, just inverted). With the AWG set to a square wave with an amplitude of 2.5V, I measure 870mVpp at this ADC ouput. 870mVpp / 46.7 Ohm = 18.6 mApp. And yes, that's only about 85% of I

_{OUTFS}.

[ My 6074BD scope is not so accurate, though, so please grant sufficient tolerance to my measurements. ]

gf