The shift in the voltage is likely not just from load to VDD1 / VDD2. I see a good chance that part of it is from current to ground. So it can really matter where the gound is taken from.
As a reference ground, I was and am using GND1 right on the NU180. That is the "A" ground on A3. I'm unclear what other grounds would be pertinent to its performance or why. There is GND2 of course, but I wouldn't call that a "ground," it's really just a negative supply.
I noticed that my plots differ from Simon's. I do not see a dip between conversion periods.
I tried raising Vdd2 from its nominal ~4.3V to 4.5 or so with a power supply, which was not a problem. As expected, once the power supply took over, the dips, etc., were gone, and the output became flat.
I also raised Vdd1 slightly, which also worked OK (Vdd2 still tracks Vdd1 here). There is more like 150 mA used on that. This still looked like the above photos, just raised slightly. As I changed this voltage, the 3458A's output (measuring + or -10V) also changed, so this voltage is important for accuracy.
I also looked at GND2, and a photo is attached below (NPLC=10, 10V measured). It shows the same sort of dips, etc., as Vdd2. Power consumption is small (<2 mA). I pulled this down slightly with a power supply, and again the output became flat.
None of the above made any clear improvement, e.g. measuring +/-10V still resulted in a small offset between them. Because of the "dips", etc., we could contemplate further regulation on Nu180, but it's not yet clear that this will help. Perhaps others can play with these voltages too and see if they can discern an improvement, or else maybe point out a test I can do that would lend further evidence one way or another.
Update: I explored the sensitivity of the output based on changing Vdd1 and GND2. (On the board I am looking at, Vdd1 is used for the switches, so Vdd2 has no impact.) All measurements were with +10V as input.
* Changing Vdd1
did change the output. When changed by +0.1V, the digitized output changed by about -17 uV.
* Changing GND2
did change the output, initially. The first 0.1V lower (from about -0.6 to -0.7) changed the output by about +23 uV. This effect rapidly diminished, though. By the time GND2 reaches -1V, the output stabilizes, with no further changes in the digitized output. (I checked if the stabilization somehow fixed the polarity offset, but no.)
Unfortunately, I do not currently have a working board that uses Vdd2 for the switches, so I can't comment about its impact vs. Vdd1.