What would be the theoretical improvement provided the 80196's ADC will be replaced by a modern 14-16bit external one?
PS: the 80196' ADC has got its 5V Vref (pin13) off the 399 via a 706 opamp - see below. I wonder whether there is something to improve around the "+5REF"..
A better ADC could improve the noise quite a bit. However the useful resolution is somewhat limited as the TC of the integrator capacitor (likely C0G and thus < 30-50 ppm/K) enters in the gain there.
So more than some 12 bits would be of limited use. I pefect ADC could reduce the noise about 3 fold for the 10 PLC case and more with shorter integration. It would still be quite some effort and more like a new design of the ADC with a few more changes to also help with other noise sources.
From my calculation I get an RMS noise for the 10 PLC mode of about 1-1.5 µV from the µC internal ADC noise and some 0.5-0.7 µV from the 1/f current noise of the OP27 as the 2 largest noise sources. There are than 3 sources that contribute some 350 nV: The resistors, non filtered part of higher frequency reference noise, jitter (mainly the HC4053). The noise adds as squares (geometric sum) and the larger ones are thus more imports.
If at all a better supply filtering could help with the 80196 to reduce the ADC noise there. Chances are it is already OK and not that much room for improvement and it could be just the intrinsic noise of the µC internal ADC.
The low hanging fruits are more the OP27, the HC4053 and adding a little filtering to the reference. Still this is only a small part of the noise and tweaking an existing ciruit is tricky. At some point is may be easier to build a separate, simple voltmeter with a similar cuircuit (e.g. use a more modern µC to replace the 80196 and ASIC).
The AC part with an analog RMS->DC converter does not work well at very low voltages. It is tricky to look at the noise of the AC mode. There is the rather nonlinear response at low voltages (different units may behave differenet in the details) and also a bandwidth that gets lower with low amplitude. So it is expected that the noise depends on the AC level and possibly the AC waveform.
The AC part has some filtering for the result and the AC readings are this a bit slow by design. When reading slow the ADC uses multiple 10 PLC conversions and averaging for DC and likely also for AC.