DMM Noise comparison testing project

[VintageNut]

Do you have the latest firmware installed?

Yes, latest FW 1.6.1 is installed

Really interesting, that you can set yours to 15.

50HZ vs 60 Hz

I am in the USA where one PLC is 1/60 of a second.

[HighVoltage]

50HZ vs 60 Hz

I am in the USA where one PLC is 1/60 of a second.

Thanks,
that might be the reason, to get almost equal integration time.

15NPLC  x 60 Hz = 900 cycles
1/60 *15 = 0,250 sec integration time

12NPLC  x 50 Hz = 600 cycles
1/50 *12 = 0,240 sec integration time
 

[Le_Bassiste]

There are four main noise sources in a DMM:
1) the ADC itself: this gives a constant noise contribution relative to the range.
2) The reference noise: this only applies if a voltage other than 0 is measured - so it does not matter for data on shorted input.
    It is proportional to the measured voltage.
3) The noise from the input Amplifier. This noise depends on the range / amplification chose. Usually a constant noise voltage and thus increasingly important in the low voltage ranges.
4) Noise from the input protection / divider: High resistance in the input protection (e.g. series resistance) gives noise, that behaves similar to the amplifier noise. A special case is the typical 9.9 M - 100 K divider resistance, that can contribute to noise especially in the 100 V range.

The noise relative to the full scale in the 0.1 V range at 1 PLC is even lower than in the 10 V range, which is a little strange, but might come from the special kind of range switching used. It's also possible to have quite some quantization noise at 1 PLC and thus a noise level that can change with offsets / the exact measured voltage.

If there would be significant amplifier noise, the 0.1 V range should show a much higher noise than the 10 V range. So the noise must be mainly the range independent ADC noise. It's also typical to have very little amplifier noise for the range without extra amplification (e.g. 10 V range).

A second factor that point towards noise from the ADC is that the noise at 1 PLC is much higher than at 20 PLC. White noise from the amplifier would make the noise go down with the square root of the integration time. With a contribution of 1/f noise the noise would go down even slower with longer time. In contrast to this the ADC can have a noise contribution (e.g quantization or determination of residual charge) that goes down inverse proportional with the integration time - this about what is found here for the 34420. So that ADC is much better at 10/20 PLC than at 1 PLC.

thx kleinstein for clarifying on the possible noise sources!

here's another one that i have observed. however, i still can't fully get my head around it as to how much contribution it may have to the overall noise level of a DMM:
1) take your beloved KEI2001, set it to 200mVDC and connect the input terminals to a 10KOhms resistor. this is, if i got the figures correctly, approximately the highest allowable source impedance to keep source voltage measurement errors at no higher than 1 ppm. thus it presents (to me) something close to the "worst case" when trying to measure DC voltages.
2) now, connect your scope in parallel to the resistor. set coupling to AC, amplitude to 100mV/div, timebase to, say, 10 usec, trigger to norm and 100 mV trig level. BW of 20 MHz is fully sufficient.
3) what do you see? can somebody explain to me whether this does actually contribute to the noise?

curious to get your comments on this!

[Kleinstein]

The input amplifier can have current noise too - this can be a problem with high impedance signal sources. One special case here can be spikes from input auto zero switching. So there can be quite some spikes, but these are in the short phase when the DMM is not actually measuring. So they may not fully contribute - but a capacitive source could stretch / delay those spikes so they will contribute.

Also a chopper stabilized amplifier like in the Keithley 2000 and a few others (solartron, datron 1281) can contribute to input current noise and show spikes.

Input current noise could be important for high impedance sources. So it would be a second parameter to measure. With good shielding it should be possible to have considerable more than 10 K source impedance - some DMMs have more internal for protection. Input current noise is a parameter less looked at - so more surprises to be expected.

[TiN]

Setup:

No filters on Keithleys.
3458A - NPLC 100, AZER ON
Keithley's - NPLC 10, AZER SYNC, no filter
Number after dash is just GPIB address.
All meters measure same KX LTZ1000 reference, direct 7V, module from Fairy-tale 1.

Stuff in the middle of the graph - just me, messing with other gear nearby, causing jumps due to knocking on cables and such.

Results so far:

3458A : <1.2 uV noise ~0.12 ppm/range
2002-4 : <2.5 uV noise ~0.125 ppm/range
2002-6 : <2.5 uV noise ~0.125 ppm/range
2001-20 : <15 uV noise ~0.75 ppm/range
2001-21 : <22 uV noise ~1.1 ppm/range

I had third 2001 in loop too, but it showed huge noise, so I think now it's faulty, even though passes self-test 
Had not enough GPIB cables to connect 4th 2001.
So I'll swap 2002's with 4th 2001 tonight.

Next step would be resistance, perhaps 1K first?

[3roomlab]

hmmm shouldnt the keithley be 10NPLC + 10 repeating average ( = 100NPLC) so that it is = HP 100NPLC. then it is apples = apples?

[Kleinstein]

For a direct comparison to the 3458 filtering (average of 10 values) would be better.  However one could still do that on the data. As a rough estimate this would reduce the noise by something like 3 fold (cold be less with large fraction of 1/f noise). So the 2002 seems to be roughly on par with the 3458 but the K2001 seems to be much more noisy. A noise reading with a short for the same meters might be interesting - so one could about separate the reference contribution. The 2001 seems to be noisy with a short too, not just with a significant voltage.

The contribution from the reference in the meter depends on the actual voltage, not on the range. So the value relative to full scale is a little misleading.

At least with the 2002 and 3458 some of the noise comes from the source. So looking at the difference between the two 2002s would also be interesting: this should be sqrt(2) times the noise of the 2002, essentially without noise from the source.

[maxwell3e10]

Hello. I've been lurking around on this board recently and caught a bit of the noise bug

I noticed that the noise on the new Keithley DMM7510 does not go down as fast as it should for longer integration times, like 10 PLC, especially on 10 V scale. Here is a comparison of the time trace for DMM7510 and Keysight 34465A on 10 V scale, 10 PLC with autozero, continuous triggering. These data are saved to internal buffer, then transferred to USB stick, no computer connection.



Even though the rms noise is similar, it is clear that DMM7510 data have much larger flicker noise component.  So, I started to look at Allan variance of the voltage (with shorted inputs) for my own meter as well as using the data that others have posted on TiN website. Allan variance tells you what is the rms scatter of the data on a particular time scale. It should go down as 1/sqrt(time) until the instrument reaches offset drift noise floor. Here are the plots for 10V range, using PLC of 1 and 10. All the data are with autozero on. Turning the line sync on or off does not make a difference.


It is clear that DMM7510 has a problem, where the Allan variance does not go down until about 30 sec timescale. It is present for three meters measured by different people. Keysight 34465A has the expected normal behavior, it goes down as 1/sqrt(time) and then reaches a floor given by the drift. One can obtain similar plots from any other long raw data files posted on TiN webste (having all the data available is really great!) For example, it would be interesting to compare to a nice long HP3458A file (I can do that if you tell me which one to use).

The really weird thing is that the noise in 7510 changes depending on measurement trigger time. I noticed that first from DrDiesel's data taken at 1 sec intervals (instead of continuous, as most other data). I found that I can get the best performance on my meter using 0.1 sec trigger and 1 PLC.  One can also notice that the noise decreases faster than 1/sqrt(time) after 30 sec. This indicates a quasi-periodic noise source, I suspect some kind of clock interference.

For reference, here are my measurements of the RMS noise for various ranges and speeds for the two meters:

Range	Speed/Average	34465A ppm	DMM7510 ppm
100 mv	1PLC	2.6	0.43
100 mV	10 PLC	1.2	0.29
100mV	100PLC From 1 PLC	1.1	0.21
100 mV	100PLC From 10 PLC	0.97	0.2
1V	1PLC	0.32	0.081
1V	10PLC	0.14	0.055
1V	100PLC From 1 PLC	0.103	0.04
1V	100PLC From 10 PLC	0.098	0.033
10V	1PLC	0.13	0.081
10V	10PLC	0.04	0.051
10V	100PLC	0.016	NA
10V	100PLC From 1 PLC	0.014	0.043
10V	100PLC From 10 PLC	0.014	0.038
10V	1000PLC from 1PLC	0.01	0.019
10V	1000PLC from 10PLC	0.0063	0.016

They are generally consistent with earlier measurements for these meters reported here. DMM7510 is better at 100 mV and 1V, but mostly worse than 34465A for main metrology range of 10V.

There is also a number of programming issues I found on the Keithley DMM7510 meter.  It gave me a blue screen of death several times (even Chinese instruments don't really do that). When using internal trigger timer, it tends to hang up and stop triggering after a few thousand to tens of thousand of points. When saving to USB, it does not warn if the file already exists.  Overall, 34465A is clearly a great deal with 7.5 digit performance. DMM7510 is questionable if its worth 2-3 times higher price, perhaps it can be improved with firmware.

[dr.diesel]

There is also a number of programming issues I found on the Keithley DMM7510 meter.  It gave me a blue screen of death several times (even Chinese instruments don't really do that). When using internal trigger timer, it tends to hang up and stop triggering after a few thousand to tens of thousand of points. When saving to USB, it does not warn if the file already exists.  Overall, 34465A is clearly a great deal with 7.5 digit performance. DMM7510 is questionable if its worth 2-3 times higher price, perhaps it can be improved with firmware.

maxwell3e10

Please lurk less often, thanks for a great post!

I have also seen the blue screen and general lock up issues, I have reported this to Keithley and they confirmed they are aware of the issue and working on a fix.  Though it's been several months with nothing so far.

I'd love to somehow make Keithley aware of your above data, but have doubts anyone there would take the time to carefully review.   

[TiN]

maxwell3e10
Very nice post, pleasure to see data being used well. If you like, DSV-file - is also very long (5 days continuous) log with 7V on six meters (3458, two 2002 and three 2001s, with one faulty).

What software you used for these beautiful graphs? I'm interested to try and also add into article as another analysis method for noise, if you let so. I think would also be interesting to see 34470A in comparison too, to see if lower noise LTZ1000 ref helps on time/noise perf.
 

[HighVoltage]

There is also a number of programming issues I found on the Keithley DMM7510 meter.  It gave me a blue screen of death several times (even Chinese instruments don't really do that). When using internal trigger timer, it tends to hang up and stop triggering after a few thousand to tens of thousand of points. When saving to USB, it does not warn if the file already exists.  Overall, 34465A is clearly a great deal with 7.5 digit performance. DMM7510 is questionable if its worth 2-3 times higher price, perhaps it can be improved with firmware.

Unfortunately, my DMM7510 is showing a blue screen of death from time to time as well.
Keithley claims they are working on a FW fix.

[Dr. Frank]

 I think would also be interesting to see 34470A in comparison too, to see if lower noise LTZ1000 ref helps on time/noise perf.

A comparison between 34465A and 34470A has been done on 10V range already:
https://www.eevblog.com/forum/testgear/keysight's-new-34465a-(6-5-digit)-and-34470a-(7-5-digit)-bench-multimeters/msg889217/#msg889217

noise, i.e. mid term stability is improved over the LM399, also the long-term stability, of course.

short term noise is identical, as the A/D path is identical.

Frank

[maxwell3e10]

Thanks for good feedback.

It would be good to make an Allan plot comparison for all the meters, that I think would be the best way to compare the ultimate limits of their performance. To make things simple, lets stick to shorted input data first. TiN, if you tell me a list of good long files with 10V, 10 PLC for 3458, 2001, 2002, any others, I will make a plot. If the DMM7510 is really unique with this noise problem, that would be a good plot to advertise until Keithley take note.

I use Genplot, you can get it for free from www.genplot.com, its a pretty good data analysis and plotting program.
Just for fun, here is the histogram plot, similar to the ones made by Dr.Frank, showing the digital resolution of the meters. Its not really crucial, as both meters have resolution much higher than necessary based on noise. But interesting to note that "6.5" digit 34465A has better digital resolution. The numbers are just how they are written to internal buffer and saved to usb.

[Kleinstein]

Normally the auto zero type of measurement should essentially eliminate 1/f noise for time scales longer than one measurement cycle. Maybe the Keithley engineers were "clever" and did averaging one the zero measurements. This way 1/f noise of the input amplifier would be suppressed only from a lower frequency on.

It might be interesting to do a short test without AZ mode - just to see how much low frequency noise is expected from the input amplifier.

If the strange extra noise at the 7510 there also in the 1 V range ?

Reference noise should not be so important for data on a shorted input. This only comes with a real (low noise) voltage source source of significant hight.

[maxwell3e10]

Yes, I agree the auto zero should take care of 1/f noise at least for analog noise after the switch. It could be some problem before the autozero switch or some quasi-periodic noise that doesn't exactly average out.  As far as I can tell, on 10PLC scale, the autozero is done between every measurement and the real measurement rate is about 2/sec. On 1 PLC scale, the autozero is every 5th measurement and the average sample rate is 47/sec (no line sync) and 25/sec (with line sync).

With autozero off, the signal just drifts off to many microvolts in seconds. I didn't bother taking systematic data.


Here are the Allan variance data on 1V scale. Here the extra bump is less pronounced, but still present. It is a factor of 10 smaller in voltage, so must couple after input amplifier. Keithley can beat Keysight on this range.

[Kleinstein]

As the extra noise is lower by nearly a factor of 10 in the 1 V range, this points to noise from behind the input amplifier. It could be something like the zero of the ADC, e.g. +Ref / -Ref balance, or the integrator itself. Normally AZ should really take care of this kind of noise - so I suspect some kind of trouble with how AZ is used. If we are lucky this could get fixed by a FW update. The HW seems to be able to do better, as shown in the curves with 0.1 s trigger.

If this would be a kind of interference / beat frequency (e.g. 100 Hz to grid, crystal to crystal), I would not expect it to be so reproducible across instruments. One should also see a single frequency in an FFT (e.g. 0.1-0.3 Hz range).

[maxwell3e10]

One possibility is that using a timed trigger makes the measurements better synchronized with the internal clock than continuous measurements. But I found that using the trigger at 0.05 sec with 1 PLC  makes the noise worse than trigger at 0.1 sec. Perhaps this points to a settling issue with the autozero switch?  When using 10PLC, I tried to set the trigger to 1 sec or 2 sec, but this did not improve it.

I haven't looked at typical schematics, is the autozero switch usually before or after the input amplifier?

[Kleinstein]

With most new meters the AZ switching is before the input amplifier. I know a few (Keithley 2000, Solartron 1071/1081, datron1280 ?) that use a chopper stabilized input amplifier and AZ switching after the amplifier, thus mainly for the ADC. One could test it, by looking at the LF noise / dirft in the non AZ mode. The versions with AZ behind the input stage are usually quite good even without AZ.
So this strongly points to AZ at the input for the DMM7510.

Here the extra noise seems to come from behind the amplifier that sets the x 10 gain for the 1 V range. With AZ active I see no real source for an analog memory up to the 2 second range - so I would really expect this to be a software problem. Doing AZ only every 5th reading needs some filtering / interpolation - with to much filtering here I would about expect what we see, as the AZ would loose its effect to reduce 1/f noise in a limited frequency range.

[maxwell3e10]

You maybe onto something. I wonder if the meter extrapolates the value of zero offset based on history of prior zero measurements. If it uses a high-degree extrapolation coupled perhaps with some bug in the implementation, it can lead to undesirable results. Maybe when one uses triggered measurements, the autozero algorithm changes, so say for 0.1sec timing autozero is done with every measurement.

Perhaps somehow related to this, I noticed an interesting behavior when the temperature of the meter quickly changes (it was brought from outside into the lab). There are abrupt periodic changes in the readings every 3.2 sec or so. This is even though the autozero measurements are done every 0.2 sec. I am not sure what it is doing, perhaps applying some correction from its temperature sensor? Notice the offset is rather large, about 10 microV with autozero on. The offset still drifts quickly and the meter tries to correct this drift with abrupt changes. Eventually the voltage drifts back to zero when the meter warms up on a 20 min time scale.

Doing Fourier transform on a long data set (when temperature is stable) reveals a few little discrete peaks, but they don't contribute much to the RMS noise, mostly there is a broad peak around 0.015 Hz, which is consistent with peak in the Allan variance at half the period, around 30 sec.

[maxwell3e10]

Here is the Allan variance comparison of the four leading DMM models. Its nice to be able to find two meters of each model on TiN list. The performance is quite reproducible for each model, except for long term behavior, which depends on the length of the data set and the temperature stability during the run.

Its nice to see that 34465A performs almost as well as the 3458A while nearly 10 times cheaper. On the other hand, Keithley meters don't look so good in this comparison to HP/Agilent/Keysight. The 2002 model also has some trouble with averaging for longer times slower than 1/sqrt(time), but not as pronounced as DMM7510.

[maxwell3e10]

I took DMM7510 10v range data with autozero off for comparison. It looks like autozero does not actually remove any noise on time scale shorter than 20 sec for continuous measurements. But it does help if the measurements are triggered at 0.1 sec intervals. I think this points to a serious firmware problem.

[dr.diesel]

I think this points to a serious firmware problem.

The even bigger question is, how to get Keithley's attention?  So far they have been pretty much ignoring bugs I send them.

[ManateeMafia]

I would consider getting your list of issues together and then post them on the Keithley forum. The questions/concerns are usually handled by a Dale C.

Post a link back to EEVblog, if it is possible, to show the issues you are describing. They may be willing to discuss what you are seeing and having it posted on their forum makes it more visible to the company.

[HighVoltage]

I have issues with Keithley DMM7510, 2450 and 2460 SMU and it took forever to even get a response. At the end I was told that with a new FW update, these issues would be solved. Who knows when that will happen.

To compile all new Keithley problems, I have opened a new thread:
https://www.eevblog.com/forum/testgear/keithley-dmm7510-smu-2450-2460-problems/

May be we can help Keithley in fixing these problems.

[Kleinstein]

The relatively high noise of the DMM7510 in the 2 s to 30 s range is likely due to a problem in the software. Having nearly the same noise without AZ active in the range shows that for some reason AZ is not able to suppress the 1/f type of noise. At 10 PLC and short times the AZ mode is also higher in noise than expected: one would expect a factor of 1.4 (about what is seen at 1 PLC), but at 10 PLC this looks even more. This suggests they are trying a kind of extrapolation and not averaging for the zero measurement. Heavy averaging would have caused the extra low frequency noise, but at least with the advantage of lower noise at short time scale. The data without AZ don't look so bad that extrapolation would do any good (except maybe under fast changing temperature).

It is not a really serious problem, but it is kind of a shame if a poor firmware degrades the possible performance. The good thing is that this problem should be relatively easy to fix, once they are willing to do so.

The second point the data show is that there is quite some 1/f type noise coming from the ADC itself (not the input amplifier) - this indicate there might also be a potential for an HW update (e.g. better OP in the integrator). So other DMMs might had the same weakness in the software (the data for the 2002 somewhat suggest that), but only the high 1/f noise from the ADC make the weakness so obvious.

The glitches / steps during the measurement with high temperature change rate could be due to automatic internal gain calibrations. It is expected that on changing temperature the will be extra cycles of measuring the internal ref. (e.g. 7 V) to compensate for gain drift. These could cause some minor upset for the AZ measurements. Overall it does not look that bad given the hard conditions. Though there might be similar adjustments under normal conditions as well - only less frequent (and smaller if made the right way).