EEVblog #683 - Rigol DS1000Z & DS2000 Oscilloscope Jitter Problems

[EEVblog]

If they fix future scopes without changing the number, then if you order a scope it will be pot luck as to whether you get a "good" or a "bad" scope.

But if the scope meets specifications and has proven to perform fine in every way on everyones scope, then what's the problem?
The real issue is here is whether or not the fix has truly fixed the problem, then if it has then I don't see the problem.
Either it's fixed or it ain't.
If the clock is still lousy but doesn't affect the performance in any way, then it's just academic to say it's a "bad" scope".

[Optisat]

Sorry, board version 0.1.1.  I have no idea where the heck I got the 0.2.3.

I think you read it from the screen!  I got the same thing on my 1054z on one occasion and the next time I looked at it it was back to 0.1.1.  There's something funky going on when displaying the board version.  Has anyone else noticed this?

I have a 1054z with SW 04.01.SP2 it first showed board version 0.1.1 later on showed board version 0.2.1 after upgrade back to board version 0.1.1.

[Lightages]

But if the scope meets specifications and has proven to perform fine in every way on everyones scope, then what's the problem?
The real issue is here is whether or not the fix has truly fixed the problem, then if it has then I don't see the problem.
Either it's fixed or it ain't.
If the clock is still lousy but doesn't affect the performance in any way, then it's just academic to say it's a "bad" scope".

I agree mostly. If the "fixed" firmware was installed from the start then nobody would have noticed the jittery PLL. If there are still problems or other artifacts then Rigol will lose some face and sales.

[tggzzz]

If they fix future scopes without changing the number, then if you order a scope it will be pot luck as to whether you get a "good" or a "bad" scope.

But if the scope meets specifications and has proven to perform fine in every way on everyones scope, then what's the problem?
The real issue is here is whether or not the fix has truly fixed the problem, then if it has then I don't see the problem.
Either it's fixed or it ain't.
If the clock is still lousy but doesn't affect the performance in any way, then it's just academic to say it's a "bad" scope".

Yes and no.

If it meets specs then you can't expect a refund. But by analogy, remember the effect of the Pentium FDIV bug in which the processor gave the right result most of the time.

If it actually doesn't affect performance or doesn't affect performance in a way that matters to you, then I agree. But that's different from "not affecting performance in any way that I have noticed up until now". People will put up with almost anything, provided they are expecting it and understand the causes.

[EEVblog]

If it meets specs then you can't expect a refund.

You'll get argument from me.
It's a case of either this fix is solid and works for everyone, or it's still potentially flaking in some way.
First case fine, 2nd case is not fine.
It's just like the overclocked ADC's in the old 1052E. Not good design practice, but there was not one single reported case I am aware of where that was a problem. They got away it.
If Rigol get away with this clock in the same way, then so be it.

[EEVblog]

I agree mostly. If the "fixed" firmware was installed from the start then nobody would have noticed the jittery PLL. If there are still problems or other artifacts then Rigol will lose some face and sales.

Exactly, just like no one knew or cared about the ADC overclock until it was discovered. And then the mud sort of stuck to Rigol, but it didn't affect the scope sales in any measurable way AFAIK. Some mud will stick because of this, but unless it proves to go sour again, I don't think it will hurt them or their sales.
It probably needs a few months of stats at least to see. But not everyone has the capability to test the clock, or desire to break their warranty seal, and that could potentially still be a problem for Rigol.
If they are smart they will get a hundred units and get the spectrum of the clock on each one, and do it over temperature.

[tautech]

I agree mostly. If the "fixed" firmware was installed from the start then nobody would have noticed the jittery PLL. If there are still problems or other artifacts then Rigol will lose some face and sales.

Exactly, just like no one knew or cared about the ADC overclock until it was discovered. And then the mud sort of stuck to Rigol, but it didn't affect the scope sales in any measurable way AFAIK. Some mud will stick because of this, but unless it proves to go sour again, I don't think it will hurt them or their sales.
It probably needs a few months of stats at least to see. But not everyone has the capability to test the clock, or desire to break their warranty seal, and that could potentially still be a problem for Rigol.
If they are smart they will get a hundred units and get the spectrum of the clock on each one, and do it over temperature.

There has been comfirmation of temperature sensitivity by 3 members pre latest firmware.
https://www.eevblog.com/forum/blog/eevblog-683-rigol-ds1000z-ds2000-oscilloscope-jitter-problems/msg577560/#msg577560
Does it still exist now, as yet it is another unkown.

[EEVblog]

There has been comfirmation of temperature sensitivity by 3 members pre latest firmware.
https://www.eevblog.com/forum/blog/eevblog-683-rigol-ds1000z-ds2000-oscilloscope-jitter-problems/msg577560/#msg577560
Does it still exist now, as yet it is another unkown.

That comment is dated pre the latest firmware?

[nixfu]

Arguing if their clock signal is not high enough quality is not worth much to me.    Lots of products have used shaky PLL clock signals and designed around them and used software compensation for temperature drift etc.   

Frankly, I am still impressed that Rigol actually worked on this issue as asked to do so by the community.

The latest version seems to be a huge improvement and even if it's not perfect for some edge cases, their response make me feel like they are still willing to keep improving.

[tggzzz]

Arguing if their clock signal is not high enough quality is not worth much to me.    Lots of products have used shaky PLL clock signals and designed around them and used software compensation for temperature drift etc.   

If software is used to keep frequency drift within tolerance, then that is a perfectly respectable and satisfactory solution.

I rather doubt software can correct jitter though!

[rf-loop]

I agree mostly. If the "fixed" firmware was installed from the start then nobody would have noticed the jittery PLL. If there are still problems or other artifacts then Rigol will lose some face and sales.

Exactly, just like no one knew or cared about the ADC overclock until it was discovered. And then the mud sort of stuck to Rigol, but it didn't affect the scope sales in any measurable way AFAIK. Some mud will stick because of this, but unless it proves to go sour again, I don't think it will hurt them or their sales.
It probably needs a few months of stats at least to see. But not everyone has the capability to test the clock, or desire to break their warranty seal, and that could potentially still be a problem for Rigol.
If they are smart they will get a hundred units and get the spectrum of the clock on each one, and do it over temperature.

There has been comfirmation of temperature sensitivity by 3 members pre latest firmware.
https://www.eevblog.com/forum/blog/eevblog-683-rigol-ds1000z-ds2000-oscilloscope-jitter-problems/msg577560/#msg577560
Does it still exist now, as yet it is another unkown.

I have previously reported temperature sensitive jitter related to guite long trigger delay times. (in my unit have not seen this "5us" (and multipliers) special jitter.  This long delay jitter was with all delay times (best visble over 50~100us delays) . It was with cold scope and then disappear after some minute warming and then start agen after warm up continue more some more minutes.

** This jitter I can not find in my individual DS1074Z at all  after update. 

** AC trigger jitter and position shift:  well fixed.

(Sidenote: And this is natural. With longer trigger delay, example with 100ms delay there is nearly 10ns "walking and jumping  around". This is natural amount of cheap oscilloscope cheap reference oscillator instability. (in my measurement 10second watching time peak to peak roughly around 1ppm after oscilloscope was well temp stabilized.)

[DanielS]

Frankly, I am still impressed that Rigol actually worked on this issue as asked to do so by the community.

I would be more surprised that Rigol let it slip for over a year since it was first reported before fixing it - time base stability is "somewhat" of a big deal on an oscilloscope.

Credibility is almost everything in the T&M industry: how many people who need accurate measurements would have trust issues with a brand that quietly shipped tens of thousands of defective scopes regardless of price point while attempting to make a push into higher-end markets? Reputation is built on consistent quality and performance standards across the entire portfolio; it takes only one mishandled problematic product to cast doubt across the whole range and here, Rigol has the whole 1000z and 2000a series being scrutinized for this clock issue.

Rigol knew about it for a year but did not fix it until it went viral... props for finally fixing it (hopefully) but also shame on you for sweeping the issue under the rug until someone tripped on the lump and brought the issue back into the spotlight.

[DanielS]

If software is used to keep frequency drift within tolerance, then that is a perfectly respectable and satisfactory solution.

I rather doubt software can correct jitter though!

Since the "jitter" must be correlated to something else to produce its 10µs periodicity, I suspect there is a 1-bit DAC involved in the PLL's control loop. If this signal is generated by the FPGA, then Rigol may have reduced their 1-bit DAC noise by increasing the output modulation frequency and modifying the bit pattern. Ex.: a naive 1-bit modulation may leave the output high for X consecutive ticks and then low for Y consecutive ticks while more complex fractional modulation would interleave highs and lows to reduce low frequency content, making the output easier to filter.

In this or similar cases, "software" could indeed fix or at least significantly improve jitter.

[tggzzz]

Credibility is almost everything in the T&M industry: how many people who need accurate measurements would have trust issues with a brand that quietly shipped tens of thousands of defective scopes regardless of price point while attempting to make a push into higher-end markets? Reputation is built on consistent quality and performance standards across the entire portfolio; ...

Precisely.

If I have difficult to identify/locate/fix problems in my equipment, the last thing I want to do is spend a lot of time on a wild goose chase due to their test equipment being faulty. And if there's any doubt about its solidity, then I've no way of knowing whether I must trace or should ignore odd behaviour - which is double plus ungood.

For similar reasons, in all labs I've worked in, putting a suspect RF cable back on the shelf was a hanging offence.

Of course if a company is only targetting the amateur market, then they might reasonably make different choices.

[tggzzz]

If software is used to keep frequency drift within tolerance, then that is a perfectly respectable and satisfactory solution.

I rather doubt software can correct jitter though!

Since the "jitter" must be correlated to something else to produce its 10µs periodicity, I suspect there is a 1-bit DAC involved in the PLL's control loop. If this signal is generated by the FPGA, then Rigol may have reduced their 1-bit DAC noise by increasing the output modulation frequency and modifying the bit pattern. Ex.: a naive 1-bit modulation may leave the output high for X consecutive ticks and then low for Y consecutive ticks while more complex fractional modulation would interleave highs and lows to reduce low frequency content, making the output easier to filter.

In this or similar cases, "software" could indeed fix or at least significantly improve jitter.

Nah! that would be a hardware redesign

More seriously, I struggle to comprehend that anybody wouldn't use the bog-standard delta-sigma techniques for a 1-bit DAC, since they give the best possible output and are extremely cheap inside an FPGA. As for increasing the clock frequency, unless they are using a very low frequency to start with, it would only make a small difference.

[Bud]

I suspect there is a 1-bit DAC involved in the PLL's control loop. If this signal is generated by the FPGA, then Rigol may have reduced their 1-bit DAC noise by increasing the output modulation frequency and modifying the bit pattern.

No nothing like that. The PLL is a single chip with an external low pass filter and a reference 25MHz clock oscillator. The only link to the (main) processor is a SPI bus for loading setup configuration on the PLL, that is all. The modulation is caused inside the loop because the PLL chip is trying to get the internal VCO to match the reference frequency (to lock) via a divider counter and it cannot, so the PLL goes into a cycle with a repeat period defined by the PLL setup and loop filter parameters. The reason the PLL is not able to lock is bad setup configuration and incorrect loop filter parameters.

[tggzzz]

I suspect there is a 1-bit DAC involved in the PLL's control loop. If this signal is generated by the FPGA, then Rigol may have reduced their 1-bit DAC noise by increasing the output modulation frequency and modifying the bit pattern.

No nothing like that. The PLL is a single chip with an external low pass filter and a reference 25MHz clock oscillator. The only link to the (main) processor is a SPI bus for loading setup configuration on the PLL, that is all. The modulation is caused inside the loop because the PLL chip is trying to get the internal VCO to match the reference frequency (to lock) via a divider counter and it cannot, so the PLL goes into a cycle with a repeat period defined by the PLL setup and loop filter parameters. The reason the PLL is not able to lock is bad setup configuration and incorrect loop filter parameters.

Wow. How on earth did that get past quality control?

[Howardlong]

...because hardly anyone had noticed the problem, and probably still wouldn't. Although I wouldn't call it an edge case, there do need to be a certain set of criteria met to see the problem. Stars need to be somewhat aligned, and the Swiss cheese doesn't need too many holes.

In some ways, Rigol are a victim of their own success, had they not had such a successful product, it probably would never have seen the light of day.

I do agree though, the operation of those AD PLLs is such that you do need to take care on the choice of the loop filter parts. They have plenty of documentation and tools to help the engineer with that, if the alleged parts chosen are indeed "wrong", then something should've gone a bit wrong during unit testing, when checking the PLL lock range. As I remember, there's a lock bit available on those devices as part of SPI register set, or maybe they're ignoring that, using it as a write-only part.

[Bud]

If the clock is still lousy but doesn't affect the performance in any way, then it's just academic to say it's a "bad" scope".

The problem is we do not know what else may be wrong. Let me through one in: Math on multiple channels, i.e. add/substruct signals. Since ADC clock is garbage, the digitized signals will contain garbage information, and the Math result may well be garbage.

FFT will be garbage too, since bad clock will get itself right there. This one may be of a less issue if used internally, since FFT on Rigol is a joke in the first place.

Measurements related to timing? May be, who the hell knows! Therefore users will remain in the dark if what they see is a approximation/indication or a proper measurement. People should ask themselves if they trust a company that cannot get a basic circuit right, does not read parts manufacturers datasheets, overclocks chips, naively lasers off chip markings to hide tracks, and does not have guts to pick up poop after itself. As far as it is today, someone called this scope a "shelf decoration". Agreed, for that use it will work darn well.

[leppie]

since FFT on Rigol is a joke in the first place

And while we at it, hi-res mode on the DS1000Z! 

[Bud]

As I remember, there's a lock bit available on those devices as part of SPI register set, or maybe they're ignoring that

That is correct, there is a pin that can be configured for lock detection. This pin is not wired to anything in Rigol, so no way for the main processor to detect lock/unlock condition. Also in the original firmware it was disabled. They enabled it in Beta and MarkL checked it and it indicated the PLL was still unlocked, and anyway because it is not connected to anything the processor would not know.

[orin]

Sorry to tell you Dave, the clock is still junk. The 68.3kHz spurs just at 35dBc is a garbage PLL. 
That is assuming your spectrum analyzer which is also happened to be of Rigol brand, is a trusted one.

I have attached the reference shots done using my 4360-7 PLL at 1GHz with a proper loop filter. Note my RBW was wider than your 300Hz, and still my screenshots are much better. In the 1MHz span shot: the pedestal is at 60dBc. The 10MHz span shot: PFD spurs +/-2.5MHz from the carrier are at 75dBc.

Sadly, does not seem Rigol is ever going to get it right. They should hire a year 1 electronics college  student to redesign the PLL for them.

I hope in your video you did not say the problem is now fixed. Please feel free to use the attached pics in the next revision of your video to inform people what a good PLL spectrum should look like. Side by side with Rigol's they could help deliver a message to Rigol in visual if they have difficulties reading.


EDIT: I took a liberty to make and added a side-by-side picture of Dave's and my PLL screenshots, scaled to the same 10dB/ vertical scale.

Before we go any further, I have to ask how Dave measured that.  Was it with an EM probe or directly connected to the PLL output?  If it was an EM probe, the 68KHz could be coming from somewhere else.  Note that the bad plots with 100KHz modulation also had 68KHz spurs.

[MarkL]

Before we go any further, I have to ask how Dave measured that.  Was it with an EM probe or directly connected to the PLL output?  If it was an EM probe, the 68KHz could be coming from somewhere else.  Note that the bad plots with 100KHz modulation also had 68KHz spurs.

On which post did you see 68kHz with the 100kHz modulation?  I'd like to take a closer look.  On some of the pre-beta plots I'm getting 62.5kHz.

The 68kHz spurs seem to be new starting with the beta.  Here's an example of the 68kHz spurs taken with direct differential probing of the PLL output (beta_main_carrier_640x480.jpg):

  https://www.eevblog.com/forum/blog/eevblog-683-rigol-ds1000z-ds2000-oscilloscope-jitter-problems/msg560788/#msg560788

They also show up in the FFTs which also says it's in the ADC clock.

It's also interesting I'm not getting exactly 68kHz from every plot, which further points to PLL instability.  The next spur over seems to be more consistent at 125kHz.

You have a good point to be suspicious about the E-field probe, but I don't think it's coming from anywhere but the PLL output.

[EEVblog]

Rigol knew about it for a year but did not fix it until it went viral...

To be fair Rigol only knew about the AC trigger coupling issue, which may not have been related at all to the ADC clock issue, we don't know.

[EEVblog]

Before we go any further, I have to ask how Dave measured that.  Was it with an EM probe or directly connected to the PLL output?  If it was an EM probe, the 68KHz could be coming from somewhere else.  Note that the bad plots with 100KHz modulation also had 68KHz spurs.

E-field probe.
I don't think the 68KHz is coming from anywhere else.