Comparing the performance of 17 different GPSDOs

[texaspyro]

I recently completed some comparison tests of the PPS and 10 MHz outputs of 17 different GPSDOs.

The tests were done with a TAPR TICC time interval counter running in timestamp mode and the data was acquired and processed with Lady Heather.   The TICC was clocked by a HP-5071A cesium beam oscillator with the high-performance tube.   The TICC has a resolution of around 60 picoseconds.

The PPS signal from the GPSDO was connected to channel A of the TICC (for those GPSDOs that have an easily accessible PPS signal).  For receivers without a PPS signal available, Heather shows ADEV, HDEV, MDEV, and TDEV data for the 10 MHz signal.   For receivers with a PPS,  ADEV is shown for the 1PPS and 10 MHz signals.

The GPSDO output frequency (10 MHZ) was connected to channel B of the TICC through a TAPR TADD-2 Mini divider to get a 1 Hz output.

Note that PPS and 10 MHz plots have a display averaging filter applied.   Without this filter the plots mostly show noisy "grass" and not any interesting details.  A side effect of the display filter is to reduce the "span" (difference between the max and min values seen) of the plot data.  Typically the spans are around twice the value shown in the plot header, but for a couple of receivers it is up to a 10x reduction of the raw span.

Each GPSDO was allowed to warm up for at least 24 hours,  or longer... until the DAC output voltage slope leveled off.

The GPSDO antenna was a L1/L2/GPS/GLONASS/BEIDOU antenna from China.  It is mounted on a 1 meter tripod and fed into an 8-way amplified L1/L2/Glonass/Beidou splitter.  With a L1/L2 survey receiver this antenna produces location error ellipses in the 5-10mm range.  The antenna is in a rather horrible location for multi-path and sky view.   Lots of tall trees and a 2 story stucco-over-wire mesh house.

Also note that the terminator on the 1PPS signal was defective... it was actually open circuit.  The BNC cables were less than 1 meter long.  A couple of tests with a good terminator after-the-fact showed little effect on the results.

This plot is of the HP-5071A connected to the TICC.  Since the TICC is also clocked by the 5071A, this plot basically shows the noise level of the measurement system.

[texaspyro]

Here are the receivers that speak Trimble TSIP messages.  These include the Thunderbolt and NTBW/NTPX/NTGS "telecom" GPSDOs.   Also a Datum Starloc II GPSDO (a sort-of Thunderbolt with the must bug ridden firmware that I have ever seen).

[texaspyro]

The UCCM receivers are a small "telecom" GPSDO that are usually rather inexpensive.   There are several versions of these made by Trimble,  Symmetricom,  and Samsung.  They tend to have surprisingly good performance.

[texaspyro]

These are receivers that have use SCPI commands.  They include the HP Z3801A and 58503B.  Also the Lucent Z3812A the XO (REF0) half of the Lucent KS-27361 GPSDO system.  The Z3812A is one of the devices where the display averaging filter reduces the actual "span" of the data by a factor of 10.

[texaspyro]

Here are some other GPSDOs... you are unlikely to come across many of these, except for the Trueposition and Star4 devices.

brandy.gif is a Brandywine GPS-4
zyfer.gif is a Zyfer Nanosync 380
rftgm-xo is the XO side of a Lucent RFTG-m GPSDO.  The output is 15 MHz instead of 10 MHz.

[cdev]

Thank you for doing all this work!

[uncle_bob]

Hi

Very cool !!!

Any idea what happened to the OSA-4530 GPSDO about half way through the run?

Bob

[cdev]

Okay, please correct me if I am guessing this wrong.

The blue and red lines basically are counting the same thing, just in different ways, and should in theory be closer to one another the smaller the short term (one second) variability of the GPSDO's own 1PPS is from the TAPR's PicDiv - the 10MHz divided by 10 million, right?

The lines start out on the left showing the higher values because there they are only averaging the shorter time, which has elapsed up to that point, and the endpoint is always the same (20,000) because thats the point you stopped the test for that unit, so then the final value is that?

The measured GPSDOs can't be better than the accuracy of the time base used to evaluate them, right?

I am wondering what the ambient temperature conditions were for the devices under test, were they all around the same?

You let them run for a minimum of 24 hours before the measurements, were they in the open air or inside some kind of box?

[uncle_bob]

Okay, please correct me if I am guessing this wrong.

The blue and red lines basically are counting the same thing, just in different ways, and should in theory be closer to one another the smaller the short term (one second) variability of the GPSDO's own 1PPS is from the TAPR's PicDiv - the 10MHz divided by 10 million, right?

The lines start out on the left showing the higher values because there they are only averaging the shorter time, which has elapsed up to that point, and the endpoint is always the same (20,000) because thats the point you stopped the test for that unit, so then the final value is that?

The measured GPSDOs can't be better than the accuracy of the time base used to evaluate them, right?

I am wondering what the ambient temperature conditions were for the devices under test, were they all around the same?

You let them run for a minimum of 24 hours before the measurements, were they in the open air or inside some kind of box?

Hi

The channel A and channel B data (mostly red and blue, but sometimes not) would be the same if the GPSDO's internal divider was doing as good a job as the external divider used.

The GPSDO numbers *could* be a lot better at the shorter time periods. The stability of the 5071 may be limiting things below the 10 to 50 second range. Once you get past that (longer time periods) it is unlikely that the Cs standard is the limiting factor.

The main measurement shown is an ADEV. Effectively it's a standard deviation of frequency differences. There is a lot of detail out on the net about just how it is computed and its basic limitations. As you get to longer times, the amount of data going into the standard deviation calculation is less and less. Thus the confidence in the value shown gets worse and worse.

Bob

[texaspyro]

Any idea what happened to the OSA-4530 GPSDO about half way through the run?

Nope,  the OSA is a rather, uhhh,  shall we say weird beastie.   The plot from Heather controlling the unit did not show any abnormalities, signal dropes, etc at the time.  I have seen similar tantrums with that unit.

[texaspyro]

I am wondering what the ambient temperature conditions were for the devices under test, were they all around the same?

You let them run for a minimum of 24 hours before the measurements, were they in the open air or inside some kind of box?

They were in open air or whatever enclosure they were in.  Room temperature varied less than 1 degree C during the runs.

[edpalmer42]

Okay, please correct me if I am guessing this wrong.

The blue and red lines basically are counting the same thing, just in different ways, and should in theory be closer to one another the smaller the short term (one second) variability of the GPSDO's own 1PPS is from the TAPR's PicDiv - the 10MHz divided by 10 million, right?

As Texaspyro stated in the initial post, the blue trace (Channel A) is the PPS and the red trace (Channel B) is the 10 MHz.  If the DUT didn't have a PPS output, he showed ADEV, HDEV, MDEV, and TDEV data for the 10 MHz signal.  I noticed that a couple of the units showed differences between the PPS and the 10 MHz outputs at low values of tau.  It would be interesting to know the explanation for that.

The lines start out on the left showing the higher values because there they are only averaging the shorter time, which has elapsed up to that point, and the endpoint is always the same (20,000) because thats the point you stopped the test for that unit, so then the final value is that?

Not really.  An ADEV graph is actually a 3-way composite graph that shows the worst of the measurement system, the reference, and the DUT.  Usually, at low values of Tau, the measurement system is the weak link.  The TICC used here has a resolution of 60 ps, i.e. 6e-11.  Notice that none of the graphs have a 1 sec. value less than 6e-11, including the system noise floor graph.  The 5071 high performance option has a 1 sec. spec. of <5e-12 so it's unlikely that it's affecting things.

The measurement system continues to dominate the measurement until the graph flattens out, usually in the 10 - 50 sec. range.  Now you're seeing the effects of the OCXO itself.  A better OCXO will show a lower plateau in this area.  There are a few graphs that show this plateau in the low-to-mid e-12 range.

The plateau ends when the graph turns and drops lower and lower for higher values of Tau.  This is the 'signature' of the GPS system.  If you draw a line from 1e-10 @ 100 sec. that goes down and to the right at a slope of 1 decade per decade (e.g. 1e-11 @ 1000 sec., 1e-12 @ 10K sec., etc.) you'll see that everyone ends up parallel to this line.  Some units run on one side of that line, some on the other.

The measured GPSDOs can't be better than the accuracy of the time base used to evaluate them, right?

Well, you can't get measurements that are better than the timebase, but that's not much of a limitation.  Over a period of a day, the drift of a Rb is low enough that it doesn't have much of an effect on the readings.  Also, an ADEV graph has lots of information that is completely seperate from the frequency accuracy.  e.g.:  How clean is the OCXO?  How are the PLL parameters affecting the stability of the output?  If you're just looking for frequency accuracy, ADEV isn't really the right tool for the job.

Ed

[uncle_bob]

Hi

...... errr .... I believe the data was done against an HP 5071 Cs standard rather than against a Rb ....

Bob

[edpalmer42]

Hi

...... errr .... I believe the data was done against an HP 5071 Cs standard rather than against a Rb ....

Bob

Yes Bob, I know.  My point was that even a Rb is good enough to get useful ADEV curves from a GPSDO when you're measuring over a period of one day.  I believe that cdev was hinting at the question as to whether an amateur who doesn't have a Cs standard can make any useful measurements in this area.  My answer is: Yes!   

Ed

[cdev]

Ed, we've communicated about this before but I had a completely different mental image of what you meant, so this explanation is very informative to me.

Okay, please correct me if I am guessing this wrong.

The blue and red lines basically are counting the same thing, just in different ways, and should in theory be closer to one another the smaller the short term (one second) variability of the GPSDO's own 1PPS is from the TAPR's PicDiv - the 10MHz divided by 10 million, right?

As Texaspyro stated in the initial post, the blue trace (Channel A) is the PPS and the red trace (Channel B) is the 10 MHz.  If the DUT didn't have a PPS output, he showed ADEV, HDEV, MDEV, and TDEV data for the 10 MHz signal.  I noticed that a couple of the units showed differences between the PPS and the 10 MHz outputs at low values of tau.  It would be interesting to know the explanation for that.

The lines start out on the left showing the higher values because there they are only averaging the shorter time, which has elapsed up to that point, and the endpoint is always the same (20,000) because thats the point you stopped the test for that unit, so then the final value is that?

Not really.  An ADEV graph is actually a 3-way composite graph that shows the worst of the measurement system, the reference, and the DUT.  Usually, at low values of Tau, the measurement system is the weak link.  The TICC used here has a resolution of 60 ps, i.e. 6e-11.  Notice that none of the graphs have a 1 sec. value less than 6e-11, including the system noise floor graph.  The 5071 high performance option has a 1 sec. spec. of <5e-12 so it's unlikely that it's affecting things.

The measurement system continues to dominate the measurement until the graph flattens out, usually in the 10 - 50 sec. range.  Now you're seeing the effects of the OCXO itself.  A better OCXO will show a lower plateau in this area.  There are a few graphs that show this plateau in the low-to-mid e-12 range.

The plateau ends when the graph turns and drops lower and lower for higher values of Tau.  This is the 'signature' of the GPS system.  If you draw a line from 1e-10 @ 100 sec. that goes down and to the right at a slope of 1 decade per decade (e.g. 1e-11 @ 1000 sec., 1e-12 @ 10K sec., etc.) you'll see that everyone ends up parallel to this line.  Some units run on one side of that line, some on the other.

The measured GPSDOs can't be better than the accuracy of the time base used to evaluate them, right?

Well, you can't get measurements that are better than the timebase, but that's not much of a limitation.  Over a period of a day, the drift of a Rb is low enough that it doesn't have much of an effect on the readings.  Also, an ADEV graph has lots of information that is completely seperate from the frequency accuracy.  e.g.:  How clean is the OCXO?  How are the PLL parameters affecting the stability of the output?  If you're just looking for frequency accuracy, ADEV isn't really the right tool for the job.

Ed

I wasn't thinking about any particular hardware, actually, just as a general principle, but the above is much more useful.

Its a shame that time-nuts or eevblog doesn't collect all of this information in one place!

[TheSteve]

Thanks for your efforts!
I feel as though I can be happy with my Trimble UCCM for 42 bucks.

[edpalmer42]

Ed, we've communicated about this before but I had a completely different mental image of what you meant, so this explanation is very informative to me.

Isn't it great when it finally 'clicks' and you understand a bit more than you did before?     Been there, done that (many times) and I hope it happens many more times for me!   

Its a shame that time-nuts or eevblog doesn't collect all of this information in one place!

+1 for this.  No, make that + a few million.  There have been a few attempts to start a wiki, but nothing has really worked.  KO4BB has a good timing wiki, but the content is far too static to make me happy.  As a result, we have to keep reinventing the wheel and asking the same questions - particularly where new Time-Nuts are  concerned.  One of the Time-Nut Masters asked me if eevblog was a good place to put his information.  I couldn't really recommend it because the forum doesn't lend itself to that type of usage.  Yes, I know that Dave has a Wiki, but it's not set up for that type of content.

All that has to happen is for one of the Masters to get hit by a bus and we'll lose a lot of priceless information.  I realize that many (most?) of the Masters are professionals who have to deal with ugly things like Non-Disclosure Agreements, but still.....

Ed

[cdev]

I'm worried about Lars, if any of you don't feel you would be imposing, please check up on him.

[tvb]

Its a shame that time-nuts or eevblog doesn't collect all of this information in one place!

True. I know. But the internet is a large place. It's ok for there to be more than one electronics forum, or test equipment forum, or GPS forum, or time & frequency forum. Each has its own unique character and content. Note that personal labor-of-love sites like febo.com (John A) and leapsecond.com (Tom VB) or ke5fx.com (John M) or the time-nuts mailing list have been around far longer than eevblog (2009). For that matter, longer than Youtube (2005) or Facebook or Twitter, etc.

Niche hobby sites with long history, solid reputations, curated content, and well-indexed by google are worth something. Not all discussion needs to be on a single site, or commercial forums like eevblog. What we try to do with time-nuts is actively promote those who have their own web sites, or blogs, or forums, or just email access. The diversity is useful. What unifies the list is the narrow topic of precise time & frequency. More details on the main time-nuts page: www.leapsecond.com/time-nuts. That said, eevblog's web-only interface, easy photo and video attachment, and topical forum model is quite nice.

Anyway, back to GPS. You'll find a nice set of ADEV plots of those 17 GPSDO at: http://leapsecond.com/u/sims/gpsdo17/

[jpb]

I've only just found this thread - I'm not sure how I missed it as I'm on the forum most days.

Thanks for all those measurements.

The Star 4+ result at 20 seconds is quite a bit higher than the value claimed in the manual (also as supplied by Texaspyro).

On page 19 of the manual they claim an ADEV of <= 10^-12 at 20 seconds which is a lot less than the measured 4.6 x 10^-11.

I unfortunately don't have a great reference for doing measurements and my counter has a one shot resolution of 50 psecs which is 2.5x10^-12 on its own.

I have been measuring a Samsung GPSDO against the Star4+ and at 20 secs the ADEV is around 120 microsecs which is 1.2 x 10^-4/10^7 = 1.2 x 10^-11 if my maths is right so if both make an equal contribution then this would be around 8.5x 10^-12 ignoring any correlation (i.e. dividing by sqrt(2)). This is with the Star 4+ time constant set to 1000 secs in Lady Heather. The Samsung is just defaults as I haven't had a chance to wire up the RS232.

EDIT : to correct my calculation (divide by root(2) not 2).
Also I've since made measurements vs an 8663 OCXO (similar to the one used in the Star 4+). At 20 secs the ADEV is almost exactly 100 microHz so 1 x 10-11 and dividing this by root(2) gives 7 x 10^-12 still a lot bigger than the claimed < 1 x 10^-12.

At 100 seconds I get 80 uHz = 5.6 x 10^-12.

[texaspyro]

The TAPR-TICC that I did the measurements with has a 60 ps resolution.   ADEV values for tau <= around 20 secs can be heavily influenced by the TICC resolution and noise.