Very Intermittent issue with my 10Mhz reference.

[Zenwizard]

I have a weird issue with my RB reference where every once in a while it will fall out of 1PPS lock with my GPSDO. I have a Samsung GPSDO that is on an outside antenna. Normally I see around 12 satellites 9 being the lowest count and 15 being the highest. The RB unit is a new SRS 725 Op 3 that I bought new for the lab direct from SRS. What is going on is about once every 4 months the TT goes to 999,999,998 sits there for 3 days then falls out of lock. The TT in this case is the time difference between the 1PPS input and the 1PPS output in nano seconds. Usually this sits around 7 when everything is running smoothly.

The 725 is set up to about a 15 Hour integration interval for slow adjustments. My gut tells me that my GPSDO might be starting to show its age as in terms of hours is has substantially more than the 725. If that is the case I have on the acquire list a Leo NTP server that can also provide the 1PPS from the GPS.

I need to keep this system working as the RB is the time basis of the lab and I do not currently have the gear to calibrate it against anything else. It is the best clock I have.
Everything will sync back up after a reboot of the 725 but I would like to not have to do that for stability sake I also don’t want to have to reboot the GPSDO if I can help it either. I should also mention that both units are on high quality full sine wave online UPS units that are functioning normally so power issues should not be the problem but I am ruling nothing out quite yet.

Curious if anyone else has any thoughts as to what might be going on.

Zen

[MIS42N]

I have no idea what is the problem. Is there a logical reason why, 3 days after the TT goes to 999,999,998, it falls out of lock. Is that part of the specification - if the difference is too large for too long then the system decides lock is lost. If that is the case, it is not a part of the problem - it is a logical outcome of the TT going to 999,999,998. Otherwise it is an additional mystery to solve.

Is the Samsung GPSDO 1pps generated internally (has holdover capability) or is it the 1pps from the GPS module - maybe the GPS module is intermittently not producing a 1pps. If the Samsung can output NMEA data from the GPS module, can you log it and when an 'incident' occurs - replay it through suitable software to see if there's an anomaly in the signal.

Like most problems, "divide and conquer". Must be really annoying it occurs so infrequently. If the output is not required 24/7 you could try inducing the problem. Disconnect the Rb from the GPSDO for a second etc. Tests that don't require either device to be powered down.

[Zenwizard]

Yea this one is going to be LONG to trouble shoot. I have a sneaking suspicion that the GPSDO is loosing 1PPS and when it is coming back the sync is almost exactly wrong. To get the TT to go ti 999,999,998. I will see if I can log the data from the GPSDO The 725 is should back in to lock but the loop time is quite long to prevent jerking around the 10Mhz ref. The loop integration is 18 hours.

It is currently at 317 but if i let it run for a few more days it will get down to about 7ns which is quite good. The 725 is set if it spends too much time out of lock to the 1PPS it will go in to holdover and rely on just the RB unit. The 725 is really a shell and distribution box for a PRS10 at it core.

But I will look in to what kind of logging I can make happen.

Zen

[MIS42N]

It's there in the specification of the FS725 Rubidium Frequency Standard manual - "Holdover 72 hours". That explains the 3 days between going out of spec and eventually saying lock is lost.

So only the jump to TT 999,999,998 needs explanation. Is this an artifact (i.e. what the system says if it can't make out what to say) or real. If it is real it is 2ns short of 1 second. That points me toward a software glitch. Is the software up to date? If it is, communicating with the manufacturer might be beneficial. The whole point of holdover is to ride through loss of 1pps and it looks (to me, many thousands of miles away) like this may not be working. Hence the test to deliberately disconnect and reconnect the 1pps from the GPSDO to the Rb. If everything is working as documented, this should not have any ill effect. [as an aside, many years ago I wrote a sliding windows serial protocol driver - the protocol was called X.PC - and to test it, just disconnected and reconnected the RS232 plug and socket during transmission. It passed, 100% received without error]

And when you do a reboot of the Rb, is that a power cycle or a software command. As I read it, using the software reset keeps power on, just resets the software. I'd do that the day you see the 999,999,998 so lock is reestablished in a timely manner without disturbing the Rb cell.

Is the 4 months gap significant? is it always the same time or just "about 4 months". I had a problem that happened when a particular cleaner was rostered to a particular room. As they left, they turned off the power (which they were all supposed to do but nobody checked). The timing was such that it glitched the software that did a computer backup at that time and it failed. This was in an office hundreds of miles away. But asking "what happens at these times" eventually tracked the problem.

[donlisms]

Huh.  In 10's complement, that's -2.  I wonder what a TT is.

[MIS42N]

Huh.  In 10's complement, that's -2.  I wonder what a TT is.

In the first post OP says "The TT in this case is the time difference between the 1PPS input and the 1PPS output in nano seconds."

[donlisms]

Thanks.  Sorry.

[Zenwizard]

I dove in to the manual for a while. It seems a high TT might be ok with this unit. TT is the time in ns relative to the incoming and outgoing 1PPS. If 1PPS inbound arrives after the 1PPS output by 5ns TT will be 5. If the inbound 1pps arrives 5ns BEFORE the outbound 1PPS TT will be 999999995. So this seems to as least be normal behavior for the 725. But now the question is why the fall out of lock. I have updated the PT parameter to 14. This will now give me a 18.5 hour loop reaction. It will slow down the tuning but will work with way more averages. So if there is any noise on my 1PPS line this might help.

I also ordered Leo's NTP server as this has a 1PPS output as well so If my GPSDO is struggling I have a second test source for 1PPS in the lab.

Zen

[MIS42N]

I have updated the PT parameter to 14.

This may not be a good idea. The larger the PT parameter, the harder it is to maintain a lock. If you have done the calibration of the SF value accurately then it may work.

I have not dealt with this equipment, but I understand the fundamentals of GPSDO design. In the manual it talks of using a GPS receiver, not a GPSDO, as the source of 1pps. So they expect a fair bit of jitter in the 1pps input. Yet they specify PT8 as a default. The LOWER the PT value the more likely you are to maintain a lock but at the expense of some phase noise from the Rb as the PLL pulls it.

I don't know what you use the 10MHz output for. If frequency errors of 10^-11 (ie 100uHz at 10MHz) are tolerable then I suggest don't mess with the default PT.

You could look at something based on the u-blox ZED-F9T for your 1pps. Not cheap but the dual band eliminates the ionospheric delay problem. Reputedly this can affect the 1pps up to 25ns in a day for single band (usually L1) receivers. Quote "Timing accuracy <5 ns (1-sigma, clear sky, absolute mode)".

[DimitriP]

I don't know what you use the 10MHz output for. If frequency errors of 10^-11 (ie 100uHz at 10MHz) are tolerable then I suggest don't mess with the default PT.

Depending on what's
"tolerable" doing away with the 1pps should be feasible too since SRS is marketing the FS725 as a "stand alone 10MHz frequency standard with an estimated 20-yar agin less than 0.005 ppm. ....The unit's low aging eliminates the need for an external frequency reference in many applications" - maybe this is not one of them    (https://www.thinksrs.com/downloads/pdfs/manuals/FS725m.pdf)

Priced at a little over 3K, I would have attempted to contact SRS before attempting to crowd-source a solution and going shopping  for best-guess replacement equipment. 

[MIS42N]

Depending on what's "tolerable" doing away with the 1pps should be feasible too since SRS is marketing the FS725 as a "stand alone 10MHz frequency standard with an estimated 20-yar agin less than 0.005 ppm. ....The unit's low aging eliminates the need for an external frequency reference in many applications" - maybe this is not one of them

The advantage of an Rb cell is its stability. The frequency may be wrong but it is consistent. Introducing 1pps from a GPS receiver allows the frequency to be corrected. The combination is a good one, the GPS ensures the frequency is accurate, the Rb cell ensures it is stable. I think most users would like both.

Ageing of 0.005 ppm is quite a lot. To put it in perspective my GPSDO is accurate to better than 0.001ppm (or 1ppb - one part in a billion) using a very cheap OCXO. The parts cost less than $50US (except for the 12V power supply). The current set up with a timing grade GPS (U-blox LEA-M8T) looks to improve this by an order of magnitude (0.1ppb). After a year of use, an Rb cell can drift that amount. So it makes sense to have a reasonable (less than say $300) cost GPSDO to ensure the accuracy of a $3K reference. The combination should be good for an accuracy of 0.01ppb. Trying to get past that is difficult with single band GPS, but 0.001ppb should be possible with a dual band GPS.

It all depends on what the user needs.

[thermistor-guy]

...
I should also mention that both units are on high quality full sine wave online UPS units that are functioning normally so power issues should not be the problem but I am ruling nothing out quite yet.
...

Zen

What type of UPS units do you have - online double conversion, line-interactive, offline?
https://www.vertiv.com/en-asia/about/news-and-insights/articles/educational-articles/what-are-the-different-types-of-ups-systems/

Do the UPS units' logs show any brown-out or switchover events?

[Zenwizard]

I am using online double conversion UPS units. They are Leibert UPS units. from before the rename to Vertiv. The GPS module is a Samsung unit. I did back the PT back down to 8 at 14 it was moving but the time frame was WAY too long. The 725 was taking over 4 days to settle. Falling in and out of lock still not fully settled. Everything is currently locked and stable. but I am suspecting the GPSDO is starting to show its age. I have a NTP server from Leo coming it was just taking a bit to make it through customs. I am interested to see how the Leo will help things, The 1PPS is specified at 60ns of jitter.

Some of my needs for the unit
Testing GPSDO units and evaluating 10Mhz references (has been requested)
Calibrating Scopes and other time bases
Yes I am aware that I am calibrating 3% tolerance scopes with a Rb units to tolerances far tighter then they need to be. But they can do it if only briefly due to drift.
I also have a need to calibrate and align some spectrum analyzers.

10Mhz reference for the lab to tighten up as much gear as I can
HP spectrum analyzers 3X one of the units tops out at 18.5 Ghz
Rigol function generators 2X
Frequency counters 2X

When looking at other GPSDO units I did not like how much the GPSDO was moving around during long term multi day measurements saw 250uHz of jitter over the course of about an hour. One GPSDO unit was even falling out of lock for a second or two.

The price was high but reasonable given the cost of a Cesium beam tube ($50,000.00 new) or Hydrogen Maser ($195,000.00 used)

Zen

[MIS42N]

When looking at other GPSDO units I did not like how much the GPSDO was moving around during long term multi day measurements saw 250uHz of jitter over the course of about an hour. One GPSDO unit was even falling out of lock for a second or two.

How well a GPSDO performs is down to a lot of factors:
1. The position and type of receiving antenna. A quadrafiliar with a clear view of the sky is best. A patch antenna indoor is a poor choice. The quadrafiliar has a better rejection of reflections that a patch.
2. The type of GPS module. A general purpose module like a U-blox NEO-M8N performs OK in a mobile situation but for fixed position a timing module like U-blox NEO-M8T performs better
3. Disciplining algorithm used. Cheaper GPSDOs often use fast PLL parameters to get a quick lock but track the signal (usually 1pps) from the module tightly, meaning the output will follow short term deviations of the GPS signal - a likely cause of short term jitter.
4. Quality of the control voltage applied to the disciplined oscillator. This could be from a DAC or PWM with limited granularity (larger voltage steps) and/or poorly regulated supply voltage.
5. Quality of the oscillator being disciplined.
If 1. and 2. are addressed well, it allows 3. 4. 5. to be lower spec and still get reasonable results. Where 1. or 2. are less than optimal, the GPS signal needs to be averaged over a longer period (point 3.), which then requires 4. and 5. to be upgraded. So it pays to put effort into acquiring the best GPS signal. Which is why a receiver like the u-blox ZED-F9T is a good choice.

Note that feeding 1pps from a good GPS receiver to the Rb unit without going via a GPSDO is a viable option. From my quick scan of the manual, it appears all of the items 3. 4. 5. are in the unit and the disciplining algorithm can be tuned. Maybe the manufacturer could give some guidance about this.

So seeing 250uHz jitter is not too bad. That's 2.5 parts in 10^-11. That's less than 0.5Hz at 18.5GHz. How good do you want it to be?

A GPSDO that falls out of lock is not a worry if it is getting poor GPS data (1. and 2. above). Otherwise, it's a fail from me.