Yet another DIY GPSDO - yes, another one

[FriedLogic]

"It's maybe not the best interview ever..."

That's quite the understatement. It's a terrible interview.

It's maybe a bit of an unfortunate combination. Sometimes even those who really know their stuff are not so good at explaining it to a general audience. A good interviewer can guide things along, but in this case Ian appears to be a bit out of his specialist area so can't that help much. Even so, it gives a good bit of context.

The name of the FacePlant engineer is Oleg Obleukhov (also the co-author of the blog post I linked to), and during the entire interview he does not refer so much as once to the essential concepts of Allan variance/deviation, just as in the blog post.

This project does all seem to be very much a work in progress, and there are not a lot measurements given. The Orolia page on their ART card also has very few details. The official announcement was very recent, so hopefully there will be more info sometime.
A statistic like Time Deviation would be needed to give an overview of the performance, but along the way these statistics are just one of many ways of looking at the data.
A time error plot is given for the SA.53s oscillator, which at least gives some indication of the holdover performance.

Clearly the team that designed the FacePlant GPSDO "Time Card" skipped on the fundamentals, tried to compensate for that lack of basic knowledge by overengineering the "clock processing logic" aka "time engine" (the FPGA), worked hard on "re-inventing the wheel" of GPSDOs, and failed.

One excerpt from the blog post: "The time engine’s job is to interpolate in nanoseconds the granularity required between consecutive PPS signals." That translates to "We reinvented a nanosecond resolution TIC using an expensive FPGA, to measure the phase difference between the oscillator and the GPS PPS".

Lars used a $0.50 4046 and a few discrete components to achieve the same thing, and the STM32 GPSDO uses Erik's version of Lars' TIC that requires a $0.10 74HC74 and a few discrete components, plus a few lines of code in the STM32 MCU. And it probably has better overall resolution than the FacePlant TIC "Time Engine" (implemented in the FPGA).

Although the basic GPSDO is important, it's only one small part of what the card has to do. Given the info in the interview, it's these other requirements that would appear to be driving the FPGA selection. The FPGA on the Orolia card looks like one of the higher pin count Cyclone 10 FPGAs.
Although there are low cost TDCs around, they seem to want the Time Card to be the start of a larger project, so I can understand why they would want to integrate it in the FPGA. It also allows a lot of flexibility.

If your $50 million datacenter cannot afford a pair of good $50 rooftop GNSS antennas then I guess yes, in your case holdover specs still matter in 2021. 

The holdover requirement is in case of any GNSS signal loss, not just a bad antenna. This can sometimes be official, like the military testing jamming or whatever, but is probably more often less official. Since the signal is such low power, something like a faulty power supply or inverter can jam GNSS signals over a significant area. If a company is providing any service that needs precise time, it needs to take this sort of scenario into account as there could be serious financial and even legal consequences if the service goes down because of something foreseeable like this. Pretty much any GPSDO supplier will show holdover as one of the specs, especially for rubidium oscillator based units.

Oh, and the most incredible "feature" of the FacePlant GPSDO "Time Card" is probably the most obvious one, and also the reason I mentioned the FacePlant project in this thread in the first place: they spent > $3,000 and are using an 8-layer PCB with plenty of empty space, but they couldn't fit a $5 32-bit ARM MCU in their design. 

The FPGA, which is needed for other things anyway, would likely make it redundant.

[AndrewBCN]

...
This project does all seem to be very much a work in progress, and there are not a lot measurements given.
...

For companies that depend on reliable, precise timing, a "work in progress" product that doesn't provide a clear service out of the box is not an option. There are fully characterized PTP servers available commercially, with guaranteed performance ; here is a ready-to-rack-and-connect example: https://www.cxr.com/en/produits/vcl2156-ntp-ptp-263.html

Disclaimer: I have zero connection with CXR.

[thinkfat]

Well, seems like they don't like the security risks connected with these time appliances. That was one of the drving forces to build an open source solution.

[MIS42N]

I watched about half the video then skipped through and sampled the rest. There didn't seem to be any solid information about why sub microsecond timing is so important. If transaction A is initiated 1us before transaction B but from a distance 1000m further away it will arrive at least 2us later. Either ignore the fact that A arrives after B and process them as if B were first, or create a queue based on time stamps. The queue solution is problematic. How long to wait for message A before processing B? Maybe there is no A, so every message must be stored for the latency of the longest delay. Or have a system that can reverse out any action if a previous stamped action is received. How far does the reversal go? It is the stuff of programming nightmares.

And once in the realm of microseconds, there's many other considerations - interrupt latency in the computer, the fact instructions are executed at discrete intervals, delays in cables.

It seems there are two sorts of time requirements. The sort that would be used for scientific purposes such as imaging a black hole. In that case, the 'time card' is not going to be good enough - the observatories would use something more accurate than a rubidium cell as their base, and try to maintain picosecond synchronisation around the globe. The other requirement is in the commercial world, and given the delays intrinsic in networks a sub millisecond accuracy would be good enough and the 'time card' is overkill.

Just my thoughts. I can see that NTP may not be good enough today. I had a look at my server and it reports variations of about 10ms (which is good enough for my wife who looks at the computer clock and decides it is time to cook dinner). However even a poor OCXO can have sub millisecond holdover for 24 hours.

So is there justification for this 'time card'? The effort around it - to have open standards and a forum - make a lot of sense. The card itself, not so sure.

[AndrewBCN]

Well, seems like they don't like the security risks connected with these time appliances. That was one of the drving forces to build an open source solution.

Uh? What "security risks" do they associate with a commercial PTP server? PTP/IEEE1588 Rev. 2 is a well defined standard, whether you publish the source files for your implementation of it, changes exactly nothing.

Also, the FacePlant PTP server "package" doesn't seem to work out of the box, actually it looks more like a nightmare to integrate into a network - if it works at all. 

[erikka]

Andre, As you were so kind to help me with the right components its was a quick task to assemble your GPSDO on a breadboard and do some experiments.
YOu code is well formatted with relevant comments so it was easy to find my way.

First questions:
Looking at this part of the code and having looked at the frequency measurements this part of the control loop is a bit puzzling:

  // check first if we have the data, then do ultrafine frequency adjustment
  // ultimately the objective is 10000000.000 over the last 1000s (16min40s)
  if ((cbTho_full) && (avgftho >= 9999999.990) && (avgftho <= 10000000.010)) {
   
    // decrease frequency
    if (avgftho >= 10000000.001) {
      if (avgftho >= 10000000.005) {
        // decrease PWM by 5 bits = very fine
        adjusted_PWM_output = adjusted_PWM_output - 5;
      } else {
        // decrease PWM by one bit = ultrafine
        adjusted_PWM_output = adjusted_PWM_output - 1;
      }
      analogWrite(VctlPWMOutputPin, adjusted_PWM_output);
    }
    // or increase frequency
    else if (avgftho <= 9999999.999) {
      if (avgftho <= 9999999.995) {
       // increase PWM by 5 bits = very fine
        adjusted_PWM_output = adjusted_PWM_output + 5;     
      } else {
      // increase PWM by one bit = ultrafine
      adjusted_PWM_output = adjusted_PWM_output + 1;
      }
      analogWrite(VctlPWMOutputPin, adjusted_PWM_output);
    }
  }


Do I read the code correctly that you do nothing if (avgftho == 10000000.000)?
That would imply that avgftho == 10000000.000 is the dead zone. And I indeed see the frequency wander between 9999999.999 and 10000000.001
Is the range of the PWM sufficient to go one decade lower?
If yes you can move ultra ultra fine down when (avgftho == 10000000.000) and up when (avgftho == 9999999.999) to avoid the dead zone.
Or am I missing something?

Next question:
After adjusting the PWM you do not flush/restart the ring buffers so at the next call to adjustVctlPWM the avgtho could still contain a large amount of data from before the PWM adjust.
Is that on purpose? I understand that if your control loop acts very slow it does not matter but if you want the OCXO to be adjusted in time with temperature changes in the room the loop should not be slower than needed. At least if you want to be as good as possible during drifting of the OCXO.

I know your chosen MCU has plenty of flash but it may be a good idea to have a critical review and elimination of duplicated code. A shorter program is often easier to fully understand.

Some small remarks from someone that tried to use your code for learning and experimenting (and if you read your own code again in some years, I've learned this the hard way):
Define symbolic names of the used MCU pins at the top of the ino and use these symbolic names in the code. THis facilitates changing pinning where allowed and acts as documentation of the HW
The control loop preferably should not have magic constants (such as  '5' in the above code) but these constants should all be defined in one place together with an explanation of their value.

Suggest to add some early debugging info using the blue led (which is always available) to help people trying to duplicate your design for error cases like: no PPS, no 10MHz, no display (if enabled), frequency adjust not working as expected

[AndrewBCN]

Hello Erik,
First, thank you very much for taking the time to be my very first beta tester, and for your kind expert comments. It is quite essential to have someone with an outside view of any complex project, and for the STM32 GPSDO, you are that person. Additionally, you have an excellent grasp of all GPSDO concepts since you have built your own using an SI5351.

Let me try to answer your questions as best as I can.

...
Do I read the code correctly that you do nothing if (avgftho == 10000000.000)?

Yes. That is because my initial objective was really to achieve 10E-9 precision. At 10000000.000 +/- 0.001Hz we are already an order of magnitude better than the initial objective, so I guess it's just bonus precision at this point.

That would imply that avgftho == 10000000.000 is the dead zone. And I indeed see the frequency wander between 9999999.999 and 10000000.001
Is the range of the PWM sufficient to go one decade lower?

I honestly don't know. There is a lot of noise in the Vctl line in the breadboard assembly, with the inexpensive power supply I am using. I am waiting to have an STM32 GPSDO assembled on a proper PCB with a good power supply to explore 10E-10 and 10E-11 territory "comme il faut". 

If yes you can move ultra ultra fine down when (avgftho == 10000000.000) and up when (avgftho == 9999999.999) to avoid the dead zone.
Or am I missing something?

No, you are 100% correct. I had not thought of that and I did not understand     at first when you mentioned a deadband around 10000000.000 but now I finally realize that the simple change you suggest would make the control loop gain one bit of resolution with a very simple change in the software. That is going into the next release for sure.     

Next question:
After adjusting the PWM you do not flush/restart the ring buffers so at the next call to adjustVctlPWM the avgtho could still contain a large amount of data from before the PWM adjust.
Is that on purpose? I understand that if your control loop acts very slow it does not matter but if you want the OCXO to be adjusted in time with temperature changes in the room the loop should not be slower than needed. At least if you want to be as good as possible during drifting of the OCXO.

You are right, and I confess again I never thought of that. The FLL control loop is still very much in its "first writing" stage, I have been procrastinating on rewriting it, basically because I wanted o do a lot more testing and collect more data. But I have noticed that even with temperature variations of +/- 5C the control loop as it is can still keep the OCXO frequency at better than 10E-9. And here again I was thinking of waiting for the PCB to collect more serious data with frequency variations vs. temperature.

And before that I want to implement the PLL control loop with your TIC circuit, and the hybrid control loop algorithm!     

I know your chosen MCU has plenty of flash but it may be a good idea to have a critical review and elimination of duplicated code. A shorter program is often easier to fully understand.

Some small remarks from someone that tried to use your code for learning and experimenting (and if you read your own code again in some years, I've learned this the hard way):
Define symbolic names of the used MCU pins at the top of the ino and use these symbolic names in the code. THis facilitates changing pinning where allowed and acts as documentation of the HW
The control loop preferably should not have magic constants (such as  '5' in the above code) but these constants should all be defined in one place together with an explanation of their value.

Suggest to add some early debugging info using the blue led (which is always available) to help people trying to duplicate your design for error cases like: no PPS, no 10MHz, no display (if enabled), frequency adjust not working as expected

Defnitely those are excellent suggestions and I'll try to focus on them for the next major bump in the STM32 GPSDO revision, when I go from 0.04 to 0.05. In particular your suggestion to use the onboard blue LED as an early error indicator is a brilliant idea (another one).     

Please do post a picture of your setup when you have the time!

[AndrewBCN]

I found a very interesting article which compares the performance of various u-blox receiver modules. Since the STM32 GPSDO uses preferably an inexpensive (10€) u-blox NEO M8N GPS receiver module, I think the choice is fully justified when we examine the data provided by the article:

https://hamsci.org/sites/default/files/publications/2020_TAPR_DCC/N8UR_GPS_Evaluation_August2020.pdf

by John Ackermann N8UR

What the data in the article essentially indicates is that the tested u-blox GPS receiver modules all perform more or less the same, when they are used to provide a 1PPS signal in a stationary GPSDO application.

[MIS42N]

That is a really informative paper and it finally explained Allan Deviation in terms I can understand. It also vindicates my decision to use the cheapest GPS units with a 25ns Tic. The sawtooth spreads the 1pps over an interval of about 22ns so if the 25ns window is exactly positioned over the 22ns sawtooth the output will read all zeros. The window only has to move 4ns one way or the other to pick up the occasional top or bottom of the sawtooth. Given the slight fluctuation in received GPS, the sawtooth itself wanders a bit so a few non zero readings are almost certain. That matches what I have observed and gives a sound basis for it. excellent.

[AndrewBCN]

I always check AliExpress for the availability of u-blox GPS modules, and recently the NEO M9N has shown up on the radar. Just to provide the context:

The M9N is > M8N > M7N > M6N, in terms of performance and in terms of price too. (see the article I linked to above)

Right now, the M9N modules cost around $40~50. The M8N modules can be found for around $10~12, the M7N for $7~10 and the bottom-of-the-barrel M6N for $5~8.

So I would say the sweet spot at this time is clearly the NEO M8N modules. I'll post an update when this changes, of course.

[Johnny B Good]

@AndrewBCN

 That was a very interesting document (downloaded and added to my ublox folder). I read it all from start to end. The CEP scatter plot for the M8N closely matched that of my very first (genuine as it happened) M8N module. The subsequent fakes (needed to replace the original on account I managed to blow up the PPS output with a 12v jolt) also showed similar CEP map deviation plots.

 It was interesting to note the almost constant observed sawtooth correction rate (which does depend on how close the tcxo just happens to be to its nominal 48MHz frequency as was later acknowledged in that document).

 If you program the PPS to 10MPPS and monitor this frequency (or better yet, the third harmonic) with an HF receiver using narrow band FM, you can quite clearly hear these saw tooth corrections as ticks which IME can vary at a rate of from 4 per second to one every 25 seconds or longer, depending on the tcxo's temperature (cooling or warming the whole module will vary the rate - a finger tip touch is usually sufficient to reveal this sensitivity to temperature).

 Fortunately, since I'm not interested in such short term (1 to 100 seconds) stability, the saw tooth corrections disappear below the noise at more sensibly chosen 1000 seconds or longer integration intervals as far as its use as a GPSDO frequency standard goes.

 Whilst I still had the M8N based MK I working to compare with the MK II GPSDO, I had the distinct impression that the M8N had a much greater phase shift modulation (up to 50ns at minutes long time scales versus the 6 or 7 ns pk-pk shifts of the M8T). However, I didn't possess a GPS antenna splitter to properly compare them at that time and by the time I'd made one up, I'd already stripped the MK I to test its "Five Volt" 13MHz OCXO at potentially destructive voltages without endangering the rest of the component parts.

 As it eventually turned out, this "Five Volt" ocxo did prove to be a 12 volt unit that just happened to be able to function perfectly fine from a 4.8 to 5.2v supply (mind you, it took a few seconds before it would start outputting a sine square wave on 4.8v which reduced to just a second when powered from 5.2v).

 It had been my very first, one and only, ocxo that, in the absence of a datasheet, I didn't care to risk burning out before getting some use out of my 4 quid purchase - hence my playing safe and assuming it might possibly be a five volt only part.

 By the time I was ready to sacrifice it on the altar of 'scientific research', I'd already gained a few clues to indicate that the experiment wasn't likely to end in tears. Indeed, I raised the test voltage to 14 volts to put its 12 voltedness beyond all doubt.

 I've been meaning to lash up a second MK II on solderless breadboard using another 10MHz ocxo with an M8N module to compare against the performance of the M8T using an antenna splitter to 'level the playing field'. Previous tests had involved an indoors antenna alongside of the external antenna, which setup had only served to confuse these initial comparative test results. Hopefully this time round, I'll be able to get a more definitive answer to the question, "Does replacing the M8N with  an M8T make as much of an improvement as I think it does?"

 BTW, if anyone's interested in my DIY Rb lab reference project, I've retired the analogue temperature controller in favour of an Arduino nano setup, the details of which I posted into a seemingly moribund Rubidium oscillator based topic as per the link below.

https://www.eevblog.com/forum/blog/eevblog-235-rubidium-frequency-standard/msg3652885/#msg3652885

 Although there haven't been any responses so far, the number of times that the attached images have been viewed suggest it's not an entirely dead topic. I'm not the first one to have made such a necro-posting as you'll see if you look about half a dozen postings upthread. Whether I'll get any replies as the first necro-posting received remains to be seen. It's early days yet

[AndrewBCN]

This post is to explain why I have decided to add an optional second MCU to the STM32 GPSDO. Actually I have a very good reason for doing so.

First things first: I am adding an optional PIC12F675 (or PIC12F629) which is one of the smallest PICs available in 8 pin format. It will be programmed to function as a synchronous divider for the 10MHz OCXO output, instead of 7 decade counters. It costs under $1.

Datasheet: https://ww1.microchip.com/downloads/en/devicedoc/41190c.pdf
Divider code: http://www.leapsecond.com/pic/picdiv.htm

Purpose: to provide an accurate 1PPS signal even in the absence of GPS reception, in other words, whether the GPSDO is operating in holdover mode or normal mode.

I still have to add the PIC to the schematic diagram and of course I have to add it to the PCB. In fact I still have to test this idea, I just ordered the PICs and should receive the ICs in two weeks or more.

[bingo600]

Hmm
Why a PIC ?

I guess the vast majority of users here , would already have Arduino (AVR) programming devices in hand.

I'll not be adding that "Option"

Here is a "PicDiv" for Avr ie (T84) , by the late Ulrich Bangert
https://www.eevblog.com/forum/projects/divide-by-10000000/msg2815024/#msg2815024


/Bingo

[AndrewBCN]

Hmm
Why a PIC ?

Because it's the smallest possible chip that does the job perfectly (8-pin DIP).

I guess the vast majority of users here , would already have Arduino (AVR) programming devices in hand.

I'll not be adding that "Option"

Here is a "PicDiv" for Avr ie (T84) , by the late Ulrich Bangert
https://www.eevblog.com/forum/projects/divide-by-10000000/msg2815024/#msg2815024


/Bingo

Choose your favorite solution!

Three reasons why I personally prefer the PIC:
1. I have carefully examined both the PIC and the AVR source code files and the PICdiv code is much, much better quality.
2. The AVR T84 is a 14 pin DIP, and 3 x more expensive than the PIC.
3. The PIC can be easily programmed using the STM32F411CEU6 Black Pill.

[bingo600]

Choose your favorite solution!

I will

[thinkfat]

Me in your shoes, I'd try to stop the feature creep at a point Anyone wanting a 1PPS output derived from the 10MHz, can easily add their own "PicDiv" themselves. Keep in mind that a 1PPS output is not worth a lot if it's not aligned to an actual reference time, e.g. UTC seconds.

[SteveyG]

Hmm
Why a PIC ?

I guess the vast majority of users here , would already have Arduino (AVR) programming devices in hand.

I'll not be adding that "Option"

Here is a "PicDiv" for Avr ie (T84) , by the late Ulrich Bangert
https://www.eevblog.com/forum/projects/divide-by-10000000/msg2815024/#msg2815024

/Bingo

PIC programming hardware is so cheap it shouldn't cause a problem. The PIC is probably a better choice for the job in this application though anyway.

[MIS42N]

Does anyone use a 1pps for anything? It does seem like feature creep. I had a quick look at my own project, I could generate a 1pps from the processor, no need to add another. I would need to use a PIC16F1459 to get an extra 6 pins and output 1pps using the second PWM. Since the processor clock is the OCXO, and the PWM is driven by internal circuitry, the leading edge is hardware timed and not dependent on an accurate software interrupt. It could be aligned with the 1pps from the GPS module within 25ns by slewing the OCXO frequency a bit. Maybe you could do that instead of adding more bits. Those 32 bit processors have lots of inbuilt goodies.

Unless there is a big demand, I'm not going down that rabbit hole.

[AndrewBCN]

...
Keep in mind that a 1PPS output is not worth a lot if it's not aligned to an actual reference time, e.g. UTC seconds.

Good point.
From the PICdiv page:
These dividers also allow for 1PPS synchronization. If pin 4 (Arm) is held low for a second the divider stops and waits. The divider synchronizes to the rising edge of pin 5 (Sync). There is a weak pullup on the Sync pin. If you never plan to stop or sync the divider connect the Arm pin to Vdd, or simply connect the Arm pin to the Sync pin.

[AndrewBCN]

Does anyone use a 1pps for anything?...

A 1PPS signal aligned with UTC has many uses, and if I can get it using a $0.90 8 pin MCU, the cost/benefit ratio is more than acceptable.

In any case, it's optional, like the sensors, display, Bluetooth interface, regulated PSU, phase detector/TIC, etc. If you don't want it, don't populate the PCB with it and don't enable the firmware subroutines for it.

[MIS42N]

A 1PPS signal aligned with UTC has many uses,

I guess that's what I was asking. What are these uses (other than industrial people that would be buying a $1000 GPS unit that provides what they want).

[bob91343]

I have a similar question.  If a 1 pps is to be accurate, its rise/fall time must be short.  For a precision of 1 ppb it would have to be known within one nanosecond.  So its rise and fall must be of that order of magnitude or better, and its amplitude and shape needs to be stable.  Any circuitry that uses it must also be stable; I suppose the precision would be allocated among both the pulse parameters and the triggered circuit constants.

The stuff I don't understand far exceeds what I think I understand.  Here, perhaps a little knowledge isn't a dangerous thing as much as it's misleading.

[thinkfat]

...
Keep in mind that a 1PPS output is not worth a lot if it's not aligned to an actual reference time, e.g. UTC seconds.

Good point.
From the PICdiv page:
These dividers also allow for 1PPS synchronization. If pin 4 (Arm) is held low for a second the divider stops and waits. The divider synchronizes to the rising edge of pin 5 (Sync). There is a weak pullup on the Sync pin. If you never plan to stop or sync the divider connect the Arm pin to Vdd, or simply connect the Arm pin to the Sync pin.

Interesting, but I am not sure if this will work. What will be the Sync signal, 1PPS from the GPS, I guess. Thats very noisy. If you want the 1PPS to be within 1ns of UTC, some trickery in software will be needed.

[iMo]

Those pic dividers are using the pic chip as a logic device (more less it mimics an fpga), not as a traditional "MCU".
The pic is clocked by 10MHz (10MHz is fed from the OCXO into the pic's master clock). Everything inside the pic happens on the rising edge of its clock and the added jitter of the pic's 1PPS output is in XX pico-seconds area, afaik.

[AndrewBCN]

All your questions have been answered here:

https://en.wikipedia.org/wiki/Pulse-per-second_signal