GPSDO phase between PPS and clock

[MIS42N]

Can there really be as much as 30 to 50ns diurnal variation? If this is true, it would neatly explain why I've been banging my head against a brick wall over the past couple of months in my attempts to eliminate a diurnal chill factor effect on my temperature stabilised (and barometrically compensated) rubidium frequency standard (RFS) project.

It is unlikely any single frequency (i.e. L1 band) GPS receiver will do better than this. The solution is to go to a dual frequency GPS receiver like the u-blox ZED-F9T. To quote from the literature "Stress testing absolute timing over 24 hours achieved a tolerance better than +/-10ns with a standard deviation under 3ns.".

Both rubidium and cesium beams are disciplining oscillators, to get better than 10ns deviation in a day is dealing with stability of better than 1 part in 10^-16 which isn't going to happen in your average back yard. The method of detecting the fine transitions at GHz frequencies is not 100%, it involves sweeping a signal from a slightly lower to a slightly higher frequency and looking for a dip or peak. The sharpness of that is affected by many factors one being the Zeeman effect.

So one could chuck the cesium or rubidium and just go for a really good oscillator. I have seen prices in the thousands of dollars. And let the GPS discipline that with a loop time of hours.

[Johnny B Good]

@MIS42N

 Many thanks for confirming my suspicions over the diurnal instability of the GPS system (L1 bound receiver limitations at any rate) and your observations on the stability of commodity class rubidium oscillators and those 'caesium in a box' lab frequency standards (both of which incidentally, are mere 'secondary atomic standards' outside of a nation's time and measurements' lab such as NIST in the USA).

 Incidentally, I'd just received a Sparkfun ZED-9FT  module yesterday morning and did observe that +/- 10ns jitter on its 1PPS (an accumulation of 2ns or so steps over 5 to ten second intervals) compared to the ruby (and my home made GPSDO for that matter!). I couldn't really detect any longer term drift between the ruby and the ZED-9FT over a two hour or so period (hence my throwing my gpsdo into the mix for good measure).

 However, this was only an initial check, using the multi band GNSS/RTK antenna I'd bought at the end of last year with just such a gps receiver upgrade in mind, using its default settings since I couldn't run the later versions of U-centre in my winXP VM needed to access its configuration interface and recognise it as a USB device).

 Unfortunately, the win7 pro 64 VM, after I'd cleared the login password that I hadn't been able to recall despite my password hint, kept complaining about missing components (even after installing the required redistributables) when trying to start version 22 of U-centre after each re-installation attempt, so I was left with no means to properly configure it'

=======================================================================================================
[EDIT 2024-11-20]

The show stopper in this case was simply due to the lack of support for USB-C  in winXP. I got around this issue by using the host serial interface on the I2C pins (unhelpfully labelled as CIPO for the MOSI/TXO and COPI for the MISO/RXI) to connect an FTDI232 serial to USB adapter which allowed U-Centre 8.16 running in the winXP VM to connect to the ZED9 (only after buying a refurbished Thinkpad running win10, of course!).

 I now have it assembled onto 100mm by 100mm single sided copper clad FR4 board that neatly fits into another 100 x 100 x 50 mm extruded aluminium case (as per its MK II M8T predecessor). At the moment, just the bare essentials, I still have to add the LiPo  and its support circuitry and the reverse polarity and over-voltage protection components before I can finally box it up.

 "Bare essentials" in this context does include the the three DC-DC converters which make up the 'power train' to facilitate internal backup power from input voltages ranging from a low of 4.8 to a high of 24 volts at a fully loaded (45mA feeding the GNSS/RTK antenna) warmed up input power of circa 1.9W (peak startup from stone cold power demand of 7 or 8 watts which allows any 2.1A rated USB power source, wallwarts, powerbanks and any other 10W rated PSU sources up to a maximum of 24v to be used)

[/EDIT]
=======================================================================================================

 The core of this and several other problems lies with the fact that my version (17.3) of Linux Mint no longer supports the many and varied dependencies required to upgrade from VBox version 5 to fix the failed EHCI USB options in my windows VMs. Not an issue in the winXP VM when it comes to running the latest (but ancient v. 8.16) compatible version of u-centre  to interact with the M8T modules but a major show stopper as far as re-configuring that nice shiny new ZED-9FT that's now in my possession. :'(

I've been putting this long overdue OS upgrade off for the past two or three years and now see that I have no choice but to face the upheaval involved in such an exercise before I can do anything else with that Sparkfun module.

 Forward planning with the original Linux Mint installations regarding the use of separate partitions for boot, root, home and swapfile, helps minimise the potentially disruptive effects of an OS upgrade exercise but even so, there'll be enough disruption to contend with assuming I do manage to avoid accidentally deleting the existing partition spaces.

 I suppose I should do a complete backup of the boot and system drive (a 240GB SSD) beforehand just to make sure any dramas remain just that (a recoverable drama rather than an irretrievable disaster). I've already created the live USB boot drive to test and install LM 22, now I just have to make sure to create a recovery image of the SSD before taking the plunge so it looks like it might be another day or two (if all goes well) before I can start properly checking out that Sparkfun module (possibly a week or longer if it doesn't go well).


=======================================================================================================
[EDIT 2024-11-20]

 About 2 or 3 weeks after posting that, I decided to make a massive hardware upgrade before making such a major OS upgrade. The reason being that one, I was due a hardware upgrade anyway and two, I might as well combine both tasks to maximise the benefit from the hassle of just doing an OS upgrade.

 I'd reviewed my OS options and had settled on Fedora 37 little realising that the kernel versions being used by the major distros at that time lacked any support for the 2.5 GBe lan ports typical of my chosen MoBo (and many others) at that time which ultimately resulted in a 5 months long protracted upgrade process only resolved by disabling the on-board lan port and fitting a 1GBe PCIe adapter.

 Unfortunately, I now find myself faced with the same OS upgrading issue by failing to notice Fedora's yearly OS update cycle and the need to use the associated upgrade feature that's only good for the next annual upgrade with no means of working around this time limit by upgrading to the next, now out of date version as a means of stepping through the previous versions you'd failed to upgrade to
[/EDIT]
=======================================================================================================

 The only good news to offer is that I seem to have finally optimised the chill factor compensation algorithm I've been trying to fine tune to stabilise the ruby against diurnal temperature variations. The night before, I left my hobby room windows open to lower the ambient temperature to test my attempts to fine tune the base plate temperature control algorithm only to discover that it was reaching the fan cut off set temperature (36 deg C) and undershooting some 200mK or so at just under 15 deg ambient.

 I took advantage of this low temperature opportunity to refine the base plate temperature control and chill factor algorithms whilst awaiting the expected delivery of the SparkFun module. The rebound back to the 36.10 deg +/-20mK region occurred when the room had warmed up to a less chilly 20 to 21 deg but crept up another 20 to 30mK at around 25 to 26 deg room ambient. A small sacrifice of control at the top end for the considerable improvement at the bottom end of the ambient range.

=======================================================================================================
[EDIT 2024-11-20]

All the above chill factor compensation nonsense has long since been discarded. In its place I have modified the innards of the enclosure by adding a couple of exhaust to intake plenum return galleries (40mm PVC pipes) with servo operated flap diverter valves to stabilise the fan intake temperature between a low of 23.5 21.5deg at a 1 -2 deg ambient up to a high of 32 deg at the upper ambient temperature limit of 32 deg.

 I have changed the set temperature to 36.050 deg and it remains stable to within +/- 1mK in the 1 to 12 deg ambient range, degrading to +/- 5mk in the 23 to 28 deg range for about 25% of the time, otherwise maintaining a steady +/- 1mK for the remaing 75% or so. I'm inclined to assume an effect due to the physics package ovens going in and out of sync with each other to disturb the fine balance of my temperature control algorithm.

 The only puzzling aspect being the major improvement at the low end of the ambient temperature range versus the earlier instability with my earlier temperature control efforts. However, a lot of water has spilled over the damn since then,  suffice to say that I have made many improvements to the temperature control algorithm in the meantime, including elimination of any need to shut the fan off and the associated kick startup code.

 One final thought about this puzzle being that it may simply the result of the diverter valves effectively sealing it off from the influence of  any random external drafts of ambient airflow. Indeed, this was the reason why I'd set the plenum target temperature so close to the upper limit in the first place (31.5 deg for the mid point of the +/- 0.5 deg plenum temperature control range of the diverter valves' end to end extremes which corresponds with an ambient temperature range of 19 to 33 deg). If that's the case, I would expect to see such tight +/- 1mK control maintained right up to around the 19 deg mark.

[/EDIT]
=======================================================================================================

 I ran the same test last night with a considerably improved result. Since you'd mentioned the impossibility of stabilising a rubidium to within 10ns drift per 24 hours, I've attached screenshots to show just how close it's possible to approach such a high level of precision in a home lab setting with a repurposed LPRO-101.
=======================================================================================================
[EDIT 2024-11-20]

 Since I've now had a chance to compare the MK II against the ZED9 based MK III gpsdo to observe the diurnal phase shift due to the ionosphere, it seems my results must have been down to pure dumb luck at that time since the MK II drifted some 33ns over each of the 24 hour test runs compared to the MK III.

 Of course, there still remain the GPS system errors and the effect of moisture in the troposphere to contend with which are a common source of error for both GPSDOs and therefore, effectively nulled out of this comparison, leaving only the effect of the ionosphere visible. It's possible that the total 24 hour phase wander with the MK II could have been as much as 40ns which rather makes my "excellent" test results very suspect .

[/EDIT]
=======================================================================================================

 Image1 was taken at 00:55 this morning followed by images captured at 2am and 3 am before I opened the windows and took to my bed. I got up just after 10:30 to check the room temperature (down to 14.5 deg) before capturing the next two images at 10:39 (second of which was after clearing the persistence and closing the windows). The final two in that sequence were grabbed at 12:42 and 13:47 respectively. Image 4 looks like it might have simply been a GPS excursion rather than the ruby deciding to take a sideways excursion (I hope!).

 I'd left the room to warm up a degree before switching on a 500W fan heater to bring the ambient back up to a less chilly 18.5 deg (it's now at 19.5 deg, some 35 minutes after that final screenshot). The XYL has just turned the central heating on so I'll keep running my test whilst that takes over from the fan heater and the temperature continues climbing over the next few hours, this afternoon and into the evening.

 Now that I've reduced the control cycle time from a whopping 420ms down to a less glacial 110ms, I've got some more optimising of the various kick start delays used in the control algorithm to overcome the reluctance of a cheap two phase bldc fan cooler to restart from a standstill so my work on the algorithm isn't quite yet done but it looks like I'm very close to the limits of that hard pressed nano 3 board I'm using.

=======================================================================================================

[EDIT 2024-11-20]

 In the light of my latest temperature control achievements, the above is now ancient and outdated history (I'm happy to say). It seems I was nowhere near being close to the limits of that "hard pressed" nano 3 and the fan controller's performance limits at that time.

 Only now that I've maxed out the cycle time to just 7ms and a +/- 1mK variation from the set base plate temperature over 1 deg (possibly even as low as -5 deg) to around the 19 deg ambient temperature mark only occasionally rising to a +/- 5mK in the 19 to 28 deg range, can I say with some confidence that I've finally reached the limits of that "hard put upon" Arduino nano3 controller.

 I'm only too well aware that holding the base plate to within a mK of a set point is not sufficient by itself when there exists the unknown quantity of internal temperature gradient variations with changes of ambient temperature due to limitations of the thermal insulation around the remaining five sides of the rubidium oscillator.

 The addition of a controlled exhaust air recirculation system helps mitigate this, but it would work more effectively without the 2cm thick foam polystyrene jacket around the LPRO 101 to allow the fan exhaust to cool those remaining five sides with the freed up space used to increase the insulation layer thickness against the interior walls of the case itself.

 However, this would be a major modification I'd rather put off until I can build another rubidium oscillator using a Symmetricom SA.22C I'd bought some two years ago. The problem is that I'd like to assemble (or buy) a similarly "oversized" project case to build this into, using my hard won knowledge and experience gained over the past three years in developing a precise control of temperature over such devices.

 Incidentally, before you ask, yes, I have incorporated barometric pressure compensation (to the tune of 0.8 E-13 per mBar drop in pressure). Indeed this was the motivation to replace my early analogue temperature control setup with a micro controller based solution since there simply weren't any cost effective analogue output barometric sensors avalable to keep this in a purely anaolgue domain and I knew full well that if I had any hope of reaching a frequency stability better than a humble 1E-12, barometric compensation would be an absolute must have feature.

 By the way, if anyone has any helpful suggestions in regard of sourcing such an enclosure (kit or ready built), I'd like it to match within half an inch or so, the dimensions of my existing enclosure which are 12 inch wide by 10 inch deep by 8 inch tall. It needs to be so big to allow me install an effective layer of insulation to minimise the effect of ambient temperature variations.

[/EDIT]

[MIS42N]

@Johnny B Good
I gave up upgrading the O/S on computers a while back. I buy second hand systems with the 'next' operating system installed. This one was 18 months old, still under factory warranty, less than half retail price with monitor thrown in for free. I have a Linux based server for storing files, shared to any computer I put on the network. It has mirrored disks (Raid 1?) so if a disk fails there's a good copy. There's backup hardware if the processor fails. Runs under Ubuntu 16.4 (upgrading from this would be "if it ain't broke, fix it until it is"). As far as possible software is open source so easy to install software on the 'new' computer.

Re my comment about "the impossibility of stabilising a rubidium to within 10ns drift per 24 hours" I think your attention to detail takes it out of the "average back yard" class. If you can achieve that sort of performance day after day you'll prove me wrong. I worked in a standards laboratory for a while, which included a cesium beam timepiece. We were happy if the room temperature was 20±0.5°C, it didn't seem to affect the beam (which probably had its own heater). But we needed to null out magnetic fields and have a Faraday cage around the room to get best results. In those days there was no GPS so long term of WWV and WWVH was the best we could do as a reference. So we never knew how accurate we were (IIRC our accreditation was -2 parts ± 7 parts in 10^-13 to the national standard, so comparison error was larger than the possible difference).

@everybody - to return to the original question

If the aim of the game is to have a frequency standard, then the phase relationship between the GPS signal and the oscillator is arbitrary. The only important thing is it is fixed. If the average phase error for period A (say - 100 seconds) and the average phase error for the following period B (the next 100 seconds) are the same, then the frequency relationship is constant - i.e. they are same. It is good practice to design a GPSDO to achieve this long term, then the GPSDO can be used as a reference for testing rubidium and cesium standards with a long time comparison. A PLL design will achieve this.

If using a PLL design, taking the phase relationship between 1pps and 10MHz doesn't work out. If the 1pps is more than 50ns early or 50ns late then the result is ambiguous. Which is why most designs divide the 10MHz down to a lower frequency before phase locking.

If the aim is to have a time standard, welcome to a world of pain. The phase relationship becomes important as does propagation delay in the antenna lead, ionospheric conditions, equipment design, etc. If a particular phase offset is required, then it can be established at power up and as long as it is maintained then both frequency and phase are correct. An early design of mine worked off an interrupt caused by the 1pps, and required it to arrive a few µs after a clock interrupt so they didn't interfere. It took a few seconds to drift the oscillator the required amount. I could have modified the startup to time the clock startup off an early 1pps but it was not necessary. Fortunately in the current build the 1pps is timed by hardware and the interrupt can be delayed without impact.

[Johnny B Good]

 I've edited my previous post to  show the progress I've made with my rubidium oscillator during the past two years along with the results of a yet to be fully completed MK III ZED9 FT based gpsdo (just the essentials to afford meaningful test results - solderless breadboard prototyping just doesn't cut it for this type of project).

This is by of a heads up on the newly edited previous post for anyone who still has enough of a passing interest to have clicked the notification button to alert them any to any new postings to this topic thread (I just realised that just editing my post might not trigger such a notification).

[Johnny B Good]

 Following up on my last post, I've since made more progress.

 During the past year or so, I'd started to notice a troubling instability in the phase of my RFS wrt to my MK II (MT8 based) gpsdo which finally degraded to the point of losing atomic lock some 3 or 4 weeks ago, leading me to shell out for another LPRO 101 (mounted on a base plate extender with a 15v to 24 v boost converter with the output wired to a separate SMA socket and a DB9M socket for power in and BITE output signal but no other connections to the monitoring and ext C field connections) with a claimed lamp voltage better than 9v.

 This lamp voltage claim proving to be somewhat false (a mere 5.87 ish volts - just 1 volt more than the original LPRO), I complained to the seller who apologised and immediately shipped out a replacement, telling me not to bother returning the first one. This arrived from Israel within just 4 days but, unfortunately also failed to meet the 9v lamp claim (it was a little better at about 6.5 ish volts). When I complained again about this, the seller offered to fully refund on a return or let me choose to accept a 50% refund and keep both units. I pragmatically accepted the refund option so I now have two working 'spares' to hand.

 As a result, I'm now running the 2nd unit in my RFS and the first one (with a modified adapter that gives me full access to all of the LPRO's interface pins) sat on an aluminium sheet metal heatsink to monitor lamp voltage which can take a day or three to properly settle down.

 When I first received these two LPROs, I'd run them to check their basic frequency and phase stability, sitting them on top of my RFS by way of an instant to hand heatsink. Both had shown no sign of any instability until I transplanted the first one into my RFS as being the quickest way to gain access to the lamp (and xtal) voltage monitoring pins with the bonus of being able to fine tune it to syntonise it to the MK III (ZED9FT based) gpsdo.

 Incredibly, I saw the exact same symptoms of phase instability, noting that it seemed to be tied to variations around the base plate set point temperature (36.050 deg). Indeed both of these units behaved exactly the same once installed into my RFS for testing, leading me to conclude that the problem was related to the control of the fan cooler.

 I discounted magnetic flux leakage and fan commutation induced ripple current effects before finally considering the possibility of fan vibration on the LPRO (which I was already only too aware had an incredibly high sensitivity to vibrational disturbance). Sure enough, as soon as I'd removed the 4 fixing screws that had been holding the fan in firm contact with its heatsink, the problem disappeared.

 In hindsight, I guess the fan vibration issue had been present all along. In the early days of developing my temperature control algorithm, the vibration frequency had not been stable enough to synchronise with the servo amp's adjustment rate long enough to build up a significant destabilising effect. It only became an ever more noticeable problem as I gradually refined the temperature control algorithm's fan speed control to the point where I finally reached a stability of +/-1mK. Such exquisitely fine fan speed control was all it took to destabilise the LPRO's output. In short, "I had been hoist by my own petard"

 Anyway, all that said, I thought I'd attach a few screenshots along with 12 hour movie condensed into a two minute timelapse showing the outputs of the MKs II and III gpsdos with the RFS and the spare LPRO mentioned above. The MK III (ZED9FT) is the trigger reference on CH1 (yellow), the RFS is CH2 (magenta), the MK II (M8T) is CH3 (blue) and the 'spare LPRO' is CH4 (green).

 I rather think this will not only prove 'entertaining' but also somewhat educational as to the weakness of the single frequency timing gps receiver modules that we've been making do with over the past couple of decades. Incidentally, please note that both gpsdos are of the G3RUH type (no additional and complex micro controller other than the very sophisticated one(s?) in the gnss receiver modules themselves).

 I'd have prefered to have posted a timelapse of a full 24 hour run but the process I'm using would require me to set up the Blue Recorder session, disable the screen saver, leave the monitor powered up and not touch the PC for the whole 24 hour period (it's a long story). The preceding screenshots are there to show the full extent of the diurnal phase wander of the M8T based gpsdo (some 35ns worth) with the final screenshot showing a continued (and to me, a surprisingly stable) phase stability of the RFS.

PS. The timelapse is best played at half speed.

PPS. Here's a picture of the extended base plate adapter
https://www.eevblog.com/forum/blog/eevblog-235-rubidium-frequency-standard/msg3714325/#msg3714325

PPPS. I've added a couple more post timelapse movie screenshots

[Johnny B Good]

Here's another 12 hour movie converted into a 2 minute timelapse.

 This time I managed to syntonise the naked LPRO into a closer frequency adjustment (just a mere 46μHz high - approximately a 2 cycle drift over the 12 hour run).

 I did this so I could better determine which was the culprit for the occasional rapid phase shifts on the RFS. Observation clearly reveals this to be the ZED9FT gpsdo. What's not clear is whether this is a weird peculiarity of the gpsdo or the unlikely possibility of a corrective feature in the gps data stream aimed specifically at dual frequency receivers to the exclusion of single frequency receivers.

 As I've already mentioned, the method I'm using to generate these timelapse movies involves the use of a screen movie capture utility called "Blue Recorder" (apparently a fork of one called "Green Recorder").

 Whilst this works ok as a hack to grab a region of screen real estate at 1fps for however many hours I need to record, it is  vulnerable to any other windows being opened that happen to intrude into the selected area (or window) as well as the effect of the screensaver cutting in and even that of switching the monitor into standby. All of which means I have to disable the screensaver and leave the monitor powered up for the duration as well as not opening any new windows in the meantime.

 What I'd really like would be a video stream capture at the network level from the SDS2104X Plus or possibly a second physical monitor (virtual desktops (and machines) just don't cut it in this use case). Now that this last run has finished, I'll be searching for a better solution along those lines. In the meantime, if anyone has already found a  solution to this problem, I'd really appreciate some advice.

 It's only this initial sourcing of the movie footage that's the big issue. Once I've gotten a clean 12/24 hour movie file, I've no problem in processing it into a timelapse movie using kdenlive (after repeating the Blue Recording process at 60fps on the VLC playback at 32 times normal speed - a mere 24 minutes for a 12 hour movie clip that had been captured at 1fps). This second run of Blue Recorder is simply to generate a 12 minute 60fps movie to simplify the timelapse conversion in kdenlive.

 There might be a better way to convert the 12hour 1fps movie directly into a 2 minute 60fps timelapse in kdenlive but I haven't been able to find it. BTW, the final final step in reducing the movie file size to a mere 2.6MB this time round involves the use of Handbrake to compress kdenlive's 271.7 MiB output file with very little loss of the original video quality.

[MIS42N]

I did this so I could better determine which was the culprit for the occasional rapid phase shifts on the RFS. Observation clearly reveals this to be the ZED9FT gpsdo. What's not clear is whether this is a weird peculiarity of the gpsdo or the unlikely possibility of a corrective feature in the gps data stream aimed specifically at dual frequency receivers to the exclusion of single frequency receivers.

Have you limited the constellations used by the GPS unit. If it is flipping between GPS and Galileo for instance that could be the problem. If your antenna has good sky exposure then try limiting to just GPS.

[Johnny B Good]

I did this so I could better determine which was the culprit for the occasional rapid phase shifts on the RFS. Observation clearly reveals this to be the ZED9FT gpsdo. What's not clear is whether this is a weird peculiarity of the gpsdo or the unlikely possibility of a corrective feature in the gps data stream aimed specifically at dual frequency receivers to the exclusion of single frequency receivers.

Have you limited the constellations used by the GPS unit. If it is flipping between GPS and Galileo for instance that could be the problem. If your antenna has good sky exposure then try limiting to just GPS.

 I've been limiting myself to just the GPS constellation for the past 4 or 5 years for just that reason. Mind you, now that the Galileo system is more or less complete, it might be worth switching to that constellation instead of the first and well established GPS. Regardless, the best advice is to avoid adding two or more GNSS into the mix and you'd be well advised to avoid the GLONASS constellation altogether.

 This policy came about as a result of my experimenting with my very first gps module (a genuine M8N as it happened) plus additional advice from others more experience than I had been at that time. My distinct impression back then had been that there was about a 2ns discrepancy between the GPS and Galileo and a whopping 8ns (or was it 18ns?) or so between GPS and GLONASS. In general, it's best to stick with just one constellation, even if the most or only readily accessible happens to be GLONASS.

 As regards the funny behavior originating with the MK III gpsdo, I rather think it's some sort of issue within it rather than some peculiarity with the gps signals. However, once I've got a more usable system in place to collect the scope's video data at 1fps to generate daily 24 hour runs as short 5 or 10 minute timelapse movie clips, I'll experiment with the Galileo and even the GLONASS constellations to confirm where in the system this peculiarity actually resides.

 Here's a single screenshot showing the current position of the RFS waveform, still within +/- 10ns of where it had been 24 hours ago.

 The fact that I've managed to achieve such stability with the RFS has now finally justified the use of the more expensive to run (54W versus the 22W of the SDS1202X-E) SDS2104X Plus where its 46 seconds boot time versus the 16 seconds of the 1202 doesn't matter so much in this case and, joy of joy, no need to enable infinite persist to tell me whether the RFS has drifted by more than a cycle overnight.

[Solder_Junkie]

GLONASS has poorer stability compared to GPS and Galileo. I can’t remember the specification, but given the political situation, it makes even less sense to use GLONASS.

Even with an outdoor “mushroom” antenna and a good sky view, I “see” more GPS than Galileo satellites.

Ublox recommend sticking with a single constellation for timing use, which also probably applies to frequency too. With my GPSDO measurements I haven’t noticed any difference between GPS, Galileo or both together.

SJ

[Johnny B Good]

 I finally figured out a way to work around the "Hands Off!" issue when recording my 1fps 24 hour movies, just add a second monitor and remember to minimise all active windows on the main display before switching it off for the night.

 Here's what will likely be my last 25.75 hours conversion into a 4 minute timelapse movie for some time (I need to select a low vibration fan from my NOS). Also, I think the weird behaviour with the MK III (ZED9FT) gpsdo was due to having the elevation mask set on the high side (30 deg). I've reduced it to 25 deg, matching the setting in the MK II (M8T) gpsdo. I may reduce it even further.

 The problem with the MK III being that there is an even greater risk of it losing sight of fully operational dual frequency SVs, forcing it to rely on SVs operating only on the L1 frequency. It's not such a great issue with the MK II since I'd allowed for the possibility that a high elevation mask could leave it seeing only out of service SVs for maybe up to half an hour or longer (hence the 25 deg setting).

 Anyway, that almost 26 hour run has revealed several issues that would otherwise have been difficult to observe. It also reveals the full diurnal phase shift caused by diurnal variations in the TEC of the ionosphere amounting to some 35ns in total.

[Johnny B Good]

GLONASS has poorer stability compared to GPS and Galileo. I can’t remember the specification, but given the political situation, it makes even less sense to use GLONASS.

Even with an outdoor “mushroom” antenna and a good sky view, I “see” more GPS than Galileo satellites.

Ublox recommend sticking with a single constellation for timing use, which also probably applies to frequency too. With my GPSDO measurements I haven’t noticed any difference between GPS, Galileo or both together.

SJ

 That advice to stick with only one constellation for timing use is just as valid for (stable) frequency reference purposes. The 12 and 24 hour timelapse movies I posted nicely reveal that both types are long term stable frequency-wise. However, a drift in phase between the two is also a transient change in frequency (mostly that of the MK II and of the order of a few tens of μHz per hour at various times throughout the diurnal cycle).

 I'm currently some 11 hours into another 12 hour run I'd started just after 3 am this morning, after powering the RFS down to try a modification I'd hoped would mitigate the fan vibration issue that caused those rapid phase transients. This is for my own use, namely to see whether reducing the elevation mask from 30 down to 25 degrees in the ZED9FT's configuration would eliminate or at least reduce the weird events seen in my timelapse movies so far.

 Powering the RFS down for the twenty minutes or so that it had taken me to apply a couple of strips of capton tape to the plenum partition divider to reduce friction in its contact with the LPRO's polystyrene jacket to attenuate the fan vibration effect proved an ineffective strategy so, selecting a low vibration example from my NOS of these fans is now on my to do list.

 Anyway, I've attached a couple of screenshots (last night's and the latest one taken just now). The RFS trace has leapt some 30ns "backwards" (ahead of the reference in terms of phase) during that 11 hours. However, a not unexpected result after such a brief shutdown and the drift rate looks to have started to slow down as the daily aging drift induced drop in frequency starts to build up.

[Johnny B Good]

 Here, for your viewing pleasure is another 24 hour timelapse. Not quite the same old same old but one with an important difference, namely that the triggering channel is now CH2 (the RFS) instead of CH1 (the ZED9FT).

 Here I'm now using the RFS to check the stability of the GPS system as well as positively identify the source of the "wobbles" on the other three traces. The ZED9FT drifted some 17ns compared against the RFS over that 24 hour period. Subsequent observation has revealed the RFS can do much better now that I have reduced the effect of the fan vibration somewhat (but not completely).

 It seems that what I'd taken to be a daily aging drift is in fact an artifact induced by the fan vibration. I did pick out a fan that seemed to have a weaker level of vibration but I could still observe its effect on the stability of the RFS. It was hard to say for sure whether its effect was less pronounced or not. However, choosing a low vibration fan will be rendered moot since it seems I need to clone the LPRO's polystyrene jacket using sponge rubber to more fully isolate it from the fan vibrations anyway.

[Johnny B Good]

 Here's another couple of movie clips (real time, not timelapse). They're examples of how u-centre v8.16 (running in a winXP VM) displays the results from the M8T and the ZED9FT gps receivers.

 Also included is a screenshot taken just 14 minutes ago.

[MIS42N]

The time lapse videos are informative. I don't get much from the images. What do they mean? what can be learned from them? a nice demo of your recording ability but to me doesn't say much about what is going on.

So, what does 17ns drift in 24 hours say? I work it out that the RFS is drifting 2 parts in 10^-13. The next question: is it consistent day to day? If it is, your RFS is really good and just needs a tweak to the C field. If not consistent then the RFS is still being affected by environment, why?

Why have you not tweaked the other RFS? I tried counting the 24 hour drift, I got 15 cycles. A drift of about 2 parts in 10^-11. Should be able to fix that.

Have you thought of a temperature compensated C field? https://pubs.aip.org/aip/adv/article/11/6/065212/992728/A-modified-C-field-circuit-for-the-rubidium-atomic

An excellent rabbit hole.

[Johnny B Good]

The time lapse videos are informative. I don't get much from the images. What do they mean? what can be learned from them? a nice demo of your recording ability but to me doesn't say much about what is going on.

It'll be easier to analyse what's going on if you run the playback at half or even one third speed (30 and 20 fps).

 I was running this video capture to make it more obvious when the fan vibration effect was kicking in by using the RFS as the trigger source. In this case, the other three traces will display the wobbling effect. Also, I wanted to get the measure of the diurnal phase wobbles that plague the single frequency timing receivers such as that M8T based MK II gpsdo of mine.

So, what does 17ns drift in 24 hours say? I work it out that the RFS is drifting 2 parts in 10^-13. The next question: is it consistent day to day? If it is, your RFS is really good and just needs a tweak to the C field. If not consistent then the RFS is still being affected by environment, why?

That amount of drift suggests that the fan vibration is compromising the frequency stability (also, it was probably still approaching its final peak of stability after the last shutdown - it can take a few days to reach its ultimate stability after just a few hours of cool down. And of course, with a recent 30 mBar boost in barometric pressure over the last few days, there's still the issue as to just how accurately I've calibrated the compensation coefficient (Datum's datasheet only specifies this as "<1E -13/mbar" and I've used a figure of 0.8 E -13/mBar taken from a Swiss research paper). Given the recent swings in air pressure, I seem to be in the right Ball Park (and even in the right sense of adjustment).

 As far as fine tuning the C field, I've got a genuine Bournes 5K Hybritron helical pot fed from a dedicated temperature stabilised 7805 (4.97896v) driven with a Beckman Duodial turns counter that allows me to tune it to within less than 1μHz. The wiper contact is connected via a 2Mohm thermally stabilised resistor to the 150Kohm input impedance of the C Field pin, giving me an effective reduction ratio of 14.333 of the 4.97896v across the pot (aka, the intelligent way to pad out a calibration pot when you have an internal coarse adjustment trimpot to set the main pot close to the middle of its range).

Why have you not tweaked the other RFS? I tried counting the 24 hour drift, I got 15 cycles. A drift of about 2 parts in 10^-11. Should be able to fix that.

 I should if ICBA but in this case, it wasn't worth the hassle since it had been powered up simply to let me monitor the lamp voltage longer term. Since I had a spare 'scope channel to hook it up to, it seemed churlish of me not to have included it. If nothing else, it nicely demonstrates the difficulty of syntonising it to within +/- 50μHz other than by pure dumb luck using the internal trimpot alone. Also, it reveals the obvious (if in this case, impressively low) tempco of an LPRO simply shoehorned into a nice case acting as a passive heatsink left to the mercy of room temperature fluctuations.

Have you thought of a temperature compensated C field? https://pubs.aip.org/aip/adv/article/11/6/065212/992728/A-modified-C-field-circuit-for-the-rubidium-atomic

An excellent rabbit hole.

 Interesting questions.

 Firstly, when I started this project, I had no idea just how deep a rabbit hole I would be journeying into.

 The initial idea to stabilise the base plate temperature had been prompted by the temperature coefficient curves of a sample of 19 LPRO s shown in the "LPRO User's Guide & Integration Guidelines" document which showed a worst case variation of 1 E-10 for a 50 degree temperature change.

 I'd figured there'd be a very good chance of holding the frequency to better than 1 E-12 if I could stabilise the base plate temperature to within +/- 0.25 deg and so my epic journey into +/- 5 mK territory began.

 Holding to a +/- 0.25 deg variation was just doable in the purely analog domain using a thermistor and comparitor circuit to switch a cooling fan on and off but once I realised the dire need to compensate for atmospheric pressure variations just to achieve a 1 E-12 level of stability with no affordable analogue barometric sensors in sight, the decision to use a microcontroller to achieve temperature control and barometric compensation became my only way forward (BMP280s being as cheap as they are had rather made this a "No-Brainer" option ).

 Little did I realise I'd ultimately be achieving temperature control down to within +/- 1mK for temperatures in the -5 to +16 degree range and +/- 5mK for temperatures in the 16 to 30 degree range. Also, I hadn't expected this level of fan speed control to bring me the grief of fan vibration induced frequency instability as I now find myself dealing with (the 20MHz crystal oscillator circuit in all of these LPRO 101s is exquisitively sensitive to mechanical vibration).

 The vibration only becomes a serious issue if it is maintained to within a few Hz of the FM frequency used in the Physics package for several seconds at a time. In the early days, due to lack of fan speed control precision, this hadn't been a problem. Now, however, I find myself "Hoist by my own petard" due to my greatly improved fan speed control.

 That article you linked to was rather enlightening with regard to the potential of thermally stabilising an RFS to within a mK or three of a set temperature (along for us earth bound experimenters, the inclusion of barometric compensation). However, the circuit in fig 5 looks to my eye rather odd with regard to the placement of the thermistor in series with the very high input impedance of a non-inverting opamp input pin (GOhms??).

 It does leave me wondering about how tightly controlled the input impedance of such opamps are in their manufacture and how this would vary with temperature. Perhaps there's more to this circuit and they've only shown a simplified version.

 I chose to stabilise the temperature on the basis that the best way to eliminate the effect of a collection of disparate thermal coefficients is to cheat by not having any temperature variations by which to trouble this collection of random temperature coefficients in the first place. However, I'm only too well aware that you also need to maintain a constant thermal gradient within this box of tricks we call a rubidium oscillator, hence the polystyrene jacket and the cooling air recirculation control to reduce the range of fan intake temperature and the airflow around the jacketed LPRO to minimise the variation of the LPRO's thermal gradients due to changes in ambient temperature.

 I also have a plan to dampen vibration and reduce the thermal gradient effect by filling the 3mm gap between the circuit board and the base plate with a 3mm thick soft silicone thermal gasket, avoiding the area underneath the crystal oscillator (risk of increased stray capacitances) and the area beneath the physics package (using a 3.5mm thick insulating sponge material to provide mechanical damping in lieu of the silicone gasket). However, I'm leaving this plan on hold until I can further minimise the fan vibration effect.

[Johnny B Good]

 I had another go at syntonising the spare LPRO (green trace) a couple of nights ago. You can see my efforts near the beginning of the attached timelapse movie. Some of you may have noticed the am noise on the MK III gpsdo (ZED9FT, yellow trace). It had become a bit of an annoyance for me since it ought to have looked as pristine as the MK II (blue trace).

 It seemed a little odd in that it was more pronounced on the troughs rather than where one might expect to see Vcc noise appearing on the peaks. The reason for this odd reversal turned out to due to CH1 having been set to invert the waveform. I corrected this oversight after allowing the capture to run to completion before taking the MK III apart to investigate this noise issue on the 74HC14 which drives the LPF feeding the 10MHz sine wave output line. I replaced the suspect 4.7 μF ceramic cap with a 9μF one and, as the next timelapse movie clip will show, this cured the noise issue.

 The attached screenshots, BTW, are before and after stills.

[Johnny B Good]

 Here's the following night's run showing the results of my repair on the MK III and with the CH 1 invert turned off. The loss of lock on the MK III (yellow trace) was due to my disabling the invert time pulse in the ZED9FT's configuration options to see whether this would correct the now 50ns phase jump. It not surprisingly to me, made no difference, hence the next loss of lock.

 This left me wondering why the MK II was more or less 180 degrees out of phase with the MK III. After all, they are both G3RUH based designs and should have only been adrift of each other by just a few nano seconds rather than the 40/60 ns they were showing.

 After closely scrutinising my hand drawn circuit diagrams of both gpsdos, I realised that I'd recruited an extra 74HC14 inverting gate into the buffer chain that drove the 4 inverters feeding the LPF, each via a 160 ohm resistor in the MK II. The simplest fix would be to divert the two wires going to the 4 inverters from the output of that second buffer gate onto its input to join the connection that fed the 74HC390 used to divide the OCXO frequency down to 100KHz.

 You can see the consequences of this modification work shortly after the lost lock events. Disconnecting the MK II will obviously flatline the blue trace and because this was powering the antenna, means the MK III lost signal and went into a much degraded holdover mode. As you see from the time displayed by the 'scope, this modification took me just under half an hour to complete.

 At this point, I had both gpsdos' elevation mask settings at 15 degrees which should have fixed the issue of the MK III completely losing sight of SVs fully functional on both frequencies and above the elevation mask threshold. However, I was still seeing the effect of it having to fall back on SVs that were only offering service on the primary frequency for the lack of good dual frequency SVs above the elevation mask setting.

 I've only just now set this elevation mask angle down to 10 degrees on the ZED9FT. The M8T doesn't need such a low setting since there's a more than ample supply of SVs fully functional in single frequency mode to virtually guarantee it will always have sight of more than the minimum of one to maintain lock.

 So far, all looks good viewing the 'scope traces in real time. I'll have to do another run before I can be certain that this measure has cured the MK III of "This Weirdness".

 I've included a before and after screenshot. Also, to follow this post is a 16 hour squashed into an 8 minute timelapse movie. It was going to be squashed down to a 4 minute timelapse clip but decided this much squashing made closer examination even at half or one third playback speed less useful. If you prefer shorter 2 or 4 minute clips, there's always the high speed playback option in you media player to satisfy this need.

[Johnny B Good]

The fan vibration effect hasn't completely disappeared as can be seen near the start of this 8 minute timelapse clip I've attached (along with before and after screenshots).

 Not only can you see the effect of this being displayed by the other three traces all moving in lock step along the X axis you will also see the MK III (yellow trace) contributing its own quirky behavior to the mix (jumping between dual and single frequency operation).

 Whilst earlier test runs with relatively high elevation mask settings comparing the MKs II and III gpsdos had suggested a diurnal phase wander of as much as 35ns variation by the MK II against the MK III, this appears to have been an artifact introduced by the MK III being obliged to downgrade to single frequency operation every time it lost sight of any fully operational dual frequency SVs that also happened to be above the elevation mask filter.

 Recent runs now suggest a diurnal variation somewhere around the 20ns mark - in line with what others (blessed with a functioning Cesium frequency standard) have reported.

 This time round, the RFS seems to have kept its phase relationship with the MK III within a 10ns variation despite the occasional interference from the fan. I think that if I can eliminate this issue, the RFS should be able to show its true colours in regard of frequency stability and do even better.

 Admittedly, this wasn't a full 24 hour run but even now, some 4 hours on, it remains within that 10ns window as the third and final screenshot shows. Indeed, it's been within that window since the after screenshot in the previous post.

[Johnny B Good]

 Here's a follow up 17 hours into 7.5 minutes timelapse movie started just after 9pm last night. Remarkably, the RFS (magenta trace) has remained within its previous 10ns wide window relative to the MK III gpsdo (yellow trace) during that run and over the next two hours since as depicted in the after screenshot (bare LPRO green trace now flatlined due to my having to clear space on the bench to fabricate a foam rubber plenum partition in which to mount the cooling fan in the RFS).

 This timelapse sequence is likely to be my last for some time since I'm likely going to be doing some extended experimentation trying out various strategies aimed at suppressing the fan vibration issue. The foam rubber version of the current expanded polystyrene partition, if it shows any promise, will likely end up being sandwiched between two sheets of roofing lead for good measure (and additional strength).

[Johnny B Good]

I fabricated a replacement plenum partition divider in sponge rubber and swapped it out with the original expanded polystyrene foam one yesterday evening, using the lead sheet with the fan firmly screwed in place and checked how good a fit it this assemblage would be.

 It seemed ok but I realised I'd need to trim a quarter of an inch off the lead sheet all round to allow the sponge rubber divider to suspend it clear of any contact with the base of the case and the 40mm pvc pipes which return the exhaust back into the plenum chamber. I had this ready for its first test run by twelve minutes to nine.

 This morning, after I'd recorded a 17 hour movie, it occurred to me to check that I'd cut enough off the lead sheeting along the bottom edge and in the corners where the PVC return pipes poked through. The clearance was rather marginal so I cut off a bit more lead to positively ensure it was floating clear of any direct contact with the enclosure.

 As far as I can tell, this seems to have put paid to the fan vibration issue. However, I'm still contemplating the sponge rubber jacket option.

 When I finally did get to review the video, I spotted a remarkable event that had occurred just before 11 am, namely the MK II M8T gpsdo (blue trace) suddenly dropping behind by 18 ns followed some 23 minutes later with a similar sudden reversal. This was obviously the result of a GPS update on the state of the ionosphere to cancel out the ToF error.

 Since this a significant factor relating to this thread's title, I thought I'd prepare a short two minute timelapse movie of this particular 1 hour segment and present it here for anyone still concerned about the original question thus posed.

 BTW, I reckon this would have been an almost instantaneous jump but for the fact the the PLL's output  in the MK II (an attempt at filtering out the minute to minute variations in ionospheric delay) has a time constant of 1100 seconds. Also bear in mind that this timelapse is running some thirty times faster than real time so if you want to get a more realistic impression, you'll need to slow the playback by a factor of 30 ( 32 will suffice in VLC media player).

[Johnny B Good]

After examining last night's 12 hour run, I could still discern some sign of fan vibration every now and again so decided to eliminate the lead sheet from the foam rubber partition divider since I suspected it may have been creating a resonance close to the critical 2308 rpm speed that destabilises the physics package lock.

 However, before doing this modification, I took the opportunity to use the original fan to excite the critical vibration frequency into the LPRO's polystyrene foam jacket by perching it on top the jacket and running it off the bench power supply with the 1202X-E monitoring the tacho pulses to measure its rpms.

I run the fan up to a maximum voltage of 15.6v (limit is set by the 16v bypass cap on the motor pcb) which as just under 2,900 rpm (the fan is rated at 12v for a nominal speed of 2500rpm +/- 10%. I found the critical speed of 2308 rpm occurred just below the 12v mark (11.9 something afair - it varied).

 Since the replacement fan was firmly screwed to the lead sheet and basically a friction fit into the foam rubber to leave the lead sheet hanging, I was going to remove the lead sheet and fan and replace it with the fan I'd been using for the above test in order to expedite the modification and keep the LPRO's down time to a minimum of just 5 minutes or so. Unfortunately, one of the glue joints in the foam rubber fan mounting / plenum partition wall had failed (my bad - I hadn't used sufficient glue) so the job took some 30 minutes or so which means another protracted settling period before I can start getting meaningful stability data.

 I've attached a short timelapse movie (4 x real speed of a 14m 24s movie clip showing the effect of the fan vibrations - 3m 38s long). As always you can opt to playback at 1/4 speed to see the effect in real time or else use double or quad speed if you just want to "cut to the chase".

 Now, getting to the heart of this issue, it's root lies in the 2G tip over effect as demonstrated by Dave in this youtube video:

https://youtu.be/zILwgQhjC_Q

 Which itself is demonstration of its use as an accelerometer sensor as described in the attached pdf on this subject.

 In essence, this is an inescapable feature of even the best quartz crystal oscillators. In this case, the problem is made some two to three orders of magnitude worse by multiplying the crystal oscillator frequency up to 6.8GHz and further exacerbated by the need to stabilise it to within a micro Hertz or better in order not to lose atomic lock.

 The 38.467 Hz fan vibration is obviously a sub multiple of the discriminator's modulation frequency (153Hz in this case) used by the physics package to detect the magnitude and sense of the error between the crystal oscillator's multiplied up frequency and the atomic resonance line.

 Random vibration frequencies, up to a limiting level, merely add phase noise (or a transient wobble in the case of an impact induced acceleration event). Even so, it's advisable to isolate the LPRO 101 (and any other rubidium oscillators for that matter) from any and all forms of vibration, not just those at an exact sub multiple (a quarter of the discriminator's operating frequency in this case).