pll loop filter design

[Etesla]

Long post, I spend a lot of words trying to describe my exact issue, but the overall question is right at the end if you wanna skip the read.

For the past couple of hours, I have been trying to get my simulation of a PLL based FM transmitter to work in a desirable way. I'm using the CD4046 as the PLL, I'm using it's pin 2 as the phase detection output, and I'm using a common collector colpitts oscillator with a varactor as the VCO. The loop filter between CD4046 pin2 and the input to my VCO is causing me some grief, I'm sure it's due to my own lack of understanding. A schematic is attached.

For the loop filter, I am currently trying to use a simple RC filter. The cutoff frequency of the filter is currently about one tenth of the frequency of the pulses coming out of pin 2 (around a 60 khz square wave). After some time, this setup almost always locks up the PLL as desired, with a characteristic sawtooth waveform on the output of the loop filter as shown in the CD4046B datasheet in figure 3. This locked waveform on node 'vco' is in one of the attachments. The obviously phase locked input signals are shown in another attachment.


The problem I now have is that I have to do a lot of frequency dividing to get the output frequency I want from the reference frequencies that are available. This means that the sawtooth waveform (about 60 khz) is happening much slower than the output of my VCO is oscillating (about 100MHz). The result of this is that the center frequency of the VCO output is indeed what it is expected to be (about 101.5MHz), but there is a TON of phase noise around that center frequency due to the huge ramping signal being fed into the VCO at 60 khz (duh). A picture of the FFT of the output of the VCO is in one of the attachments, you can obviously see the unacceptable amounts of phase noise. Note that when I just power the VCO from a DC source (no PLL stuff), its output frequency is absolutely specific (bandwidth of less than 1 kHZ as opposed to several MHz)

Now, to me, the obvious solution is to reduce the cutoff frequency of the RC loop filter (make R and/or C bigger). This would average out the pulses from the phase detector more, making the sawtooth waveform have a smaller amplitude, and decreasing its impact on the phase noise of the output of the VCO. However, Upon trying this in simulation, lowering the cutoff frequency lead to to unstable behavior, and the PLL no longer locks. The output of the loop filter gets into an oscillating steady state (shown in "steady state oscillation"). It seems like the changes on the output of the loop filter just aren't fast enough to counteract the difference in the reference and VCO frequencies and the PLL gets into the situation of chasing its own tail (shown in "PLL chasing its own tail")...

So, with all that out of the way, I would like to know the following:
Why does the cutoff frequency of the loop filter in a PLL have a minimum allowable value to retain stability, at least according to my sims? Why can't it be arbitrarily low (like 10 Hz cutoff frequency) and still lock eventually? Is this the issue being described on page 3-125 in the CD4046 datasheet in the frequency capture section? How do I reduce the amplitude of that triangle wave but maintain a frequency lock?!

Side questions:
Any recommendations on resources for PLL loop filter design for this type of phase detector?
Whats the benefit of using pin 13 on the CD4046 as the phase comparator instead of pin 2?
I divide the VCO frequency by about N = 1600. Is this division just impractically large and doomed to fail no matter what?

You're a hero if you made it through this whole post. Thank you so much!

[moffy]

You are better off keeping the carrier frequency(FM center frequency) much, much higher than the signal you are trying to demodulate. What is the FM bandwidth, 250kHz? If it is just a standard FM audio demodulator wouldn't you be better off using a standard chip?

[Benta]

If you are using pin 2, you have a type 1 PLL (frequency locked, with phase offset) and you can get away with just using an RC filter.
If you use pin 13, it will be a type 2 PLL (frequency locked with 0 phase offset) and this needs an active loop filter.

That being said, you really need to do the maths and do a proper stability analysis of your loop. I suspect you have too much gain in your VCO (= it's too sensitive to input voltage change).
Also, setting the filter cut off to just a tenth of the input frequency is wildly optimistic. A hundredth would be more to the point.

Last, if you do a proper analysis, don't forget the hidden pole in the VCO, which you'll need to estimate.

[duak]

It looks to me that you have too much loop gain and when you reduce the filter corner frequency the error builds up to a greater value.  The oscillator itself is going to be quite stable for the short term.  What you want to do is limit the maximum change in frequency to a gentle nudge that corrects the long term drift.
 
To build on what Benta says, you'll have to analyze the loop.  If you don't want to do this, then I would first set the oscillator so that it runs as close as possible to the desired frequency and then fiddle with the gains, filter values and correction voltage span until it works the way you want.  I would also use the type II phase detector.

Does this make sense?

An example of what is possible, when I was starting in my career in the 70's I had to modify the design of a digitizer for a satellite ground station to work with a different satellite.  The format was roughly the same but the line rate and sample rate were different.  Because the format was analog rather than digital, the sampling rate had to be phase locked to the line rate.  The sampling clock was something like 1 MHz and line rate was 6 Hz or so.  The VCO was actually a VCXO - a Voltage Controlled Crystal Oscillator - that could be varied in frequency by maybe a KHz.  It was quite neat to watch this thing lock to the signal.  First it found the precursor which identified Start of Line then it started to pull the sampling clock into lock.  The process took maybe a second or so.  When locked, the correction voltage was essentially zero and the VCXO phase noise was at its minimum.  If it lost lock, (should something like an aircraft pass in front of the dish) it could freewheel for a few lines and then snap back into lock within a line or two.

[Benta]

First, you are only using one part of the CD4046, which makes me wonder why you've chosen this part. A single-function XOR gate would bring the same result:
https://www.onsemi.com/products/standard-logic/logic-gates/mc74vhc1g86

Attached a general PLL block diagram with the loop parameters.

For a PLL with an XOR gate as phase detector:

K_P = V_DD / pi
F_C(s) is the loop controller (aka loop filter) transfer function. You'll need to calculate/synthesize this.
F_V(s) is the VCO transfer function. I've no idea what you have, normally it would be (2pi*f_MAX) / V_DD.
 For a frequency limited VCO it is (2pi*(f_MAX-f_MIN)) / (V_DD * s)
K_N is the divider ratio, in your case 1/1600

The final transfer function is:

[K_P·F_C(s)·F_V(s)] / [1 + K_P·F_C(s)·F_V(s)·K_N]


Have fun