EEVblog Electronics Community Forum

Electronics => Beginners => Topic started by: pion on August 29, 2020, 07:59:20 am

Title: How to debug crystal oscillator startup failures?
Post by: pion on August 29, 2020, 07:59:20 am
I'm manufacturing a PCB for a simple microcontroller project using the Voltera V-One (https://www.voltera.io (https://www.voltera.io)). While the MCU runs fine under its internal RC oscillator, I'm having a lot of trouble getting it to work off my external 12MHz crystal. The crystal fails to start oscillating in nearly all cases and I've only gotten a stable clock a handful of times out of many hundreds of power cycles.

I've spent a while searching and reading about the basics of oscillator operation, but what I've found lacking is procedures & tips specifically for debugging XTAL startup failures. I know that this is a fairly simple problem so I am hoping that a few folks from this community of experts can share either your firsthand knowledge or some links to good resources online.

Further details on my board for those interested: I'm using an NXP LPC812 MCU in a TSSOP-16 package with a 3.3V supply on a 2-sided board that I designed and printed with the Voltera. The XTAL is a 12 MHz quartz crystal in a through-hole HC49/S package. I am using two 27pF load capacitors in a 0603 SMD package (I took the values from a reference design that used a 12 MHz XTAL). I've tried to position all four of these components as close to one another on the PCB as reasonable, but I don't have enough experience to know whether I'm introducing too much stray impedance, propogation delay, or ground loop currents for this circuit to be stable.

As I mentioned, I have gotten what I believe to be a good signal a few times. Measured at either of the XTAL/EXTAL pins with a 1GHz active probe on a 200 MHz 5GS/s scope, it's an almost-clean sine wave at 12 MHz and around 1.5V amplitude, which I believe is reasonable from the LPC812 datasheet (https://www.nxp.com/docs/en/data-sheet/LPC81XM.pdf (https://www.nxp.com/docs/en/data-sheet/LPC81XM.pdf), pp. 59-61). Given that a clock signal does get generated at least some of the time, I feel that I can safely rule out misconfiguration of the MCU registers and focus solely on the hardware side of things. I can confirm via SWD that my firmware hangs in its oscillator startup code, waiting for PLL lock that obviously never comes.

I know that I might get advice on adding a feedback resistor, but the LPC812 already has an internal feedback resistor so this is unnecessary. My only other guess is that I have too much capacitance for some reason, so I replaced the 27pF load caps with 2.7pF. However, I have never successfully gotten this smaller capacitance load to oscillate, so I feel like I was closer with the 27pF. I'm reluctant to start arbitrarily trying different valued caps (if for no other reason than SMD soldering is still an annoying exercise for me), so it would be very helpful to me if someone could suggest a more directed approach to debugging this.

Thanks for reading! All suggestions are welcome.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 29, 2020, 08:14:49 am
Show your layout. And what specific crystal you are using?

In my experience oscillators are pretty hard to screw up so they don't even start.
Title: Re: How to debug crystal oscillator startup failures?
Post by: floobydust on August 29, 2020, 08:16:45 am
Check the crystal's series resistance and load capacitance values are reasonable. Datasheet Table 26 offers specs of 18 or 39pF for the two caps.
LPC81x errata (https://www.nxp.com/docs/en/errata/ES_LPC81XM.pdf) it seems to be finicky about VDD ramp up time sect.3.7 and osc issues below 1.9/2.3V (not relevant at 3.3V).
I use the finger test to check if an oscillator is weak, touch OSC IN and it should keep running. Or observe the start-up time with a scope with various loading caps.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on August 29, 2020, 08:43:20 am
Show your layout. And what specific crystal you are using?

Layout added to my original post now. I'm using an unknown 12 MHz crystal that I found in my components bin.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 29, 2020, 08:44:32 am
I'm using an unknown 12 MHz crystal that I found in my components bin.
That is your problem most likely. Try a crystal that you actually have a datasheet for.

Also, I don't get this layout. Where is the crystal and capacitors and traced to the MCU?

Ok, I think I found those. The layout is not ideal, but it would probably not be the source of the crystal not starting entirely.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on August 29, 2020, 09:01:35 am
Check the crystal's series resistance and load capacitance values are reasonable. Datasheet Table 26 offers specs of 18 or 39pF for the two caps.
I suppose I can desolder the crystal and check capacitance and ESR with an LCR meter. I have the Keysight U1733C meter which ranges from 100Hz to 100kHz for testing capacitance - should I be using the low or the high end of the measurement?

I want to try 18 and 39pF but the closest value caps I have on hand are 10, 22, 27, and 68pF. Do you think that the trace lengths & ground plane might add stray capacitance? (The traces are silver conductive ink and through-holes / vias are rivets and I don't know if these add more capacitance than standard copper traces & PTH.) I will try 22pF next as that seems closest to 18.

LPC81x errata (https://www.nxp.com/docs/en/errata/ES_LPC81XM.pdf) it seems to be finicky about VDD ramp up time sect.3.7 and osc issues below 1.9/2.3V (not relevant at 3.3V).
I did see the errata previously and noted both of those points. My supply voltage is good and I'm able to break into the firmware with my debugger and observe the loop waiting for PLL lock, so that tells me that the MCU is indeed starting up and executing code correctly.

I use the finger test to check if an oscillator is weak, touch OSC IN and it should keep running. Or observe the start-up time with a scope with various loading caps.
My problem is it's almost never running and it's seemingly random as to when it does run.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on August 29, 2020, 09:06:38 am
I'm using an unknown 12 MHz crystal that I found in my components bin.
That is your problem most likely. Try a crystal that you actually have a datasheet for.
Shoot – I have several hundred crystals and none of them have datasheets. Am I totally out of luck and need to order & wait for a new crystal?

Also, I don't get this layout. Where is the crystal and capacitors and traced to the MCU?
Ok, I think I found those. The layout is not ideal, but it would probably not be the source of the crystal not starting entirely.
Would you be willing to offer specific suggestions for improving my layout? I am quite new to PCB design and it would be very helpful to get some expert input. I will also probably spin a new version of the board so I would definitely take your feedback into my next design.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 29, 2020, 09:11:02 am
Shoot – I have several hundred crystals and none of them have datasheets. Am I totally out of luck and need to order & wait for a new crystal?
What is the source of those crystals?

You don't necessarily need to order new crystals, but if I were you, I would be changing the crystal, not the capacitors. You don't even need to solder the new crystal in, just hold the pins in the holes and see if it starts.

The issue with random crystals is that not all of them are the same. It may be some strange 3rd harmonic crystal, which will not work in an oscillator designed for the fundamental crystal. Chances are low, but a random bin of parts may contain anything.

Would you be willing to offer specific suggestions for improving my layout?
Try to place the crystal as close to the MCU as possible (shortest traces). Avoid VIAs if possible. Do not have that spaghetti of traces and broken ground planes cross the crystal traces. 

You don't show the mechanical dimensions of your crystal, so it is hard to judge, but in this case it looks like a better placement for it would be where you have the capacitors now - close to the IC. And capacitors could go on the other side using holes from the crystal as vias.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on August 29, 2020, 09:23:54 am
Shoot – I have several hundred crystals and none of them have datasheets. Am I totally out of luck and need to order & wait for a new crystal?
What is the source of those crystals?
They're all surplus parts that I've acquired from many years of collecting components – no idea about their individual origins.

The issue with random crystals is that not all of them are the same. It may be some strange 3rd harmonic crystal, which will not work in an oscillator designed for the fundamental crystal. Chances are low, but a random bin of parts may contain anything.
Yeah, fair point. I really prefer to use existing components rather than buying new ones when possible, though. Do you think building a simple Colpitts circuit or something like it for testing crystals is a good idea? Or is it really more trouble than it's worth? I would understand for high frequencies (hundreds of MHz+) that I'd need tight tolerances, but 12 MHz seems quite low doesn't it?

You don't necessarily need to order new crystals, but if I were you, I would be changing the crystal, not the capacitors. You don't even need to solder the new crystal in, just hold the pins in the holes and see if it starts.
Are you saying that I can keep the old crystal soldered into place but just touch a new crystal to the old leads? Doesn't the parallel capacitance affect it?

Would you be willing to offer specific suggestions for improving my layout?
Try to place the crystal as close to the MCU as possible (shortest traces). Avoid VIAs if possible. Do not have that spaghetti of traces and broken ground planes cross the crystal traces. 

You don't show the mechanical dimensions of your crystal, so it is hard to judge, but in this case it looks like a better placement for it would be where you have the capacitors now - close to the IC. And capacitors could go on the other side using holes from the crystal as vias.
Got it. Wonderful suggestions all. Thank you! I will be revising my layout to take all of these points into account.
Title: Re: How to debug crystal oscillator startup failures?
Post by: atmfjstc on August 29, 2020, 01:57:49 pm
Could also be that the crystal is damaged? If it was dropped at some point, the crystal could have chipped or cracked, and who knows how it might behave then.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 29, 2020, 06:11:21 pm
Are you saying that I can keep the old crystal soldered into place but just touch a new crystal to the old leads? Doesn't the parallel capacitance affect it?

No, remove the old one, I suspect is is bad anyway. Just try a few new ones without soldering.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on August 29, 2020, 11:56:19 pm
Could also be that the crystal is damaged? If it was dropped at some point, the crystal could have chipped or cracked, and who knows how it might behave then.

Are you saying that I can keep the old crystal soldered into place but just touch a new crystal to the old leads? Doesn't the parallel capacitance affect it?
No, remove the old one, I suspect is is bad anyway. Just try a few new ones without soldering.

I've removed the old crystal and replaced the load caps with 22pF. I have tried five other 12 MHz crystals (two HC49/S and three HC49/U) by pressing their pins into the solder pads, and all exhibit the same behavior.

I want to understand the root cause of the failure before trying a workaround such as starting over with a new PCB. Is there some debugging that I could do with my scope or other instruments (or perhaps even build a diagnostic circuit) to try to understand the ultimate cause of the failure to oscillate? Is there a way to tell whether it is excess capacitance or something else?
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 29, 2020, 11:59:17 pm
Capture with the scope the very beginning of the oscillator enable process. It is typically pretty easy to do since XOUT line goes high when oscillator is enabled. See if oscillations start and then die down, or don't start at all.

Also what code do you use? I'm not familiar with NXP part, but some parts have automatic gain control, but it needs to be turned on only after oscillator is stable. Some parts just have settable gain, and you need to set it to the correct value.

Ok, it looks like oscillator here is pretty primitive with no configuration options. You still need to follow things described in the Section 4.3.3 of the User Manual. Pay attention to disabling of the pull-up/pull-down resistors.
Title: Re: How to debug crystal oscillator startup failures?
Post by: jmelson on August 30, 2020, 12:10:05 am
The classic (modern) oscillator is a CMOS inverter connected to the two crystal pins.  Then, you put a reisistor of some value across the crystal to put the inverter into the middle of its input range.  The selection of the resistor value needs to be adjusted to get the best startup.  There will also be resonating capacitors, typically from each crystal pin to ground.  The series combination of these two capacitors should be equal to the manufacturer-specified resonating value, minus the stray capacitance of the board and IC.

I made boards with the Xilinx XC9500 (5 V) CPLD family for years.  Then, I was forced to move to the 9500XL family (3.3 V) and they added a "weak keeper".  This is a non-inverting gate that feeds the pin back to itself with an ~ 20 K resistor, to prevent input pins from floating.  This fouled up my crystal oscillator circuits.  I had to go down to a 10 K resistor across the inverter to get it to start reliably.

Jon

Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 30, 2020, 12:12:48 am
That LPC chip already has that resistor, it should not need anything other than the crystal and load capacitors,

Also, OP, can you show us your soldering? You don't have a mess of flux all over the place by chance? Some fluxes are conductive.
Title: Re: How to debug crystal oscillator startup failures?
Post by: David Hess on August 30, 2020, 02:45:53 am
CMOS gate oscillators can be picky about crystal losses because of low gain which makes discrete transistor oscillators more reliable.  Check page 5 of TI's application note available here:

https://www.ti.com/lit/an/snla290/snla290.pdf?ts=1598754949238 (https://www.ti.com/lit/an/snla290/snla290.pdf?ts=1598754949238)
Title: Re: How to debug crystal oscillator startup failures?
Post by: JoeyG on August 30, 2020, 04:39:58 am
Good article on  how to measure  Xtal  characteristics https://www.youtube.com/watch?v=y-rCgumTn4Q (https://www.youtube.com/watch?v=y-rCgumTn4Q)
Title: Re: How to debug crystal oscillator startup failures?
Post by: T3sl4co1l on August 30, 2020, 05:04:26 am
I don't know about the LPC family specifically, but ST has excellent info in their appnotes:
https://www.st.com/resource/en/application_note/cd00221665-oscillator-design-guide-for-stm8afals-stm32-mcus-and-mpus-stmicroelectronics.pdf (https://www.st.com/resource/en/application_note/cd00221665-oscillator-design-guide-for-stm8afals-stm32-mcus-and-mpus-stmicroelectronics.pdf)

I think in part because ST seems to prefer a weaker oscillator, thus requiring a higher ESR crystal, which has smaller loading capacitors.

The oscillator is an analog circuit, of course; the traditional inverter circuit simply has a maximal amount of transconductance (peaking around its transition level, and it stays biased there thanks to the feedback resistor).  The crystal is a three-terminal network, which has to have a ground connection in order to have phase shift (which, using a 2-terminal crystal, implies the loading capacitance needs to be greater than zero).  As a network, it has impedance and gain parameters.  Impedance times transconductance times network gain is loop gain, and it must be greater than 1 to sustain oscillation.  For equal loading capacitors, we can assert the crystal network's gain is about 1, so we're mostly concerned with impedance and transconductance.

Nearly consequential to this, is the loading capacitance itself.  This will have reactance comparable to the impedance of the network at resonance.  And typically, crystal ESR is a modest fraction of this.

Crystal resonant frequency also depends on capacitance; you can pull the resonant frequency (over a narrow range) by changing the loading capacitance, of course only if you don't mind that you're changing impedance as well.  This is why crystals are specified with an equivalent parallel loading capacitance.  Use the same capacitance in circuit, and you will get nominal frequency within the stated accuracy.  (Use the incorrect capacitance, or incorrect resonant mode -- crystals can be series and parallel resonant, with most oscillators using the parallel mode -- and you'll have some error.)

So, to some extent, you can deal with a low transconductance oscillator, by reducing the loading capacitors -- perhaps yours will work with 20, 15 or 10pF capacitors; but perhaps not 2.7pF, because it's just too little for the crystal you're using.

Most crystals have modest ESR, 50-100 ohms; as you can see, ST prefers somewhat higher, ca. 200 ohms, and these types specify smaller loading capacitors, which will give a higher network impedance, leading to more loop gain.  Makes sense.

Don't forget to include pin and crystal holder capacitances -- the parallel equivalent capacitance equals the two terminal capacitances to ground, in series.  That is, if you have two 20pF capacitances from either end of the crystal to ground, they act in series as the crystal sees it, for a total 10pF equivalent.

Also, by extension, very low frequency (tuning fork) crystals, typically use quite small capacitances to begin with (~5pF), and combined with the low frequency, they have quite high impedances indeed, 100s kohms.  Keep this in mind when using one of these (e.g. for a RTC).  They are sensitive to ambient noise, PCB contamination, etc.

Tim
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on August 30, 2020, 11:27:40 am
Capture with the scope the very beginning of the oscillator enable process. It is typically pretty easy to do since XOUT line goes high when oscillator is enabled. See if oscillations start and then die down, or don't start at all.

I've examined many traces of XTAL so far and only a handful showed oscillation during my original experiment prior to swapping capacitors; all subsequent traces indicate no hint of it. I've attached representative shots of the latter:
As you can see, there is no indication that any oscillation is being started.
[attach=1][attach=2][attach=3]

Also what code do you use? I'm not familiar with NXP part, but some parts have automatic gain control, but it needs to be turned on only after oscillator is stable. Some parts just have settable gain, and you need to set it to the correct value.

Ok, it looks like oscillator here is pretty primitive with no configuration options. You still need to follow things described in the Section 4.3.3 of the User Manual. Pay attention to disabling of the pull-up/pull-down resistors.

The firmware is a basic "hello world" test in C generated by MCUXpresso. I've taken care to use the IDE's Config Tools to properly set up my pin routing and clocks and I have inspected the generated code to confirm that it's doing proper initialization according to Section 4.3.3 (enable power, disable pull-* resistors, enable 1-bit for XTAL*, disable bypass, add software delay). With SWD I observe all of these steps getting executed prior to the PLL startup code which then hangs waiting for signal lock. I read through the register definitions, too, and did not see anything that allows me to control gain or impedance compensation.

The classic (modern) oscillator is a CMOS inverter connected to the two crystal pins.  Then, you put a reisistor of some value across the crystal to put the inverter into the middle of its input range.  The selection of the resistor value needs to be adjusted to get the best startup.  There will also be resonating capacitors, typically from each crystal pin to ground.  The series combination of these two capacitors should be equal to the manufacturer-specified resonating value, minus the stray capacitance of the board and IC.

I made boards with the Xilinx XC9500 (5 V) CPLD family for years.  Then, I was forced to move to the 9500XL family (3.3 V) and they added a "weak keeper".  This is a non-inverting gate that feeds the pin back to itself with an ~ 20 K resistor, to prevent input pins from floating.  This fouled up my crystal oscillator circuits.  I had to go down to a 10 K resistor across the inverter to get it to start reliably.
That LPC chip already has that resistor, it should not need anything other than the crystal and load capacitors,

Yes, as ataradov says, the MCU already has a feedback resistor. Prior to my original post, I did experiment with putting various value resistors across the crystal leads to see if that helped, but it did not.

Also, OP, can you show us your soldering? You don't have a mess of flux all over the place by chance? Some fluxes are conductive.
[attach=4][attach=5][attach=6]
I've attached photos of the relevant portion of the board. The messy soldering is due to repeated capacitor and crystal swaps, sorry. (I decided to re-solder the crystal because I apparently had an open contact while testing different crystals, whoops.) The flux residue is almost entirely Chipquik SMD291 (http://www.chipquik.com/datasheets/SMD291.pdf (http://www.chipquik.com/datasheets/SMD291.pdf)), which is nonconductive, but I have removed as much as I easily can with IPA.

I should point out that I measured the resistance between each XTAL pin on the back of the PCB and the corresponding lead of the MCU on the front of the PCB, and I saw about 1.4 Ohms each. This seems quite high to me - might this be a significant issue, or is it low enough to be safe?

CMOS gate oscillators can be picky about crystal losses because of low gain which makes discrete transistor oscillators more reliable.  Check page 5 of TI's application note available here:
https://www.ti.com/lit/an/snla290/snla290.pdf?ts=1598754949238 (https://www.ti.com/lit/an/snla290/snla290.pdf?ts=1598754949238)

Thanks David, that's great info. I'd like to measure the stray capacitance on my board but so far my LCR meter readings have been all over the place and I think I need to learn how to use it properly.
Good article on  how to measure  Xtal  characteristics  (URL)

Thanks, that inspired me to run a frequency response analysis of the unpowered crystal circuit:
[attach=7]
Unfortunately the labels on the graph only extend to two decimal points, but the table data shows the peak around 12,000,322 Hz. So, at least while driven by my scope's waveform generator with the rest of the PCB unpowered, the oscillator circuit works 100% of the time at quite close to nominal frequency.

I don't know about the LPC family specifically, but ST has excellent info in their appnotes:
https://www.st.com/resource/en/application_note/cd00221665-oscillator-design-guide-for-stm8afals-stm32-mcus-and-mpus-stmicroelectronics.pdf (https://www.st.com/resource/en/application_note/cd00221665-oscillator-design-guide-for-stm8afals-stm32-mcus-and-mpus-stmicroelectronics.pdf)

I think in part because ST seems to prefer a weaker oscillator, thus requiring a higher ESR crystal, which has smaller loading capacitors.

The oscillator is an analog circuit, of course; the traditional inverter circuit simply has a maximal amount of transconductance (peaking around its transition level, and it stays biased there thanks to the feedback resistor).  The crystal is a three-terminal network, which has to have a ground connection in order to have phase shift (which, using a 2-terminal crystal, implies the loading capacitance needs to be greater than zero).  As a network, it has impedance and gain parameters.  Impedance times transconductance times network gain is loop gain, and it must be greater than 1 to sustain oscillation.  For equal loading capacitors, we can assert the crystal network's gain is about 1, so we're mostly concerned with impedance and transconductance.

Nearly consequential to this, is the loading capacitance itself.  This will have reactance comparable to the impedance of the network at resonance.  And typically, crystal ESR is a modest fraction of this.

Crystal resonant frequency also depends on capacitance; you can pull the resonant frequency (over a narrow range) by changing the loading capacitance, of course only if you don't mind that you're changing impedance as well.  This is why crystals are specified with an equivalent parallel loading capacitance.  Use the same capacitance in circuit, and you will get nominal frequency within the stated accuracy.  (Use the incorrect capacitance, or incorrect resonant mode -- crystals can be series and parallel resonant, with most oscillators using the parallel mode -- and you'll have some error.)

So, to some extent, you can deal with a low transconductance oscillator, by reducing the loading capacitors -- perhaps yours will work with 20, 15 or 10pF capacitors; but perhaps not 2.7pF, because it's just too little for the crystal you're using.

Most crystals have modest ESR, 50-100 ohms; as you can see, ST prefers somewhat higher, ca. 200 ohms, and these types specify smaller loading capacitors, which will give a higher network impedance, leading to more loop gain.  Makes sense.

Don't forget to include pin and crystal holder capacitances -- the parallel equivalent capacitance equals the two terminal capacitances to ground, in series.  That is, if you have two 20pF capacitances from either end of the crystal to ground, they act in series as the crystal sees it, for a total 10pF equivalent.

Also, by extension, very low frequency (tuning fork) crystals, typically use quite small capacitances to begin with (~5pF), and combined with the low frequency, they have quite high impedances indeed, 100s kohms.  Keep this in mind when using one of these (e.g. for a RTC).  They are sensitive to ambient noise, PCB contamination, etc.

This is tons of great info, Tim. Thank you. I need to chew on this a bit more before responding. I did switch back to 22pF but so far have not gotten any signs of life.

---
(EDIT:) How the hell do I get my attachments to show up as inline? The square brackets around "attach=X" don't seem to work.
Title: Re: How to debug crystal oscillator startup failures?
Post by: T3sl4co1l on August 30, 2020, 01:16:45 pm
Did you by any chance do a connectivity/shorts test on that PCB?  The printing looks awfully sloppy for TSSOP footprints...

The freq response is encouraging.

And yeah, inline attachments is disabled while we figure out why they were corrupting.  In the mean time, edit and embed a link to the attachment like so:

(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1056830;image)

Tim
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 30, 2020, 06:05:54 pm
This is a very strange device. From the device datasheet I can't even tell what step actually enables the oscillator. There does not seem to be a defined "enable" bit. It looks like just the pin configuration does this. This is very strange.

It looks like your oscillator tries to do something while VDD is still rising. Why is your VDD so slow?

I would add an empty delay loop at the very beginning of the program before the clock initialization:
Code: [Select]
for (int i = 0; i < 100000; i++)
  asm("nop");

Basically I want to see the crystal startup procedure when VDD is already stable.

And also, trigger on the XOUT, not VDD.
Title: Re: How to debug crystal oscillator startup failures?
Post by: jmelson on August 30, 2020, 09:18:46 pm
Wow, your board photo looks like both leads of the crystal are on the same copper land that goes all over the board.
Do you have a blank board or a CAD drawing of the PC board to see where the crystal ends go?

Jon
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 30, 2020, 09:21:01 pm
Do you have a blank board or a CAD drawing of the PC board to see where the crystal ends go?
First post has layout pictures.
Title: Re: How to debug crystal oscillator startup failures?
Post by: floobydust on August 30, 2020, 09:46:08 pm
I don't see an NXP logo on the part, is it from a reputable distributor? That looks like something from the gutters of guangdong.

The PCB layout is not great when you take ground-pours and have slices in them, and connect them on one end. It totally ruins them. Lack of vias just leaves islands that I am not a fan of. The very long trace from the MCU GND to the south end of the board, curling to J1-7 is not ideal. I don't think it's enough to upset the crystal oscillator but would it make the board unreliable wrt EMC.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on August 30, 2020, 09:48:32 pm
It is not supposed to have a logo. The device is marked correctly according to the datasheet.
Title: Re: How to debug crystal oscillator startup failures?
Post by: floobydust on August 30, 2020, 10:08:28 pm
I automatically get the creeps from that soulless laser etching with no manufacturer's logo... lots of pics showing one.

I agree the VDD ramp up is way too slow. This is interesting:
"8.20.5   Wake-up process
The LPC81xM begin operation at power-up by using the IRC as the clock source. This allows chip operation to resume quickly. If the SysOsc, the external clock source, or the PLL is needed by the application, software must enable these features and wait for them to stabilize before they are used as a clock source."

Thanks NXP for telling us how long that takes lol. I've seem 10's msec for xtal osc to startup.
OP could also try HF osc mode to increase gain.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on September 01, 2020, 12:53:34 am
Did you by any chance do a connectivity/shorts test on that PCB?  The printing looks awfully sloppy for TSSOP footprints...
Yes, I think I've nearly worn out the continuity buzzer on my meter since I started manufacturing this PCB :)  I've been super careful about shorts and opens and have tested the crap out of this board each time I make a change. I know that the printing looks sloppy (I've had to re-solder that IC multiple times) but I have made sure that every lead is wired to its trace and I've used an Xacto knife to etch grooves between some of the leads that had shorts in them.

And yeah, inline attachments is disabled while we figure out why they were corrupting.  In the mean time, edit and embed a link to the attachment like so:
Ahh got it. Thanks for the tip!

This is a very strange device. From the device datasheet I can't even tell what step actually enables the oscillator. There does not seem to be a defined "enable" bit. It looks like just the pin configuration does this. This is very strange.
The actual power-up of the XTAL is accomplished by clearing bit 5 of PDRUNCFG (UM10601 section 4.6.32), after which it is expected for software to induce a 500us delay.

It looks like your oscillator tries to do something while VDD is still rising. Why is your VDD so slow?
An excellent question. To find out, I first tried bypassing my 3v3 LDO and powering the MCU directly from my power supply but I saw the same slowness. I then suspected my 3.3uF bypass cap on the 3v3 side might be too large, so I desoldered it, but the waveform remained the same. Finally, I replaced my power supply (Keysight E36313A) with a square wave output from my signal generator and this time I saw a much tighter rise to Vdd, about 200us, as opposed to >40ms when using the power supply (wow! I had no idea that the slew time was so glacial on such an advanced power supply. What the crap, Keysight?).

Nevertheless, the now-improved Vdd slew time did nothing to fix the oscillator.

I would add an empty delay loop at the very beginning of the program before the clock initialization:
Code: [Select]
for (int i = 0; i < 100000; i++)
  asm("nop");

Basically I want to see the crystal startup procedure when VDD is already stable.

And also, trigger on the XOUT, not VDD.
I've added the above delay and I'm toggling a GPIO as a marker to indicate start and end of clock init. Here are scope shots at increasing levels of magnification:
No signs of oscillation anywhere. It concerns me that the expected 500us delay is turning out to be 2ms. I'm going to try to figure that out, but it doesn't seem like that would be directly related to the failure here.

(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057652)
(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057656)
(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057660)
(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057664)
(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057676)

Wow, your board photo looks like both leads of the crystal are on the same copper land that goes all over the board.
When manufacturing the PCB I cut to shape a layer of Kapton tape and stuck the crystal leads through it, so that there is an insulating film between the crystal body and the ground plane pour on the PCB. I've repeatedly tested to make sure that there is no conductivity through the layer.

I don't see an NXP logo on the part, is it from a reputable distributor? That looks like something from the gutters of guangdong.
I automatically get the creeps from that soulless laser etching with no manufacturer's logo... lots of pics showing one.
I totally get why you got that impression. I did probably eight or 10 rounds of hot air reflow on this chip to get it correctly mounted without shorts/opens and it looks like the heat might have eroded some of the package text. In addition, I was using the ring light on my microscope to illuminate the board. These are genuine chips that I bought from DigiKey a while back and here's what they look like new and in different lighting:
(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057672)

The PCB layout is not great when you take ground-pours and have slices in them, and connect them on one end. It totally ruins them. Lack of vias just leaves islands that I am not a fan of. The very long trace from the MCU GND to the south end of the board, curling to J1-7 is not ideal. I don't think it's enough to upset the crystal oscillator but would it make the board unreliable wrt EMC.
Roger that, definitely getting the message that my layout needs to be redone. I will be making a new version in the near future but I want to see if I can still get this one working right now.

I agree the VDD ramp up is way too slow. This is interesting:
"8.20.5   Wake-up process
The LPC81xM begin operation at power-up by using the IRC as the clock source. This allows chip operation to resume quickly. If the SysOsc, the external clock source, or the PLL is needed by the application, software must enable these features and wait for them to stabilize before they are used as a clock source."

Thanks NXP for telling us how long that takes lol. I've seem 10's msec for xtal osc to startup.
I believe this is what the 500us software delay is for. At any rate, I am seeing no indication that any oscillation is occurring whatsoever, so I don't think that more settling time would help in this case.

OP could also try HF osc mode to increase gain.
Great suggestion. I set bit 1 in SYSOSCCTRL and retried; no change, however.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on September 01, 2020, 01:06:11 am
After I/O pins init the XOUT goes to some strange level of 2.5 V.

It is also strange that XOUT went to 0 in a steady state. See what happens on the XIN at the same time. And then try to drive XIN from the external source. Does XOUT show inverted waveform?
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on September 01, 2020, 08:53:14 am
After I/O pins init the XOUT goes to some strange level of 2.5 V.
Yeah, the 3.3V->2.5V transitions happens when the Switch Matrix (SWM) is used to route the functionality of pins P0_8 and P0_9 to XTALIN and XTALOUT, respectively. At the time that this happens, the pull-up resistor is still activated (it gets disabled later in the sequence), and being an analog pin now with a lower operating point it doesn't surprise me that the reference high voltage is lower than 3.3V. This is just pure speculation on my part though.

It is also strange that XOUT went to 0 in a steady state. See what happens on the XIN at the same time.
Here is a capture of XIN vs XOUT. Note that XIN was captured using a passive probe while XOUT was with an active probe.

(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057820)


And then try to drive XIN from the external source. Does XOUT show inverted waveform?
I'm assuming you mean with an unpowered PCB. Here's what it looks like when I drive XIN with a 3.3V 12 MHz sine wave. I captured statistics for ~1k samples. The frequency and quality of the waveform vary quite widely, but I think that that's in large part due to my long injection leads causing significant inductance. I'm a bit surprised to see XOUT Vpp >9V though! I know that this is due to resonance, because setting my wavegen to 11.99MHz or 12.01MHz attenuated the signal by more than one order of magnitude. But I was not expecting the voltage to triple like that.

(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1057824)
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on September 01, 2020, 10:34:45 pm
The driving experiment needs to be done with with the device powered. You don't need high frequency, a few kHz will work. You can also use a series resistor to limit the current.

Basically I want to check that inverter is actually working as an inverter and produces correct results.

Without the crystal XIN and XOUT should reach the same "middle" voltage. In your case it looks like XIN is pulled-up, so XOUT is 0, but the feedback resistor is not strong enough to override that pull-up. So may be try adding a stronger external feedback resistor to see what happens.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on September 01, 2020, 11:31:10 pm
The driving experiment needs to be done with with the device powered. You don't need high frequency, a few kHz will work. You can also use a series resistor to limit the current.

Basically I want to check that inverter is actually working as an inverter and produces correct results.

Without the crystal XIN and XOUT should reach the same "middle" voltage. In your case it looks like XIN is pulled-up, so XOUT is 0, but the feedback resistor is not strong enough to override that pull-up. So may be try adding a stronger external feedback resistor to see what happens.
Ok, here are captures of XOUT on a powered PCB with the crystal still attached and 68pF load capacitors this time (from another failed attempt last night to get the board working):
It looks like the inverter works well from LF to about 1.2MHz. Beyond the -3db point it appears that the signal attenuates and dephases rapidly. Do you think that this is my problem?

May not be relevant but I'm curious about the sudden reversal of trend for the phase around 100kHz. What would explain that?

(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1058196)
(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1058200)
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on September 02, 2020, 12:43:11 am
The trend is probably due to capacitors.

The amplitude on the running oscillator would be lower, it will not be a full swing signal.

68 pF is a lot, I would drop that. Try something like 12 pF, or even try without any capacitors.

Other than this, I'm out of ideas. I have never had any issues with crystal oscillators.
Title: Re: How to debug crystal oscillator startup failures?
Post by: floobydust on September 02, 2020, 12:56:18 am
If you can keep it oscillating, try the test and see what you get: Raltron Measure Negative Resistance/Oscillator Margin (https://www.raltron.com/wp-content/uploads/2017/04/RT-DWG-Measure_Negative_Resistance-Circuit_Margin-complete-version.pdf)

I looked at 400 12MHz crystals and they have loading capacitance of 4-32pF and ESR from 15-300 ohms. They vary widely and ruling out if the problem is your crystals is very tough. Try any other crystals you have lying around.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on September 05, 2020, 10:25:34 am
I tried 27pF caps again because that was the only value that ever gave a good signal and I have a limited selection of 0603 caps in the 1-100pF range. When that didn't work, I swapped out yet another crystal, which didn't help either. (I have no way of knowing ESR or other motional parameters for my inventory of crystals as I do not own a crystal impedance meter and I couldn't find a schematic to build one myself.)

Each time I rework this board, it causes more damage to the traces due to the properties of the silver conductive ink in the Voltera PCB printer and now the board is becoming hard to work on. So between being out of ideas and the PCB rework getting more difficult, I think it's time for a new board.

Using the layout suggestions graciously offered by ataradov and floobydust, I started from scratch and laid out a brand new board, this time routing 100% by hand. Among many differences from the previous version, I am mounting bypass and load caps on bottom and there is no ground plane around the crystal. I also added space for two more load caps so I can mix & match parallel cap values (in 0402 size, where I have different values available) and test points to help tune crystal resonance.

I would love to have some input on this design, paying particular attention to the critical oscillator section. Thanks!

One question I have is whether I should put a small ground plane zone on the front, north of the XTAL extending only to the two GND test points, to help isolate some noise from the load caps that are on the other side.
Title: Re: How to debug crystal oscillator startup failures?
Post by: floobydust on September 05, 2020, 04:57:54 pm
After all this, I have to question the impedance of the Voltera conductive ink  :-// What do we know about how it behaves at 12MHz and the Skin Effect? It might be the issue here.
The new pcb layout looks much better.
I would use thicker traces to the crystal portion. It's OK to have the caps right near the crystal pins and the ground-pour closer. You have it backed-off at the crystal pads but not at the test-points or capacitors, so what was the point. I always use ground-pour on MCU crystal traces and the pF it adds is not a problem.
The ground pour has some skinny slices which I never like, they wreck the whole concept. So I try to keep those two areas from being thin.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on September 05, 2020, 05:19:12 pm
Yes, I did not realize that this was printed with the ink. I assumed the PCB was routed.

The first thing to do is to measure the actual resistance of the track from the crystal pin to the MCU pin.

And then make the traces to the crystal as thick as you can, and still be prepared to solder the wire in top of them. You can actually do that for the existing design - just double the tracks with thin wires. Use individual strands from the multi-stranded wire.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on September 05, 2020, 11:29:13 pm
I would use thicker traces to the crystal portion. It's OK to have the caps right near the crystal pins and the ground-pour closer. You have it backed-off at the crystal pads but not at the test-points or capacitors, so what was the point. I always use ground-pour on MCU crystal traces and the pF it adds is not a problem.
The ground pour has some skinny slices which I never like, they wreck the whole concept. So I try to keep those two areas from being thin.
And then make the traces to the crystal as thick as you can, and still be prepared to solder the wire in top of them.
Thanks guys, I've updated to v1.3.1:
After all this, I have to question the impedance of the Voltera conductive ink :-// What do we know about how it behaves at 12MHz and the Skin Effect? It might be the issue here.
12 MHz is a low frequency relative to what Voltera ink can handle (https://community.voltera.io/t/using-the-v-one-for-rf (https://community.voltera.io/t/using-the-v-one-for-rf)). Also, doing our FRA and signal injection experiments above suggested to me that even my subpar layout performed adequately at that frequency. Or do you have a different interpretation?

The first thing to do is to measure the actual resistance of the track from the crystal pin to the MCU pin.

I did this earlier:

I should point out that I measured the resistance between each XTAL pin on the back of the PCB and the corresponding lead of the MCU on the front of the PCB, and I saw about 1.4 Ohms each. This seems quite high to me - might this be a significant issue, or is it low enough to be safe?

You can actually do that for the existing design - just double the tracks with thin wires. Use individual strands from the multi-stranded wire.
I haven't tried this yet. I'll report back if this ends up fixing it, but I'm pretty doubtful at this point. I think I need to make a new board.

(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1060806;image)
(https://www.eevblog.com/forum/beginners/how-to-debug-crystal-oscillator-startup-failures/?action=dlattach;attach=1060810;image)
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on September 05, 2020, 11:54:49 pm
1.4 Ohms is not significant if it is indeed the case.

The layout looks ok, I guess, for the limitations of the technology.
Title: Re: How to debug crystal oscillator startup failures?
Post by: floobydust on September 06, 2020, 08:05:03 pm
I'm not familiar the Voltera (https://www.voltera.io/product) materials and process for thin films. It looks like fun but a learning curve.
Surprised one person went to 5GHz with the material, but it seems there is a chance for variability... Application thickness (ink height), curing, wash, flux, tin bismuth silver soldering etc. This isn't copper.
"As the {baking} ink temperature increases, chemical reactions are triggered and the metallic particles will form a conductive matrix and fuse together. Many conductivity and soldering problems occur because the ink was either not cured right away, at the right temperature, or long enough."

Copper rivets (https://www.voltera.io/support/working-with-rivets#main) for via's seem odd to mix with silver paste. Another connection to test, as how does ink do much on copper, I'm used to soldering.
I would be curious of the (flux?) wash process, and measure leakage currents and capacitance between some trace runs, in case that is happening.
It's just that a new board would not really put us further ahead, whether it works or not, as to knowing what went wrong.
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on September 09, 2020, 01:32:42 pm
An update on this: I've printed, assembled, and tested the new board. It, too, failed to oscillate most of the time, but I was able to get a good crystal sine waveform and buffered CLKOUT square output from the MCU one time. I measured both at precisely 12.000 MHz. The square wave looks very clean and a mask test showed that less than 10 samples out of ~300,000 deviated from the nominal 83.33ns period by more than 1ns. So it is a very stable and accurate clock – the one time that it worked. This shows me that the board has good steady-state signal integrity and the firmware has the correct initialization sequence (presuming no hidden race conditions are present).

All previous and subsequent resets and power cycles failed to get the oscillator running, however. The XIN and XOUT behavior was identical to that of the previous board. Notable differences in this board are the new layout as well as using 39pF load capacitors (which I discovered as part of an entire booklet of SMD caps that was buried in my workshop, so I have more values in the 1-100pF range to try now, if necessary).

Resistance between each XTAL pin and the MCU is 400-500 mOhm, better than the old board and very reasonable IMO. Capacitance between each XTAL pin and ground is 50-100pF (hard to get a repeatably precise number with my DMM and I can't use my LCR meter because I don't know how to get it to measure capacitance at DC).

I've done two experiments so far: Set the high-frequency mode bit in SYSOSCCTRL and added feedback resistors (20k and 680k). There's been no change. I'm loathe to start reworking the board again to try different valued caps and especially different crystals. I feel that since none of that made any difference in the scope output on the old board, I need to be thinking about other possible problems instead. But I'm stumped.

"As the {baking} ink temperature increases, chemical reactions are triggered and the metallic particles will form a conductive matrix and fuse together. Many conductivity and soldering problems occur because the ink was either not cured right away, at the right temperature, or long enough."
I cured right away and used the default settings on the Voltera.

Copper rivets (https://www.voltera.io/support/working-with-rivets#main) for via's seem odd to mix with silver paste. Another connection to test, as how does ink do much on copper, I'm used to soldering.
It seems to work ok for me and presumably for the majority of Voltera users. I do continuity tests on all vias and I add a little solder between the rivet and the trace if there's an open.

I would be curious of the (flux?) wash process
For hand soldering and reflow, I use minimal amounts of (nonconductive, no clean) ChipQuik SMD291. For all three sources of flux (the ChipQuik, rosin core in my hand solder, and the solder paste) I wipe away most of it with a Kimwipe and some 91% isopropanol.

measure leakage currents and capacitance between some trace runs, in case that is happening.
Haven't done this yet but that will be my next step.

It's just that a new board would not really put us further ahead, whether it works or not, as to knowing what went wrong.
I agree, I was just getting frustrated about getting nowhere with the old board and had also reworked it too many times for it to be viable. (Un)fortunately the new board seems to suffer from the same issue, so I can continue debugging on that.
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on September 09, 2020, 06:16:20 pm
The fact that Xout is squared is strange to me. Typically when oscillations are stable  both Xin and Xout are closer to sine waves. Xout is a bit higher amplitude.

I don't know id this is some feature of the NXP oscillator or something else is wrong.

Can you make it start again? Can you capture Xin/Xout at the same time?
Title: Re: How to debug crystal oscillator startup failures?
Post by: T3sl4co1l on September 09, 2020, 06:35:24 pm
Is it the kind with logic-level XOUT, and you need a series damping resistor?  Or it was just configured for such?  (Sometimes there are multiple drive levels; this might be a setting used for ceramic or LC resonators.)

Or is CLKOUT a separate pin?

Tim
Title: Re: How to debug crystal oscillator startup failures?
Post by: pion on September 09, 2020, 09:45:50 pm
The fact that Xout is squared is strange to me. Typically when oscillations are stable  both Xin and Xout are closer to sine waves. Xout is a bit higher amplitude.

I don't know id this is some feature of the NXP oscillator or something else is wrong.

Can you make it start again? Can you capture Xin/Xout at the same time?
No, sorry, perhaps my wording wasn't clear. XIN and XOUT were both sine waves. CLKOUT is the system clock output, which is a buffered copy of the internal system clock and consequently a square wave.

Or is CLKOUT a separate pin?
It's an internal signal which I routed to a physical pin via the MCU's Switch Matrix (SWM).
Title: Re: How to debug crystal oscillator startup failures?
Post by: ataradov on September 09, 2020, 11:00:49 pm
I think the only way here is to try different capacitors. And may be see if there are some known NXP specific issues with that oscillator.

This is very strange. I connected crystals to a lot of different MCUs, sometimes without any additional capacitors, and I never had any issues.
Title: Re: How to debug crystal oscillator startup failures?
Post by: T3sl4co1l on September 10, 2020, 04:11:55 am
As long as we're thinking about trace resistance, are the power pins nice and quiet, well bypassed?  (Should be able to probe between the pins, may need some fine wire to get a stable connection to them.)

Tim
Title: Re: How to debug crystal oscillator startup failures?
Post by: floobydust on September 10, 2020, 05:30:09 am
I thought one scope trace looked like it oscillated for about 10 cycles before dying out?
I would try change VDD, going up to the 3.6V limit or 4.6V (don't kill the jumpered LDO) which should increase the osc gain (or go down to lower it). The VDD rise time is also the old 2018 silicon errata issue, so I would try speed it up.

You could try would post the problem in the NXP TechSupport forums. (https://community.nxp.com/t5/forums/searchpage/tab/message?advanced=false&allow_punctuation=true&q=crystal) There's a few crystal oscillator problems, but their answers are sometimes off the mark a bit unfortunately. NXP does not seem to specify the crystal's parameters on datasheets for many MCU parts beyond "use the dev board example". Really?
The LPC812 dev board uses 27pF with ABLS2-12.000MHZ-D4Y-T (ESR 50R CL=18pF), so 27-30pF agrees with my math.