Author Topic: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL  (Read 1997 times)

0 Members and 1 Guest are viewing this topic.

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
I've spent a dozen or more hours exhausting my options to the best of my ability and have started resorted to pulling and replacing ICs near the circuit in question, so I figure it's probably past time to ask if anyone else has experience with this.

I'm working on an SRS SR530 lock in amplifier, the manual (including schematic and theory of operation) is here: https://www.thinksrs.com/downloads/pdfs/manuals/SR530m.pdf , and I'm working on the circuit on page 85.  The symptom is that the unit shows reference unlock intermittently, but only at lower frequencies - 100 Hz or 1kHz reference input are often unlocking, but 10 kHz+ is relatively error free and 50kHz I have yet to see an unlock on.  Input is being fed directly from a signal generator, generally 1Vpp HiZ, but I've tried a few combinations.  Looking at the PLL waveforms on the scope, they will stay stable and locked for a bit, then will jump slightly and sort of rebound back slowly towards the correct frequency (excursions on both sides of the intended frequency) and then sometimes keep being unstable and sometimes relock.  The Unlock LED on the front panel reflects this, and the other 2 PLLs in the unit which are both referenced from the comparator at the end of the loop of the first show the same kind of instability and are time correlated to this one (so I think this first PLL is the issue.)

Best I can tell, the front end for the reference input is fine, a stable square wave of the right frequency arrives at the pin 14 input of U306 (CD4046), and I've verified that the pin 3 phase comparator input receives a signal and that the ouput pin 13 is adjusting with pulse widths that correspond to the phase difference between the two inputs (and are positive or negative from the output bias point), so I think the PLL controller is working.  The VCO does sometimes get the right frequency, and the the triangle wave is consistent (pin 4 U309) and speeds up and slows down as the PLL controller is adjusting the loop.

I haven't super comprehensively checked, but it doesn't appear that the external (to this page of schematic) analog switch control lines are toggling at all when the instability happens.  I've checked the comparator (U314) biasing and it is stable and consistent with printed voltages.  Removing said comparator results in a much faster but stable PLL frequency with the controller not trying to adjust anything.  While it's a loop and I know it's tricky to point a finger at a part when everything is feeding back on itself, I feel like I've checked as many signals as I can think of and replaced half the ICs involved already, so I'm wondering where the instability is hiding or if there's some kind of trick/test I can muster to figure out what's the cause.
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Hello DaJMastaTopic,

I've no experience on the SR530, but here are a few hints.
- Check your power supplies when locked and unlocked to see if there's a difference
- reflow each solder joint on the motherboard
- Use freeze-spray to check if it is a thermal issue that triggers the unlocked status
- replace all electrolytics
- measure all resistors and diodes
- replace the most error /partial failure prone parts unconditionaly (and use top notch DIL sockets, to ease replacement) : analog switches, opamps, logic ICs (in that order).

In tedious situations where locating to root cause is really difficult (like closed-loop systems^^), a succesful repair is not only a time investement, but it has also a financial cost.

This SR530 seem to use rather vanilla parts. Replacing all of the semiconductors, buy sockets, first tier electrolytics and two freeze-spray cans should be around 150-250 bucks.
It's up to you to decide if it's a reasonable investment or not.
« Last Edit: December 07, 2024, 08:58:04 am by timeandfrequency »
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
Well I had hoped there was an option that was less brute-forcey than what I had already started resorting to - I am not interested in a blanket recap, or individually measuring every component.  While I want it fixed, I'm seeking understanding too, and a substantial expense on a pretty common not so expensive unit doesn't sound appealing.

It's not working yet despite a few more replacements and a lot more investigating.  I found an LM311 in the second PLL (U329) that had no real output, because while there was signal on the input, it was more negative biased than it should have been and never reached the threshold.  The circuit was partially functional, but it turned out to be one of the output buffer transistors in U322 seemed to be malfunctioning, and replacing the chip has brought the waveform back (the phase controls on the front panel now maintain the right frequency instead of the PLL freerunning into a much faster mode).  Now this didn't fix it, but the red unlock light is on less often.

I poked around a bit more and started suspecting these transconductance amp chips (perhaps there are some good reasons they were EOL'd?), and I've found another which may be suspect - U307.  The output shows a little activity shortly after power on and while pin 1 shows a very small pulsed signal after that, but the output is just railed negative (though it gets there gradually and shows some response to the pulses in the first couple of seconds).  Trouble is, I don't have a replacement on hand, the previous was an SMD chip on a carrier with bent pins to fit in a socket, so I'll have a few days before I can try and replace it.
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
It took some time for USPS to deliver the replacements, but I got them in, swapped parts... and it's still broken.  So I checked with the thermal camera again, no outliers (though CA3140s are power hungry!), and probed around some more.  I've isolated it to the first reference PLL (not the quadrature PLL or the off-the-main board PLL) by pulling the chip and board, respectively, so I reflowed every pad in the physical area of that first PLL (two rows of ICs with the surrounding passives) - no change.

So I probed around some more, using the quadrature PLL as an apples-to-different-kind-of-apples comparison, and I narrowed things down.  The fault I'm seeing looks somewhat like switching of the triangle wave (pin 4 of U309), where it will be stable and will change frequency appropriately, but when the unlock light goes off, you see a vertical artifact - it looks almost like you switch from one triangle wave to another mid waveform, but only briefly.  Farther upstream, at pin 6 of U308 (Vvco), you can see normal adjustment pulses, a couple volts or so excursions from a fixed offset, but then when the unlock light is on, sometimes there are brief pulses that rail to the positive rail, a behavior that doesn't exist in the quadrature VCO.  I've checked the other ends of the Vvco connection, and while the same signal is apparent on the near side of the resistor, it's not present past it, so it is being generated by this PLL.  I've also replaced virtually every IC in the loop, save U306, the PLL controller, since it looked like its output is normal, but I suppose it's the next suspect even though at a glance, its output is normal and it starts looking strange at U308.  U308, U307, U311, U310, U309, U314, U315, and U312 have all been socketed and replaced, testing with other pulled and new ICs along the way.  The small holes on the board make IC extraction really timely, so it's been an annoying slog so far.

So I'm truly baffled.  It's still intermittent and brief, so I don't know how a passive or a short could do it.  It doesn't change pressing on any part or flexing the board, so I don't think it's a cracked trace.  I've replaced almost every active part and checked the pins talking to other portions of the board and ruled them out.  I've monitored the rails for noise, abnormality, or intermittent issues and seen nothing.

I suppose in making this post, I've just convinced myself to swap U306 because it's the last active part in the loop, but at least without time correlating inputs and outputs (I don't remember if I did this with the quadrature loop or this one in the past, but I know I've done a little), maybe there's something about the timing of its normal switching behavior that's resulting in wonky outputs of U307 which is quickly discharging the capacitors generating the triangle wave.
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
I replaced U306, no difference.  I probed around and captured some synchronized waveforms, and this is the troublesome one I'm seeing.  Blue trace is pin 3 of U306, the comparator input,  pink trace is pin 1 of U301, the !Q output, and the yellow trace is pin 4 of U309, the triangle wave.  This captured glitch is the issue I see, and you can see, there are two distinct slopes going on, and I'm not sure what's initially causing it.  I've now replaced every IC in the loop and the issue persists.



I wonder if C310 between pins 2 and 3 of U314 could be allowing too much signal through and causing improper triggering or what, but I've checked all the resistors in the loop now and the two integration caps attached to U310 - no problems so far.  It's worth mentioning that as a result of the work in the last post, the unlock errors are less frequent, but their nature is the same.  It could be that I fixed something in a secondary loop or something that was slightly contributing which is now working better.

It's the wrong behavior, it's not a behavior present in the quadrature loop, and all of the faults I can think of that could cause it would have been fixed by replacing the ICs.  Any guesses?
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Hello DaJMasta,

I see that you made significant work and analysis to narrow down the issue.

I suppose in making this post, I've just convinced myself to swap U306 because it's the last active part in the loop, but at least without time correlating inputs and outputs (I don't remember if I did this with the quadrature loop or this one in the past, but I know I've done a little), maybe there's something about the timing of its normal switching behavior that's resulting in wonky outputs of U307 which is quickly discharging the capacitors generating the triangle wave.

I replaced U306, no difference.  I probed around and captured some synchronized waveforms, and this is the troublesome one I'm seeing.  Blue trace is pin 3 of U306, the comparator input,  pink trace is pin 1 of U301, the !Q output, and the yellow trace is pin 4 of U309, the triangle wave.  This captured glitch is the issue I see, and you can see, there are two distinct slopes going on, and I'm not sure what's initially causing it.  I've now replaced every IC in the loop and the issue persists.


Updated version

You write "pink trace is pin 1 of U301, the !Q output" : can you confirm ?
Because Pin 1 of U301 is the output of an LF353 opamp.



It might be interesting to probe U306 Pin 13 (output of the phase comparator II).
This signal is of logic level (it's a kind of PWM signal related to the instaneous phase difference of both input signals).
Usually, the phase comparator output is then converted to an analog value. C336 and probably U307 (*) are in charge of that conversion.
This should be reflected at pin 12 of U317 U307. Do you see a smooth analog trace at pin 12 ?
U317 U307 might be used as a VCA because of the 3 2 switches that drive pin 16.
Of course, I don't know if I'm right but I see a 'Vvco' label at pin 6 of U308 (CA3140). VCO drive voltage has to be very smooth without any glitch.


It's the wrong behavior, it's not a behavior present in the quadrature loop, and all of the faults I can think of that could cause it would have been fixed by replacing the ICs.  Any guesses?
Yes you changed every IC in the loop and there's still the same flaw.

What about the logic signals /HI, /LO, /f  and /2f  that are on the right side of the schematic ? Are they stable and always show a valid logic voltage (< 0,8V or > 2,4V) ?
Did you check the '10.24V REF' signal ?

Update #2 (20241217)
(*)  and also C306
U317 -> U307
3 switches - > 2 switches
« Last Edit: December 17, 2024, 08:07:17 am by timeandfrequency »
 

Offline fenugrec

  • Regular Contributor
  • *
  • Posts: 239
  • Country: ca
Intermittent problems are awful. You seem to be on the right track, you probably addressed these already but in case:

How did you capture that interesting glitch in the triangle generator, are you triggering your scope on an Unlock comparator / LED / something else ? That will be most helpful. Nothing worse than staring at a rolling display waiting for a glitch to happen...
Is this circuit built with SMT components ? MLCC caps can sometimes fail intermittent / partial short, although I doubt they used them in critical areas e.g. integrators / triangle wave gen.
I guess any passive could behave intermittently, but I see that separating cause and consequence will be difficult in such a closed loop.
I think you mentioned you checked supply rails; if you have a good trigger source, did you also test locally on critical ICs ? For the off chance that some of the power distribution is flaky...

Is there any way of isolating parts of the loop , e.g. triangle generation, and (if needed) stimulate it from an external source for a period of time and see if that glitches ?
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Is this circuit built with SMT components ? MLCC caps can sometimes fail intermittent / partial short, although I doubt they used them in critical areas e.g. integrators / triangle wave gen.
Unless there were more recent boards produced, this seems to be only through hole. Shahriar shows it's guts.
Even if this TE works at rather low frequency (< 100 kHz ?), it's a complicated gear : 13 pages of schematics, about 1400 parts.

What definitely raises the level is the deep interaction of many logic signals driving analog switches located inside the different functional blocks. You can watch signals at a few hunderts Hz and the loop can go bersek because of a logic glitch of less than 100 ns.

Typical Ton & Toff for the 'DG' series is about 400 ns, but for the control signal tr & tf < 20 ns is required.
If the control signal is too slow and only 'ramps up', unpredictable behaviour occurs.

@DaJMasta seems to have replaced all of the IC's in the loop, and there's only a moderate behaviour improvement. So next direction could be looking at the signals coming from outside of the loop and that influences it's behaviour (dynamically or only during a range change). Any instability or glitch in any control signal could deeply bother the loop.

@fenugrec's suggestion of careful triggering and then trying to identify anything weird is for sure useful in the current situation.
In the picture you provided, something plainly uncommon happens : if this is exactely the moment when the loop unlocks, you're on the right track.

If nothing shows up, you will need to start replacing the passive parts.


PS : I updated my previous post. Changes are mentionned at the bottom of the text.
« Last Edit: December 17, 2024, 08:10:38 am by timeandfrequency »
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
It is all through hole, two sided board, pretty old school through and through.

For clarification, the pink trace was pin 1 of U306 - long night of debugging showing up there.  With a little more thoughtful analysis, you can see that the triangle wave slope is proportional to the pulse width shown on that trace - so this is the result of comparing the comparator's input to the PLL controller (pin 3 of U306, light blue trace) with the steady input signal (not shown).  The yellow waveform is captured after the triangle wave generation circuitry suggesting that part actually works fine - it should be responding with a different sloped triangle wave based on the controller's phase comparison.  Since the controller is comparing the reference and comparator input waveform correctly and the following circuitry is converting that pulse width information to an accordingly sloped triangle wave, the problem must lie with the comparison stage at the end, so that there's something wrong with the data being driven into pin 3 of U306.  At least, that's my understanding.

For the analog switches involved in this loop, there is no toggling when this behavior is occurring.  I've also looked at the "phase voltage" control line, and while it's not exactly the 5.12V or -5.12V they seem to indicate, changing through the phase controls on the front panel does make it jump about 5V and neither position seems to make it more or less stable.

Since it was perhaps a comparator issue, I took a look at the +5V line that biases some of the outputs and I do see some noise where other rails I see none.  In fact, it's close to 300mV peak to peak, so I went snooping around.  The caps involved in this seem to be ok (no heating of the tantalums, the 47uF C329 measures 43uF and 5.4 Ohms ESR at 100Hz), but the noise is present on the sheet side of R371 whereas the supply side of R371 is not noisy at all.  I went and searched through the sheet for all the connected nodes to this local 5V rail and there are quite a few, and a bunch of them are relatively low impedance (through 1k Ohm resistors) directly to switching signals, so the noise seen seems reasonable, though perhaps still slightly suspect.  The one oddity I could spot on that rail was pin 14 of U305 - a high trigger level input on the input conditioning circuit - where the 330k Ohm resistor to this suspect 5V rail puts pin 14 at only about 1V, meaning around 12uA of draw through it.  While not the source of noise itself, certainly, that value is 20x over the datasheet input current for that pin, which makes me wonder if it could be operating in a way that's sort of thrashing the +5V rail.

As for triggering, the unlocks are frequent enough that I've been just running longer captures and stopping and zooming in.  My test frequency is around 100Hz, so I can build up a few seconds worth of traces, visibly see noise on them, then stop for a single capture and zoom in to have a look.  When not unlocked, the output pulses don't show the narrow pulses pictured in the trace above.  Probing the unlock LED bus itself gives you an analog voltage, as it's the three VCO unlocks summed and put through a low pass filter.  You can monitor the VCOs somewhat individually on the  cathode sides of the diodes (on this sheet D301 and D302).

I'm out of time tonight, but maybe I can just pull the now socketed U314 and inject a comparator output to see if I can stabilize the loop... I don't think the chip is bad as it's already been replaced, but perhaps it could show me something interesting about whatever passives around it which may be interfering, or maybe it will show me some wonky behavior elsewhere when given a consistent, stable signal.
 

Offline fenugrec

  • Regular Contributor
  • *
  • Posts: 239
  • Country: ca
involved in this seem to be ok (no heating of the tantalums, the 47uF C329 measures 43uF and 5.4 Ohms ESR at 100Hz),

I very much doubt this would cause all your issues, but that ESR looks high for a tant ? Opened a random datasheet here and I'm seeing dissipation factors of ~ 6% , and if I'm mathing correctly, is equivalent to ~ 2.2ohm ESR at 100Hz.

Quote
pin 14 of U305 - a high trigger level input on the input conditioning circuit - where the 330k Ohm resistor to this suspect 5V rail puts pin 14 at only about 1V,

If you pull U305 out of its socket, does that voltage go back to 5V ? Else C305 could be bad (excessive leakage)
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #10 on: December 18, 2024, 04:28:03 pm »
Hello DaJMasta,

You seem to be very close to find the issue.

[...] the problem must lie with the comparison stage at the end, so that there's something wrong with the data being driven into pin 3 of U306.  At least, that's my understanding.

For the analog switches involved in this loop, there is no toggling when this behavior is occurring.  I've also looked at the "phase voltage" control line, and while it's not exactly the 5.12V or -5.12V they seem to indicate, changing through the phase controls on the front panel does make it jump about 5V and neither position seems to make it more or less stable.  [...]
I could not find in your posts any information that you changed U313, which sometimes is also 'in the loop'.

As you mentionned ealier, C310 (22 pF)  presence is not easy to explain. Why put an capacitor between the inputs of a comparator ? HF noise reduction ?
Try to replace it or even try another value (lower or higher value ?)

Did you check the '10.24V REF' signal ? or better the 1.42 VDC reference signal (U314 pin 2) ?


You say "For the analog switches involved in this loop, there is no toggling when this behavior is occurring". I would still give a deeper look in that. IMO, there's something 'external' that is jolting the loop. But for the moment you cannot see it.
As you record very long sequences of a 100 Hz signal, your sampling rate (12.5 MSa/s) avoids catching plainly anything shorter than 160 ns (Nyquist). Even if the 'DG's are rather slow (400 ns), comparators and vanilla logic gates or flip-flops usualy switch within in 10 ns.

I would suggest to improve your trigger condition to be spot on the moment the triangle goes bersek :

- trigger if blue trace (pin 3 of U306) ='1' for less than 1 ms
or
- trigger if pink trace (pin 1 of U306) ='1' for less than 200 us

Then crank up the sampling rate at the maximum value your scope allows and carfully probe again each logic control signal and analog value :
e.g.
pins 8 & 9 of U312,
signals /HI  /LO,
 /f , /2f,
1.42 VDC ref voltage,
U311 pin 16,
U311 pin 8,
U311 pin 1,
Vvco,
U310 pins 9 & 16  (the others are /HI and /LO),
U306 pin 14


Good luck !
 

« Last Edit: December 18, 2024, 06:15:43 pm by timeandfrequency »
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #11 on: December 19, 2024, 12:16:09 am »
The ESR for C329 seems reasonable to me - it's the one in that group that isn't a tantalum, and I looked up comparably sized and rated electrolytics and around 5 ohms seems normal.  U305 is currently not socketed, but its function is the signal lost timeout (helps reset the PLL when disconnected and tells another board) and that signal seems to be operating properly.

I think C310 is probably for high frequency noise immunity - it's on a few comparators in similar configurations, and at least the way I think about it, a low value capacitor will be an open until higher frequencies where it will slowly act more like a short, so it balances the two inputs more for higher frequency edges, making it less likely to trigger on that noise.  I can't say my explanation is 100%, but SRS has some serious analog design folks designing these, so I am happy to chalk it up to them knowing better than I some esoteric footnote of the LM311's performance.

The 1.42V reference is correct and the 10.24V ref is correct.

I did some scoping around to try to trigger on the narrow secondary pulse and captured some traces.

This is a check of the comparator U314 to see if it's operating correctly.  The yellow and blue traces at the bottom are the inputs to the comparator, the purple is its output, and the green up top is the reference input (pin 14 of U306).  Looks normal, I'm not seeing the comparator triggering on noise or artifacts that don't seem to be in there intentionally.  I don't know if I entirely understand the purpose of the feedback path or the small signal it puts on pin 2, but I don't think it's malfunctioning.

So if the comparator isn't it, then why is it trying to generate a faster sloped triangle wave?  I scoped the PLL chip and its output more particularly:

In this one, the top three signals are the same (green is reference input, purple/red is comparator, blue is triangle wave output), but the yellow trace is right on pin 13 of U306, the control line for the programmable current source.  Again, this bit of the circuit is a little beyond my understanding, but I think I should be checking values of C336, R316, R317, and C306.  Don't have the time to immediately, but maybe tonight.
« Last Edit: December 19, 2024, 05:29:25 am by DaJMasta »
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #12 on: December 19, 2024, 06:20:29 am »
Checked those four passives and the 3 1k resistors nearby.  Everything measured good, but the ESR of C336 bounced around a bit at low frequency and then seemed fairly low in comparison to other ceramics I had on hand, so I tried replacing it with an X7R modern ceramic 470pF cap... no real difference.  Maybe slightly more stable, maybe that's just my own noticing biases, but the behavior of the triangle wave is unaltered and there are still intermittent errors.
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #13 on: December 20, 2024, 06:55:29 am »
Probed all the digital control of the analog switches today and the only narrow pulses I saw on any of them were the ones generated by the PLL controller's !Q output, not even notable noise otherwise.  Pulled a bunch more passives and confirmed them, a few will measure odd in circuit but then be totally fine out of circuit.

So I took a more blanket approach and cleaned the bottom of the board with IPA a few times - after the first couple I actually saw less locking behavior, but it was back to normal fully dry and a bit more clean and at the moment it's acting the same again.  Pushed in every IC to the sockets, visually inspected and retouched a few joints on the underside of the board.

I tried removing resistor R329 and replacing it with various values, in case somehow the hysteresis for that main comparator had gone out of whack, but while it seemed slightly better with a slightly lower value (39k was better I think but not 27k), it wasn't fixed with any higher or lower value I tried.

Also tried replacing U306's current CD4046BE with the removed CD4046AF, but to no difference.

I'm definitely grasping at straws now.
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #14 on: December 20, 2024, 12:32:19 pm »
I'm definitely grasping at straws now.
It seems still that you're very close to find the issue.


Indeed, the latest screenshots show very interesting things.

Just quicky,

About screenshot #2
SR = 2 GSa/s
TB = 1 ms/div
The yellow trace shows a significant HF noise oscillation (about 300 µs wide). This needs more investigation.
The green trace (at the top), which (at the left side of the screen) was NOT in phase with the triangle wave (blue,) now goes negative synchronously, perhaps with a small glitch (100 µs).
At the exact moment of that green negative pulse, the rate of change of the triangle wave (blue) becomes much steeper. Here somethings's going on, but I don't know yet if it is a cause or a consequence.
We can also see that  the pink trace makes this time only a very short low pulse (400 µs), the previous was 2,1 µs wide.




About screenshot #3
SR = 2 GSa/s
TB = 2 ms/div
See also my 'updated' version of this screeshot (attached picture)
Yellow trace : we see the positive and negative pulses, from the phase comparator II.
The two yellow positive pulses clearly produce a significant frequency/period change on the triangle wave (blue).
On the left of the screenshot, the triangle wave has a period of 10.2 ms (between 1 green dot and 2 green dots).

After that, it seems that the frequency of triangle wave becomes quite twice the former value. If we interpolate the triangle wave (dashed red line) we keep the former shape and almost the same period (11.2 ms, between 2 green dots and 3 green dots). This small period difference (+ 1 ms) might show up because the loop has been jolted and cannot immediately return to the old period value.

At this moment, the phase comparator (yellow pulses) tries to compensate with two positive pulses, due to the immense phase difference that occurs when the frequency is doubled.

So the question is now : why does the triangle wave significantly change it's rate of change, and why does the whole loop truly double it's frequency at this moment ?
Has it something to do with the logic signals /f,  /2f  and 1/2  U313 (the left one) ?
As I'm not fluent with Norton OPAMPs, it's not easy to guess which capacitor is charged and dischard by a constant current in order to generate the triangle wave. Can you see the triangle wave on C306, C307 or C308 ?
« Last Edit: December 20, 2024, 01:26:13 pm by timeandfrequency »
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #15 on: December 23, 2024, 08:00:43 pm »
I've checked and rechecked the logic signals, they aren't changing.

I started trying a more experimental approach - soldering extra capacitors to existing ones and seeing if the fault changed.  I tried adding various small and larger capacitances to C336, C306, C308, C310, and C312.  While some changes sort of seemed to change the frequency of the fault (and I got the PLL to stop reacting a couple of times), it wasn't by a lot, and the faults were of the same nature.

So I captured the timing of the pulses around the PLL to try and determine if it was the comparator or the PLL that was acting first, and while it seems like it is a reaction to the comparator that causes the PLL to adjust in an unhelpful way (and not the other way around), I captured one event that was clearly bad:


It's noise.  Light blue is the comparator input (pin 3 U306), red and yellow are the PLL controller outputs (1 and 13 U306), and dark blue is the signal input (pin 14 U306) and while this is before the short pulse glitch, it shows the PLL controller being incorrectly triggered by noise on the two input lines.  Zooming in, it's just a jagged mess, so while mostly common mode between the two inputs, clearly at least some of it triggers a switch.

So where was it coming from?  I had suspected noise from outside interference and clapped on the chassis covers before, but it hadn't made a difference.  I did the same again, then left only the sig gen connected to eliminate any potential ground loop issues... no dice.  Still see it with minimal external connections and the behavior doesn't change when I connect up the scope.  I don't have an isolated scope to be able to take that entirely out of the equation, but I'm not really expecting it to be external issues at this point.

I think my next steps are remeasuring the rails specifically looking for high frequency noise that correlates to transitions of the PLL output that have the glitch.  I'll also probably just add a small cap across pins 3 and 14 of U306 (maybe 22pF as a start?) for common mode noise reduction.
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #16 on: December 23, 2024, 10:08:26 pm »
Hello DaJMasta,

Indeed, this 4th snapshot is interesting : we now see the culprit.
Yes you can check the power rails and also replace all of the caps in the power supply assembly.


But I would also suggest to make a test with VERY clean power supplies.
Lift the power supply pins of U306 and U314 (not the ground pin).
For U306  : use one  9V 6F22 battery + LM7805 regulator + 2 x 100 nF connected to the regulator (see its datasheet)
For U314 : use 2 x 6F22  9V batteries (so supply is ± 9VDC)
Keep the wires from the batteries as short as possible : not more than 10 cm, and twist them together.
Do NOT use lab power supplies for this test.
And then check the loop again to see if it still unlocks.

You have never confirmed if you also changes U313.
« Last Edit: December 25, 2024, 07:35:31 am by timeandfrequency »
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #17 on: December 29, 2024, 06:16:40 am »
U313 has been swapped.

I've taken a closer look at the power and grounding and was lead down a rabbit hole by the grounding because the shell of the reference input is directly grounded to the chassis with its own wire and mount, which is not indicated in the manual.  Sure enough, disconnecting that ground and letting it only go through the 2k resistor makes it not work, but it means 0-10 ohms between the reference input BNC shell and the circuit grounds and other BNC shells, which took a lot of continuity checking to finally understand.

As for the power rails, I had checked slower speed dips and temperatures for regulators and caps but I checked the high speed noise by time correlating the narrow pulses that are causing issues with each rail - +15V, -15V, +5V, +7.5V, -7,5V and even the 10.24V reference and there is no evidence that noise is present on any of them except for the 5V as identified before.  That +5V rail lower speed stepped noise is in phase with the glitches, where its lowest value step is immediately preceding the glitch start.  I checked both ends of R330, R336, and R366 to see if the comparator output was dragging down the voltage notably and while all of them look about right, the waveform at the output of U315 (R336) has the lowest high voltage level.  Tracing it around, it goes to the signal input of the quadrature (second stage) PLL (U316) and then off board to U708 with a few input pins - maybe the extra fanout is dragging it down a little... but maybe it's more than it should be, since the HC157 input bias currents should be very low.  I may swap U708 if I can't find something more suspect.  I may also try some lower value bypass caps on these resistors for the +5V rail to see if they will help.

I could also try messing with the capacitor values on the common mode noise reduction and feedback caps on U315, since I was messing with the U314 ones before.

To be thorough, I rechecked every input to the sheet to look for noise or activity correlating to the glitch and there's basically none.  !HI, !LO, !f, !2f, !QUAD, !IN PHASE, Phase Voltage, !TRIG2, TRIG2, !TRIG1, and 488Hz measure as to not having any glitch correlated noise, but I see the slightest bit on !RESET.  Now this slightest bit is a tiny negative leaning glitch (<50mV) right around ground and not actual toggling, and this part of the circuit really doesn't have to do with the problem I'm seeing, but interestingly, pin 2 of U711 measures normal (low output, no glitch), but perhaps if I'm out of ideas I could try swapping U305.

I've checked and rechecked soldering, I've probed enough relevant points to probably have noticed an intermittent break in a trace, and I am once again scraping the bottom of the barrel to try and find an option.
 

Online DaJMastaTopic starter

  • Super Contributor
  • ***
  • Posts: 2389
  • Country: us
    • medpants.com
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #18 on: December 30, 2024, 05:52:58 am »
Did some more investigating and technically made progress but it doesn't work yet.  I checked on the 5V rail because it was showing that periodic noise, which I would sort of expect from the design, but which didn't seem good.  First I checked what it looked like at different frequencies and when lower, it is more pronounced, and approaching 1kHz, it goes away almost entirely (an order of magnitude less or more).  Interestingly, the overall lock behavior had changed too, as it was no longer working at 10kHz while the top end of the frequency range was good and the bottom end was even worse than 100Hz (10Hz basically never was locked according to the indicator light).  So the noise presence didn't really correlate with the fault, but I slapped on a 2200uF capacitor to see what it did to the noise, and yes, it removed it, but no, it didn't fix the problem.

I went investigating the !HI and !LO lines that go off the board wondering if something on the other end was dragging them down, and though it sort of looks like an output port, it's being driven by U717/U718!  What's more, !LO is just an inverted version of !HI and wasn't toggling with it, so I replaced U718, got the toggling back, and nothing changed about the behavior of the lock, best I can tell.

I saw a strange waveform at pin 14 of U327 which really looked wrong, so I replaced U327.  Nothing changed, seems like the output of the LM311 is open collector, so the high impedance of the DG211 means that when the switch is switched off, there's basically no waveform on the output.  I then rotated some of the pulled chips out with the ones in circuit, the DG211s, the CD4046, and the LM13600s all got rotated to see if something would change and it didn't, so I think the ones I have all probably work normally.

I checked the phase voltage line and while stepping by 90 degrees sort of makes an odd sticking behavior, I think it's compensating for this instead of constantly toggling through software by using the quadrature PLL's outputs which are also being switched in at the same time, and using the fine phase adjust makes the voltage sweep as expected.

I haven't tried messing with the hysteresis capacitor on U315, so maybe that's still something, but since I've been able to "solve" the +5V rail noise problem with bulk capacitance and it hasn't made a difference, I don't know what help that would be.  It feels like I need to look off this page to find something, and the third PLL on the vertical board may be the leading candidate since Vvco is part of the feedback loop of the first and connects directly to it.

I say that it's the leading candidate, but I want to reiterate how little confidence I have in that, it just seems like the next most closely related thing to try after literally trying everything else I can think of.  I may be 50-60 hours into this repair at all and it is wearing on me, for sure.
 

Online Kleinstein

  • Super Contributor
  • ***
  • Posts: 15076
  • Country: de
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #19 on: December 30, 2024, 12:57:01 pm »
The LM311 output is open collector - so that is correct.
The 4046 PLLs are known to cause trouble when the input signal at pin 14 is not ideal. This input is not a normal logic input and may react odd when the signal somehow exceeds the supply only a little. On the other side it may handle smaller signals OK. Different manufacturer's 4046 may also behave slightly different in this aspect.
 

Offline fenugrec

  • Regular Contributor
  • *
  • Posts: 239
  • Country: ca
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #20 on: December 30, 2024, 01:39:58 pm »
Earlier you mentioned there was a strange 1V level on U305.14, on the RC input pin and I found that unusual. Have you looked more into this ? An easy test (before replacing U305 + C305) would be to clip, say, 100k in parallel with R309, see if it affects the voltage on that node and the frequency of the glitches.
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Re: Troubleshooting intermittent fault in an SRS SR530 reference oscillator PLL
« Reply #21 on: December 30, 2024, 03:26:45 pm »

I saw a strange waveform at pin 14 of U327 which really looked wrong, so I replaced U327.  Nothing changed, seems like the output of the LM311 is open collector, so the high impedance of the DG211 means that when the switch is switched off, there's basically no waveform on the output.  I then rotated some of the pulled chips out with the ones in circuit, the DG211s, the CD4046, and the LM13600s all got rotated to see if something would change and it didn't, so I think the ones I have all probably work normally.
IMO, the design around U328, U329 & 1/2 U327 is not that nice because of the open collector output of the comparators, as Kleinstein underscored.
SRS designers put the pull-up R366 after the switch and as consequence, the input of the lattter can be Hi-Z. All floating lines catch noise.
I would suggest to add two 22 to 47 k pull-ups (toward 5V) to the pins 7 of U328 & U329.


[ Update #1] U305 pulse duration should be 3.3 sec  (330 k & 10 µF). Pretty long for a reset pulse.
« Last Edit: December 30, 2024, 09:05:10 pm by timeandfrequency »
 

Offline timeandfrequency

  • Regular Contributor
  • *
  • Posts: 237
  • Country: fr
Hello DaJMasta,

Let me first wish you a very happy new year 2025 and success in repairing the SR530.


I haven't tried messing with the hysteresis capacitor on U315, so maybe that's still something, but since I've been able to "solve" the +5V rail noise problem with bulk capacitance and it hasn't made a difference, I don't know what help that would be.  It feels like I need to look off this page to find something, and the third PLL on the vertical board may be the leading candidate since Vvco is part of the feedback loop of the first and connects directly to it.

I say that it's the leading candidate, but I want to reiterate how little confidence I have in that, it just seems like the next most closely related thing to try after literally trying everything else I can think of.  I may be 50-60 hours into this repair at all and it is wearing on me, for sure.

As the noise seems now gone on the +5 VDC rail, are there still other signals/location where you can find a noise burst at the moment the loop unlocks ?

The VVco signal is pretty interesting : it is generated by the output of U308 pin 6.
From there, it is fed towards multiple Norton OPAMPS, most of the time via 1% resistors.
1/2 U307 pin 16  via R314 300 k 5% or  R315 680 k 5%  (selection switch is 1/2 U311)
2/2 U307 pin 1 via R322  249 k 1%
1/2 U309 pin 16 via R318 10 k 1%
2/2 U309 pin 1 via  2/2 U307 and then R323 6.8 k 1% ; there's also R324 6.8 k 1%  towards GND (*)
1/2 U1010 pin 1 via R1013 750 k 1%

If any of the sub-assembly connected to VVco alters it's value (back injection), the PLL around U306, U307, U308, U309, U310, U314, 1/2 U313 and 1/2 U312, goes bersek for sure.


(*) The reference VCO around 1/2 U309 + 2/2 U309 is directly inspired by the NS reference design at 'FIGURE 15. Triangular/Square-Wave VCO'.
The formula to calculate the output frequency is provided. But there's something weird in the SR530 schematic : if 3/4 U312 switch is open, Pin 1 of 2/2 U309 is set to GND via R323 (6.8 k) & R324 (6.8 k). So I assume that IA = 0 and Fosc tends to infinite (!?).


The fosc formula appplied to U309 also indicates that, if the rate of change of the triangle wave changes (as we see on the snapshots when the PLL unlocks), this means that C has changed (C307 & C308 connected via switch U310) or IA or IC has changed.
Hence, it would be interesting to measure the current injected into Pin 1 of 2/2 U309 (corresponds to IA) by monitoring the voltage across R323 (6.8 k) (**), at the precise moment when the PLL unlocks.


(**) To measure the current, a differential/floating mesurement is compulsory. Use 2 channels/probes of your scope and set a math trace that calculates ([CHA]-[CH B])/6800   so 1 V vertical division of the math trace should be equal to IA = 1 mA.
The A-B method is safe in this case because the whole board is powered by low value isolated DC voltages. It cannot be used on live/mains circuits.


#Update #1 : typo
« Last Edit: Yesterday at 09:39:26 am by timeandfrequency »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf