So I have a mint-condition Agilent 3458A ...

[bdunham7]

First, 91 ppm on a 10V signal is a ~0.91 mV error. This does not sound remotely reasonable, whether "drift" or not.

It is indeed a ludicrous amount of drift or whatever you call the error.  I'm still hoping for some ideas from the more experienced people that have been down this road before arriving at any conclusions.

Next, do the "10V pre-ACAL" measurements represent accumulated error over multiple ACAL's, or are they "reset" to a degree after each ACAL? If it's "reset", as I have been presuming, then the measurements above appear to show that the so-called "drift" is accelerating, which doesn't sound like other people's experiences with "drift."

Yes, if I've shown a pre-ACAL measurement it would represent only the drift since the last ACAL.  I do have some more data that I'll post soon, but I've turned the meter off so as to try a cold start and see how it behaves.  Yes, the drift appears to be rapidly accelerating.

Then, I can't find a reference for what "cal72" specifically represents. Can someone clear that up for me?

CAL72 is the active 10VDC calibration constant.  CAL 70-74 are the in-use DC calibration constants for the 100mV, 1V, 10V, 100V and 1kV ranges respectively.  A cal constant of 0.985xxxx means that if you lost all your cal data the meter would be off by about 1.5%, which I believe is about what you observed.  These CAL70-74 constants are the ones actually used when displaying the readings and are updated by the ACAL process, the ones set during the artifact calibration are elsewhere and do not update during ACAL.

Also, yeah, using diode drops to go from 18 to 15V is surprisingly cheesy.

This was a very common design technique from that era for devices that needed a lot of different supply voltages at fairly low power.  Using a 3V zener is a bit unusual though, but if the output isn't critical I suppose it works.

Responding to thoughts by coppercone2: I have nearly 40 years of experience with IC design and packaging. I would not expect "drift" to be due to bond wires failing. The "strain" idea is more interesting and plausible, but U180 is in a robust ceramic package, and so again I would not expect that to be an issue. Electromigration (I think that's what you meant) - distortions or open-circuits in metal traces caused by high current vs. inadequate design - is at least a remote possibility, but in this case it's implausible since it's barely been used.

Anything is possible, but since it's a virtual-virgin of an instrument, I'd have a hard time believing that any IC would be failing. That would include U180, except I don't know what's on or in it. Has anyone de-lidded one and taken photos? Does it contain more than a single IC die?

Illya Tsemenko (user TiN) of xDevs.com is no longer on this forum but you might want to contact him.  He's put a lot of effort into this issue and additional thoughts from someone with your experience might be helpful.  I've no idea about the specific issue here, but custom hybrid ICs have a long and storied history of known failure patterns on high-end test gear. 

I did one more test before I shut the meter off last night and that was one suggested originally by TiN, using a shorter aperture.  I set NPLC = 0.005 and observed that the accumulated error decreased.  I'm not sure exactly why--something to do with the slope drivers--but apparently that is confirmation that the problem is actually U180. 
[/quote]

[Zondar]

Thanks for the link. That page only shows the IC die and not the resistor array. Although I can't be sure of the technology being used, it does not look looks like CMOS.

It does use gold bond-wires. Aside from the color, you can tell because there's a melted "ball" attachment to the bond pads rather than a wedge. That all looks fine.

You can also see marks on the bond pads from a test-phase prior to packaging and bonding.

Interestingly, it uses aluminum bond wires to connect between the IC and the resistor array rather than gold. I wonder if that was to suppress bi-metal contact issues vs. aluminum wires on aluminum pads, or if it was simply a matter of convenience.

[bdunham7]

Updated data log:

DATE       TIME      10V preACAL    10VpostACAL      CAL72postACAL      TEMP    AMBIENT             NOTE

7/29        2PM       --------------     --------------       .985426363           ?           ? (~27C)           Drifted +6ppm during the time it took to do the AC SCAL procedure!  ACAL done again.
7/30        8AM       10.0000201      10.0000003       .985424165           41.0C     ?                      Had drifted up to nearly +5ppm after ACAL the night before, but drifted back down overnight.
7/30        2PM       10.0000558      10.0000023       .985418895           41.5C     26.2C               
7/30        8PM       10.0000988      10.0000030       .985407328           41.8C     27.8C               Was noticeably drifting up after ACAL.  It seem to gain 2ppm or so right after ACAL, within 10-20 minutes.
7/31        8AM       10.0001000        9.9999985       .985394132           38.9C     25.0C     
7/31        2PM       ---in use----      10.0000051       .985310891           40.3C     26.5C                DMM was used in ratio mode for some other work w/ multiple ACALs.  84ppm drift in 6 hours!
7/31        9PM       10.0009787      10.0000088       .985220166            42.1C    27.0C                Over 91ppm accumulated drift since last ACAL (7 hours).  Note that there is now some drift even before ACAL is done.

There seems to be about 226ppm total drift since the start of the log.  The change in the displayed output is noticeable even in less than 1 minute and since the ACAL process does other ranges after 10V, there is now almost 1ppm of drift just waiting for the rest of the ACAL process to finish. 

[voltsandjolts]

Some more details on U180 here;
https://www.eevblog.com/forum/metrology/hp-3458a-ad-board-spare-part-availibilty-and-cost/msg3682681/#msg3682681

[bdunham7]

Thanks for the link. That page only shows the IC die and not the resistor array. Although I can't be sure of the technology being used, it does not look like CMOS. Otherwise, it looks fairly normal.

Here are links to the xDevs page.  You can scroll down to the repair A/D drift sections and one of those has some additional pictures.  I'll post links to any other info I find, I think there is a better photo somewhere of the entire assembly but I can't find it right now.

https://xdevs.com/fix/hp3458a/

https://xdevs.com/fix/hp3458a/#a3drift

[Zondar]

I'm not sure exactly why--something to do with the slope drivers--but apparently that is confirmation that the problem is actually U180. 

I'm not ready to accept anything as "confirmed." 

[iMo]

Doublecheck all +/-15V voltages. When a dropper zener fails it gets usually shorted and you will get 18.5V there and that may cause further damages..

[Zondar]

Doublecheck all +/-15V voltages. When a dropper zener fails it gets usually shorted and you will get 18.5V there and that may cause further damages..

I'll open it up and look at everything I can think of when Brian is finished with his experiments. I may need a stable reference, though.

Ironically, if the "drift" continues to increase, it may make things easier to diagnose or confirm.

[bdunham7]

I'll open it up and look at everything I can think of when Brian is finished with his experiments. I may need a stable reference, though.

Ironically, if the "drift" continues to increase, it may make things easier to diagnose or confirm.

The schematic I posted is the original design and there have been multiple updates so those zeners might not even be there.  The xDevs page has extensive information on measuring the 3 reference voltages and observing them drift. Your 34465A will be sufficient for this, especially at the rates of drift we're seeing.  At much lower rates of drift--say 0.5ppm/day--you might need a month.  Given that the rate of drift is so extreme, I still think there's an outside chance that something else has gone awry.  If anyone has seen these rates let us know.

I'm pretty much done.  The cold start simply revealed that it drifts down during the warmup and then starts going back up again once stabilized.  So for a period during warmup the effects counteract each other.  So it's time to pop the lid off....

[iMo]

..I would not expect "drift" to be due to bond wires failing. The "strain" idea is more interesting and plausible, but U180 is in a robust ceramic package, and so again I would not expect that to be an issue. Electromigration (I think that's what you meant) - distortions or open-circuits in metal traces caused by high current vs. inadequate design - is at least a remote possibility, but in this case it's implausible since it's barely been used.

The ceramic packages with that metal lid may get an air leakage. When not tested after manufacturing on that leakage it may easily happen the chip like U180 will slowly "die" (corrosion or surface contamination)..
I saw somewhere 2 tests - they put the ceramic packages into pressure chamber (gas) and then looked at bubbles when in liquid, or, they put the chips into liquid under pressure and weighted them afterwards, heavy chips were trashed..

[coppercone2]

I need to understand a few things:

First, 91 ppm on a 10V signal is a ~0.91 mV error. This does not sound remotely reasonable, whether "drift" or not.

Next, do the "10V pre-ACAL" measurements represent accumulated error over multiple ACAL's, or are they "reset" to a degree after each ACAL? If it's "reset", as I have been presuming, then the measurements above appear to show that the so-called "drift" is accelerating, which doesn't sound like other people's experiences with "drift."

Then, I can't find a reference for what "cal72" specifically represents. Can someone clear that up for me?

Also, yeah, using diode drops to go from 18 to 15V is surprisingly cheesy.

Responding to thoughts by coppercone2: I have nearly 40 years of experience with IC design and packaging. I would not expect "drift" to be due to bond wires failing. The "strain" idea is more interesting and plausible, but U180 is in a robust ceramic package, and so again I would not expect that to be an issue. Electromigration (I think that's what you meant) - distortions or open-circuits in metal traces caused by high current vs. inadequate design - is at least a remote possibility, but in this case it's implausible since it's barely been used.

Anything is possible, but since it's a virtual-virgin of an instrument, I'd have a hard time believing that any IC would be failing.

Edit: I found a die photo of U180. As many here would already know, it is a hybrid containing an IC and a separate resistor array. I struggle to think how this part would result in such an error solely due to age (in this case).

Twenty year-old electrolytics could easily be a problem, but assuming that they are the polarized capacitors in that diagram (not sure), none seem positioned to be a target of suspicion unless they were grossly failing. I also wouldn't expect resistors or ceramic capacitors to be a problem.

Any counter-arguments? What is left?

That is what I think too. I just read about IC's for a long time but I never worked with design or manufacturing.

I keep thinking of Au20/Sn80 solder vs Au80/Sn20 solder. The price difference is an order of magnitude. Both are for die attach. But if you read about it you hear of voiding. To me if there is voiding and crack propagation under it, then the crack depending on how it forms can laterally thermally isolate certain sections of the circuits that allow for islands of heat to form, heat that would normally be sinked strait down into the substrate. I think this could make strain on resistors and such and cause offset and gain drifts that are erratic.



What about the size of the die compared to other dies? I know bonding problems occur when dies get bigger. Same for gluing parts on. Large ceramic parts (i.e. heater) are known to get cranky when they get glued, so normally people like to use paste which decouples them mechanically.

[Zondar]

The ceramic packages with that metal lid may get an air leakage.

Chips are packaged in air anyway, and I seriously doubt this one is any different, so "leakage" shouldn't be a cause.

[coppercone2]

I thought they are at least packaged in clean dry air?

Could it be moisture ? But at most that would corrode the bond pads I think.

But 1ppm to me... seems in line with what you would expect from strain gauge effect? Its small but not that small. I know its dependent on the geometry of the traces



One weird idea is that maybe there is a wet tantalum or electrolytic nearby that happens to have a seal fail around the same time as a u180 seal and then the u180 absorbs some of the chemical emissions from the part. But I don't know what that would do other then destroy a bond wire, which should be very easy to find by anyone actually doing a failure analysis. I imagine standard procedure would be to re-bond it?

[coppercone2]

If someone actually understands the IC, is there like a region of circuits that dissipates significantly more power then the rest? I wonder if you can use a tiny ultrasonic probe to probe on top of a known 'hot spot' of the IC to see if the die attach failed under that point or has a discontinuity at least. My whole idea hinges on the idea that the die itself might not be a great 'heat spreader' and requires direct contact on 100% of its surface area to function normaly.


Or maybe its possible to de-lid the part and look at the top with a thermal microscope? I managed to get about 6x magnification on my TI by building a lens adapter and a stand for it that lets it work as a thermal microscope with fine adjust. I don't know how much zoom you can get from it. Maybe enough with a few lens to see if something is wrong or suspicious at least?


And another test might be to use a dial indicator to see if its attached crooked, by sweeping the chip interior and sweeping the die.


I hate IC problems, to me its like having a malfunctioning toilet (fluidic ASIC). It feels so god damn hopeless. But there is ALWAYS the chance its just the 'wax ring' and that its possible to fix without any damaged ceramic structures!!! often you can just lift it up and put a new ring.


I still have dreams about possibly making some kind of basic wire bonder that uses micrometers and those bendy hinges that sits on a old microscope. I mean a small cheap slow one, not a giant apparatus...


I wonder how low tech it can be, if you are willing to make it so you need to do 30 minutes of fine adjustments per bond wire. I always suspected there is a big cost cutting that can be done on these machines that would make their usability go down enough that it would be way too ridiculous for commercial customers to use it, but it would be possible for a hobbyist to use it since they are willing to spend the entire sunday to rebond 3 pads. We can tolerate more stress then commercial users (once the six packs come out). I.e. is it possible to make some kind of single use 'cartridge' that is manually assembled with a bit of wire rather then developing a whole feeder mechanism (black hole). Like if you can do 1 pad, then take the head apart to insert another short strand of gold, instead of trying to make it so its 'belt fed'. Like a bolt action instead of a autocannon. Along with other... design changes. 

[Kleinstein]

U180 has 2 parts. A resistor array with the resistors heavily interleaved. So even if 1 resistor is getting hot, the power is still spread out over quite some area. The power should not be that much: like some 10 K and 12 V as one of the highest power ones. This would be some 15 mW and maybe some 50 mW overall.

The other part is JFET switching with a few flip flops and gate drivers. Here it is not clear if this is more CMOS or more TTL like gates. Chances are the JFETs itself have very little power and the main power would be the synchronization flip-flops. These would likely be fast and may thus be a bit power hungry.
Silicon is a good thermal conductor and thus no so much localized high temperature or dependence on good thermal contact everywhere. The resistor substraight may not be such a good thermal conductor, but should still not get locally that hot. A loose die attachment could still cause a change in the mechnical stress.

While the drift of the ADC is in most cases a problem with U180, there is a small chance that other issues (like the supply) can also cause drift. So it would be worth checking the supplies.
If one has a good meter, one could also check the +-12 V ref. voltages - at least in some cases with a bad U180 the drift already effected these voltages. So an issue in the resistor part (could still be more trouble too).

[Zondar]

That is what I think too. I just read about IC's for a long time but I never worked with design or manufacturing.

I keep thinking of Au20/Sn80 solder vs Au80/Sn20 solder. The price difference is an order of magnitude. Both are for die attach. But if you read about it you hear of voiding. To me if there is voiding and crack propagation under it, then the crack depending on how it forms can laterally thermally isolate certain sections of the circuits that allow for islands of heat to form, heat that would normally be sinked strait down into the substrate. I think this could make strain on resistors and such and cause offset and gain drifts that are erratic.

What about the size of the die compared to other dies?

One of the problems in explaining a change from in-spec to "drifty" is the fact that it's apparently a permanent change that creeps up over time (I haven't heard anyone complaining about a sudden change). This mostly rules out things like transient temperature-related changes.

Another big question is whether drifting is inevitable - does it eventually happen to all of them, and the non-drifty ones are just late to the party? I don't think the answer is clear, is it?

HP/Agilent would not be scrimping on materials for such an exotic and no-doubt very expensive chip. It's hard to say if solder or conductive epoxy was used in this case from what little I've seen.

I discount the crack idea because that would probably cause a discontinuous change or a failure of the entire chip. Voids can and do occur when using high-volume, high-speed packaging, but probably not with the care that HP would have put into these parts.

Strain on the resistor network is plausible. It would change the resistor values slightly, but the question is by how much, and how much a change would affect performance. For example, matched resistors on the same die and under the same strain would likely stay well matched.

Also, such strain would have to be new and permanent for the performance to change from normal spec to "drifty." Temperature changes would be the obvious source of a change in strain, e.g. by mismatched TC's, and that would be seen on its own if it's large enough - but I don't see how it would cause permanent changes. What else is could?

Another possibility is poisoning of the resistor network by outgassing from the attachment material (e.g. solder) or the lid-attach material. However, I presume that the resistor network includes a good protective passivation layer. That might not be good enough over lengthly time, of course, but I mostly doubt this one too.

The size of the die could be estimated by measuring the width of the package cavity vs. the die. I haven't done that, but it's pretty easy. Larger dies are more susceptible to issues than small ones, but I don't see explanatory power here since they are all the same size anyway.

The fact that this machine has barely been used might offer a clue.

[coppercone2]

Won't it be gradual if there is a crack  propagating between tiny bubbles ? And the longer it gets the worse it gets.

I am imagining like tearing foam slowly

and I thought all the solder used for this application is either 80 20 or 20 80 and has no flux (vacuum bonded)

I also wonder if there could be wiskers in high tin die bond solder. Could a tin wisker grow in a void and 'jack up' the die ?

Wiskers are known to grow basically by themselves without much external stimulus


I have not yet seen a flux that is for ceramics or carbon. All the solder used is either vacuum melt or ultrasonic. Flux actually prevents it from working. They do a gold coating on things before gold soldering, but its also done in a vacuum (as far as I can find out)


You could have residual.... acid? from the polishing step I guess (if they do chemical mechanical die polishing), but you would think all the bake outs they do would take care of that.



One thing I did notice is that some people use a strong chemical treatment for ceramics before soldering or plating, I think a boil in phosphoric/sulfuric acid to 'etch' alumina slightly. Like 36+ hours. But its hard to imagine any chemical being the problem, the processes should be similar for all the HP ASIC chips they have
https://www.sigmaaldrich.com/US/en/product/aldrich/667447


There would have to be some nuance to the chip that makes it get a different treatment. I think the bulk 'processing' steps must all be pretty much the same for all of their silicon ASIC devices.*

*I have seen manufacturing problems that relate to objects of unusual size going through the same process, for instance if you have something that is too large to fit in the vessel you like to use, and there is some alternate thing made up for it that is of low quality. Kind of like how for most of your DIY de-rusting your use a bucket, but then you come across some object of irregular size and it ends up being laid out in a garbage bag pool inside of a lamp cover! Or something deep getting incorrect torque because it requires an adapter no one bothered to calibrate. That is why I am so interested if there is anything special about that chip that might make it go through a alternative path to other ASICS that HP uses, which seem to be generally OK (maybe aside from the analog MUXs that seem to break alot). Another one is changing ovens, there is always chaos when an alternative oven is used for something. We don't actually know how ovens work. There is theory about temperature and airflow and whatnot but I see its about as useful as tea-leaves when the curing oven is changed for a well tuned manufacturing process.

[Zondar]

Responding to a few other comments:

I had presumed that the "digital" portion contains more than just digital logic. Otherwise, I'm curious why JFET technology was used when CMOS would have been far cheaper and easier. The supply voltage looks like it's 5V, too.

There are what looks like capacitors on the die, but I don't think they are active components. And then there are what looks like some mix of strong amplifiers or buffers and possibly switches. Anyway, I'm sure they had their performance-related reasons for JFETs.

The full-resolution die photo is excellent. It's good enough that some simpler areas can easily be reverse-engineered. However, the densest sections look to be very difficult and probably too much so. Still, has anyone attempted this?

If it's purely digital logic, then the logic style wouldn't be terribly important. The final amps or buffers could be an exception.

I'd expect the resistor substrate to be silicon too.

Interleaving the resistor array (this is common when precision matching is needed) reduces the probability of things like temperature or strain causing problems.

Yes, of course U180 is the most likely source of these problems, but I'm curious: has anyone desoldered one and replaced just the chip? Or has a chip ever been studied in isolation? Related, I note that a few of the resistors could possibly be measured in in-situ if certain jumpers or connectors are opened. I doubt it would reveal much, though.

I'm slightly warming up to some sort of poisoning idea. Hmm.

[coppercone2]

This photo right? 100mb
https://xdevs.com/doc/HP_Agilent_Keysight/3458X/u180/result_1.jpg

[bdunham7]

The full-resolution die photo is excellent. It's good enough that some simpler areas can easily be reverse-engineered. However, the densest sections look to be very difficult and probably too much so. Still, has anyone attempted this?

Yes, of course U180 is the most likely source of these problems, but I'm curious: has anyone desoldered one and replaced just the chip? Or has a chip ever been studied in isolation? Related, I note that a few of the resistors could possibly be measured in in-situ if certain jumpers or connectors are opened. I doubt it would reveal much, though.

I'm slightly warming up to some sort of poisoning idea. Hmm

Yes, people have started down this path but AFAIK nobody has gotten terribly far.

Illya (xDevs) has desoldered and transplanted U180 chips for testing, but only from scrap and used sources.  A new, genuine U180 is pure unobtanium for any of us mere mortals.  At one point Illya did make a cobbled-up network of external resistors and said that 0.2ppm/hour was the best he could get as far as drift goes.  He gets an A for effort, but at the end of the day he could only replace the chip or board as was available.

Maybe they had a production issue that they just don't want to discuss and they also ended up with a huge stockpile of them to forestall obsolescence. 

[Zondar]

Maybe they had a production issue that they just don't want to discuss and they also ended up with a huge stockpile of them to forestall obsolescence.

I can't comment about today, but I think integrity was important during the golden old HP days. So I wouldn't believe they would continue to sell something that they know to be actually defective.

That raises the question of whether they, then or now, ever considered it a significant defect. Hard to say. Maybe they do think a certain level of planned obsolescence is acceptable? More likely they just expect to be servicing the units on a yearly basis, and it, along with any other problems, will be caught and fixed if and when they happen. They knew the NVRAM batteries absolutely would fail, and they didn't consider that a defect, right? Plus, no-one buying these new and expecting to have it serviced regularly is terribly worried about money.

 

[coppercone2]

What if you made a distributed project where a bunch of volunteers take a grid section of the die and identify the parts in it. It could be a fun thing where you get together volunteers and have say a 3 month period for everyone to get their analysis together and stitch it into a picture.

I think the problem is burn out . So everyone that can read it takes 2 square inches and translates it


Or a program that randomly gives a web user part of the die on request that is unsolved.


Kind of how you do archeological digs, you make a grid map and assign people to dig at different 'squares'.


I feel like it could work in a Wikipedia type thing, just assign carthesian coordinates to each 'page' and the picture could be 'interactive' much like the picture of the table of elements they had up there somewhere where you can click a box.

Once things are identified like 'resistor' 'amplifier' then people less skilled in semiconductor level engineering can analyze it with a normal EE method


It's too big and boring for one person. That is like eating a crate of matza

[Zondar]

What if you made a distributed project where a bunch of volunteers take a grid section of the die and identify the parts in it. It could be a fun thing where you get together volunteers and have say a 3 month period for everyone to get their analysis together and stitch it into a picture.

The problem is that the dense areas are so crowded that it's difficult to make out what's connected to what with any certainty. This is compounded by the fact that the wavelength of light imposes a limit on what can be seen, in this case making the jumble a little blurry, too.

[coppercone2]

Is that photo at the limit of what you can do optically? I don't have a feel for such things.


Anyway if it was all explained on a big circuit diagram it would at least limit the amount of unknowns


maybe if it was narrowed down to a few unknown patches it could get studied with some kind of better equipment by someone capable.

Unless the whole thing is like that.


Its like one of the most powerful DMM and its being held up by some little flaw no one understands. kind of ridiculous considering how important people consider it to be for electronics

[Zondar]

Is that photo at the limit of what you can do optically? I don't have a feel for such things.

Yes, it's pretty much at the limit.

It's possible to etch away the insulating layers, leaving the metal traces and contacts suspended in air, and then preferably use an electron microscope to photograph it. Electrons have a far smaller wavelength than visible light and hence can image much finer features. But even then, you won't be able to see beneath any metal with an EM, and it looks like there are broad swathes of second-level metal covering portions of the circuitry.