Is "integrated circuit burn-in" a thing?

[IDEngineer]

I've got a weird situation going on here.

Several of my ongoing projects involve MEMS sensors. I'm qualifying a new (to me) one from ST, the LSM6DS3, which is very popular and very deeply stocked at multiple large distributors so they move a lot of volume. In other words, it's not some obscure part that nobody's heard of.

When I powered up the first two proto PCB's with this part, the LSM kept exhibiting these failure modes. It would take anywhere from 30 minutes to 2 hours, but within 2 hours you could pretty much guarantee it would fail. By "fail" I mean the part would change operational modes with no external stimulus. For example, if the part was configured (via SPI) for an update rate of 400Hz, after a while it would switch to (say) 12.5Hz. You could see this on a scope. The even weirder aspect was that reading the configuration registers back out showed the PROPER VALUES for 400Hz (or any other value, I tried several) were still in the registers. And yes, I tried rewriting those proper values back again on the chance that this would cause the chip to re-evaluate them and go back to proper operation - but no luck, the chip stayed in its self-configured mode. The only way to restore proper operation was to invoke its "soft reset" SPI command or power cycle the device.

There was another error condition where the SPI interface itself became unresponsive and the ONLY way to regain control of the device was to cycle power. Ugh.

I wrote a bunch of failsafe routines to detect and gracefully handle everything but the SPI failure, and over a few days got it to the point where no more failures occurred (at least in the sense that if they did occur, my firmware caught them and dynamically restored proper operation). Then, separately, I discovered that the SPI problem appears to be that the internal digital low pass filters require too much processing power. This led me to suspect that the internal filters are implemented in firmware, not hardware, and when they hit some edge case they take out the internal CPU. By disabling all possible internal filtering and going back to my own filters in my own firmware, the SPI errors also disappeared. At this point two prototypes ran for a couple of weeks with no errors at all.

(I'd like to add that every MEMS device I've worked with has weirdnesses like this. They're finicky, and their poor documentation doesn't help. I don't know why they all seem to be like this, but they're simply a PITA to get running. I feel like I could write an aftermarket cheat sheet on the ones I've sleuthed through. However, they're darned useful so I just budget extra time whenever I start working with a new one, knowing I'll be basically writing my own appnote to get them going.)

Yesterday, I fired up a fresh proto PCB and left it running overnight. By early this morning, it had entered one of the failure modes! Since my firmware can handle anything but the SPI lockup, all evidence points to the LSM6DS3's internal firmware having locked up again. I didn't have the unit instrumented (I thought we were past this) so I couldn't dig any deeper beyond knowing that my recovery code couldn't handle it, and it handles everything I know about except the SPI failure. Cycling power instantly fixed it, exactly as with the first two units, which further supports the SPI failure theory. Since then it's been running flawlessly for about 8 hours so far today.

Thinking about differences between this latest board and the earlier ones, the only one is clock time. They're all running the same firmware, powered by the same supply, flashed by the same programmer, etc. Those first two boards got dozens of hours on them while I was working through the error handling, and their error rate dropped as I discovered more failure modes and wrote firmware to handle them. But maybe those parts were just "burning in" and my code had little to do with it. This latest one had never been out of its antistatic bag before I flashed it and let it run overnight, and within hours it failed such that commanding a reset over SPI would not work (the classic symptom of SPI failure on this part). I've never considered "burn-in" of integrated circuits to be a "thing" before, but this sure feels like it. And these are MEMS devices, which brings in a mechanical aspect that most IC's don't have.

Is there anything in the industry about IC burn-in? Perhaps specific to MEMS devices in particular?

[AndyC_772]

ST's MEMS accelerometers can be very fragile and easily damaged by soldering heat. Were the boards soldered by hand? Have you tried buying a preassembled evaluation board and wiring across to it instead of fitting the bare device to your PCB?

[IDEngineer]

Thanks for replying!

ST's MEMS accelerometers can be very fragile and easily damaged by soldering heat.

How would that situation self-correct with time, as appears to be happening here?

Were the boards soldered by hand?

No, they were assembled by a contract manufacturing house that we use all the time. Very high end shop, leading edge pick-and-place and reflow ovens, etc. They do all of our protos and most of our production. I'm confident they followed ST's guidelines from the spec sheet. But again, even if they didn't, anything "damaged by soldering heat" doesn't usually spontaneously repair itself. These parts seem to stop self-reconfiguring after a while, and the two protos that were originally having problems now work perfectly.

[Whales]

I'd try and contact ST.

Even if the parts are very popular, there could be bad batches or other shenanigans. 

[IDEngineer]

I'd try and contact ST. Even if the parts are very popular, there could be bad batches or other shenanigans.

I did that, first thing (like two weeks ago). ST has stopped having formal trouble tickets and now directs you to their online forums. I have two lengthy threads there, complete with scope screenshots where applicable, but the responses I've gotten have nothing to do with my questions!

Example: Here's a paragraph excerpted from one of my messages on the ST forum:

...The bug appears to be related to LPF2, the digital filter block after the A/D. If you look at Figure 5, page 32 of the LSM6DS3's spec sheet, the "Digital LP Filter" and the "Composite Filter" are configurable filters that post-process the output of the A/D converter. Extensive testing suggests that if those digital filters are completely disabled, the LSM6DS3 becomes reliable. But enabling those filters - with any of a variety of configuration options - risks the LSM6DS3 silently switching its ODR (best case) or simply locking up (worst case)....

...and here's the response:

HI IDEengineer, thank you for your effort and the related detailed description of the activity. Sorry for our delay in the answers, I would double check the LPF2 filter behavior, but consider that all the filters introduces some delays and you should have to wait some ms to get a stable value each reading. Regards

I talk about disabling filters to improve reliability, and they answer that digital filters have delays. Non-sequitor!

Here's what I wrote back, in part:

...I understand your point, that filters introduce delay, but that is not what is happening. There is a bug in the LSM6DS3, and our research suggests it is related to the workload caused by enabling LPF2 and its related filter chain. Since we disabled LPF2 and its filter chain, we have not been able to reproduce the bug. In our opinion, the LSM6DS3 should be treated as a bare IMU as if it did not contain LPF2 nor anything past it in the signal chain. LPF2 and all further filters should be forcefully disabled at startup if reliable operation is desired.

...and there's been no response since. So it appears ST themselves are not going to be a source of useful assistance.

[SilverSolder]

Any glitches /ground bounce / etc. etc. that could cause the comms or power to be marginal in any way?

Is there a demo board available from the manufacturer just to help prove that the chip works at all?

[IDEngineer]

I worried about noise too, so one of the first things I checked was the signals and supply rails with one of those little homemade "wrap the wire around the probe ground" tricks. Incredibly clean, both signals and power, right at the chip.

I don't know if there's a demo board, but remember: The devices that weren't reliable at first are now 100% reliable for days at a time. I don't know how a demo board will affect that.    It really is like the chips need to burn in for a while before they get reliable.

Update: I just confirmed that the overnight error was due to the SPI failure mode. I have diagnostic LED's on the board that flash in certain patterns so I get a limited amount of debug data when an error occurs. The farther along in the process the code gets, the greater the number of flashes. Crude but effective. I was getting a single flash, which is the MCU telling me that it detected an LSM6DS3 error but, while trying to resuscitate the chip, it is now not getting a valid WHO_AM_I value back from the chip. That's its first test to confirm that the SPI interface is working... and once the SPI fails, there's no recovery except power cycling. (Speaking of power cycling, did you know that there was a MEMS device a while back where cycling power was the RECOMMENDED SOLUTION to an initialization error? I couldn't believe it, but there it was right in the spec sheet. I guess they imagined you'd include a high side PFET and have the MCU selectively cycle power to the device until it initialized properly.  )

[IDEngineer]

More: One of the reasons I selected the ST LSM6DS3 is because it is footprint and pin compatible with the Bosch-Sensortec BMI160. They are not firmware compatible at all, but I've already written the routines for each (and reading back the WHO_AM_I/CHIP_ID values allows me to dynamically discern which chip is installed on a given board). I had the shop build a few units stuffed with the BMI160 too, and so far those have not demonstrated any of these errors. I'm going to start a long-term test on a couple of BMI160 boards to see if anything shows up, but so far this appears to be an ST-specific bug.

Why did I worry about second sources? Because on earlier MEMS projects we've run into supply chain problems where a sole-source device went out of stock everywhere with 26 week lead times from the factory. THAT was really fun, so I worked long and hard to find two footprint compatible devices for this project. I can tell ya, there's zero coordination between manufacturers and I was darned lucky to find these two parts to be physically compatible - I can accommodate differences at the SPI/firmware level. (And no, there's no room for a second footprint on the PCB.)

[SilverSolder]

Maybe the high side PFET for surgical power cycling is not such a bad idea?  -  How much power does the MEMS chip use, maybe it can live off the current from an I/O pin on the MCU, so no PFET is needed?

[IDEngineer]

Maybe the high side PFET for surgical power cycling is not such a bad idea?  -  How much power does the MEMS chip use, maybe it can live off the current from an I/O pin on the MCU, so no PFET is needed?

I actually did consider that during schematic design when I was using that part a couple of years ago. The MCU runs on 5V0 while all MEMS chips run on something lower... 3V3, 3V0, 2V5, 1V8, etc. So then there'd have to be a regulator in series, which further increases current consumption. As I recall it was still technically possible but I got a little queasy thinking about connecting the input and output caps that regulators need directly to an I/O pin (with the resultant current spikes). Even if I used a regulator that didn't require an input cap, you're still putting the ESR of the I/O pin in series with your supply with who knows what noise effects and transient response (I never actually wired it up). I got deep enough into it that I convinced myself it was a bad idea.

Leaving the MEMS supply separate and selectively applying power via a FET that is controlled by the MCU is definitely doable. Given what I'm finding now, I may add that to a future rev of a MEMS board. It just seems so... second-rate. Maybe the best answer is to avoid parts that can only be recovered with power cycling, but as I seem to be learning, they don't always publish that fact in the spec sheet! Some seem honest enough to say so, others are just silent about it.

These chips are used in millions of cellphones and tablets. I find it hard to believe they're being power-cycled all the time. The chips have very aggressive power-saving modes that are openly and specifically intended for phones and tablets, to save power. Heck, this LSM6DS3 brags about "Compliant with Android K and L" as the fourth bullet item on the front page of its spec sheet. There's no mystery what market they're targeting. Do I think they added an FET to each cellphone to cycle power on these devices? No, I don't. Their internal power saving modes can drop their consumption to almost nothing... a quick glance shows the LSM goes to 6uA in "power down" mode.

One wild, hare-brained theory I have is that the LSM fails if motionless for too long. I've never had one fail while I was actually working on it, only when they've been sitting for extended periods on the bench with nothing happening to it. I might be just a couple feet away cranking on firmware, but there it sits, powered on, sometimes for hours at a time, then I turn to it and voila - dead. And certainly overnight the DUT's aren't moving. Maybe there's some sort of power-saving mode that detects extended periods of motionlessness and seeks to save power, but then doesn't come back out of it. I've researched all the various power modes and done everything I can to switch it into "normal" or "high performance" modes with no power savings at all, but maybe the device's internal firmware has some sort of "we're smarter than you" feature. Hard to test for that, due to the unpredictable timing of these errors.

[thermistor-guy]

...

Several of my ongoing projects involve MEMS sensors. I'm qualifying a new (to me) one from ST, the LSM6DS3, which is very popular and very deeply stocked at multiple large distributors so they move a lot of volume. In other words, it's not some obscure part that nobody's heard of.
...

Is there anything in the industry about IC burn-in? Perhaps specific to MEMS devices in particular?

This part has a moisture sensitivity level of 3, according to Digi-Key:
https://www.digikey.com/product-detail/en/stmicroelectronics/LSM6DS3TR/497-15383-1-ND/5180534

I would check that your board assembler is receiving these parts in unbroken moisture barrier bags; or if not, that the board assembler is dry-baking these parts before assembly.

That's normally the case, but stuff-ups happen.

I can remember a case where batches of Xilinx chips were mysteriously failing. It turned out that the supplier was shipping them to our board assembler in unsealed bags. These parts had been sitting on the shelf in a humid warehouse, for months, prior to shipment. The moisture absorbed by the packages caused havoc during automated soldering. Once our Production and Purchasing people got strict about moisture sensitivity safeguards, the problem went away.

[IDEngineer]

Interesting thought on the moisture sensitivity. I wonder if being powered on for a while would bake excess moisture out (for that matter, I wonder if the package is hermetic). If the problem is solely moisture interfering with a good SMT reflow, that is certainly not going to improve with power-on time.

I'll confirm with the assembly shop and report back. Thanks!

[SparkyFX]

Is there anything in the industry about IC burn-in? Perhaps specific to MEMS devices in particular?

That question reminds me of something: Have the parts been passing an X-Ray while crossing a border or such?

There are quite a few things that can damage such sensitive parts.

[AndyC_772]

I know that, with some of these parts, the board layout is critical to avoid asymmetric forces being applied to the device at the point the solder solidifies during reflow. (Might be Freescale rather than ST who specify this).

What does your layout look like? Is there anything strongly asymmetrical around the device and the traces immediately leading to it?

Maybe there's some residual thermal stress 'baked' into the device after reflow, which reduces over time as the PCB pads are able to move slightly?

[Whales]

I did that, first thing (like two weeks ago). ST has stopped having formal trouble tickets and now directs you to their online forums. I have two lengthy threads there, complete with scope screenshots where applicable, but the responses I've gotten have nothing to do with my questions!

...

Customers not good enough to help directly?    Then you're not good enough for customers.

My sincere apologies IDEngineer, it sounds like they're fools.  That's not a good way of keeping customers.

[amyk]

These chips are used in millions of cellphones and tablets. I find it hard to believe they're being power-cycled all the time. The chips have very aggressive power-saving modes that are openly and specifically intended for phones and tablets, to save power. Heck, this LSM6DS3 brags about "Compliant with Android K and L" as the fourth bullet item on the front page of its spec sheet. There's no mystery what market they're targeting. Do I think they added an FET to each cellphone to cycle power on these devices? No, I don't. Their internal power saving modes can drop their consumption to almost nothing... a quick glance shows the LSM goes to 6uA in "power down" mode.

I suspect this could be a case of "popular part with little-used feature has a bug" --- it might be worth finding some Android source code from some device that uses this IC to see if it uses the same options (filters, etc.) that you're using.

[IDEngineer]

I know that, with some of these parts, the board layout is critical to avoid asymmetric forces being applied to the device at the point the solder solidifies during reflow. What does your layout look like? Is there anything strongly asymmetrical around the device and the traces immediately leading to it? Maybe there's some residual thermal stress 'baked' into the device after reflow, which reduces over time as the PCB pads are able to move slightly?

You're absolutely correct about mechanical stress affecting such parts, and in the case of the ST LSM6DS3 the spec sheet does explicitly discuss that. But they say, and we've confirmed, that the effect is to impart a bias to measurements in one or more axes. It's interesting to watch the baseline values change as you twist the PCB. I've already written calibration routines into the firmware for exactly this reason. Soldering the parts to the board definitely "locks in" some stress, but twisting the PCB even slightly has a much greater effect. (Note this means the device will need calibration AFTER installation in its final location since mechanical mounting will likely impose some degree of stress on the board.) My calibration routines can be remotely accessed via a software command to make that easier.

No, there's nothing inherently asymmetrical about the layout. But it IS an interesting thought that, with time and power applied, the board might warm enough to allow solder migration. However, as I said, the effects stated by ST and seen by us seem to be limited to a measurement bias, not to (firmware?) lockups.

[IDEngineer]

That question reminds me of something: Have the parts been passing an X-Ray while crossing a border or such?

I have no way to know what they might have encountered along the way, but neither we nor our assembly house x-rayed them!

[IDEngineer]

Customers not good enough to help directly?    Then you're not good enough for customers.

In fairness, their technical staff is supposed to be monitoring and responding on the forums. They didn't just abandon technical support, they got rid of the "private trouble ticket" system. I understand the advantages of that, including 1) someone else might answer faster, and 2) the answers become accessible to others with the same question. However, it's the abysmal quality of the "answers" that I'm getting that's so discouraging. As I illustrated above, it's like they don't even comprehend my question. Maybe they just don't believe their part could have such a problem, or that this particular customer is an idiot. The latter may be true, but I guarantee the former is too!

[IDEngineer]

I suspect this could be a case of "popular part with little-used feature has a bug" --- it might be worth finding some Android source code from some device that uses this IC to see if it uses the same options (filters, etc.) that you're using.

That's just it - this bug isn't in some little-used feature, or even an optional feature like the internal filters. This bug shuts down the SPI interface, the only way to interact with the device! To be clear, I have managed to eliminate several other bugs by disabling every filter and feature over which you have any sort of control, which I now presume lessens the processing burden on the internal firmware. But this one appears to correct itself with enough power-on time. I suppose I can just insist that Production build a burn-in rack with the appropriate connectors and give them 24? hours, but I'd sure like some sort of confirmation that we're treating the actual disease and not just a symptom.

[IDEngineer]

It's early this morning, but the result of the overnight test with the BMI160's is that neither of them failed overnight. Like the most recent LSM6DS3 test, that's one board with many hours of power applied and one board literally fresh out of the antistatic bag. When I did this with the LSM's, the new one failed overnight. But now neither of the BMI's show any ill effects. I'll continue letting them run, but it's not looking good for ST....

[ogden]

No, they were assembled by a contract manufacturing house that we use all the time. Very high end shop, leading edge pick-and-place and reflow ovens, etc. They do all of our protos and most of our production. I'm confident they followed ST's guidelines from the spec sheet.

It is important that you/they followed spec sheet "Surface mounting guidelines for MEMS sensors", TN1198 (en.CD00134799.pdf)

But again, even if they didn't, anything "damaged by soldering heat" doesn't usually spontaneously repair itself.

It's usually about mechanical stress which may relax after some time or after some number of thermal cycles.

[IDEngineer]

It's usually about mechanical stress which may relax after some time or after some number of thermal cycles.

Right, but I would be surprised if mechanical stress could selectively fail the SPI interface. Impose a bias on the MEMS sensor readings, sure. But cause the SPI interface to stop working, after it DID work for 30-120 minutes? Then keep the power applied and "after a while" this error stops occurring and the SPI interface becomes reliable? Hmm....

[David Hess]

Is there anything in the industry about IC burn-in? Perhaps specific to MEMS devices in particular?

Burn-in is a very expensive process so only used on parts which can support the cost.  Test time is measured in 10s of cents to dollars per second so it is used as sparingly as possible.

Exceptions include high cost parts which support built in self test and grading like microprocessors and memory.  The highest precision voltage references may go through extended burn-in.  Many military and aerospace qualifications require burn-in.

That question reminds me of something: Have the parts been passing an X-Ray while crossing a border or such?

There are quite a few things that can damage such sensitive parts.

Exposure would have to be 10s of minutes to hours to matter.  X-rays roughly act like short wavelength UV exposure except they can penetrate through the packaging.  You can use an x-ray generator to erase floating gate memory but as with a UVEPROM eraser, it takes a while.

[AndyC_772]

Even so, burn-in is normally used to detect early failures, not to somehow magically make faulty parts start working. I'm not aware of any physical mechanism which would cause a damaged part to start working just because it's been switched on for a while.