Author Topic: I2C fault injection  (Read 892 times)

0 Members and 1 Guest are viewing this topic.

Online tom66Topic starter

  • Super Contributor
  • ***
  • Posts: 7397
  • Country: gb
  • Professional HW / FPGA / Embedded Engr. & Hobbyist
I2C fault injection
« on: October 17, 2024, 01:59:13 pm »
I've got a bug on a product where about 1 in 100k to 1 in 1 million requests to an I2C device on my bus fail.

This is of course very difficult to reproduce.  It seems to be a result on some analog signal variation, perhaps a NAK interpreted as an ACK or a glitch filter violation.

I would be interested to know if anyone has any suggestions for a device that can inject I2C errors into an active bus.  Injecting a NAK could be difficult but by inserting the device in line with the bus it might be possible to make the microcontroller see something that isn't there, or generate obviously bad responses based on a pattern.

Could also try things like enabling clock stretching, stuck SDA, etc.  and ensure that my system can recover from all of these fault conditions.
 

Offline AndyC_772

  • Super Contributor
  • ***
  • Posts: 4324
  • Country: gb
  • Professional design engineer
    • Cawte Engineering | Reliable Electronics
Re: I2C fault injection
« Reply #1 on: October 17, 2024, 02:15:47 pm »
I think I'd approach this from the other direction. It might be good design to have a system that can recover from bus errors, but unless there's some specific design feature that makes errors inevitable (eg. long cables that pick up noise, or multiple asynchronous masters that might clash), you should expect I2C to be error free.

Instead I'd concentrate on trying to generate a trigger condition for your scope in the event of an error. Put some debug into your code which says something along the lines of "if the value read over I2C is out of range, then pulse this GPIO", then set your scope to record the I2C interface and trigger off that pulse.

It might then show you something unexpected that's happening on 'bad' accesses - maybe a particularly slow clock edge or an invalid logic level. Or you might discover that the transaction is perfectly OK and the bug is in the CPU after all (eg. an unfortunately timed interrupt). Maybe the slave device you're accessing reports garbage if it's read at exactly the wrong time (eg. when an update is taking place).

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 15984
  • Country: fr
Re: I2C fault injection
« Reply #2 on: October 17, 2024, 02:23:23 pm »
Yes I'd also focus on being robust to occasional errors, but being able to reproduce problems is also important.

I would probably design a tool that would act as an I2C slave device that randomly gives bad replies (unexpected NAKs or other errors). Could be done on a small MCU as long as it supports I2C slave.
 

Offline eutectique

  • Frequent Contributor
  • **
  • Posts: 457
  • Country: be
Re: I2C fault injection
« Reply #3 on: October 17, 2024, 02:36:14 pm »
So, what exactly constitutes a bug or a fail? Is it reading wrong data from a device? Or the the device is stuck and no further transfers are possible?

The usual recovery from the latter condition is switching host SCL and SDA into GPIO output mode and issuing a series of pulses to simulate the transfer. This will unblock the device.

Also, a good way to create electrically-noisy environment is to fire the gas piezo lighter (the kitchen variety) near your I2C wires. Improves the chances from 1:1M to 1:1.
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 15984
  • Country: fr
Re: I2C fault injection
« Reply #4 on: October 17, 2024, 03:16:57 pm »
Also, a good way to create electrically-noisy environment is to fire the gas piezo lighter (the kitchen variety) near your I2C wires. Improves the chances from 1:1M to 1:1.

Yes that's the poor man's EMI generator many of us have used. (Note: that tends to trigger some hilarity from non-EEs, you've been warned.)
Also works as a poor man's ESD gun.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf