I2C glitch challenge - Cannot figure out how to clean this up.

[rx8pilot]

Hi All,

I have an existing project where the various systems are connected with I2C at 100Khz. The master is an 8bit AVR Mega164P and there are 5-7 slaves at any given moment from various manufacturers. I largely wrote the i2c (TWI) code starting a few years back and have not had any real trouble until now.

The challenge is that there is a glitch during the ACK. From what I can tell, this was not a problem with all the slaves I have used so far so I did not try to fix it. I have added a new slave device from Maxim DS2484 which is an 12c to 1-wire driver. This device tends to lose control when the glitch happens during the ACK.

What I think is happening is that there is a timing error when the MASTER releases the SDA so that the SLAVE can issue the ACK. It's hard to tell what is pulling what since any device on the line can have a problem. This is especially true for a non-expert in i2c like myself.

In the scope screenshots the SDA and SCL are connected to the digital inputs and are decoded. The bottom half of the image is a ZOOMED in view. The Yellow trace is the analog channel looking at the SDA. In the first image, I believe the master is releasing the SDA  about 2.3us before the the SLAVE is sending the ACK. In this case the SLAVE does not care and everything moves on.



The next image is when the glitch is wide enough to be read, the SLAVE (in think) goes off the rails. This seems to happen about 50% of the time, so I would guess that it is right on the pulse width threshold of that particular slave device.



More of a zoom to see that the SDA is fighting for control during the ACK.



There does not seem to be any issues with rise time. Pull-Ups are 3k. Vcc is 5v. Traces are rather short - under 10in from end to end. The software uses the TWI (i2c) hardware in the AVR so I am not aware of any option to adjust the ACK timing so that I can reduce or eliminate the glitch. I also tried to adjust bus speed from below 100Khz to a little over 400Khz but that made no difference.

Is anyone aware of an option to adjust the timing on the AVR side? As far as I can tell, the internal hardware timing is fixed - At least I have not found anything in the data sheet yet. I am hoping someone has solved this before and has a simple solution.

[nctnico]

I've found some I2C devices are very picky about the right start/stop order and don't like to see a new start before a stop.

[hans]

I believe sometimes generating a start and then a stop code can reset the state machines of some I2C devices.

Maybe this is useful work-around if the glitch keeps happening at the same point in the transactions.

[nctnico]

Also the saw tooth bothers me. Can it be a power supply problem? Are all I2C devices powered from 5V because I2C is not designed to allow mixing logic levels.

[rx8pilot]

This one seems to be struggling with the ACK, before the STOP is even issued by the master.

If all was by the book.....to write a byte....
The master would:
Send a START
Send a 7-bit device address
Send a '0' bit to indicate a WRITE is about to happen
Release the SDA line so that the slave device can ACK by pulling it low during the ACK SCL pulse
Send a DATA byte
Wait for ACK from slave by releasing the SDA line and letting the slave pull it low.
Send a STOP

This one addresses the slave and sends the WRITE bit. The master immediately releases the SDA line but the slave has not yet pulled it down to ACK. There is about a 600ns delay from when the Master releases the line and the slave pulls it back down again. Sometimes the resulting glitch is long enough to send the bus into chaos and sometimes it is ignored.

All the other slaves see a similar glitch and ignore it. This particular device is clearly more sensitive to an errant pulse. Looking for a way to change how fast the AVR master releases the SDA after the data is sent. If I can delay the SDA release by about 1us or so, this would be all fine. It is a very frustrating problem for sure.

I believe sometimes generating a start and then a stop code can reset the state machines of some I2C devices.

Maybe this is useful work-around if the glitch keeps happening at the same point in the transactions.

The glitch happens at the same place every time, immediately following the last bit of data but before the slave has ACK'd. 

[tatus1969]

when you look closely at all the rising edges you notice some up and downs. Looks like you have some source of interference there. Could be the supply, or some crosstalk. Does the I2C run through wires? You need to get rid of that, because this can cause false triggering on some devices. Many implement low pass / glitch filtering but not all. Apart from that I cannot see anything wrong. These short pulses before ACK are normal.

[nctnico]

Perhaps using larger pull-ups can fix the problem. 3k Ohm is rather low; I usually use 10k.

[rx8pilot]

Also the saw tooth bothers me. Can it be a power supply problem? Are all I2C devices powered from 5V because I2C is not designed to allow mixing logic levels.

The sawtooth is very weird, I really don't know how that is happening. I very closely checked power and it seems to be clean with no more than a few mV of noise. All 5V, however there are 2 different sources of the 5V.

when you look closely at all the rising edges you notice some up and downs. Looks like you have some source of interference there. Could be the supply, or some crosstalk. Does the I2C run through wires? You need to get rid of that, because this can cause false triggering on some devices. Many implement low pass / glitch filtering but not all. Apart from that I cannot see anything wrong. These short pulses before ACK are normal.

Yes, I took a close look at the rising edges and they all have small ups/downs on the rising edge. They are very consistent and I cannot correlate them with any other system. There are a lot of high-current lines close by, but they are not active. There is also a few DC-DC converters close by, but I would not expect them to be in perfect sync with the rising edge of the i2c. These glitches are very fast and I wold be surprised if they have any impact.

[AndyC_772]

This doesn't look like any kind of interference to me. If it were, you'd see it superimposed continuously, even when there's real data being transferred.

I2C is open drain and uses a passive pull-up. It shouldn't be possible for two devices to 'fight' for control of the bus; either:

- no device drives low, in which case the bus is pulled up to VCC by the pull-up, or
- at least one device drives low, in which case the bus is driven to GND.

The intermediate voltage you're seeing, and the burst of oscillation, shouldn't be able to happen. The fact that it does implies that some device is actively driving high, which it should never do. Perhaps the AVR pin isn't set to open drain?

[Edit]: removed duff information about clock edges.

I'm afraid I'm not familiar with the AVR, but its I2C driver is the first place I'd look. Can you single step through the code to correlate events on the bus with interrupts or changes of state in its I2C peripheral?

Also, most microcontrollers' I2C peripherals that I've come across suck. They're a congenitally awful breed, and since the AVR is master, it may be a better idea to just program the pins as GPIO and write your own I2C driver.

One of the better descriptions I've come across on how the I2C bus works, is in the data sheet for the PCA9501 I2C EEPROM/GPIO expander. Have a read of that and you'll be in a good position to spot other anomalies in your system.

Even if you don't manage to fix the problem, at least you've had the chance to post some screen shots from your new toy. Don't tell me you didn't take a little pleasure from that opportunity 

[tatus1969]

some devices have transistors from output to vcc that get activated for a short moment after the low side is switched off, in order to help the pullup resistor rising the voltage. The idea is to have short rise times even in open drain / pullup configuratuon. Maybe that is counter productive here and can be disabled. Check for things like pullup boost or alike.

[Sal Ammoniac]

Since you said this started happening after you added the Maxim DS2484 to the system, can you remove all other I2C devices from the bus and see if this still occurs? Then you'll know if the issue is with the new Maxim device.

It's pretty obvious that something on the bus is driving SDA high, which is something that should never happen on an I2C bus. If it's the Maxim part, talk to your Maxim FAE and see if they have any advice.

Can you try another (non-AVR) I2C master for comparison purposes?

[AndyC_772]

I think the glitch in your first post is a non-issue.

I2C data (SDA) is allowed to change, arbitrarily, when SCL is low. The glitch is simply due to the propagation delay between the master relinquishing control of the bus on the falling edge (which it generates), and the slave device detecting the falling edge and pulling SDA low.

Since I2C devices sample on the rising edge of SCL, no device should see this glitch at all.

I guess it's possible that some device sees the change of SDA while SCL is (locally) still high, and misinterprets this as a start condition? That still doesn't explain some device apparently driving SDA high actively.

[rx8pilot]

The first is that I2C is open drain and uses a passive pull-up. It shouldn't be possible for two devices to 'fight' for control of the bus; either:

It could be a single device that is pulsing the SDA very fast which would result in a sawtooth. It is hard to sort out what device is contributing though. The system design will make it difficult to completely isolate the other slaves, but it only happens when this Maxim is addressed.

The intermediate voltage you're seeing, and the burst of oscillation, shouldn't be able to happen. The fact that it does implies that some device is actively driving high, which it should never do. Perhaps the AVR pin isn't set to open drain?

The AVR is using the internal TWI (I2C) hardware that takes control of the pins and the programmer cannot do much (that I am aware of). If the slave device, for whatever reason, was pulsing the SDA into a low state very fast and just fighting the pull-up resistor - it may result in the sawtooth without anything actively pulling it up. Why the DS2484 would ever behave that way is beyond me, so maybe the source is something else. Just to eliminate variables, I could swap the DS2484 with a new one in case this one was damaged during soldering or some other issue.

I'm afraid I'm not familiar with the AVR, but its I2C driver is the first place I'd look. Can you single step through the code to correlate events on the bus with interrupts or changes of state in its I2C peripheral?

Also, most microcontrollers' I2C peripherals that I've come across suck. They're a congenitally awful breed, and since the AVR is master, it may be a better idea to just program the pins as GPIO and write your own I2C driver.

I can only step through so much, the I2C hardware automatically deals with the ACK after sending a byte of data. There are some outputs of the AVR connected to the scope that I am pulsing to correlate the code with the results. It is not too helpful though since I cannot control the timing of the ACK.

Even if you don't manage to fix the problem, at least you've had the chance to post some screen shots from your new toy. Don't tell me you didn't take a little pleasure from that opportunity 

Absolutely! A proud moment indeed. It is very easy to control and grab frames from my programming workstation too. Aside from the excitement of winning the scope, it has been remarkably useful in real life challenges. It is very intuitive and easy to find difficult details.

Since you said this started happening after you added the Maxim DS2484 to the system, can you remove all other I2C devices from the bus and see if this still occurs? Then you'll know if the issue is with the new Maxim device.

It's pretty obvious that something on the bus is driving SDA high, which is something that should never happen on an I2C bus. If it's the Maxim part, talk to your Maxim FAE and see if they have any advice.

Can you try another (non-AVR) I2C master for comparison purposes?

I may try to rig a simpler system to further isolate the problem. The only other I2C master I have is a Bus Pirate which I think is a PIC of some sort. That could help although I don't know how to use it very well - tried it once about 2 years ago and have not used it since.

[rx8pilot]

I think the glitch in your first post is a non-issue.

I2C data (SDA) is allowed to change, arbitrarily, when SCL is low. The glitch is simply due to the propagation delay between the master relinquishing control of the bus on the falling edge (which it generates), and the slave device detecting the falling edge and pulling SDA low.

Since I2C devices sample on the rising edge of SCL, no device should see this glitch at all.

I guess it's possible that some device sees the change of SDA while SCL is (locally) still high, and misinterprets this as a start condition? That still doesn't explain some device apparently driving SDA high actively.

This is what has me confused - all the other slaves ignore the glitch and there is not an issue. The DS2484 seems to have the issue only half the time. The 1st screenshot is in fact the DS2484 being addressed with no problem. Maybe I should probe the pins directly on the DS2484 to see it there is anything happening locally - currently probing close to the master. It is behind an I2C buffer that facilitates hot-plugging to the bus. There are a few other I2C slaves operating normally on this side of the buffer, so I would be surprised to find a problem there. At this point, I am looking at anything at all. This is day two of the investigation and it is getting old. Hoping to turn a corner soon.

[nfmax]

It looks to me as if the slave might not be pulling down on SDA as 'hard' as the master. If you look at the first screenshot, you can see the SDA line drop in voltage slightly after the falling edge of the clock following the ACK. Then in the second screenshot the pulldown is so weak the master logic is getting confused and going into some sort of metastable oscillation. This might be because of a bad connection to the SDA pin on the slave (dry joint?) or maybe ESD damage - probably not helped by the relatively low value of pull-up resistance, but one that ought to work. The steps on the rising edges of SDA seem to be at the same voltages on every pulse, regardless of which device has been driving the line low, so they probably aren't directly related. They don't correspond exactly to the sawtooth voltages.

I doubt anything is actually driving the SDA line high.

Try replacing the DS2484 - it may be bad

[rstofer]

I assume all devices on the bus are 5V logic, right?
Your pull-up resistors could be as low as 2.2k before they can handle an extensive bus with distributed capacitance.  There's actually a formula a graph on page 42:
http://www.cs.unc.edu/~stc/FAQs/Interfaces/I2C-BusSpec-V2.1.pdf

That glitch when the master releases the bus and the slave hasn't decided to handle it is expected behavior.  The only thing that matter is the value of SDA when SCL transitions.

The long half-voltage thing isn't normal.  Maybe smaller pull-up resistors, maybe something else.

So, unhook everything except the Maxim device and see how it looks.

[AndyC_772]

Some devices do have oddly large capacitance. Might be worth changing your pull-up resistors if you haven't already? I normally use 2k on I2C as a matter of course; I've had way more issues with pull-ups being too weak than too strong.

Perhaps the I2C buffer you mention is a clue. What part is it? How does it determine which direction to drive in at any given moment in time? Does it require separate pull-ups on each bus segment? How does the signal integrity compare between the two segments? Could *it* be the thing that's getting confused by the change in bus direction at the ACK time?

[nctnico]

It is behind an I2C buffer that facilitates hot-plugging to the bus.

Aha! I'd remove that chip first and see how it works then. These chips have to guestimate which direction the data should flow and the saw tooth could indicate the buffer can't decide what to do.

[rx8pilot]

Perhaps the I2C buffer you mention is a clue. What part is it? How does it determine which direction to drive in at any given moment in time? Does it require separate pull-ups on each bus segment? How does the signal integrity compare between the two segments? Could *it* be the thing that's getting confused by the change in bus direction at the ACK time?

I think you are right - this is the place to look. I had been probing on the master side of the bus, until now. When I looked at both sides of the buffer - problem on the master side while totally clean on the other side. Duh, the problem is between those two points which is a TI TCA4311A. I can stop looking at the code, AVR, and the DS2484 for a minute and focus on this element.
http://www.ti.com/lit/ds/symlink/tca4311a.pdf

There does not seem to be any issues with other parts on the island bus, so I am looking for something that would trigger an oscillation. It does require it's own pull ups for the isolated i2c bus and I had those at 3k. I swapped for 7.5k and see no difference. Next, I will try to add some capacitance to the bus to see if that will smooth out high-frequency in case that is the trigger. Also, reading the data sheet more closely to see if I find any clues.

This is progress! Thank you all for the help. I can feel a solution coming this way.

[rx8pilot]

Aha! I'd remove that chip first and see how it works then. These chips have to guestimate which direction the data should flow and the saw tooth could indicate the buffer can't decide what to do.

The microscope now knows where to focus. This has had me running is circles.

[nctnico]

The datasheet seems to imply the chip has been designed around 10k pull-up resistors.

[AndyC_772]

Right. It's an interaction between the buffer IC and the bus capacitance. A lower pull-up value may well sort it.

An I2C buffer has to determine which device has driven each wire on the bus low at any given moment.

Say device 'A' drives low. The buffer must drive device 'B' low, but also remember the direction of communication. When 'A' releases the bus and allows it to go high again, the buffer releases 'B' and vice versa. During normal data communication, this is fine.

The problem comes with ACK, because the bus goes straight from being driven low by the master, to being driven low by the slave.

In this case, what happens is:

- master drives low
- buffer drives slave low
- master stops driving low
- positive glitch inevitably appears on master side, because the buffer is driving master>slave and not slave>master
- buffer stops driving slave low
- buffer determines that slave side is still low, so starts driving master low
- when slave relinquishes SDA, the buffer sees the slave side go high, and it relinquishes the master side.

This is all OK, provided you don't mind the glitch (it's harmless!).

But, add in some bus capacitance and it all goes horribly wrong.

Suppose the buffer is passing a logic low from A to B.

A starts to go high, so the buffer stops driving B low. But because bus segment B has capacitance, the buffer sees B still low, and starts driving A low again, on the basis that B must now be 'driving' the bus.

B gradually floats high, and the buffer stops driving A low. But A has capacitance too, so the buffer swaps direction again, and the cycle repeats. You have an oscillator.

Stronger pull-ups. You owe me one beer 

[Sal Ammoniac]

Post your question to https://e2e.ti.com/support/interface/i2c/f. TI people frequently respond to posts there and may have some insight for you.

[nuno]

(...) and there are 5-7 slaves at (...)

Man, you're BRAVE!!

[rx8pilot]

(...) and there are 5-7 slaves at (...)

Man, you're BRAVE!!

It doesn't seem so bad does it?