i2c bus weirdness on Linux

[hpmaxim]

I'm using a Cubieboard2 (similar to RPi2 or RPi3) running Linux and wrote some code to access a MEMS device via i2c.

The MEMS device has been acting erratically in an intermittent way.  I have gone back and forth with the manufacturer and finally shared scope traces of SDA/SCL and got the following response:

"Each I2C transaction should start with a I2C start condition. which is correct in your case.  On multiple byte read, each byte read should be followed by an ACK from the master. when you don't want to read more data, master should generate NAK signal on the last byte read indicate the end of read. And master should not send out more invalid clock signals. A STOP should be issued from the master that the communication is over.

In your read operation, you didn't give NAK to slave, and keep sending out the clock without the STOP, so the sensor will continue output the next FIFO sample -- Even though the clock is not complete."

My code is pretty simple:

int fd;
unsigned char i2c_address = 0x6a;
unsigned char outbuffer[2] = {0x3c,0};
unsigned char inbuffer[2];

fd = open("/dev/i2c-1", O_RDWR);
ioctl(fd, I2C_SLAVE, i2c_address);
write(fd, outbuffer, 1);
read(fd, inbuffer, 2);

Anyone got any ideas on why it's not handling NACK correctly or otherwise being flaky?

[Nominal Animal]

Let's see.  There is a summary of the i2c-dev functionality at kernel.org/doc/html/latest/i2c/dev-interface.html.

If you install the i2ctools package (using the distribution repository; however, the sources are here), you can use the i2ctransfer tool to check if it performs the transfers correctly:
    i2ctransfer 1 w1@0x6a 0x3c r2@0x6a

If that behaves exactly like your own code, then the problem is in the i2c-dev kernel driver for Cubieboard2.  (I don't think this is the case, but if it is, I'd need to know the exact kernel version to help further; i.e. the output of uname -rv.)

If that works (and your own code does not), then the problem is that you need to use the I2C_RDWR ioctl() instead, to combine I2C reads and writes with correct ACK/NACK'ing.

(I apologise for not giving a direct answer, because I'm too lazy to set up one of my SBCs with my "scope" (Analog Discovery 2; I'm too poor to have a proper scope) and actually check.  I do think it is better to show how to triage-diagnose-fix the problem, rather than just give a final answer, because these threads are indexed by search engines, and others may encounter the same or a similar problem later: better overall results for everyone.)

You can find the sources for the i2ctransfer utility here; the I2C_RDWR ioctl is simple to use.  Writing offhand –– so I haven't checked this at all! –– I think the corresponding code to yours would be

Code: [Select]

    const char *devpath = "/dev/i2c-1";
    int  fd = open(devpath, O_RDWR);
    if (fd == -1) {
        fprintf(stderr, "%s: %s.\n", devpath, strerror(errno));
        exit(EXIT_FAILURE);
    }

    struct i2c_msg  msg[2];
    struct i2c_rdwr_ioctl_data  op;
    unsigned char  rbuf[2], wbuf[1];

    wbuf[0] = 0x3c;

    msg[0].addr = 0x6a;
    msg[0].flags = 0;
    msg[0].len = 1;
    msg[0].buf = wbuf;

    msg[1].addr = 0x6a;
    msg[1].flags = I2C_M_RD;
    msg[1].len = 2;
    msg[1].buf = rbuf;

    op.msgs = msg;
    op.nmsgs = 2;

    int n = ioctl(fd, I2C_RDWR, &op);
    if (n < 0) {
        fprintf(stderr, "%s: Could not communicate: %s.\n", devpath, strerror(errno));
        close(fd);
        exit(EXIT_FAILURE);
    } else
    if (n != 2) {
        fprintf(stderr, "%s: No response.\n", devpath);
        close(fd);
        exit(EXIT_FAILURE);
    }

    /* Success, data in rbuf */

    close(fd);


[DiTBho]

Usually these problems happen due to too high clock frequency used in the i2c transaction, or due to too high/too low pull-up resistors on the i2c lines.

Check out these two points.

[Nominal Animal]

Usually these problems happen due to too high clock frequency used in the i2c transaction, or due to too high/too low pull-up resistors on the i2c lines.

True; I completely forgot it is impossible to tell exactly why that ACK/NAK/STOP (final) bit is in the wrong state from the 'scope trace alone!

Another common problem with I2C is that some bidirectional level translators aren't really suited for I2C use (TXB0108 is one example!), and can cause similar issues (invalid first/last bits when changing direction).  (Which is why I like to use the ones SparkFun and Adafruit use – pairs of MOSFETs like BSS138 –, and SN74LVC1T45 for UART level shifting.  Both approaches can go down to 1.8V, which is used on some SBCs on signal lines directly connected to the SoC, like the Odroid HC1 I have.)

The i2c-dev documentation explicitly states that

Note that only a subset of the I2C and SMBus protocols can be achieved by the means of read() and write() calls. In particular, so-called combined transactions (mixing read and write messages in the same transaction) aren’t supported.

which is why I believe the core problem is using read()+write() instead of I2C_RDWR ioctl.  It's just been a while since I did I2C on an SBC (I prefer to use microcontrollers for I2C and SPI and sensors, and a higher-level protocol between the SBC and the microcontroller, usually USB or UART)...

[DiTBho]

I prefer to use microcontrollers for I2C and SPI and sensors, and a higher-level protocol between the SBC and the microcontroller, usually USB or UART

Yup, So I do 

In the current Mooka project I have to read and write an ISO7616 Card. The card like a "smart-card" but it's a true simple I2C EEPROM, which needs to be managed by a Linux router, so it could be an I2C-device attached to the router I2C-chain, but I don't like this way, and I put a MPU for its use. The MPU talks to the card directly over I2C, and then talks to the router over a serial line.

In this way the events are handled by the MPU, while for the router the ISO7616 card is a char-dev serial device connected to / dev / ttyS5 which saves me from having to deal with other levels of complexity with the kernel and things that can go wrong with an SBC.

Just to say, before we talk about kernels and userspace, you need to configure the pins for I2C on the SBC. Pins are multi-functional on modern SoCs nowadays, so you need to configure them in the DTS session before booting the kernel. In my experience with Allwinner's SoCs, this part was problematic, and I wasted two weeks because the DTS parser in their firmware (uBoot) was not able to correctly setup pins.

[hpmaxim]

Thanks for all the great info, let me address a few points:

1) I've used two different boards. I designed a custom board which takes 5V in, and drops it to 3.3V with a linear regulator.  The linear regulator powers the MEMS device and is what I use for the pull-ups, which I think end up being around 3.3k ohms.  After I couldn't get that to work (and before I found out it was the i2c), I got a devboard off eBay for $10.  That used, I think about 5k pull ups, and relied on a 3.3V supplied by the CubieBoard.  Looking at the traces, you can actually see that the "low-level" for SDA isn't completely stable.  I suspect that is because the pull-down on the CubieBoard and the pull-down on the MEMS device aren't the exact same strength and so the different level tells you which is doing the pull down.

The custom board actually has an ESP32-S2 on it (which if I were doing now, I'd use an S3).  The intent was to have two I2C busses, one that was local to the board and went between the MEMS devices (there are a bunch of them on the board, but this one in particular is the only one misbehaving) and the ESP32, and one between the ESP32 and the CubieBoard.  This board is designed as a semi-drop-in replacement for an old board which had different MEMS devices and no uC on it -- so I'd prefer to stick with I2C, but I suppose that I could potentially use SPI or some other more advanced protocol to get information off the custom board.  The custom board also included jumpers which allowed me to short the two I2C busses together, and that's what I'm currently doing.  I'm just ignoring the ESP32, shorting the busses together and giving the Cubieboard direct access to the MEMS on the other i2c bus.

Point is, I think all the IO everywhere is at 3.3V, and level shifters probably aren't involved.

2) I'm not sure how to adjust SCL frequency, but just visually looking at the scope traces, SCL appears to be nearly square/50% duty cycle at about 100kHz. When I zoom in more, I can easily see the rising edge time constant behavior from the pull-up.  It looks pretty reasonable, but I don't know what typical values are.

3) I'll try out the i2c transfer tool and the ioctl I2C_RDWR (which looks a bit ugly, but not too painful).  The big problem on my end is looking at my scope trace, it's not clear to me what he saw that was wrong, so I'm not sure if I saw the scope trace from i2ctransfer if I'd know if it was working or not.  Luckily, the read/write seems to fail after a couple minutes or less, so if I re-write the code to use i2c_rdwr, hopefully I should see the problem sooner or later.

[Nominal Animal]

The big problem on my end is looking at my scope trace, it's not clear to me what he saw that was wrong, so I'm not sure if I saw the scope trace from i2ctransfer if I'd know if it was working or not.

See the Wikipedia article on I²C, especially the line state table.

Here is basically what we expect to see on the scope for the write part (I think; I usually let Sigrok/PulseView decode I²C for me):
SDA: ▔▔▔▔ ▔╲▁▁▁ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳▁▁▁╳ ╳▁▁▁╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳▁▁▁╳ ╳▁▁▁╱▔ ▔▔▔▔
SCL: ▔▔▔▔ ▔▔▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁▁╱▔▔▔ ▔▔▔▔
     Idle Start Addr0 Addr1 Addr2 Addr3 Addr4 Addr5 Addr6 WRITE  ACK   W0    W1    W2    W3    W4    W5    W6    W7    ACK   Stop  Idle
but the thing is, during the ACKs, it is the MEMS device that pulls SDA low (with host side SDA floating).  If the MEMS device does not pull SDA low, you get a NAK, and the write fails.
(During ╳, you can have "glitches" or "spikes", especially at ACK/NAK, because that's where the other side is supposed to pull the SDA line down.  During the middle horizontal line, the signal can be high or low, but should stay so without glitches.)

The read part can be separate, or instead of a Stop, you can get a repeated Start.  Then, the same seven address bits, and then READ (SDA high instead of low), followed by eight data bits, ACK, eight data bits, NAK, and STOP.

If you do not have one, I recommend you get one of the cheap "saleae logic" clones off eBay –– they're direct implementations of the Cypress FX2 application schematic, and work fine under sigrok.  (Sigrok uses its own fx2lafw firmware for these, so when you are using such a clone, you're not actually violating any Saleae's rights; just rip off any stickers that have "Saleae" on them.)
These are 8-channel logic analyzers with max. 24 MHz sample rate, for about $15 or so shipped, and when used with Sigrok, rely on the Cypress appnote/example circuit and completely open software ONLY.  For I2C, Sigrok has a protocol decoder, so the user interface will show a graph of the logic levels (digital, so high or low only), and interpret it for you also (it even tells you which are address and which are data bits, what the address or data value is, and so on).

You can get a "proper" logic analyzer, but for I2C and UARTs and SPI for up to say 4 MHz or so, a cheap Cypress FX2 one has sufficed for me.  (Although I use Linux exclusively, I've been told the PulseView release binaries Sigrok offers work fine in Windows also.)

[DavidAlfa]

Everything is weird in Linux 

[Nominal Animal]

Everything is weird in Linux 

If I get to pick, I will always choose the weird and evolving over the sterile, clean, static.  At least that way there is a chance of making something better.
One step at a time, my friend. 

[DiTBho]

Everything is weird in Linux

Yup, and regarding I2C, there are more weird things:
I2C-DMA ( I2C_M_DMA_SAFE O_M_G )
I2C/SMBUS Fault Codes and fault injection

 

[nctnico]

@ the OP: please post your oscilloscope traces here as well.

[hpmaxim]

Well, some good news...  I re-wrote my code using Nominal Animal's suggestion which appeared to be absolutely correct other than that I also needed to add:
#include <linux/i2c.h> since I only had #include <linux/i2c-dev.h> and...

It seems to work... I have done about 3 million reads without any issue at this point, prior to that, I was lucky to get 150k.

However, looking at the traces I don't see much difference.  The manufacturer specifically called out the ? on the decode as indicator I was doing something wrong, but it's still there.

The traces in this post are all before implementing the fix.

[hpmaxim]

And these are traces after implementing the fix.  Note that the second trace is just a zoomed in view of the first.

[cv007]

It seems to work... I have done about 3 million reads without any issue at this point, prior to that, I was lucky to get 150k.

Your original posted code was not checking return values for the called functions, with the new code now you are unless you removed the checks. If that was a/the problem with your posted code, then now since presumably checking return values you should be able find out if any of the functions fail for some reason, and how often. If there are no failures ever with the new code, then I guess that explanation does not hold.

[Nominal Animal]

Here's how I interpret the two first old scope traces:



Instead of a single transaction, note how the read and the write parts are separate transactions.  The write part works fine, but the read part fails at the end of the first byte read with NAK.  (We know that it is a fail and not a successful one-byte read, because we do a two-byte read.)

Compare to the new scope trace:

Here, we have a single transaction.  I've marked the restart point with RS.  Because we know that we only want to read two bytes, we can be pretty sure that the final NAK is by our host end: it is how the controller tells the device during a read operation that the controller is not interested in more bytes.
(RS is restart, R is Read, A is ACK, N is NAK.)

The reason your scope shows ? for NAK, is because it is ambiguous: it could be an error, or it could be expected; only someone who knows the expected read lengths knows for sure.  (A NAK at the end of a write always indicates an error; it is just that for reads, a NAK can be caused by the device to indicate an error, or by the controller to indicate it has all the bytes it is interested in.)

Oh, and the protocol is such that we do indeed expect a NAK after the second byte has been read (it's how the host tells the MEMS device the host has everything it needs), followed by STOP.  The new trace looks okay to me.

[Nominal Animal]

If I were you, I'd check the MEMS datasheet whether the write and the read must be a single transaction, or whether it can be done in two separate transactions.  If two separate transactions are allowed, I'd check if there is a maximum interval between the two transactions.

My guess here is that the MEMS device actually wants a single transaction.  In Linux, that means using I2C_RDWR ioctl on the i2c-dev.
For OP, it just happens that when the read transaction follows closely enough to the write transaction, the device somehow just deals with it.

If the MEMS datasheet says a single transaction is needed, then the case is closed: the controller side did the wrong thing.  (To do a write and a read in a single transaction in Linux, you have to use I2C_RDWR ioctl; read() and write() always do separate transactions.)

[nctnico]

Well, some good news...  I re-wrote my code using Nominal Animal's suggestion which appeared to be absolutely correct other than that I also needed to add:
#include <linux/i2c.h> since I only had #include <linux/i2c-dev.h> and...

In that case I assume the compiler was also throwing warnings about implicitely defined functions? In such a case C assumes the argument is an integer which can lead to a lot of trouble if the argument has a different size. IOW: if those warning appears, fix them!

[Doctorandus_P]

I've also got a Rigol scope, but I much prefer to use an EUR10 "Saleaeaeeae clone" Logic Analyser with Sigrok / Pulseview.
* The small box hardly takes up any space at all on your desk.
* Short dupont wires are much easier to manage then clumsy big Oscilloscope probes.
* It can sample upto a few MHz, which is plenty for I2C and lots of other protocols (Pulseview has 100+ software decoders).
* Pulseview can add much more useful information by stacking decoders for specific chips on top of I2C.
* PC monitor with many pixels is a better UI then a scope screen for a logic analyzer.
* With 8 channels you can keep track of multiple simple serial busses at the same time.
* It's also great for catching Debug data (for example from a spare serial port), as it preserves timing in relation to other signals.
* With a generic CY7C68013A uC board (and the righte I2C EEprom data) you can even go upto 16 channels (But these generic boards lack input protection).

These EUR10 boxes do have some limitations:
* Sample rate is not too high,
* It's digital only, Especially for I2C it's good to check the RC time constants  with the pullup resistors with your scope to check if this is all right.
* Bus conflicts when different nodes try to drive a signal in opposite directions are not caught.

But overall, especially considering the price point of these things, I reccomend for everybody interested in writing software for uC's  to get (at least) one of these. even before you buy an oscilloscope.
These things cost less then an (overpriced) "original" arduino.

[DiTBho]

Zero Lap-C LAs are better, compatible with Sigrok, and not so expensive. I bought mine for 60 euro.

Ok, 6x expensive than "Saleaeaeeae clone", but it's not in the range of 300 euro required by other professional stuff and it offers more features.

You can buy a DSO with software analysis modules, basically what you see on Pulseview is performed directly by the DSO.

You can also buy a MSO, basically a DSO + LA all in the same tool. Not too bad, in my opinion, and probably my next purchase.

I have a very old Rigol DSO, 2Ch @ 50Mhz which is also too old for the software analysis modules, it's not supported, but newer models have this possibility, for i2c, spi, can, uart, .. and you have to buy a license-key for each sub-module.

[DiTBho]

p.s.
Some i2c-ADC shoot out 18 bit, hence 3 bytes.
Some i2c-MEMS shout out 2 bytes. But some even up to 3 bytes.

You have to check the datasheet here, there are weird beasts out of there

[DavidAlfa]

These EUR10 boxes do have some limitations:
* Sample rate is not too high

The $7 "Saelaeaeae" is 24Msps(8ch) / 12Msps(16ch) waaaay more than required to debug i2c.
I'd only get the $300 one if I really need it, otherwise it's a waste of money.

[nctnico]

These EUR10 boxes do have some limitations:
* Sample rate is not too high

24Msps(8ch) / 12Msps(16ch) is more than great to debug i2c. $7 vs $300 is no brainer. Get the $300 only if you really need it, otherwise it's a waste of money.

Bad advice. When dealing with these kind of serial bus problems, always look at the signals in the analog domain first. You wouldn't be the first to get stuck on a problem that stems from the analog domain. And having a feature to decode the serial bus while you are at it, just makes live easier. There is no cheap way out.

Several years ago I was called over to solve a problem. The guy was looking at an RS485 bus (for a device he developed) with a logic analyser and was stuck for 2 weeks already. Hooking the signals up to an oscilloscope revealed the problem instantly and it was fixed within 30 minutes.

In addition: once the digital bus is working, it is much easier to debug communication at a higher level (printing the values in a meaningful form to a serial port for example). So all in all the cheap logic probes are far less useful then you think.

[DavidAlfa]

For the analog domain you only need a single shot on the DSO. All rising/falling edges should be the same, it's impossible that the device or the pullup is faster with some bits and slower with others.

[Doctorandus_P]

Bad advice. When dealing with these kind of serial bus problems, always look at the signals in the analog domain first. You wouldn't be the first to get stuck on a problem that stems from the analog domain. And having a feature to decode the serial bus while you are at it, just makes live easier. There is no cheap way out.

Several years ago I was called over to solve a problem. The guy was looking at an RS485 bus (for a device he developed) with a logic analyser and was stuck for 2 weeks already. Hooking the signals up to an oscilloscope revealed the problem instantly and it was fixed within 30 minutes.

I completely agree with this.
If you have an oscilloscope always first verify that you have decent logic levels, and only then go further by decoding the data with a logic analyzer.

[nctnico]

Bad advice. When dealing with these kind of serial bus problems, always look at the signals in the analog domain first. You wouldn't be the first to get stuck on a problem that stems from the analog domain. And having a feature to decode the serial bus while you are at it, just makes live easier. There is no cheap way out.

Several years ago I was called over to solve a problem. The guy was looking at an RS485 bus (for a device he developed) with a logic analyser and was stuck for 2 weeks already. Hooking the signals up to an oscilloscope revealed the problem instantly and it was fixed within 30 minutes.

I completely agree with this.
If you have an oscilloscope always first verify that you have decent logic levels, and only then go further by decoding the data with a logic analyzer.

Why go to the trouble of using a logic analyser if you have decoding functionality in your oscilloscope? Hook-up once, measure once and you have all the info you need. I have a very high end Tektronix logic analyser with a boatload of bells & whistles but it has been years since I used it. The MSO sitting on my bench is just as suitable for 99% of the tasks a logic analyser does.