Author Topic: i2c bus weirdness on Linux  (Read 3134 times)

0 Members and 1 Guest are viewing this topic.

Offline hpmaximTopic starter

  • Regular Contributor
  • *
  • Posts: 132
i2c bus weirdness on Linux
« on: April 28, 2022, 08:42:21 pm »
I'm using a Cubieboard2 (similar to RPi2 or RPi3) running Linux and wrote some code to access a MEMS device via i2c.

The MEMS device has been acting erratically in an intermittent way.  I have gone back and forth with the manufacturer and finally shared scope traces of SDA/SCL and got the following response:

"Each I2C transaction should start with a I2C start condition. which is correct in your case.  On multiple byte read, each byte read should be followed by an ACK from the master. when you don't want to read more data, master should generate NAK signal on the last byte read indicate the end of read. And master should not send out more invalid clock signals. A STOP should be issued from the master that the communication is over.

In your read operation, you didn't give NAK to slave, and keep sending out the clock without the STOP, so the sensor will continue output the next FIFO sample -- Even though the clock is not complete."

My code is pretty simple:

int fd;
unsigned char i2c_address = 0x6a;
unsigned char outbuffer[2] = {0x3c,0};
unsigned char inbuffer[2];

fd = open("/dev/i2c-1", O_RDWR);
ioctl(fd, I2C_SLAVE, i2c_address);
write(fd, outbuffer, 1);
read(fd, inbuffer, 2);

Anyone got any ideas on why it's not handling NACK correctly or otherwise being flaky?
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #1 on: April 29, 2022, 04:59:09 am »
Let's see.  There is a summary of the i2c-dev functionality at kernel.org/doc/html/latest/i2c/dev-interface.html.

If you install the i2ctools package (using the distribution repository; however, the sources are here), you can use the i2ctransfer tool to check if it performs the transfers correctly:
    i2ctransfer 1 w1@0x6a 0x3c r2@0x6a

If that behaves exactly like your own code, then the problem is in the i2c-dev kernel driver for Cubieboard2.  (I don't think this is the case, but if it is, I'd need to know the exact kernel version to help further; i.e. the output of uname -rv.)

If that works (and your own code does not), then the problem is that you need to use the I2C_RDWR ioctl() instead, to combine I2C reads and writes with correct ACK/NACK'ing.

(I apologise for not giving a direct answer, because I'm too lazy to set up one of my SBCs with my "scope" (Analog Discovery 2; I'm too poor to have a proper scope) and actually check.  I do think it is better to show how to triage-diagnose-fix the problem, rather than just give a final answer, because these threads are indexed by search engines, and others may encounter the same or a similar problem later: better overall results for everyone.)

You can find the sources for the i2ctransfer utility here; the I2C_RDWR ioctl is simple to use.  Writing offhand –– so I haven't checked this at all! –– I think the corresponding code to yours would be
Code: [Select]
    const char *devpath = "/dev/i2c-1";
    int  fd = open(devpath, O_RDWR);
    if (fd == -1) {
        fprintf(stderr, "%s: %s.\n", devpath, strerror(errno));
        exit(EXIT_FAILURE);
    }

    struct i2c_msg  msg[2];
    struct i2c_rdwr_ioctl_data  op;
    unsigned char  rbuf[2], wbuf[1];

    wbuf[0] = 0x3c;

    msg[0].addr = 0x6a;
    msg[0].flags = 0;
    msg[0].len = 1;
    msg[0].buf = wbuf;

    msg[1].addr = 0x6a;
    msg[1].flags = I2C_M_RD;
    msg[1].len = 2;
    msg[1].buf = rbuf;

    op.msgs = msg;
    op.nmsgs = 2;

    int n = ioctl(fd, I2C_RDWR, &op);
    if (n < 0) {
        fprintf(stderr, "%s: Could not communicate: %s.\n", devpath, strerror(errno));
        close(fd);
        exit(EXIT_FAILURE);
    } else
    if (n != 2) {
        fprintf(stderr, "%s: No response.\n", devpath);
        close(fd);
        exit(EXIT_FAILURE);
    }

    /* Success, data in rbuf */

    close(fd);
 
The following users thanked this post: hpmaxim, DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3918
  • Country: gb
Re: i2c bus weirdness on Linux
« Reply #2 on: April 29, 2022, 06:44:59 am »
Usually these problems happen due to too high clock frequency used in the i2c transaction, or due to too high/too low pull-up resistors on the i2c lines.

Check out these two points.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 
The following users thanked this post: Nominal Animal

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #3 on: April 29, 2022, 07:13:40 am »
Usually these problems happen due to too high clock frequency used in the i2c transaction, or due to too high/too low pull-up resistors on the i2c lines.
True; I completely forgot it is impossible to tell exactly why that ACK/NAK/STOP (final) bit is in the wrong state from the 'scope trace alone!

Another common problem with I2C is that some bidirectional level translators aren't really suited for I2C use (TXB0108 is one example!), and can cause similar issues (invalid first/last bits when changing direction).  (Which is why I like to use the ones SparkFun and Adafruit use – pairs of MOSFETs like BSS138 –, and SN74LVC1T45 for UART level shifting.  Both approaches can go down to 1.8V, which is used on some SBCs on signal lines directly connected to the SoC, like the Odroid HC1 I have.)

The i2c-dev documentation explicitly states that
Quote
Note that only a subset of the I2C and SMBus protocols can be achieved by the means of read() and write() calls. In particular, so-called combined transactions (mixing read and write messages in the same transaction) aren’t supported.
which is why I believe the core problem is using read()+write() instead of I2C_RDWR ioctl.  It's just been a while since I did I2C on an SBC (I prefer to use microcontrollers for I2C and SPI and sensors, and a higher-level protocol between the SBC and the microcontroller, usually USB or UART)...
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3918
  • Country: gb
Re: i2c bus weirdness on Linux
« Reply #4 on: April 29, 2022, 08:12:42 am »
I prefer to use microcontrollers for I2C and SPI and sensors, and a higher-level protocol between the SBC and the microcontroller, usually USB or UART

Yup, So I do  :D

In the current Mooka project I have to read and write an ISO7616 Card. The card like a "smart-card" but it's a true simple I2C EEPROM, which needs to be managed by a Linux router, so it could be an I2C-device attached to the router I2C-chain, but I don't like this way, and I put a MPU for its use. The MPU talks to the card directly over I2C, and then talks to the router over a serial line.

In this way the events are handled by the MPU, while for the router the ISO7616 card is a char-dev serial device connected to / dev / ttyS5 which saves me from having to deal with other levels of complexity with the kernel and things that can go wrong with an SBC.

Just to say, before we talk about kernels and userspace, you need to configure the pins for I2C on the SBC. Pins are multi-functional on modern SoCs nowadays, so you need to configure them in the DTS session before booting the kernel. In my experience with Allwinner's SoCs, this part was problematic, and I wasted two weeks because the DTS parser in their firmware (uBoot) was not able to correctly setup pins.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline hpmaximTopic starter

  • Regular Contributor
  • *
  • Posts: 132
Re: i2c bus weirdness on Linux
« Reply #5 on: April 29, 2022, 02:06:19 pm »
Thanks for all the great info, let me address a few points:

1) I've used two different boards. I designed a custom board which takes 5V in, and drops it to 3.3V with a linear regulator.  The linear regulator powers the MEMS device and is what I use for the pull-ups, which I think end up being around 3.3k ohms.  After I couldn't get that to work (and before I found out it was the i2c), I got a devboard off eBay for $10.  That used, I think about 5k pull ups, and relied on a 3.3V supplied by the CubieBoard.  Looking at the traces, you can actually see that the "low-level" for SDA isn't completely stable.  I suspect that is because the pull-down on the CubieBoard and the pull-down on the MEMS device aren't the exact same strength and so the different level tells you which is doing the pull down.

The custom board actually has an ESP32-S2 on it (which if I were doing now, I'd use an S3).  The intent was to have two I2C busses, one that was local to the board and went between the MEMS devices (there are a bunch of them on the board, but this one in particular is the only one misbehaving) and the ESP32, and one between the ESP32 and the CubieBoard.  This board is designed as a semi-drop-in replacement for an old board which had different MEMS devices and no uC on it -- so I'd prefer to stick with I2C, but I suppose that I could potentially use SPI or some other more advanced protocol to get information off the custom board.  The custom board also included jumpers which allowed me to short the two I2C busses together, and that's what I'm currently doing.  I'm just ignoring the ESP32, shorting the busses together and giving the Cubieboard direct access to the MEMS on the other i2c bus.

Point is, I think all the IO everywhere is at 3.3V, and level shifters probably aren't involved.

2) I'm not sure how to adjust SCL frequency, but just visually looking at the scope traces, SCL appears to be nearly square/50% duty cycle at about 100kHz. When I zoom in more, I can easily see the rising edge time constant behavior from the pull-up.  It looks pretty reasonable, but I don't know what typical values are.

3) I'll try out the i2c transfer tool and the ioctl I2C_RDWR (which looks a bit ugly, but not too painful).  The big problem on my end is looking at my scope trace, it's not clear to me what he saw that was wrong, so I'm not sure if I saw the scope trace from i2ctransfer if I'd know if it was working or not.  Luckily, the read/write seems to fail after a couple minutes or less, so if I re-write the code to use i2c_rdwr, hopefully I should see the problem sooner or later.
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #6 on: April 29, 2022, 03:37:50 pm »
The big problem on my end is looking at my scope trace, it's not clear to me what he saw that was wrong, so I'm not sure if I saw the scope trace from i2ctransfer if I'd know if it was working or not.
See the Wikipedia article on I²C, especially the line state table.

Here is basically what we expect to see on the scope for the write part (I think; I usually let Sigrok/PulseView decode I²C for me):
SDA: ▔▔▔▔ ▔╲▁▁▁ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳▁▁▁╳ ╳▁▁▁╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳───╳ ╳▁▁▁╳ ╳▁▁▁╱▔ ▔▔▔▔
SCL: ▔▔▔▔ ▔▔▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁╱▔╲▁ ▁▁╱▔▔▔ ▔▔▔▔
     Idle Start Addr0 Addr1 Addr2 Addr3 Addr4 Addr5 Addr6 WRITE  ACK   W0    W1    W2    W3    W4    W5    W6    W7    ACK   Stop  Idle
but the thing is, during the ACKs, it is the MEMS device that pulls SDA low (with host side SDA floating).  If the MEMS device does not pull SDA low, you get a NAK, and the write fails.
(During ╳, you can have "glitches" or "spikes", especially at ACK/NAK, because that's where the other side is supposed to pull the SDA line down.  During the middle horizontal line, the signal can be high or low, but should stay so without glitches.)

The read part can be separate, or instead of a Stop, you can get a repeated Start.  Then, the same seven address bits, and then READ (SDA high instead of low), followed by eight data bits, ACK, eight data bits, NAK, and STOP.

If you do not have one, I recommend you get one of the cheap "saleae logic" clones off eBay –– they're direct implementations of the Cypress FX2 application schematic, and work fine under sigrok.  (Sigrok uses its own fx2lafw firmware for these, so when you are using such a clone, you're not actually violating any Saleae's rights; just rip off any stickers that have "Saleae" on them.)
These are 8-channel logic analyzers with max. 24 MHz sample rate, for about $15 or so shipped, and when used with Sigrok, rely on the Cypress appnote/example circuit and completely open software ONLY.  For I2C, Sigrok has a protocol decoder, so the user interface will show a graph of the logic levels (digital, so high or low only), and interpret it for you also (it even tells you which are address and which are data bits, what the address or data value is, and so on).

You can get a "proper" logic analyzer, but for I2C and UARTs and SPI for up to say 4 MHz or so, a cheap Cypress FX2 one has sufficed for me.  (Although I use Linux exclusively, I've been told the PulseView release binaries Sigrok offers work fine in Windows also.)
 

Offline DavidAlfa

  • Super Contributor
  • ***
  • Posts: 5954
  • Country: es
Re: i2c bus weirdness on Linux
« Reply #7 on: April 29, 2022, 05:07:58 pm »
Everything is weird in Linux  :-DD
Hantek DSO2x1x            Drive        FAQ          DON'T BUY HANTEK! (Aka HALF-MADE)
Stm32 Soldering FW      Forum      Github      Donate
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #8 on: April 29, 2022, 05:18:04 pm »
Everything is weird in Linux  :-DD
If I get to pick, I will always choose the weird and evolving over the sterile, clean, static.  At least that way there is a chance of making something better.
One step at a time, my friend.  :-+

 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3918
  • Country: gb
Re: i2c bus weirdness on Linux
« Reply #9 on: April 29, 2022, 06:16:42 pm »
Everything is weird in Linux

Yup, and regarding I2C, there are more weird things:
I2C-DMA ( I2C_M_DMA_SAFE O_M_G )
I2C/SMBUS Fault Codes and fault injection

 ;D
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 27003
  • Country: nl
    • NCT Developments
Re: i2c bus weirdness on Linux
« Reply #10 on: April 29, 2022, 06:47:08 pm »
@ the OP: please post your oscilloscope traces here as well.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline hpmaximTopic starter

  • Regular Contributor
  • *
  • Posts: 132
Re: i2c bus weirdness on Linux
« Reply #11 on: April 30, 2022, 06:37:17 am »
Well, some good news...  I re-wrote my code using Nominal Animal's suggestion which appeared to be absolutely correct other than that I also needed to add:
#include <linux/i2c.h> since I only had #include <linux/i2c-dev.h> and...

It seems to work... I have done about 3 million reads without any issue at this point, prior to that, I was lucky to get 150k.

However, looking at the traces I don't see much difference.  The manufacturer specifically called out the ? on the decode as indicator I was doing something wrong, but it's still there.

The traces in this post are all before implementing the fix.

 

Offline hpmaximTopic starter

  • Regular Contributor
  • *
  • Posts: 132
Re: i2c bus weirdness on Linux
« Reply #12 on: April 30, 2022, 06:38:30 am »
And these are traces after implementing the fix.  Note that the second trace is just a zoomed in view of the first.
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 828
Re: i2c bus weirdness on Linux
« Reply #13 on: April 30, 2022, 07:23:25 am »
Quote
It seems to work... I have done about 3 million reads without any issue at this point, prior to that, I was lucky to get 150k.
Your original posted code was not checking return values for the called functions, with the new code now you are unless you removed the checks. If that was a/the problem with your posted code, then now since presumably checking return values you should be able find out if any of the functions fail for some reason, and how often. If there are no failures ever with the new code, then I guess that explanation does not hold.
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #14 on: April 30, 2022, 08:26:32 am »
Here's how I interpret the two first old scope traces:



Instead of a single transaction, note how the read and the write parts are separate transactions.  The write part works fine, but the read part fails at the end of the first byte read with NAK.  (We know that it is a fail and not a successful one-byte read, because we do a two-byte read.)

Compare to the new scope trace:

Here, we have a single transaction.  I've marked the restart point with RS.  Because we know that we only want to read two bytes, we can be pretty sure that the final NAK is by our host end: it is how the controller tells the device during a read operation that the controller is not interested in more bytes.
(RS is restart, R is Read, A is ACK, N is NAK.)

The reason your scope shows ? for NAK, is because it is ambiguous: it could be an error, or it could be expected; only someone who knows the expected read lengths knows for sure.  (A NAK at the end of a write always indicates an error; it is just that for reads, a NAK can be caused by the device to indicate an error, or by the controller to indicate it has all the bytes it is interested in.)

Oh, and the protocol is such that we do indeed expect a NAK after the second byte has been read (it's how the host tells the MEMS device the host has everything it needs), followed by STOP.  The new trace looks okay to me.
« Last Edit: April 30, 2022, 08:32:29 am by Nominal Animal »
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #15 on: April 30, 2022, 08:40:29 am »
If I were you, I'd check the MEMS datasheet whether the write and the read must be a single transaction, or whether it can be done in two separate transactions.  If two separate transactions are allowed, I'd check if there is a maximum interval between the two transactions.

My guess here is that the MEMS device actually wants a single transaction.  In Linux, that means using I2C_RDWR ioctl on the i2c-dev.
For OP, it just happens that when the read transaction follows closely enough to the write transaction, the device somehow just deals with it.

If the MEMS datasheet says a single transaction is needed, then the case is closed: the controller side did the wrong thing.  (To do a write and a read in a single transaction in Linux, you have to use I2C_RDWR ioctl; read() and write() always do separate transactions.)
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 27003
  • Country: nl
    • NCT Developments
Re: i2c bus weirdness on Linux
« Reply #16 on: April 30, 2022, 10:26:20 am »
Well, some good news...  I re-wrote my code using Nominal Animal's suggestion which appeared to be absolutely correct other than that I also needed to add:
#include <linux/i2c.h> since I only had #include <linux/i2c-dev.h> and...
In that case I assume the compiler was also throwing warnings about implicitely defined functions? In such a case C assumes the argument is an integer which can lead to a lot of trouble if the argument has a different size. IOW: if those warning appears, fix them!
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Online Doctorandus_P

  • Super Contributor
  • ***
  • Posts: 3385
  • Country: nl
Re: i2c bus weirdness on Linux
« Reply #17 on: April 30, 2022, 01:50:06 pm »
I've also got a Rigol scope, but I much prefer to use an EUR10 "Saleaeaeeae clone" Logic Analyser with Sigrok / Pulseview.
* The small box hardly takes up any space at all on your desk.
* Short dupont wires are much easier to manage then clumsy big Oscilloscope probes.
* It can sample upto a few MHz, which is plenty for I2C and lots of other protocols (Pulseview has 100+ software decoders).
* Pulseview can add much more useful information by stacking decoders for specific chips on top of I2C.
* PC monitor with many pixels is a better UI then a scope screen for a logic analyzer.
* With 8 channels you can keep track of multiple simple serial busses at the same time.
* It's also great for catching Debug data (for example from a spare serial port), as it preserves timing in relation to other signals.
* With a generic CY7C68013A uC board (and the righte I2C EEprom data) you can even go upto 16 channels (But these generic boards lack input protection).

These EUR10 boxes do have some limitations:
* Sample rate is not too high,
* It's digital only, Especially for I2C it's good to check the RC time constants  with the pullup resistors with your scope to check if this is all right.
* Bus conflicts when different nodes try to drive a signal in opposite directions are not caught.

But overall, especially considering the price point of these things, I reccomend for everybody interested in writing software for uC's  to get (at least) one of these. even before you buy an oscilloscope.
These things cost less then an (overpriced) "original" arduino.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3918
  • Country: gb
Re: i2c bus weirdness on Linux
« Reply #18 on: April 30, 2022, 03:26:02 pm »
Zero Lap-C LAs are better, compatible with Sigrok, and not so expensive. I bought mine for 60 euro.

Ok, 6x expensive than "Saleaeaeeae clone", but it's not in the range of 300 euro required by other professional stuff and it offers more features.

You can buy a DSO with software analysis modules, basically what you see on Pulseview is performed directly by the DSO.

You can also buy a MSO, basically a DSO + LA all in the same tool. Not too bad, in my opinion, and probably my next purchase.

I have a very old Rigol DSO, 2Ch @ 50Mhz which is also too old for the software analysis modules, it's not supported, but newer models have this possibility, for i2c, spi, can, uart, .. and you have to buy a license-key for each sub-module.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3918
  • Country: gb
Re: i2c bus weirdness on Linux
« Reply #19 on: April 30, 2022, 03:28:32 pm »
p.s.
Some i2c-ADC shoot out 18 bit, hence 3 bytes.
Some i2c-MEMS shout out 2 bytes. But some even up to 3 bytes.

You have to check the datasheet here, there are weird beasts out of there :-//
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Offline DavidAlfa

  • Super Contributor
  • ***
  • Posts: 5954
  • Country: es
Re: i2c bus weirdness on Linux
« Reply #20 on: April 30, 2022, 06:59:30 pm »
These EUR10 boxes do have some limitations:
* Sample rate is not too high
The $7 "Saelaeaeae" is 24Msps(8ch) / 12Msps(16ch) waaaay more than required to debug i2c.
I'd only get the $300 one if I really need it, otherwise it's a waste of money.
« Last Edit: April 30, 2022, 07:03:54 pm by DavidAlfa »
Hantek DSO2x1x            Drive        FAQ          DON'T BUY HANTEK! (Aka HALF-MADE)
Stm32 Soldering FW      Forum      Github      Donate
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 27003
  • Country: nl
    • NCT Developments
Re: i2c bus weirdness on Linux
« Reply #21 on: April 30, 2022, 07:06:00 pm »
These EUR10 boxes do have some limitations:
* Sample rate is not too high
24Msps(8ch) / 12Msps(16ch) is more than great to debug i2c. $7 vs $300 is no brainer. Get the $300 only if you really need it, otherwise it's a waste of money.
Bad advice. When dealing with these kind of serial bus problems, always look at the signals in the analog domain first. You wouldn't be the first to get stuck on a problem that stems from the analog domain. And having a feature to decode the serial bus while you are at it, just makes live easier. There is no cheap way out.

Several years ago I was called over to solve a problem. The guy was looking at an RS485 bus (for a device he developed) with a logic analyser and was stuck for 2 weeks already. Hooking the signals up to an oscilloscope revealed the problem instantly and it was fixed within 30 minutes.

In addition: once the digital bus is working, it is much easier to debug communication at a higher level (printing the values in a meaningful form to a serial port for example). So all in all the cheap logic probes are far less useful then you think.
« Last Edit: April 30, 2022, 07:07:46 pm by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 
The following users thanked this post: DiTBho

Offline DavidAlfa

  • Super Contributor
  • ***
  • Posts: 5954
  • Country: es
Re: i2c bus weirdness on Linux
« Reply #22 on: April 30, 2022, 07:22:32 pm »
For the analog domain you only need a single shot on the DSO. All rising/falling edges should be the same, it's impossible that the device or the pullup is faster with some bits and slower with others.
Hantek DSO2x1x            Drive        FAQ          DON'T BUY HANTEK! (Aka HALF-MADE)
Stm32 Soldering FW      Forum      Github      Donate
 

Online Doctorandus_P

  • Super Contributor
  • ***
  • Posts: 3385
  • Country: nl
Re: i2c bus weirdness on Linux
« Reply #23 on: April 30, 2022, 07:49:08 pm »

Bad advice. When dealing with these kind of serial bus problems, always look at the signals in the analog domain first. You wouldn't be the first to get stuck on a problem that stems from the analog domain. And having a feature to decode the serial bus while you are at it, just makes live easier. There is no cheap way out.

Several years ago I was called over to solve a problem. The guy was looking at an RS485 bus (for a device he developed) with a logic analyser and was stuck for 2 weeks already. Hooking the signals up to an oscilloscope revealed the problem instantly and it was fixed within 30 minutes.

I completely agree with this.
If you have an oscilloscope always first verify that you have decent logic levels, and only then go further by decoding the data with a logic analyzer.
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 27003
  • Country: nl
    • NCT Developments
Re: i2c bus weirdness on Linux
« Reply #24 on: April 30, 2022, 08:28:10 pm »

Bad advice. When dealing with these kind of serial bus problems, always look at the signals in the analog domain first. You wouldn't be the first to get stuck on a problem that stems from the analog domain. And having a feature to decode the serial bus while you are at it, just makes live easier. There is no cheap way out.

Several years ago I was called over to solve a problem. The guy was looking at an RS485 bus (for a device he developed) with a logic analyser and was stuck for 2 weeks already. Hooking the signals up to an oscilloscope revealed the problem instantly and it was fixed within 30 minutes.

I completely agree with this.
If you have an oscilloscope always first verify that you have decent logic levels, and only then go further by decoding the data with a logic analyzer.
Why go to the trouble of using a logic analyser if you have decoding functionality in your oscilloscope? Hook-up once, measure once and you have all the info you need. I have a very high end Tektronix logic analyser with a boatload of bells & whistles but it has been years since I used it. The MSO sitting on my bench is just as suitable for 99% of the tasks a logic analyser does.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Online SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14541
  • Country: fr
Re: i2c bus weirdness on Linux
« Reply #25 on: April 30, 2022, 08:45:50 pm »
Using a decent logic analyzer (doesn't even have to be expensive) is still much more comfortable usually. (And a scope doesn't always have protocol decoding functionalities - sometimes comes with expensive options.) While a true scope is often more comfortable to use than a PC-based scope for all other tasks, for logic analysis, I find PC-based solutions beat the crap out of a scope. Having a decent GUI with a large screen and mouse is a lot more productive for this.

I use a DSView Plus for most logic analysis tasks and it's a time saver really.
 

Offline hpmaximTopic starter

  • Regular Contributor
  • *
  • Posts: 132
Re: i2c bus weirdness on Linux
« Reply #26 on: April 30, 2022, 11:06:31 pm »
I thought I had posted a response, but it doesn't appear I did.

I'm a bit confused as to why there is so much consternation over using a Scope.  I'm using a scope, because I have a scope.  While, I realize that a logic analyzer might be the "ideal" tool, given the advanced triggering functionality and decoding on the Rigol I see no real benefit to a logic analyzer over the scope.  I have a 500MHz HP scope, but it's older and I don't use it, because it's bigger, has less memory, has a monochrome display, no USB, less advanced triggers, no decoding etc.  In short, if you are operating below 100MHz, the Rigol is better in pretty much every way imaginable (I wish it had WiFi though).  I didn't buy the scope because I was looking for analyzer, I bought it because I was looking for a newer, more modern scope.  I have plenty of older HP logic analyzers, which despite the fact that they have tons of functionality (I have a 1661 and a 16500B with pretty much every plug in module you can imagine), I consider them boat anchors.  They're big and require a lot of effort to setup and use.

Nominal Animal...  The device in question is an ST LSM6DSM.  Datasheet can be found here: https://www.st.com/resource/en/datasheet/lsm6dsm.pdf

Page 43 states: "The slave address is completed with a Read/Write bit. If the bit is ‘1’ (Read), a repeated START (SR) condition must be issued after the two sub-address bytes; if the bit is ‘0’ (Write) the master will transmit to the slave with direction unchanged."

I'll take it you think that's the operative sentence there... That sending the write followed by a separate read is a violation that the ioctl(I2C_RDWR) corrects, rather than either the I2C_SLAVE or I2C_RDWR were invalid I2C messages?

Side question...  Is there a way to adjust SCL?  I appear to be transmitting at 100kHz, and that actually seems to occupy a lot of time.  Pretty much everything seems to support 400kHz.

FWIW, I've now transmitted approximately 22M samples over the I2C bus without an issue.
 

Offline cv007

  • Frequent Contributor
  • **
  • Posts: 828
Re: i2c bus weirdness on Linux
« Reply #27 on: May 01, 2022, 04:18:56 am »
In reply #13, this is what I was getting at (from the datasheet)-
Quote
If a slave receiver doesn’t acknowledge the slave address (i.e. it is not able to
receive because it is performing some real-time function)...
Since you were not checking the return values in your original post, any failure of the device to ack an address in the read/write transactions along with your code proceeding as if it did, could get you some strange results. For example, a write that was not address ack'd, the following read that succeeds is now returning some 'other' register value if register increment is enabled (default). Or if the read did not address ack, you are now (presumably) reading 'inbuffer' which also would then contain invalid data.

Which could explain-
Quote
The MEMS device has been acting erratically in an intermittent way
Its probably unusual for the mems device to nack the address, but its possible (says the datasheet). If your 'new' code is checking return values, then its not really a 'fair' comparison even though the new code may have solved some other problem.

Your original posted code could be tested with checking the return values, and acting accordingly. This could be a bad 'diagnosis' by me, but is easy enough to test. If wanted.

 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #28 on: May 01, 2022, 03:41:09 pm »
For what it is worth, I think using a scope to check the transitions and levels first is a good approach.  When those are ruled fine, using a cheap logic analyzer with sigrok's Pulseview –– for which you can write your own protocol analyzers in Python, if it does not already have one; it does for I2C –– means you can catch even rare errors by running it continuously for quite some time.  And I suggested this before OP posted the scope traces, showing their scope already has a I2C protocol analyzer.

It's not like there is just one way to do this stuff.

Also, I kinda-sorta assumed hpmaxim did check read() and write() return values, because in Linux, they have to be.  Only when you do not care if the data is read or written at all or if an error occurs, is it okay to not check the return value.  (The ioctl(fd, I2C_RDWR, &op) returns the number of successfully completed messages, not the byte count.)  I sometimes have heated arguments with people who claim it shouldn't be necessary with say ordinary files; I say it always is necessary, because even running out of disk space is something you need to be prepared to deal with (and cannot verify reliably beforehand, TOCTOU).  I hate the idea of a program I'd use just silently ignore an error, losing or garbling my precious data.

The device in question is an ST LSM6DSM.  Datasheet can be found here: https://www.st.com/resource/en/datasheet/lsm6dsm.pdf

Page 43 states: "The slave address is completed with a Read/Write bit. If the bit is ‘1’ (Read), a repeated START (SR) condition must be issued after the two sub-address bytes; if the bit is ‘0’ (Write) the master will transmit to the slave with direction unchanged."

I'll take it you think that's the operative sentence there... That sending the write followed by a separate read is a violation that the ioctl(I2C_RDWR) corrects, rather than either the I2C_SLAVE or I2C_RDWR were invalid I2C messages?
Yes, exactly.

If you think about how the state machine is implemented in the LSM6DSM, it is much easier to require the write and the following read to be a single transaction; otherwise, it would have to save the sub-address bytes somewhere, and deal with partial sub-address bytes and such.

Now, it is a completely separate question whether I'm right or wrong.  As a honest uncle bumblefug hobbyist, I'm often wrong.
(I love being pointed out exactly how I'm wrong, because that's the best opportunity to learn.  I only get irate, when I'm told I'm wrong, but not exactly why or how.  I can't learn anything from that, just wastes my time forcing me to re-examine what I said..  and as nctnico recently noted elsewhere, I have my own pitfalls I can get stuck into.)

What cv007 wrote does apply, I believe.  It's just that because I believe the separate transactions produces unreliable results in all cases anyway, I focused on the I2C_RDWR ioctl instead.  (Logic being, it is of tertiary interest at best as to what kind of "data" one obtains via the erroneous approach.  Make sure the approach itself is correct first, then deal with any coding details involved.)

The reason I posted my own interpretations of the traces is so that if my understanding is wrong, it would be easier for others to point it out.

Is there a way to adjust SCL?  I appear to be transmitting at 100kHz, and that actually seems to occupy a lot of time.  Pretty much everything seems to support 400kHz.
Yes, by modifying the Device Tree.  In your case, it would be the &i2c1 section, clock-frequency parameter (defaults to 100000).

I believe you have a /boot/dtbs/sun7i-a20-cubieboard2.dtb file.
You copy this somewhere for backup purposes, then run
    dtc -I dtb -O dts /boot/dtbs/sun7i-a20-cubieboard2.dtb > sun7i-a20-cubieboard2.dts
to decompile (the "device tree blob" back into "device tree source").
Edit it in your favourite editor (vim, emacs, nano), changing the clock-frequency = <100000>; to clock-frequency = <400000>; in the &i2c1 section, then run
    dtc -I dts -O dtb sun71-a20-cubieboard2.dts /boot/dtbs/sun71-a20-cubieboard2.dtb
to compile the source back to blob at the expected place.  Reboot, and /dev/i2c-1 should run at 400 kHz.

Before you ask: No, I do not believe there is a way to change it programmatically.  If you consider the typical use cases of single-board computers (SBCs), it is typically safer to choose the I2C bus clocks only once during boot, preferably as early as possible, and not trust individual applications or kernel I2C-using code to do it correctly.  It is easy enough to modify the device tree blob instead.
« Last Edit: May 01, 2022, 03:43:40 pm by Nominal Animal »
 
The following users thanked this post: DiTBho

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3918
  • Country: gb
Re: i2c bus weirdness on Linux
« Reply #29 on: May 01, 2022, 04:02:02 pm »
I do not believe there is a way to change it programmatically.  If you consider the typical use cases of single-board computers (SBCs), it is typically safer to choose the I2C bus clocks only once during boot, preferably as early as possible, and not trust individual applications or kernel I2C-using code to do it correctly.  It is easy enough to modify the device tree blob instead.

that's what I meant in my previous post, when I say that DTS is so problematic and annoying with things like SPI and i2c that I prefer to implement these functionality in an MPU and then attach the MPU to the SoC over a serial line.

Code: [Select]
._____________
|             |
|     SoC     |
| ARM64-LE-HF |
|  Allwinner  |
|             |
|  /dev/ttyS0 |==== serial line0 ==== console
|     ...     |                        _______
|             |                       |       |
|  /dev/ttyS2 |==== serial line2 ==== |  MPU  | ==== i2c IMU
|_____________|                       |_______|


For me, if I want to change the clock frequency of the SPI master, or of the i2c master, what I have to do is issuing a command over the serial

echo "clock=400Khz" > /dev/ttyS2
done  :D

Of course, you have to implement for the MPU the specific i2c driver for the device you want to use, while Linux offers a lot of stuff already done, but dealing with an MPU is (for me) easier to verify that things are correctly working because you don't have to deal with all the extra-complexity introduced by the Linux kernel + userspace.

The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 
The following users thanked this post: Nominal Animal

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6308
  • Country: fi
    • My home page and email address
Re: i2c bus weirdness on Linux
« Reply #30 on: May 01, 2022, 04:37:01 pm »
that's what I meant in my previous post, when I say that DTS is so problematic and annoying with things like SPI and i2c that I prefer to implement these functionality in an MPU and then attach the MPU to the SoC over a serial line.
For non-built-in things, yes.  For built-in things, like the laptop I'm typing this on (HP EliteBook 840 G4) having a built-in accelerometer, an ST LIS3LV02DL (connected via SPI or I2C), that the Linux kernel exports as a simple input event device, the DTS is really nice.  Everything is done in-kernel (in this laptop case, as directed by ACPI tables: the x86-64 BIOS equivalent, more or less, of Linux-specific Device Tree stuff.)

It's a compromise, for sure.

For development, using a dedicated microcontroller is much faster.  So much so that if I were to develop something like an accelerometer for a laptop, I'd for sure do the very initial test prototype using an MCU.  I really like using Teensy microcontrollers and ATmega32U4-based "pro micro" clones for such, because they have native USB, lots of libraries and support, and if something like USB Serial or USB Human Interface Device suits the use case, I'll just use that, and not have to write any OS support code.  (My first microcontroller project was an arcade-style joystick and buttons embedded in a large birch plank.)

Even there, having a scope to check (especially whether the I2C pullup-pulldown resistors are correctly sized), and some kind of logic analyzer to examine the actual contents ("long term", over millions of transfers), makes a lot of sense.  In my opinion, it even tends to be fun; especially if you automate the uninteresting parts.  Maybe a 28BYJ-48 stepper with an ULN2003 driver to rock the sensor according to input provided by a test program, which also records the accelerometer data.  Maybe stick a RasPi camera module to take pictures, but store only those that turn out to be of interesting cases.

(On Youtube, Matthias Wandel often does various sorts of el-cheapo measurement contraptions like this.  Most recent one (of this writing) is comparing construction adhesives for woodworking [when your shop is too cold to use normal wood glue].)
 
The following users thanked this post: DiTBho

Offline hpmaximTopic starter

  • Regular Contributor
  • *
  • Posts: 132
Re: i2c bus weirdness on Linux
« Reply #31 on: May 01, 2022, 09:44:56 pm »

Quote
The MEMS device has been acting erratically in an intermittent way
Its probably unusual for the mems device to nack the address, but its possible (says the datasheet). If your 'new' code is checking return values, then its not really a 'fair' comparison even though the new code may have solved some other problem.

Your original posted code could be tested with checking the return values, and acting accordingly. This could be a bad 'diagnosis' by me, but is easy enough to test. If wanted.

The code I initially posted was a highly simplified version of what I was actually doing, but I think it contained the operative issue, and Nominal Animal correctly identified the issue that I should have been using I2C_RDWR not I2C_SLAVE.  I'm using test code which I was constantly tweaking...  At different times I did check or didn't check the results, however, my experience has been it either works or it doesn't, in terms of the return value.  That is whenever I have checked the return value of write() or read() if I was getting data out, the results were always correct -- even when I saw the erratic behavior.  This is a red herring.
« Last Edit: May 01, 2022, 09:47:48 pm by hpmaxim »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf