I designed a board containing a number of MEMS sensors (LSM6DSM, LPS22HH, etc), and had it manufactured by PCBWay.
The board is intended to hook up to a Single Board embedded Linux machine via an i2c bus. The board also incorporates an ESP32S2, which at this time is not being used. There are actually two i2c busses, one internal to the board, and one between the ESP32 and the Linux machine, however, the busses are shorted together so that the Linux machine can directly talk to the sensors.
From what I can tell, the sensors seem to be working okay, except the LSM6DSM (tri-axis accelerometer/gyro) is acting... funny. I'm using it in continuous FIFO mode, at 108Hz, with no decimation. I'm configuring it as follows:
{0x12,0x44}, // CTRL3_C set to BDU + IF_INC
{0x08,0x09}, // FIFO_CTRL3 set to no decimation (accel+gyro)
{0x0a,0x2e}, // FIFO_CTRL5 set to continuous mode, 104 Hz (accel+gyro)
{0x10,0x40}, // CTRL1_XL set to 104Hz, +/- 2g, ODR/2
{0x11,0x42}, // CTRL2_G set to 104Hz, +/- 125 dps
{0x14,0x60}}; // CTRL5_C set to rounding for accel+gyro
Every 150ms, I run a function that does the following:
read FIFO_STATUS1/2 (0x3a/0x3b) to get the num_samples
for (i=0;i<num_samples;++i) {
read FIFO_STATUS3 (0x3c) to get the FIFO_PATTERN
read FIFO_DATAOUT_L/H(0x3e/0x3f) to get data
printf ("%d %d %d\n",i,FIFO_PATTERN,data);
}
What should be happening is that the the FIFO_PATTERN should be incrementing from 0 to 5 and then returning to 0. Pattern 5 should be Accel in Z direction (i.e. 1G, or ~16384).
This is what usually happens, but if you run the program long enough, you'll see FIFO_PATTERN skip to a different value (sometimes not even between 0 and 5). Skips are typically either jump by one in the pattern, or a completely random number (outside the 0-5 range). When it skips to a random number, it'll keep giving random numbers for maybe 20 samples, and then get back to 0-5. When it skips to a different value that is in 0-5, it returns to the increment by 1 pattern, but it is no longer synchronized with the data.
I've also seen data skipped. So for example, you should see 16384 every 6 samples, but sometimes it'll miss 1 and you'll see it 11 samples later. So both the data and pattern can get desynchronized. However, it is interesting to note that when the data skips, it is ALWAYS by a complete sample (16-bits), not 8-bits.
The behavior is repeatable but somewhat random (i.e. you may have to run the program for 20 seconds, or 5 minutes, before it happens, but it will happen). I had four boards built up, they all exhibit the same behavior. The LPS22 seems to be working okay, although I haven't tested it nearly as extensively. The sensor is giving about 560 uG's of noise, which is less than the quoted typical number of ~750 uG's at 108Hz, and I went to great pains to keep the power supplies clean.
Sending a software reset (0x01 to address 0x12) seems to fix it (temporarily). I also tried an experiment where I sent a reset, initialized everything and read for a few seconds, stopped, and then restarted many minutes later (repeatedly) for a few seconds. Every time, it worked fine. In other words, the problem appears to be occurring as a result of reading or somehow interacting with the LSM6DSM, it doesn't just get screwed up on its own.
I tried looking at the i2c bus on my scope (DS1054Z), there are occasional undershoots to about -1.1V on falling edge transitions of the data line, and some (+/-150mV) on cross-talk, but generally the signal is pretty clean on rising edges of the clock. When using the scope in decode mode, I am seeing exactly what I think I am sending (and receiving) on the scope. Although the scope produces a "?" (failure to acknowledge) after the last byte of each read -- which I think is expected behavior.
FWIW, I realize that the way I'm reading the FIFO out is unusual. I was initially reading the number of samples, the FIFO_PATTERN, and then reading out all the data in the buffer in one go, i.e.
Read 2 bytes at 0x3a to get num_samples
Read 1 byte 0x3c to get FIFO_PATTERN
Read num_samples * 2 bytes at 0x3e to get data
Which is obviously *WAY* more efficient... However, this exhibited the same problem, so I am now reading them individually to try to improve my ability to debug.
I really appreciate you taking the time to write such a thoughtful reply. I'm sure my reply to you is going to be disappointing.
Couple of quick thoughts somewhat tangent to your particular code / board:
* Have you tried running your code talking to a LSM6DSM development board vs. your target sensor board?
* Have you tried running your code on another MCU / SOC platform Arduino or some MCU devkit or whatever?
* Have you checked to see if you have any race conditions if your code or the IO is multi-threaded or something?
* Since this is an embedded LINUX machine do you know what would happen correctly or incorrectly if sometimes the LINUX machine running your I2C acquisition code stalled the running of your process by X amount of time where X can be NNN microseconds or I suppose even several milliseconds sometimes maybe depending on your system. If you're trying to keep up with real time streaming data going into the FIFO and for some reason you don't, what is the response going to be, and how do you recover from that condition per. the sensor data sheet?
* That is kind of a huge undershoot to -1.1V on the falling edge. Maybe it is partly just because of the scope wiring / probe vs. ground wiring etc. But I guess I would look at your board's layout to see if something is really not so good like crossing a split in the ground plane or other separation of the signal path and the return path.
* Also several of the common I2C sensor chips also support SPI and some of them have some really funky multiplexed pin usage between I2C mode and SPI mode and depending on the signal patterns you apply to the pins it might be possible to switch to the wrong mode. Double check all of the I2C chips' data sheets connected to your I2C bus and verify that there are no incorrect levels applied to their control pins, that your I2C addresses are all non-overlapping, that some other chip isn't going to be in some undefined or bad (SPI) mode at the same time the LSM6DSM is getting data read from it and cause a bus conflict or something.
* Also re check the way you are doing timing if there is any aspect of that in your code vs. the chip control. Try it down at 100 kHz I2C bus rate or similarly low rate and generally be conservative on delays and timings.
* Maybe disconnect the other ICs from the I2C bus while you talk to the LSM6DSM if you use your target board or use a LSM6DSM dev board or something and see if there is still a problem.
Re: using the DS1054 or other logic analyzer -- maybe try to record the sequence of I/Os for a long pattern or figure out a triggering mode and signal you can devise to detect when the problem happens and then look at the data span for NN milliseconds before the "the error happened" trigger point to see if anything looks off electrically or in data flow etc. Maybe if your host can turn on a spare output pin to be used as the scope trigger or something to know when the error detection happened if it takes a long sequence to get to the problem.
Maybe see if you can find an error talking to I2C on the LSM6DSM other than FIFO related -- maybe just loop continually reading some constant register with a known correct value like a CHIP ID or something and see if you can do that 100k times or whatever with zero errors or if every now and then something bad happens. Similar idea for a writable and read-back able register -- write, then read in a loop.
1) I don't have a development board, so haven't tried anything but the target.
2) The target board is a formfactor and mostly pin compatible replacement for another target sensorboard which had different sensors (and lacked an ESP32). I suppose theoretically I could hook up a Raspberry Pi and try to wire between them, but then I'm not sure what the point is.
3/4) I can't guarantee there is no race condition, I have actually seen the machine just go out to lunch for as much as a few hundred milliseconds under certain circumstances (which didn't occur). However, I'm not sure that would affect I2C behavior since that's presumably at a device driver level which may have completely different priority.
I think it's important to remember, I actually recorded the I2C bus on the scope during a failure and looked at in the protocol decoder. The decoder saw nothing unusual (except perhaps the lack of a acknowledge on the last receive byte). I saw nothing unusual looking at the traces (admittedly, I'm not super knowledgeable about I2C), and the decoder showed valid writes of everything I wrote, and my reads matched that of the decoder.
5) I agree that -1.1V is a huge amount of overshoot. In fact, I find it hard to believe there are ESD diodes that would kick in if a digital input goes below ground, seems hard to imagine it dropping more than 800mV. I think the layout is pretty good. I didn't actually do the layout, but it's a 4 layer board with a dedicated ground plane and a lot of stitching to it. Pretty much all grounds everywhere were tied directly to the ground plane with 1 or 2 vias. To monitor the bus I soldered 2-3" 30 gauge wires to the jumpers which shorted the buses together. The ground was connected to the ground pad at the bottom of the ESP32 footprint. The pad was about 30mm from the jumpers. I tend to believe this is measurement error.
6) You're right about most of your pinout comments. I2C mode is selected by tieing the "CS" pin high, and the "SD0/SA0" pin low. I can confirm that is the case in the schematic (and layout). However, I am trying to operate in "Mode 1", and according to a table on page 53 datasheet, the "SDX" and "SCX" pins should be tied to ground or VDDIO -- and wouldn't you know it, they're floating... And of course, they're under the chip (it's an LGA).
7) i2c seems to be running at 100kHz. All timing seems to be within spec.
Can't disconnect the other parts easily. They're mostly LGA.
9) Already did that... Not a great solution, but I just triggered on the address (0x6a), detected the fault and halted the program, with a sufficiently long capture buffer, I can see the final transaction and numerous transactions prior. Like I said, it looks fine.
10) There is a WHO_AM_I register. I could try reading that a million times, ensuring I get back the right answer, and then read from the FIFO and make sure the data is still valid. Whether it succeeds or fails, I'm not sure how much it would tell me...
If I had to guess, I'd say the floating pins are the wildcard here.
Let me save you a ton of time and frustration: DUMP THE LSM6 SERIES IMMEDIATELY. Save yourself. Learn from our experience. No reason to relive the pain of others.
Back in 2019 we did exhaustive verification of the LSM6 series. We found the same erratic behavior that you describe. The symptoms vary a bit but the erratic, unpredictable nature is the same. The part would generate errors, or utterly invalid data, or simply become unresponsive and could not be revived without a power cycle. We have an MCU-controlled PFET controlling power to our MEMS devices on every board to this day as a result of the LSM6 series against the threat of ever finding another MEMS device that behaves the same way. In other words, the LSM6 nightmare continues to haunt our designs even now.
We did identify a few conditions which we kinda sorta thought maybe were reproduceable, including:
* If the devices were held motionless (no changing X/Y/Z data) they appeared to lockup sooner. Keeping them in even slight motion appeared to "help" them for some reason.
* The ODR rate seemed to have an effect. For example, 12.5Hz seemed more reliable than 208Hz.
* Trying to use the digital filters made the problem(s) worse. The more post-processing you configured on-chip, the more likely it was to lock up. Best behavior resulted from simply taking the raw data from the sensor and disabling ALL internal processing - but even this didn't make it reliable, just less unreliable.
The effects were often bizarre. Sometimes we would continue to get interrupts but the resulting output data was all 0's. Sometimes all output would cease (no interrupts, no MISO state changes, nothing) but it would still respond to an SPI-commanded soft reset and resume operation. But other times it became completely unresponsive and the only recourse was to fully cycle power to the chip... and you must also make certain you ground all I/O lines to/from the chip during this power-off period since the die consumes so little power it can retain state just from current available on its I/O pins. Yes, we did do that and Yes, it did work in terms of resetting the chip and No, we did not enjoy it.
Modern MEMS chips basically have an integrated MCU and firmware. (Don't believe me? Check out the latest Bosch Sensortec MEMS, they've dropped all pretense of being standalone and now require you to upload an ~8KB firmware image to them at every powerup! They don't have any default, standalone operation at all. WTF?!?) Our best theory was that the MCU portion of the LSM6 could get overloaded (if asked to do too much DSP) and/or had some lockup conditions affecting the SPI interface. A second theory was that the SPI hardware itself had flaws, maybe the core was still running but it couldn't communicate.
No matter. The parts are utterly unsuitable for anything but a child's toy where random behavior might be interpreted as a feature. Junk them. Run away. Don't waste more of your time nor money, it will only get more frustrating. Just reading back over my notes to type this response has me twitching from the memories.
I've not used this particular device, but I have experience of other similar MEMS accelerometers.
I've found they can be very fragile, easily damaged during the soldering process. They may be OK in volume production, but in a lab environment being reflowed by hand with hot air, they'd randomly fail.
I've found they can be very fragile, easily damaged during the soldering process. They may be OK in volume production, but in a lab environment being reflowed by hand with hot air, they'd randomly fail.
They also are very sensitive to 1) temperature (which is why so many of them have an integrated thermal sensor) and 2) mechanical stress. The latter seems especially troublesome in the Z axis and can be caused by stresses frozen into the part by soldering. I can imagine that reflow, where the part is sort of "floating free", would yield fewer stresses than hand soldering where you might be holding the part in place with tweezers, etc.
MEMS can be wonderful but they require careful handling and you must find a reliable specific part. We've had great luck with some parts from ST while others from ST were dismal failures. Bosch was wonderful for a while, we consumed tens of thousands of their BMI160, but their later parts have been both garbage (by design, see above!) and unobtainium (see the countless threads and comments on Bosch's recent behavior). That's why we did such a thorough qualification of the LSM6, because basically every time you need a new part you must go through the entire process of researching all available parts from all potential vendors, identifying candidates, bringing in samples, and qualifying until you find one or more that seem reliable. It's an arduous process, and the final rotten part is that MEMS manufacturers retire those chips faster than any other parts I've ever seen. I've seen total lifetimes from first samples to EOL/NFD of 24 months.
Sorry I haven't written back earlier...
I'm just going to generically respond to a few things and give updates.
I spoke to ST (in fact, on going conversation), they claimed the floating SCx/SDx lines shouldn't cause a problem. They eventually pointed out that there is a programmable pull-up (i.e. I write a bit to a register) the pulls them up, so I tried that. There was no difference.
A number of people have mentioned how "fragile" MEMS are. They may be, but I'm seeing no evidence that the MEMS portion is failing, it appears to be the digital logic. I haven't seen any ridiculous data out of the unit. What I've seen seems to be an inability to keep track of the location in the FIFO buffer. When it gives me a wrong value, it appears to have simply jumped around in the FIFO. Again, the FIFO_PATTERN which provides information on what sample I'm reading should be counting between 0 and 5 and then back to 0, but sometimes it jumps one. That's not a MEMS issue. The only potential thing that I can think would be causing this is some sort of glitch on the input or the power supply, but that seems exceedingly unlikely. I am continuously operating my LPS22HH on the same board for far longer. They completely share the inputs. The power supply design is identical, but they are physically different parts. Now, perhaps the parts are bad, but I have 4 different boards, and I've tested 3 and all are exhibiting the same problem on the LSM6DSM. What are the chances that three boards are exhibiting the exact same problem on the same part, but none of the others?
It's really hard for me to wrap my head around this. This appears to be a fault in their digital logic. Maybe a race condition? What are the chances of that? Seriously... Digital logic is simulated extensively, it goes through STA to ensure timing is what you expect, etc, etc.
IDEngineer, you aren't kidding about fast retirement. I started off looking at the LPS22HB, only to find out that their new hot chip was the LPS22HH. And after getting the parts back from PCBWay, I found out they're now pushing the LPS22DF. The good news is they are pin compatible and almost software compatible so it's a pretty painless transition. The guy at ST recommended switching to the LSM6DSR for new designs from my LSM6DSM.
I wrote some code to exercise the board and show the error, and was able to create a log of it. I submitted this to ST, and their AE wrote similar code (they showed it to me), and ran for longer than I could run without an error. I just got (yesterday) and LSM6DSL "eval board" from eBay, so I'm going to try to wire it up in place of my board and see if I can replicate it with that.
I chose the LSM6DSM after discussions with Kris Winer who at least claims to have extensively tested a number of sensors. He recommended the LPS22HH, LSM6DSM and MMC5983MA.
What are the chances that three boards are exhibiting the exact same problem on the same part, but none of the others? It's really hard for me to wrap my head around this. This appears to be a fault in their digital logic. Maybe a race condition?
You're rediscovering the digital logic problems I documented above. There is definitely something wrong in the LSM6 series. My vote goes with the internal "MCU-like" logic block, or the firmware that runs it. It definitely appears that you can overload it and it loses state. Sometimes (as I doc'd above) it continues to respond to input (MOSI) data, but other times it's simply locked up and the only fix is a total power cycle.
The one part I really don't understand is why motion delays the effect. If it's purely logic, with no MEMS aspect, then motion shouldn't matter. But it definitely does, we were able to delay the onset of the problem by keeping the device(s) in small constant motion. Perhaps there's a problem with internal digital filtering where a long string of unchanging values causes a rollover in the filtering math... I don't know. And ST isn't talking.
I have worked with single LSM6DS3's in the past. Without wanting to deny the previous answers about their quality, i always had a quiet pleasant time with them.
In some development projects i had multiple breakout boards connected with ridiculous long i2c lines (10m or so), which was working barely at the edge of i2c spec. The project has finished some years ago, but the system is still up, running and delivering plausible values
I remember also having some trouble with the FIFO and jumping or repeating values. I do not remember every detail of the problem, but i ended up with the following implementation which was a mixture of a "solution" and performance optimisation for my use case. The following example assumes that only the accelerometer is enabled, so you get 3 fifo values, Accel_x, Accel_y, Accel_z, for one sensor sample.
1. Waiting for the FIFO's Storage to be almost full
2. Stopp all Sensors that write to FIFO
3. Flush FIFO until next FIFO Type is 0
(Dependant on your sensor configuration, you can check if your FIFO Type changes to 0 within a maximum value of flushed fifo samples, e.G. 3 in this case. If this condition isn't met, there is something wrong with the sensor and you can throw an exception)
4. Check, how many samples are still stored. Integer Division of this value by count of fifo values you expect from every sensor sample (e.G. 3 in this case)
6. For-Loop for the Result of Last-Operation while always reading 3 values from fifo. For better performance, from this point on i do not check the fifo type anymore. While checking the code of course i had some debug output to verify the FIFO Type is always as expected.
7. Optionally, after reading all values check if the current FIFO Type is what you expect. Like 3, if this isn't met, something is wrong and you may throw an exception and discard your last read
8. Finally, and in my experience this seemed to be crucial: Always drain the FIFO to 0 saved samples and reset it properly according ST's Application Note.
9. Restart FIFO and Sensors and jump back to 1.
This implementation of course only allows you to get chunks of samples, but not a continuous stream. I do not know if this may be sufficient for your application.