Confusion about sensor data rates with Arduino Code

[LoveLaika]

I'm using the Arduino Portenta to read from a sensor, the LIS3MDLmagnetometer. This is all through the Arduino software, and I'm using a QWIIC converter cable to connect from the Portenta to the sensor through I2C. I'm reading data, so there's nothing wrong with the wiring. The problems that I'm facing have to do with the data rate and the timestamp.

According to Adafruit's tutorial, you can set the LIS3MDL's data rate from 0.625 Hz to 1 kHz. This is supported with ST's datasheet. However, changing the data rate doesn't seem to correspond with an increase in data points. For instance, when running my code, after 11 seconds, I only get 6461 data points, and that is when I set the LIS3MDL data rate to 1 kHz. By my calculations, assuming a data point every cycle, I should get ~11000 data points after 11 seconds. Likewise, setting it to lower frequencies results in the same behavior. Why am I getting less data than expected?

Regarding the timestamp, I'm using Arduino's millis() function to get a time stamp of sorts (based on how many milliseconds have elapsed since the program has started), but what is this function based on? Is it tied to the Arduino processor, or is it tied to something else? Could the LIS3MDL have some sort of time feature that is screwing up my time stamp? But that shouldn't be possible since my time stamp doesn't involve reading from the sensor...

If it helps, below is code that I'm using to diagnose coding with the sensor, though this isn't my final code. It's more or less no differerent than the code that Adafruit provides.

Code: [Select]

#include <Wire.h>
#include <Adafruit_LIS3MDL.h>
#include <Adafruit_Sensor.h>

Adafruit_LIS3MDL lis3mdl;
#define LIS3MDL_CLK 13
#define LIS3MDL_MISO 12
#define LIS3MDL_MOSI 11
#define LIS3MDL_CS 10

// Set up Function
void setup(void) {
  Serial.begin(115200);
  while (!Serial) delay(10);     // will pause Zero, Leonardo, etc until serial console opens
  Serial.println("Adafruit LIS3MDL test!");
 
  // Try to initialize!
  if (! lis3mdl.begin_I2C()) {          // hardware I2C mode, can pass in address & alt Wire
    Serial.println("Failed to find LIS3MDL chip");
    while (1) { delay(10); }
  }
  Serial.println("LIS3MDL Found!");

  // Set LIS3MDL Performance mode to HIGH performance
  lis3mdl.setPerformanceMode(LIS3MDL_HIGHMODE);

  // Set LIS3MDL to operate continuously
  lis3mdl.setOperationMode(LIS3MDL_CONTINUOUSMODE);

  // Set LIS3MDL Data Rate to 1 kHz
  lis3mdl.setDataRate(LIS3MDL_DATARATE_1000_HZ);

  // Set LIS3MDL Gauss Range to 4 Gauss
  lis3mdl.setRange(LIS3MDL_RANGE_4_GAUSS);

  Serial.println("Set time to 0 microseconds");
  Serial.println("Portenta H7");
  lis3mdl.setIntThreshold(500);
  lis3mdl.configInterrupt(false, false, true, // enable z axis
                          true, // polarity
                          false, // don't latch
                          true); // enabled!
}

// Global Variables
int ne = 0;                     // Number of events that have occurred
unsigned long timet;            // Var. for the current timestamp

void loop() {
 
  // Increase the number of events by 1
  // and get the # of milliseconds since the program ran
  ne = ne + 1;
  timet = millis();



  // Print the current event and timestamp
  Serial.print(ne);
  Serial.print(",");
  Serial.print(timet);
  Serial.print("~~");
 
  lis3mdl.read();

  /* Get a new sensor event, normalized to uTesla */
  sensors_event_t event;
  lis3mdl.getEvent(&event);

  /* Display the results (magnetic field is measured in uTesla) */
  Serial.print("\tX: ");
  Serial.print(event.magnetic.x);
  Serial.print(",");
  Serial.print(" \tY: ");
  Serial.print(event.magnetic.y);
  Serial.print(",");
  Serial.print(" \tZ: ");
  Serial.print(event.magnetic.z);

  delay(0);
  Serial.println();
}

[iMo]

The millis() comes from MCU clock.
One thing is the LIS datarate, the other is the actual I2C speed set.
With 100kHz I2C (for example) you limit the LIS sensor's datarate by its I2C speed..

[LoveLaika]

Datasheet says the IC supports standard I2C at 100 kHz or Fast I2C at 400 kHz. Shouldn't that be enough if the sensor is calibrated for 1 kHz?

[Kasper]

Try commenting out some parts in your loop to see if it'll go any faster.

When you reduce the data rate does the data change more often than you expect or does it just print more often than you expect, the same values multiple times in a row?

[perieanuo]

they say here https://circuitpython.readthedocs.io/projects/lis3mdl/en/latest/api.html : performance mode is low power for 1000Hz ("RATE_1000_HZ   1000 HZ ( Sets PerformanceMode to MODE_LOW_POWER)")
not sure if events no=data points no, maybe you have duplicated data points because you print events and no each data of interest from the sensor (generally, i understand event as change in sensor data, so you can have 100 sensor data values with only 3 different values so maybe you have 3 events ). maybe i'm wrong, maybe no.
insert in your code some sensor values read with the approach they recommend in the library simplest examples(mag_x, mag_y, mag_z = sensor.magnetic)
for every thing like this, you need to know the library by heart first
change delay always to 10 when testing your code, some devices are sensitive to how quick the interrogations occur, that's good practice before you have the right data with your desired timings

always try to shorten your code, instead

Code: [Select]

Serial.print(event.magnetic.z);
Serial.println(); just use

Code: [Select]

Serial.println(event.magnetic.z);here https://github.com/adafruit/Adafruit_LIS3MDL/blob/master/examples/lis3mdl_demo/lis3mdl_demo.ino they gave example of reading sensor data OR events, that must read some bells 

[iMo]

Datasheet says the IC supports standard I2C at 100 kHz or Fast I2C at 400 kHz. Shouldn't that be enough if the sensor is calibrated for 1 kHz?

How long does it take to transfer 1000 data off the LIS with I2C set to 100kHz?

[LoveLaika]

Try commenting out some parts in your loop to see if it'll go any faster.

When you reduce the data rate does the data change more often than you expect or does it just print more often than you expect, the same values multiple times in a row?

The data doesn't change (then again, I'm not changing it by running a magnet over the sensor). However, I do notice that as I reduce the data rate, I'm getting approximately the same amount of datapoints in a fixed time interval (~10 minutes) as I do when the sensor's data rate is set for 1 kHz.

[LoveLaika]

they say here https://circuitpython.readthedocs.io/projects/lis3mdl/en/latest/api.html : performance mode is low power for 1000Hz ("RATE_1000_HZ   1000 HZ ( Sets PerformanceMode to MODE_LOW_POWER)")
not sure if events no=data points no, maybe you have duplicated data points because you print events and no each data of interest from the sensor (generally, i understand event as change in sensor data, so you can have 100 sensor data values with only 3 different values so maybe you have 3 events ). maybe i'm wrong, maybe no.
insert in your code some sensor values read with the approach they recommend in the library simplest examples(mag_x, mag_y, mag_z = sensor.magnetic)
for every thing like this, you need to know the library by heart first
change delay always to 10 when testing your code, some devices are sensitive to how quick the interrogations occur, that's good practice before you have the right data with your desired timings

always try to shorten your code, instead

Code: [Select]

Serial.print(event.magnetic.z);
Serial.println(); just use

Code: [Select]

Serial.println(event.magnetic.z);here https://github.com/adafruit/Adafruit_LIS3MDL/blob/master/examples/lis3mdl_demo/lis3mdl_demo.ino they gave example of reading sensor data OR events, that must read some bells 

Thanks. My code was based around Adafruit's example code. I'll dig deeper into their libraries and see how they function. Perhaps I can use their library functions in a way that can reduce some overhead.

[radiolistener]

According to your code, you're using serial interface at speed 115200 to transfer samples to the PC.

Speed 115200 is about max 115200/10 = 11520 bytes per second. As you understand, 11500 letters per second is not enough speed to transfer 11000 samples per second in text format, which also includes a lot of other garbage information (like timestamp, variable names, commas and new line separators). 

[Kasper]

they say here https://circuitpython.readthedocs.io/projects/lis3mdl/en/latest/api.html : performance mode is low power for 1000Hz ("RATE_1000_HZ   1000 HZ ( Sets PerformanceMode to MODE_LOW_POWER)")
not sure if events no=data points no, maybe you have duplicated data points because you print events and no each data of interest from the sensor (generally, i understand event as change in sensor data, so you can have 100 sensor data values with only 3 different values so maybe you have 3 events ). maybe i'm wrong, maybe no.
insert in your code some sensor values read with the approach they recommend in the library simplest examples(mag_x, mag_y, mag_z = sensor.magnetic)
for every thing like this, you need to know the library by heart first
change delay always to 10 when testing your code, some devices are sensitive to how quick the interrogations occur, that's good practice before you have the right data with your desired timings

always try to shorten your code, instead

Code: [Select]

Serial.print(event.magnetic.z);
Serial.println(); just use

Code: [Select]

Serial.println(event.magnetic.z);here https://github.com/adafruit/Adafruit_LIS3MDL/blob/master/examples/lis3mdl_demo/lis3mdl_demo.ino they gave example of reading sensor data OR events, that must read some bells 

Thanks. My code was based around Adafruit's example code. I'll dig deeper into their libraries and see how they function. Perhaps I can use their library functions in a way that can reduce some overhead.

Adafruit examples can be surprisngly bad, not just poor practices but they have functions that do not work.  That being said, that's not the first thing I'd try, it can be time consumming. I also wouldn't put a 10ms delay in your loop and expect it to run any faster than 100Hz.

I'd start with a quick check of removing some printing and seeing if it runs faster.  It seems what you think is data rate of sensor is actually data rate of your loop aka how long it takes your loop to run.  As radiolistener points out, serial.print takes time and you are doing a lot of it.

[LoveLaika]

They really went out of their way to write their own libraries. Having a look at their library for the LIS3MDL, it revolves around their own libraries, and that just feels like a deep rabbit hole. I did see a where someone demonstrated an I2C sensor, and they wrote without using Adafruit libraries. It's certainly harder, but I wonder if doing it this way will be beneficial in the long run (or at the very least, I write my own functions like this).

Thanks for letting me know about the delay. I'll try and see what happens when I remove it, but since it's in the setup function and not in the loop function (delay is 0 in loop function), shouldn't that be okay?

So, if Serial.print() is so slow, what would be the best way of reading/storing data? If you can't read it out to console, do you normally write it to a file or something and have a separate program do stuff with it? What do people usually do for real-time events?

[rstofer]

Make a short loop, toggle an LED at the start of each pass.  Get rid of all print statements and every other function call that isn't necessary for taking a reading.

Now you can just measure the LED on time and off time to figure out how long a reading takes.  An oscilloscope would work well here...

Or, make your measurements in a tight loop and put the values in an array.  After you are done sampling, print the array contents.  You can time the entire sample loop with your event time.

In other words, get rid of all the code that isn't involved with measurements.

[perieanuo]

they say here https://circuitpython.readthedocs.io/projects/lis3mdl/en/latest/api.html : performance mode is low power for 1000Hz ("RATE_1000_HZ   1000 HZ ( Sets PerformanceMode to MODE_LOW_POWER)")

Thanks. My code was based around Adafruit's example code. I'll dig deeper into their libraries and see how they function. Perhaps I can use their library functions in a way that can reduce some overhead.

Adafruit examples can be surprisngly bad, not just poor practices but they have functions that do not work.  That being said, that's not the first thing I'd try, it can be time consumming. I also wouldn't put a 10ms delay in your loop and expect it to run any faster than 100Hz.

I'd start with a quick check of removing some printing and seeing if it runs faster.  It seems what you think is data rate of sensor is actually data rate of your loop aka how long it takes your loop to run.  As radiolistener points out, serial.print takes time and you are doing a lot of it.

well, the library examples work, for beginner's needs it ok, everyone starts from that. those libs are not perfect, but they work 99% and the beginner can't just write his 'library' or function without some practice (this is our case here).
and maybe you don't get the idea, the OP doesn't understand why he can't read enough points. understand this, if you interrogate any sensor in the world 6000 times, you expect 6000 data points, REGARDLESS of interrogation speed 100k, 400k, 100G or whatever (or the sum between the validated data points and missed sensor interrogations due to any kind of read error).
re-read my post with this in mind.
but someone noticed using serialprint slows down not the data acquisition, but data export to com port. fair point. maybe sensor data should be inserted in a matrix and counted, then printed on serial port or whatever the op wants.
so i still afirm op should interrogate for sensor values (this is very important) and not for events, they may be less than the data points, no argue here.
and yes, he can use 10ms delay until he gets the number of data points he want, then he can focus on how quick he can go with his hw and his sw choice. you can't just deal as beginner with 2 things at the time, better cut in two

[LoveLaika]

For what it's worth, I found a Sparkfun tutorial as well as some Github libraries someone has written for the HMC5883L, which don't involve the Adafruit libraries. While I'm running tests, I'm having a look at them to get some inspiration on how to write my own code. Granted, it may not be perfect (not to mention some inconsistencies I see with the code and the part datasheet), but it does give me a place to start, and it's surprisingly simple to follow compared to Adafruit's layers of abstraction.

[Kasper]

For what it's worth, I found a Sparkfun tutorial as well as some Github libraries someone has written for the HMC5883L, which don't involve the Adafruit libraries. While I'm running tests, I'm having a look at them to get some inspiration on how to write my own code. Granted, it may not be perfect (not to mention some inconsistencies I see with the code and the part datasheet), but it does give me a place to start, and it's surprisingly simple to follow compared to Adafruit's layers of abstraction.

Learning to write your own device drivers can be like re-inventing the wheel but it also can be a good skill to have and it is empowering to not need to rely on libraries.

[Kasper]

they say here https://circuitpython.readthedocs.io/projects/lis3mdl/en/latest/api.html : performance mode is low power for 1000Hz ("RATE_1000_HZ   1000 HZ ( Sets PerformanceMode to MODE_LOW_POWER)")

Thanks. My code was based around Adafruit's example code. I'll dig deeper into their libraries and see how they function. Perhaps I can use their library functions in a way that can reduce some overhead.

Adafruit examples can be surprisngly bad, not just poor practices but they have functions that do not work.  That being said, that's not the first thing I'd try, it can be time consumming. I also wouldn't put a 10ms delay in your loop and expect it to run any faster than 100Hz.

I'd start with a quick check of removing some printing and seeing if it runs faster.  It seems what you think is data rate of sensor is actually data rate of your loop aka how long it takes your loop to run.  As radiolistener points out, serial.print takes time and you are doing a lot of it.

well, the library examples work, for beginner's needs it ok, everyone starts from that. those libs are not perfect, but they work 99% and the beginner can't just write his 'library' or function without some practice (this is our case here).
and maybe you don't get the idea, the OP doesn't understand why he can't read enough points. understand this, if you interrogate any sensor in the world 6000 times, you expect 6000 data points, REGARDLESS of interrogation speed 100k, 400k, 100G or whatever (or the sum between the validated data points and missed sensor interrogations due to any kind of read error).
re-read my post with this in mind.
but someone noticed using serialprint slows down not the data acquisition, but data export to com port. fair point. maybe sensor data should be inserted in a matrix and counted, then printed on serial port or whatever the op wants.
so i still afirm op should interrogate for sensor values (this is very important) and not for events, they may be less than the data points, no argue here.
and yes, he can use 10ms delay until he gets the number of data points he want, then he can focus on how quick he can go with his hw and his sw choice. you can't just deal as beginner with 2 things at the time, better cut in two

I haven't used a lot of Adafruit libraries, I'd say about 90% of the ones I have used had all their basic functions working.

OP asked for help getting to 1kHz, you recommended a 10ms delay and didn't warn OP that would make it impossible to get 1kHz so I warned OP.

OP is wondering why data rate settings aren't having an effect.  I think the answer is that the OP is looking at loop speed, not sensor speed.  Step 1 can be to learn that and cleanup the loop so that 1kHz is possible.  This is a common problem, people print a lot and run out of time and memory so it seems like a good place to start.  Step 2 is get the sensor to tell us when it has updated data, using interrupt if possible or to poll it at frequencies that are appropriate for the desired data rate or write code to wait for changes for in data like this:

Code: [Select]

Serial.print("waiting for new data ");
while(newData == oldData){
  Serial.print(".");
  newData = ReadSensor();
}
Serial.println(newData);
oldData = newData;
Note: this code example is simplified, you ought update 'newData' in whatever method works for you.

For all I know, the library includes a delay to wait for updated sensor value in which case all the op has to do is step 1 and their problem will be solved.  That doesn't seem to be the case though but it is one more reason that cleaning up the loop is a good place to start.

[LoveLaika]

Oh yeah. I'm definitely feeling empowered. Running your own code that you wrote yourself without relying on third party libraries, and that anticipating feeling where it's a coin flip to whether it works or not. It's something that I've forgotten about.

[LoveLaika]

According to your code, you're using serial interface at speed 115200 to transfer samples to the PC.

Speed 115200 is about max 115200/10 = 11520 bytes per second. As you understand, 11500 letters per second is not enough speed to transfer 11000 samples per second in text format, which also includes a lot of other garbage information (like timestamp, variable names, commas and new line separators).

If I may run some math here, looking at the sensor and MCU together (no Serial UART stuff transferring to console for now), let's assume the I2C clock is 100 kHz and that the sensor is set to refresh at a rate of 1 kHz, so in 1 second, the sensor 'records' its data 1000 times. Roughly speaking, let's say a measurement is 6 bytes altogether (2 bytes per axis). According to the datasheet, following the transfer command of multiple bytes, it takes ~21 I2C clock cycles to read and store 1 measurement to a variable/struct/etc on my MCU (let's round it to 20 for simplicity). Assuming just 1 sensor, hypothetically, since the clock is so much faster than the sensor refresh rate, I should be able to take measurements 5 times before the sensor refreshes itself, and if my logic is correct, those 5 measurements will all be the same because the sensor hasn't refreshed itself yet.

Now, with Serial, 115200 baud equates to 11520 bytes per second. Assuming 6 bytes of data plus two spaces and a new line char, that's 9 chars or 9 bytes of data (again, let's round it to 10 to make it nice and even). No units, no fancy names, just pure data measurements. With the baud rate, I could transfer 1152 measurements to my serial console in a second. That's a little faster than the sensor refresh rate, so that would just barely make the cutoff (and at such close rates, I don't really feel confident in that). Does this sound right for a rough, back of the hand calculation?