Python gurus, questions about variable access speed (RPi 5 GPIO adventure time)

[John B]

As my python program grow to a few thousand lines, I'm going back through and cleaning up code. I understand that global variables are generally avoided for the risk of errors, but also in python specifically, it should be slower to look up global variable names, with local variable lookups being the fastest. I have been using dictionaries to pack a range of globally needed constants that are used in all classes/objects

The question is in regards to performance when passing in data to functions or classes.

First example, global dictionary is accessed within a class. Least performant.

Code: [Select]

//Globally defined dictionary
global_dict = {'key': 'val'}

class SomeClass:
    def __init__(self):
        //Use global_dict here


Next example, the dictionary is passed into the class at construction, and a local object is created. It will only be a reference to the dictionary, not a copy of it.

Code: [Select]

//Globally defined dictionary
global_dict = {'key': 'val'}

class SomeClass:
    def __init__(self, _global_dict):
        //Create a reference(?) to the global dictionary, but doesn't copy data
        self.local_dict = _global_dict

someclass = SomeClass(global_dict)


Last example, a local object is created and the entire dictionary is duplicated in memory.

Code: [Select]

import copy

//Globally defined dictionary
global_dict = {'key': 'val'}

class SomeClass:
    def __init__(self, _global_dict):
        //Create a local dictionary and copy the global data into a new dictionary
        self.local_dict = copy.deepcopy(_global_dict)

someclass = SomeClass(global_dict)

I am guessing the latter 2 are better than the first example, but is there any benefit or difference in the last 2 examples? Ie is there any benefit in creating a new local object with it's own memory location, vs simply creating a reference to the global dictionary?

[ejeffrey]

True local variables (not class members) are the fastest.  If you are repeatedly accessing an object from a long term scope it's fastest to assign it to a local variable.  This is because you avoid name lookups.

Class member access is only faster than global variable access if you use __slots__.  Otherwise it's basically the same.

Using copy.copy vs just reference assignment makes no difference in terms of access speed.

[brucehoult]

Why guess? Just write some typical code you care about, put it in a loop and run it a million or a billion times, and time it. Unlike a C compiler, Python isn't going to remove code you don't use the result of, or anything like that.

[ledtester]

Last example, a local object is created and the entire dictionary is duplicated in memory.

Using copy.copy vs just reference assignment makes no difference in terms of access speed.

There are other side effects of copying... it uses more memory and if you use a lot of these objects it will reduce cache hits.

You can retain the flexibility of using a reference or a copy by having the code which invokes the creation of the object decide if a copy should be made. I.e., the class code just stores a reference, but if you want a copy just have the calling code do:

... = SomeClass(copy.deepcopy(somedict))

instead of:

... = SomeClass(somedict)

[John B]

Some interesting results based on reading variables only:

Code: [Select]

from time import perf_counter
import copy

#Globally defined dictionary
global_dict = {'testkey': 'testval'}

class SomeClass1:
    def __init__(self):
        #Create a local dictionary and copy the global data into a new dictionary
        self.local_dict = global_dict
        self.local_var = global_dict['testkey']
    def read_dict(self):
        return self.local_dict
    def read_dict_key(self):
        return self.local_dict['testkey']
    def read_var(self):
        return self.local_var   



class SomeClass2:
    def __init__(self, _global_dict):
        #Create a local dictionary and copy the global data into a new dictionary
        self.local_dict = _global_dict
        self.local_var = self.local_dict['testkey']
    def read_dict(self):
        return self.local_dict
    def read_dict_key(self):
        return self.local_dict['testkey']
    def read_var(self):
        return self.local_var

class SomeClass3:
    def __init__(self, _global_dict):
        #Create a local dictionary and copy the global data into a new dictionary
        self.local_dict = copy.deepcopy(_global_dict)
        self.local_var = self.local_dict['testkey']
    def read_dict(self):
        return self.local_dict
    def read_dict_key(self):
        return self.local_dict['testkey']
    def read_var(self):
        return self.local_var
   
class SomeClass4:
    __slots__ = ['local_var', 'local_dict']
   
    def __init__(self, _global_dict):
        #Create a local dictionary and copy the global data into a new dictionary
        self.local_dict = _global_dict
        self.local_var = _global_dict['testkey']
    def read_dict(self):
        return self.local_dict
    def read_dict_key(self):
        return self.local_dict['testkey']
    def read_var(self):
        return self.local_var

someclass1 = SomeClass1()
someclass2 = SomeClass2(global_dict)
someclass3 = SomeClass3(global_dict)
someclass4 = SomeClass4(global_dict)


def time_function(_function, _number = 10000000):
    print(f'Running {_function.__qualname__}')
    print(f'Number of runs: {_number}')
    # Start the stopwatch / counter
    t1_start = perf_counter()
    for i in range(_number):
        _function()
    # Stop the stopwatch / counter
    t1_stop = perf_counter()
    print(f'Elapsed time: {t1_stop - t1_start}')
    print('\n')

time_function(someclass1.read_dict)
time_function(someclass1.read_dict_key)
time_function(someclass1.read_var)

time_function(someclass2.read_dict)
time_function(someclass2.read_dict_key)
time_function(someclass2.read_var)

time_function(someclass3.read_dict)
time_function(someclass3.read_dict_key)
time_function(someclass3.read_var)

time_function(someclass4.read_dict)
time_function(someclass4.read_dict_key)
time_function(someclass4.read_var)


Output:

Code: [Select]

Running SomeClass1.read_dict
Number of runs: 10000000
Elapsed time: 0.3098224690038478


Running SomeClass1.read_dict_key
Number of runs: 10000000
Elapsed time: 0.39295347400184255


Running SomeClass1.read_var
Number of runs: 10000000
Elapsed time: 0.3147745200112695


Running SomeClass2.read_dict
Number of runs: 10000000
Elapsed time: 0.31072110700188205


Running SomeClass2.read_dict_key
Number of runs: 10000000
Elapsed time: 0.39065061799192335


Running SomeClass2.read_var
Number of runs: 10000000
Elapsed time: 0.31075479699939024


Running SomeClass3.read_dict
Number of runs: 10000000
Elapsed time: 0.30780015198979527


Running SomeClass3.read_dict_key
Number of runs: 10000000
Elapsed time: 0.3865430949954316


Running SomeClass3.read_var
Number of runs: 10000000
Elapsed time: 0.35015562300395686


Running SomeClass4.read_dict
Number of runs: 10000000
Elapsed time: 0.32216166899888776


Running SomeClass4.read_dict_key
Number of runs: 10000000
Elapsed time: 0.38827769200725015


Running SomeClass4.read_var
Number of runs: 10000000
Elapsed time: 0.3130907129961997

It would seem the only real benefit is when accessing whole objects (ie returning a whole dictionary), or returning individual variables. The dictionary lookup step adds a measurable amount of time.

Not sure if I just have NFI what I'm doing with __slots__, but it doesn't seem to have any effect.

[ejeffrey]

Two things, first, the function call overhead is pretty large.  Second, mostly what you are trying to save is a single dictionary lookup.  If you are doing explicit dictionary lookups, thats going to also add cost that makes the variable lookup less significant.

One thing I also forgot is that "implicit" globals are slower than explicit ones.  Implicit globals need to walk the scope checking at each scope for the existence of the named variable.  Explicit globals do not.

I modified your example to run a loop inside the function call, to show that with repeated access within a single function you get a more significant change.  I removed the copy version because any performance change there I would expect to be due to the allocations, not the access itself.  Here are 5 versions to illustrate the differences:

Code: [Select]

#!/usr/bin/env python3
from time import perf_counter
import copy

#Globally defined dictionary
global_dict = {'testkey': 'testval'}
global_var = global_dict['testkey']

class SomeClass1:
    def __init__(self):
        # Access module globals directly
        pass
   
    def read_dict(self):
        for _ in range(1000):
            x = global_dict
        return x
    def read_dict_key(self):
        for _ in range(1000):
            x = global_dict['testkey']
        return x
    def read_var(self):
        for _ in range(1000):
            x = global_var
        return x


class SomeClass2:
    def __init__(self):
        # Access module globals directly, explicity label as globals
        pass
   
    def read_dict(self):
        global global_dict
        for _ in range(1000):
            x = global_dict
        return x
    def read_dict_key(self):
        global global_dict
        for _ in range(1000):
            x = global_dict['testkey']
        return x
    def read_var(self):
        global global_var
        for _ in range(1000):
            x = global_var
        return x



class SomeClass3:
    def __init__(self, _global_dict):
        # Create a local dictionary reference to the global dictionary
        self.local_dict = _global_dict
        self.local_var = self.local_dict['testkey']
    def read_dict(self):
        for _ in range(1000):
            x = self.local_dict
        return x
   
    def read_dict_key(self):
        for _ in range(1000):
            x = self.local_dict['testkey']
        return x
    def read_var(self):
        for _ in range(1000):
            x = self.local_var
        return x
   
class SomeClass4:
    def __init__(self, _global_dict):
        # cache the variables as locals at the beginning of each function
        pass
    def read_dict(self):
        local_dict = global_dict
        for _ in range(1000):
            x = local_dict
        return
    def read_dict_key(self):
        local_dict = global_dict
        for _ in range(1000):
            x = local_dict['testkey']
        return x

    def read_var(self):
        local_var = global_dict['testkey']
        for _ in range(1000):
            x = local_var
        return x
   
class SomeClass5:
    __slots__ = ['local_var', 'local_dict']
   
    def __init__(self, _global_dict):
        # Store the variables in slots
        self.local_dict = _global_dict
        self.local_var = _global_dict['testkey']
    def read_dict(self):
        for _ in range(1000):
            x = self.local_dict
        return x
    def read_dict_key(self):
        for _ in range(1000):
            x = self.local_dict['testkey']
        return x
    def read_var(self):
        for _ in range(1000):
            x = self.local_var
        return x

someclass1 = SomeClass1()
someclass2 = SomeClass2()
someclass3 = SomeClass3(global_dict)
someclass4 = SomeClass4(global_dict)
someclass5 = SomeClass5(global_dict)


def time_function(_function, _number = 100000):
    print(f'Running {_function.__qualname__}')
    print(f'Number of runs: {_number}')
    # Start the stopwatch / counter
    t1_start = perf_counter()
    for i in range(_number):
        _function()
    # Stop the stopwatch / counter
    t1_stop = perf_counter()
    print(f'Elapsed time: {t1_stop - t1_start}')
    print('\n')

time_function(someclass1.read_dict)
time_function(someclass1.read_dict_key)
time_function(someclass1.read_var)

time_function(someclass2.read_dict)
time_function(someclass2.read_dict_key)
time_function(someclass2.read_var)

time_function(someclass3.read_dict)
time_function(someclass3.read_dict_key)
time_function(someclass3.read_var)

time_function(someclass4.read_dict)
time_function(someclass4.read_dict_key)
time_function(someclass4.read_var)

time_function(someclass5.read_dict)
time_function(someclass5.read_dict_key)
time_function(someclass5.read_var)

And the results.  In each case, the loop body is executed 10^8 times.

Code: [Select]

Running SomeClass1.read_dict
Number of runs: 100000
Elapsed time: 2.4651851789967623


Running SomeClass1.read_dict_key
Number of runs: 100000
Elapsed time: 3.501273279005545


Running SomeClass1.read_var
Number of runs: 100000
Elapsed time: 1.8243163709994406


Running SomeClass2.read_dict
Number of runs: 100000
Elapsed time: 1.77801652099879


Running SomeClass2.read_dict_key
Number of runs: 100000
Elapsed time: 2.6004149750078795


Running SomeClass2.read_var
Number of runs: 100000
Elapsed time: 2.181095936000929


Running SomeClass3.read_dict
Number of runs: 100000
Elapsed time: 1.314446283999132


Running SomeClass3.read_dict_key
Number of runs: 100000
Elapsed time: 1.7895739699888509


Running SomeClass3.read_var
Number of runs: 100000
Elapsed time: 1.2826121679972857


Running SomeClass4.read_dict
Number of runs: 100000
Elapsed time: 1.09882029700384


Running SomeClass4.read_dict_key
Number of runs: 100000
Elapsed time: 1.6744724690070143


Running SomeClass4.read_var
Number of runs: 100000
Elapsed time: 1.0445217109954683


Running SomeClass5.read_dict
Number of runs: 100000
Elapsed time: 1.2537094279978191


Running SomeClass5.read_dict_key
Number of runs: 100000
Elapsed time: 1.8082258610083954


Running SomeClass5.read_var
Number of runs: 100000
Elapsed time: 1.3392351689981297

As you can see, copying the variable you need to the local scope (SomeClass4) is the fastest option.   When just accessing a simple local variable, each loop iteration takes 10 nanoseconds.

At the slowest, SomeClass1, accessing an implicit global dictionary, then fetching an item from it takes 35 ns.

I didn't do any statistical analysis, so you probably want to do that if you are really optimizing.  It would be better to run each test 10 times and report the standard deviation as well.

[ejeffrey]

I forgot something else.

Code: [Select]

for _ in range(1000):

Integers from -5 to +256 are "interned" which means that they are constructed at startup time and stored in a special table.  Whenever the result of a computation is in that range, the object from the table is used rather than constructing a new object.  So this example, iterating up to 1000 ends up recreating and destroying the objects for integers 257..999. 

If I change the function to loop up to 200 and repeat that 50,000 times, it is another significant improvement:

Running SomeClass1.read_dict
Number of runs: 500000
Elapsed time: 1.399502450003638


Running SomeClass1.read_dict_key
Number of runs: 500000
Elapsed time: 2.143058228000882


Running SomeClass1.read_var
Number of runs: 500000
Elapsed time: 1.661943153012544


Running SomeClass2.read_dict
Number of runs: 500000
Elapsed time: 1.6162579140072921


Running SomeClass2.read_dict_key
Number of runs: 500000
Elapsed time: 2.089768796999124


Running SomeClass2.read_var
Number of runs: 500000
Elapsed time: 0.9104458929941757


Running SomeClass3.read_dict
Number of runs: 500000
Elapsed time: 0.9809876910003368


Running SomeClass3.read_dict_key
Number of runs: 500000
Elapsed time: 1.3673355460050516


Running SomeClass3.read_var
Number of runs: 500000
Elapsed time: 0.9164341279974906


Running SomeClass4.read_dict
Number of runs: 500000
Elapsed time: 0.7548851610044949


Running SomeClass4.read_dict_key
Number of runs: 500000
Elapsed time: 1.2996211470017442


Running SomeClass4.read_var
Number of runs: 500000
Elapsed time: 0.6924092390108854


Running SomeClass5.read_dict
Number of runs: 500000
Elapsed time: 0.9721332789922599


Running SomeClass5.read_dict_key
Number of runs: 500000
Elapsed time: 1.4019307380076498


Running SomeClass5.read_var
Number of runs: 500000
Elapsed time: 0.9558670469996287

The fastest version is still SomeClass4, and it's down to about 7 ns per iteration.

This level of micro-optimization is _generally_ not worthwhile in python.  The most effective way to optimize python is generally to figure out how to spend less time executing python and more time executing native libraries.  Most of these cases are somewhat pathalogical, it's not normal to access a single variable 1000 times without doing anything else that is going to dominate the run time.  But it's helpful to know what the interpreter is doing, and it's good to know when something you are doing might hit one of these cases particularly hard.

[John B]

That makes sense, ie caching frequently accessed variables within a function, which is something I thought anyways. Unfortunately I don't have any functions that would really benefit from that. At most I could eliminate some table lookups by copying the needed global dictionary keys into class variables.

On another note, copy.deepcopy() seems to be quite an expensive function. I have to use that operation quite a lot on dictionaries ranging from about 20-40 key/vals

Because the keys and/or number of keys wont change during runtime, I can manually specify and copy the values. This is 25x faster than copy.deepcopy()

[golden_labels]

If one asks about performance, but didn’t mention profiling results, the question is wrong. This is where this post should end.

If we ignore above, though we should not and mind it remains relevant, steandard rules of optimization apply. That means the kind of optimization you asked about is among the last things to care about. It’s also least effective, often counterproductive, and detrimental to code quality. It’s however very attractive, because it gives the feeling of control and comprehension, while requiring little actual mental effort, understanding, or experience. The priority goes as follow: system model, program model, algorithms and data structures, code-level changes, employing techniques below language level. Did you already handle the first three?

Going deeper, again with the note that the previous paragraphs remain more important, note Python is a language. There are multiple implementations, each with different performance characteristics. I hope you see, how that complicates your question. What worked in one environment may have a very different effect in another. This is a common issue in JIT environments and heavily-optimizing static compilers: people spend a lot of time micro-optimizing their lines, just to learn that in the next release of the tool the modification has an opposite effect. Or even while run on another CPU (for native programs).

As noted above, local variables are generally the fastest in Python. This comes directly from the language model and how the name look-up has to be done. Python shares performance penalty with JavaScript: the dynamic nature of everything requires each access to behave as if a complete, multi-level search for a name was performed. So caching a reference in a local variable may be preferred, where it’s found the access impacts performance. This may be particularly important for properties of some objects, where they invoke __getattr__ and/or go through property objects. Using __slots__ may (not must) help, if it doesn’t affect semantics. But remember it’s not a silver bullet. Being smart about repeated dict accesses helps. Python made a bad API decision with dict, compared to e.g. Java’s Map. Insertion methods can’t in one go determine, if the insertion actually happened or avoid overwriting. That often forces the programmer to do unneeded double search. dict.setdefault provides some relief, but only in some situations.

But, circling back to the begining: profile, profile, profile. And profile. There is no optimization without profiling and understanding profiling data. CPython has a built-in cProfile module for that. It can be invoked from command line:

Code: [Select]

python -m cProfile script.pyMay be used for a complete program, an isolated fragment of a program, and for micro-benchmarks. While doing the last group, be sure to not fall to common mistakes. The costs are so tiny, that they can be easily dwarfed by data generation, memory access patterns, or cold start. That can be reproducible, leading to wrong conclusions. They may also have huge variance. A parameter notoriously ignored, but in this case very relevant: any apparent difference between alternative solutions, even if consistent, may actually be still within noise level.

[John B]

I did end up profiling the script using cProfile and implementing some of my experiments, and rerunning the test to see improvements.

Most of the runtime is in the scripts idle state, where it scans GPIO pins and iterates through a bunch of lists.

Initial tests showed the main loop running around 5500 times per second. Each loop currently has a fixed 100us sleep per loop to manage the CPU usage.

I replaced some range() objects with lists, as they iterate slightly faster than constructing a new range() object each time. Range is still very fast, but iterating a list or tuple is about a 7% improvement. Interestingly, if you declare a range() object ahead of time, the iteration is an 87% improvement. Unfortunately range() is a single shot generator and so a new object needs to be constantly created. From a baseline 'for i in range(x)' type loop of 1.00, iterating through a list represents about 0.93 of the time taken, while a pre-declared range() object only takes 0.13 of the time to iterate.

The other benefit is that the list can be pruned as needed to skip redundant iterations (as in my case).

I optimised infrequent operations (again in the iterations), but they represent a tiny fraction of the runtime.

I also condensed some functions to reduce lookups, along with multi-step dictionary lookups.

The total improvement shows the loop now running a bit over 5700 times per second, and 9-10% cpu usage down from a bit over 11%


I didn't really go into depth on the purpose, but this runs on a RPi5 and is used to connect an RS485 network to the local network via MQTT. I'll have to remeasure the response time, but the last measurements I took showed around a 1.6ms response time to a serial device's read request.

[ledtester]

Most of the runtime is in the scripts idle state, where it scans GPIO pins and iterates through a bunch of lists.

Initial tests showed the main loop running around 5500 times per second. Each loop currently has a fixed 100us sleep per loop to manage the CPU usage.
...
I didn't really go into depth on the purpose, but this runs on a RPi5 and is used to connect an RS485 network to the local network via MQTT.

Sounds like your code is doing a lot of what is called "busy waiting".

What are you doing that requires you to poll the GPIO pins so much? There might be a better way.

[John B]

I am scanning a pair of interrupt pins on 2x MCP23017s IO expanders in order to trigger I2C read operations.

I am using the python implementation of the gpiod / libgpiod library, then smbus for I2C

Previously I had looked into whether there was a more efficient way to read the GPIO, like a state change callback or something, but with no luck.

[ledtester]

From the datasheet:

https://ww1.microchip.com/downloads/aemDocuments/documents/APID/ProductDocuments/DataSheets/MCP23017-Data-Sheet-DS20001952.pdf

the IC has the ability to set the pin INTA when a pin on port A changes (and ditto for INTB and port B).

This page describes how Python code can be called when a RPi GPIO line changes state:

https://roboticsbackend.com/raspberry-pi-gpio-interrupts-tutorial/

So instead of polling all the time you should be able to have Python code respond to a pin change on the MCP23017.

How fast can the GPIOs on the MCP23017 change state? What are they connected to? Can you just use RPi GPIO pins instead of the MCP23017s?

[brucehoult]

I replaced some range() objects with lists, as they iterate slightly faster than constructing a new range() object each time. Range is still very fast, but iterating a list or tuple is about a 7% improvement. Interestingly, if you declare a range() object ahead of time, the iteration is an 87% improvement. Unfortunately range() is a single shot generator and so a new object needs to be constantly created. From a baseline 'for i in range(x)' type loop of 1.00, iterating through a list represents about 0.93 of the time taken, while a pre-declared range() object only takes 0.13 of the time to iterate.

Yes, obviously this is much faster:

Code: [Select]

range_x = range(x)
while true:
    for i in range_x:
        ...


Most of the runtime is in the scripts idle state, where it scans GPIO pins and iterates through a bunch of lists.

Scanning GPIO pins on a Raspberry Pi is not something that needs some unicorn Python library. That is an obvious candidate to do in C, and dead easy to do so, even if it's C called from the Python.

I don't have a Pi Pi running right now, but on a LicheePi 3A ...

Code: [Select]

for x in range(1000000):
    with open("/sys/class/gpio/gpio44/value", "r") as f:
        gpio_value = int(f.read())

print(gpio_value)

... took 131.4 seconds, while in C ...

Code: [Select]

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

int read_gpio44() {
  char buf[2];
  int fd = open("/sys/class/gpio/gpio44/value", O_RDONLY);
  if (read(fd, buf, 1) != 1)
    printf("Read error");;
  close(fd);
  return buf[0] - '0';
}

int main() {
  int tot = 0;
  for (int i=0; i<1000000; ++i)
    tot +=read_gpio44();
  printf("%d\n", tot);
  return 0;
}

... took  25.2 seconds.

So the Python is 131µs per CPIO read while the C is 25µs.

The actual OS / file system stuff takes significant time, but the C is still more than 5 times faster.

However, moving the open/close outside the loop ...

Code: [Select]

#include <stdio.h>
#include <fcntl.h>
#include <unistd.h>

int read_gpio(int fd) {
  char buf[2];
  lseek(fd, 0, SEEK_SET);
  if (read(fd, buf, 1) != 1)
    printf("Read error");;
  return buf[0] - '0';
}

int main() {
  int fd = open("/sys/class/gpio/gpio44/value", O_RDONLY);
  int tot = 0;
  for (int i=0; i<1000000; ++i)
    tot +=read_gpio(fd);
  close(fd);
  printf("%d\n", tot);
  return 0;
}

... takes 2.2 seconds, over 10x faster again. I *think* this will pick up changes, but haven't tested it.

BUT .. if you're just waiting for one of a set of GPIO pins to change then I expect you can use the OS's "poll" function and not busy-wait yourself.

https://man7.org/linux/man-pages/man2/poll.2.html

[John B]

It's already setup in line with what you're describing. Each MCP23017 has INTA and INTB configured as open drain pins, and are connected in parallel to a single RPi GPIO. Then I only need to read that GPIO pin to register any state changes on the IO expander (which then triggers a read operation of both banks in one go).

Unfortunately any documentation in regards to the RPi.GPIO library is now obsolete as that library won't work with the Pi5.

The gpiod library is a fast C library (with a python wrapper in this case). Unfortunately the documentation isn't great and there's not a lot of examples out there, but I might search around again.

[ledtester]

You might look at lgpio:

https://rpi-lgpio.readthedocs.io/en/latest/#

It has the add_event_detect method:

https://rpi-lgpio.readthedocs.io/en/latest/api.html#RPi.GPIO.add_event_detect

and there's a lot of evidence the module it works on a RPi5.

[John B]

It's been a while since I looked at the GPIO side, but things are a little clearer now. lgpio can be used as a library directly, alternatively there are some front end libraries that use lgpio, such as gpio zero, or an RPi.GPIO library designed to be an equivalent drop in for the old RPi.GPIO, which is no longer a working option for the pi 5.

At the time, gpiod was advertised as a faster library written in C, evidenced by reading/writing, pin toggling experiments etc compared to gpiozero. I think gpiod operates through the linux kernel, operating in much the same way as bruce's example, just defining the main functions like setting input/output pin type, reading or writing state.

Perhaps lgpio has seen improvements in speed since then, while gpiod has stayed fairly constant, so it's probably worth running some experiments. The questions is whether either library uses actual hardware interrupts to register pin changes, or whether they just have blocking while loops that just constantly check for changes. They may need to run in a separate thread, and so I will need to test how that fits into the main loop in a thread safe way.

It's been a while since I did some measurements, so I was happy to see an improvement in response time. Documented for future performance testing:

Measured on one of the RS485/MODBUS devices
Yellow - input button press
Purple - Read request (which is detected by the RPi IO expander)
Blue - RX from RPi (can also just make out the crosstalk/coupling of the microcontroller TX response. I would stick it on CH4, but I was already holding 3 probes with 1 hand)

[John B]

In addition to bruce's example, the old interface using /sys/class/gpio/gpioXX no longer exists on the pi 5, and is the reason for the old RPi.GPIO library no longer working. Now the device file is a single gpiomemX or gpiochipX device file.


Edit: I realised I was using the system gpiod library ~V1.6, whereas the python gpiod library is now up to V2.3

In a classic footgun moment, I installed it in the virtual environment, but it's a completely different API, so I will have to spend a bit of time getting the existing code working again. I think there's "wait for line event functions" I can experiment with.

[ledtester]

I'm still curious what kind of bit twiddling your python code is doing. It seems you are sending and receiving MODBUS messages.

Are you creating some sort of gateway between the Internet/Ethernet/Wifi and a MODBUS RS-485 network?

This project:

https://github.com/3cky/mbusd

is such a gateway. It uses a cheap RS-485 TTL to USB serial converter like this:

https://medium.com/@boonsanti/raspberry-pi-3-model-b-modbus-rs-485-to-modbus-tcp-ip-gateway-9ed14ce2c08e

[John B]

It's to connect a series of independent modules/devices, like LED controllers, relays, digital inputs, motor blind controllers etc. They ultimately connect to home automation software through MQTT, across local networks devices. MODBUS in this case is just the protocol to connect each device to the RPi host/master device. The registers don't necessarily reflect individual values, for example a 16 bit register will hold both 0-255 dimming and colour values for an LED PWM output. The IO expanders also store their states as a single 16bit register. I'm even using them as trigger registers, rather than holding values.

Both the micros and RPi python script handle the packing / unpacking of data, and the python script converts the raw registers into a far easily human-readable dictionary of values for each device. Each physical device has an associated python class, and you can easily configure whatever devices are present on each input channel (0-31, or modbus address 1-32)

My approach is probably a bit novel, and perhaps something like CAN would have been a better choice. However at the time I started designing everything, the component shortage made CAN stuff hard to find, and existing stock was very expensive. Each device having it's own "interrupt" has worked pretty well. Around 350us response time from the RPi is better than I expected, but I havent done measurements like that in a while. When I started, I was using a RPi 4 with much older libraries, and that's reflected in the overall system latency.

[John B]

Well I knew I wouldn't be able to put this project down for long 

I updated my code to use gpiod V2.3.0 with a 1:1 code functionality with some interesting but disappointing first impressions. Using the same GPIO polling method, the code retains the same level of latency as per my scope screenshot (~350us from a micro read request to the RPi starting a modbus transaction), however CPU usage jumped from 9-10% to 13-14% and slowed the main loop from 5700x per sec to 5200x per sec.

This finally motivated me to update the interrupt pins to an edge event triggered approach as opposed to a basic polling method. Now each interrupt pin has it's own thread with a looping blocking method that waits for falling edge events. This approach does work with pros and cons, CPU usage at idle has now dropped to 3.5-4.5%, but latency increased from ~350us to ~700us. It's not hugely critical as microcontroller read requests are a result of human timescale events, like button presses or motion sensors. It's been reliable in operation, although depending on abnormal inital board states, I could get the interrupts stuck where there will be no edge events, but it hasn't read the IO expanders to reset them.

I have a couple of periodic timers doing infrequent functions anyways, so I could easily attach a manual interrupt read every 1-5 seconds just to be on the safe side. That will un-fubar itself in that case, with no CPU penalty.

I'll experiment with lgpio at some future stage, but I did notice that the last release of that was March 2024, whereas gpiod was Feb 2025. Carrying on from Bruce's previous comment about a "unicorn" library for python, in fact gpiod is just a python binding of the libgpiod C library, and is in active development. Hence why I think it's still a solid choice to build around. From my understanding lgpio still has to work in a similar way, ie requests to and through the linux kernel, which is a safer way to interact with the GPIO to ensure different threads don't interfere with one another.

Next I might refactor how I am taking ownership of GPIO lines. Previously, each requested line was a separate object (which I've duplicated), but recent gpiod versions can bulk request ownership of lines and configure them in a single object. Things like input / output, edge detection mode etc. Im hoping that way will reduce system calls and reduce the latency again.

[John B]

This is more of a think out aloud type situation, but for others reference I think gpiod >V2.0 for python now gets the green light

The older V1.6 had faster individual pin read functions, but now I mainly employ blocking functions that wait for updates through the linux kernels file descriptor system. Manual pin reads are very infrequent now.

The additional latency I was getting was due to how data was being passed between threads. It was also highly variable, anywhere from 0.7-1.5ms.

I changed the interrupt pins thread so that it calls processing functions from there directly, and trimmed a bit of debugging bloat. I've now regained the 320-350us read latency, while also being consistent in that range. Idle CPU usage is down around 3.3-4%

Who said python was slow