GUI with gauges in python

[Simon]

What is the issue with system() ?

Am I better off using some sort of batch file that runs on start up for these things? I believe there is some file in linux for this?

With notify and listener, am I supposed to create the threads for them or do they handle this themselves?

[Marco]

One thread per request is usually an antipattern.

The available crutches are ever evolving, stackless, fibers ... or just ignoring it because you don't have million task concurrency.

[paulca]

What is the issue with system() ?

Am I better off using some sort of batch file that runs on start up for these things? I believe there is some file in linux for this?

With notify and listener, am I supposed to create the threads for them or do they handle this themselves?

From a quick scan of the docs it looks like, unless you need multi-threading in YOUR application code, you just leave it up to it.

There is suggestions in the "AsyncIO" section that "Bus" objects have "a thread" by default.

The "thread safe" variant is if you have multiple application threads (like timers) which want to call "send()" and "recv()".  They differ in that these calls are independently synchronised to be "one at a time" internally.  So it three threads all call "send()" on the bus, they go in order.  Without the thread safe variant, if you have two threads and both call send() then it doesn't look like they define what happens.

Thus, I think it's a good assumption that the Notifier will run in the "Bus thread".  Thus when it calls your listener code it will be the Bus thread, which you will want to return promptly.  Think ISR.****

There are other Threading uses internally for broadcast messages, assuming that if you broadcast to a dozen devices it launches threads to sequence them out rapidly?

From the docs being very quiet about threading otherwise, I would assume it's a "Most people don't need to know or care."

... until they do.  I would proceed with that assumption and start with copy pasta code from the docs or ChatGPT.  Not as an end goal, but as a vehicle towards understanding the tools.

EDIT **** - Caution.  Multiple buses means multiple threads and potential concurrent/re-entrant calls to your listeners if they are shared.  I fell foul of this in my own project with a producer/consumer for bus subscriptions which where cycled on a restart so it re-subs on disconnect.  However when I added a "standby" peer for high availability the two MQTT client threads beat the living shiz out of each other in that method.

[ledtester]

What is the issue with system() ?

With system() you have construct a command string which is then passed to a shell (like bash, csh, zsh, ...) which will parse it before invoking the desired executable. That means in constructing the command string you need to escape characters which have special meaning to the shell.

You can avoid the shell by using a call like subprocess.Popen() which will invoke the target executable directly.

[paulca]

This example looks like it's synchronous.  recv(1) will block until a message arrives.
https://github.com/hardbyte/python-can/blob/main/examples/receive_all.py

This example shows how the "Printer" listener, which I assume by default prints to commandline stdout, can be used.
Note the bus has "receive own" turned on, so it sends 3 messages and waits for 1 second.  the print_listener instance will be called, we are assuming by the bus thread.
https://github.com/hardbyte/python-can/blob/main/examples/print_notifier.py

What am I not seeing is a helpful, "run_blocking()" which means if your main thread exits, so does the bus thread, we are still somewhat assuming it exists.

So you might have to have something like this to hold your thread:

(pigeon python)

Code: [Select]

try:
   while running:
       running = findReasonsToNotRun()
       doSomethingUselessForASecond()
except KeyboardInterrupt e:
   print(e)
   break


[paulca]

What is the issue with system() ?

With system() you have construct a command string which is then passed to a shell (like bash, csh, zsh, ...) which will parse it before invoking the desired executable. That means in constructing the command string you need to escape characters which have special meaning to the shell.

You can avoid the shell by using a call like subprocess.Popen() which will invoke the target executable directly.

The commands in this case are system administrator commands requiring sudo.  sudo is not that standard across distros and may prompt for a password, which python may or may not honor properly and sudo might object on some systems anyway.

Admin commands should be handled elsewhere by the OS at boot.  The file in question depends on the distro.  Often there is a gui or /etc/network/config or the like where you can define interfaces.  You can start with a batch script just.  Might be worth looking into "Virtual Env" which is almost essential for managing dependencies during dev without having to install them all into your own system (and any users system) manually each time.  I believe it also has the ability to run "init scripts".  Thus when you "go into" that project, you run ". bin/activate" and it sets up your shell, environment, pythong interp, install the required deps for the project, runs your init script to make sure your interfaces are there etc.

The use of shell commands from within the application, hard couples the script to the shell, hardware and setup in question.  It should be sufficient to make it an init check in the code.

Can I open the canbus interface?  Does it exist?   No?  Exit with "Read the readme.txt bozo".

For my ROM programer for instance, the first things it does is open the file you gave it OR fail, then open the serial device you passed it as an arg OR fail.  Fail first, fail fast, simplify the remaining puzzle.

It's something you can refine as you go though.  There may be permissions issues if you want to refrain from running python as root.  Such as you user needs to be in the group able to access the /dev/whatevers for your canbus interface.

EDIT: You know what I find amazing.  The difference between "PoC", tutorial, chatgpt code and "complete" code.  Even just build environment setup generates work and maintenance.  Then you have things like checking and detainting parameters, system dependency detection, user errors, usage pages.  Permissions.  Different shells.  Windows support so yo can dev on the corporate laptop.  Handling actual command line arguments, using conventional -- and - "gopt" etc.  Then your first user says, "But I'm on Solaris" and your boss says, "No, you can't just tell him where to go.".  Logging code.  Auditing code.  Unit tests, intergration tests, test harnesses, test data, readmes, installer metafiles, repository meta files.  Your "one file with 50 lines" very rapidly becomes a git repo with 100 files and 10Mb.

The good news is, if the code is only for you and you don't care, then don't bother.  I have PoC scripts that have been running 24/7 with 99.99999% uptime for years and they still do their job.  I have copy pasta ESP32 code written in about 1 minute that has been sending me data about my heating from a kitchen cupboard for 5 years straight and it's never once complained it didn't have proper error handling or unit tests.  Usually dumb and simple works for simple things.  Software does not scale well in complexity and the human mind though.  So even hobby projects can get out of hand and impossible to work on rapidly as they "evolve", so "code hygene" and "best practice" should not be entirely overlooked.  Also.  Always be keen to refactor and redesign.

[Simon]

Well the libraries own example of notifier fails:

Code: [Select]

#!/usr/bin/env python

import time

import can


def main():
    with can.Bus(interface="virtual", receive_own_messages=True) as bus:
        print_listener = can.Printer()
        with can.Notifier(bus, listeners=[print_listener]):
            # using Notifier as a context manager automatically calls `Notifier.stop()`
            # at the end of the `with` block
            bus.send(can.Message(arbitration_id=1, is_extended_id=True))
            bus.send(can.Message(arbitration_id=2, is_extended_id=True))
            bus.send(can.Message(arbitration_id=1, is_extended_id=False))
            time.sleep(1.0)


if __name__ == "__main__":
    main()


It fails with: TypeError: 'Notifier' object does not support the context manager protocol

[paulca]

I would check your module version against the docs version, if you took my link, it's probably a random version.

If you checkout the repo somewhere you can try the examples that shipped with the code release.  However even thats no gaurantee.

Yeah the joys of python dependencies awaits you. 

[Simon]

Also when I try to do it as not context managed I get:

AttributeError: 'SocketcanBus' object has no attribute 'Notifier'

So I guess this is really the issue here? I basically have to make my own?

Code: [Select]

import os
import can
import time


os.system('sudo ip link set can0 type can bitrate 1000000')
os.system('sudo ip link set can1 type can bitrate 1000000')

os.system('sudo ifconfig can0 up')
os.system('sudo ifconfig can1 up')

bus0 = can.Bus(channel = 'can0', interface = 'socketcan')
#can1 = can.Bus(channel = 'can1', interface = 'socketcan')

#time1 = time.clock_gettime(time.CLOCK_MONOTONIC)

def my_listener():
    print("Message received")

def main():
    with can.Bus(channel = 'can1', interface='socketcan') as bus1:
        bus1.Notifier(bus1, listeners=[my_listener])
        bus0.send(can.Message(arbitration_id=1, is_extended_id=True))
        bus0.send(can.Message(arbitration_id=2, is_extended_id=True))
        bus0.send(can.Message(arbitration_id=1, is_extended_id=False))
        time.sleep(1.0)


main()
   

[Simon]

I would check your module version against the docs version, if you took my link, it's probably a random version.

If you checkout the repo somewhere you can try the examples that shipped with the code release.  However even thats no gaurantee.

Yeah the joys of python dependencies awaits you. 

It came from here: https://python-can.readthedocs.io/en/stable/notifier.html#


simon@pi5:~/Documents $ sudo apt install python3-can
python3-can is already the newest version (4.5.0-1).
Summary:
  Upgrading: 0, Installing: 0, Removing: 0, Not Upgrading: 0

The documentation is at 4.6.1

[paulca]

Stable and docs are 4.6.1, apt has given you 4.5.0

Consider.

Code: [Select]

paul[member=183778]linux[/member]-dev-vm:~$ mkdir test
paul[member=183778]linux[/member]-dev-vm:~$ cd test
paul[member=183778]linux[/member]-dev-vm:~/test$ python3 -m venv .
paul[member=183778]linux[/member]-dev-vm:~/test$ . bin/activate
(test) paul[member=183778]linux[/member]-dev-vm:~/test$ pip3 install python3-can
ERROR: Could not find a version that satisfies the requirement python3-can (from versions: none)
ERROR: No matching distribution found for python3-can
(test) paul[member=183778]linux[/member]-dev-vm:~/test$ pip3 install python-can
Collecting python-can
  Downloading python_can-4.6.1-py3-none-any.whl (276 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 277.0/277.0 KB 4.9 MB/s eta 0:00:00
Collecting wrapt~=1.10
  Downloading wrapt-1.17.3-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl (81 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 82.0/82.0 KB 5.0 MB/s eta 0:00:00
Collecting packaging>=23.1
  Using cached packaging-25.0-py3-none-any.whl (66 kB)
Collecting typing_extensions>=3.10.0.0
  Downloading typing_extensions-4.15.0-py3-none-any.whl (44 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 44.6/44.6 KB 6.2 MB/s eta 0:00:00
Installing collected packages: wrapt, typing_extensions, packaging, python-can
Successfully installed packaging-25.0 python-can-4.6.1 typing_extensions-4.15.0 wrapt-1.17.3

(test) paul[member=183778]linux[/member]-dev-vm:~/test$ pip3 show python-can
Name: python-can
Version: 4.6.1
Summary: Controller Area Network interface module for Python
Home-page:
Author: python-can contributors
Author-email:
License:
Location: /home/paul/test/lib/python3.10/site-packages
Requires: packaging, typing_extensions, wrapt
Required-by:

Then also consider:

Code: [Select]

(test) paul[member=183778]linux[/member]-dev-vm:~/test$ deactivate
paul[member=183778]linux[/member]-dev-vm:~/test$ pip3 show python-can
WARNING: Package(s) not found: python-can
paul[member=183778]linux[/member]-dev-vm:~/test$ source bin/activate
(test) paul[member=183778]linux[/member]-dev-vm:~/test$ pip3 show python-can
Name: python-can
Version: 4.6.1
Summary: Controller Area Network interface module for Python
Home-page:
Author: python-can contributors
Author-email:
License:
Location: /home/paul/test/lib/python3.10/site-packages
Requires: packaging, typing_extensions, wrapt
Required-by:
(test) paul[member=183778]linux[/member]-dev-vm:~/test$

[Simon]

I'm completely confused. I just use apt install as pip or pip3 never seem to work

[Simon]

presumably I have to do all of this and run each project in one of these new fangled virtual environment thingy's
https://www.raspberrypi.com/documentation/computers/os.html#python-on-raspberry-pi

[Simon]

Well, same result:

Code: [Select]

(env) simon@pi5:~/Documents $ /home/simon/Documents/env/bin/python "/home/simon/Documents/simple can.py"
RTNETLINK answers: Device or resource busy
RTNETLINK answers: Device or resource busy
Traceback (most recent call last):
  File "/home/simon/Documents/simple can.py", line 29, in <module>
    main()
    ~~~~^^
  File "/home/simon/Documents/simple can.py", line 22, in main
    bus1.Notifier(bus1, listeners=[my_listener])
    ^^^^^^^^^^^^^
AttributeError: 'SocketcanBus' object has no attribute 'Notifier'
SocketcanBus was not properly shut down
(env) simon@pi5:~/Documents $


[Simon]

Ok it might be working now having gone back to their original example and swapped out a few things, will see how long before I break this one.

[Simon]

Well that was short lived, now thew file runs and exits and nothing seems to happen.

[paulca]

presumably I have to do all of this and run each project in one of these new fangled virtual environment thingy's
https://www.raspberrypi.com/documentation/computers/os.html#python-on-raspberry-pi

Yes and no.  It's an extra step yes, but the purpose is to give you a separate python environment to your host OS.  That means that you don't need to mess about with your main OS python setup, which is used system wide by scripts and admin stuff.  It also mean you can trial and remove versions for the specific project.

When you come to "release" the code, the venv will produce a list file of dependencies for the installed 'wheel' package.  Such when your user goes to install your python app, it can auto install the deps.  Python packaging is a whole other monster though.

There are options.  venv is probably the most common built in method.  There is also "Anaconda" which is a more heavy weight package manager. 

One possibly easier option is to run pip3 to install deps as your normal user.  This will install the deps into your /home folder and they are only in scope for you.

[paulca]

Well that was short lived, now thew file runs and exits and nothing seems to happen.

Did the interface being busy cause issues with the lower levers so that Notifier couldn't start maybe.

I would also look to see how you set it's "Logging level".  It may default to something like error only and not debug full output.

BTW.  I find some of the python modules a bit lack luster.  I have abandoned several projects because I couldn't get the python libs to work.  Abandoned or rethought.

[paulca]

If you are using a text editor I would suggest an IDE, it will help you a lot as it will lead you with autocomplete etc.

PyCharm is free in community edition and probably the "mutts nuts".  Thonny is a very basic IDE meant for beginners and kids, it's on power with the ArduinoIDE.

Although...  PyCharm on a PI.  Hmm.   Is it a P4?  PyCharm can be a beast using 1Gb of RAM or more.

[Simon]

If I try to use pip or pip3 I get a message saying that this is externally managed and that I should use sudo. So I used:

sudo apt install python3-can

This installs version 4.5

If I activate my environment and do:

pip install python-can

It installs version 4.6.

Do I need to setup the CAN hardware and start the bus from a system wide prompt or will running the setup commands from the python environment work?

I'm working on a Pi5 with 8 GB of RAM, I've just had a Pi500+ turn up at home with 16GB which I'll setup when I get back. So far I have used VS Code.

[paulca]

Do I need to setup the CAN hardware and start the bus from a system wide prompt or will running the setup commands from the python environment work?

I'm working on a Pi5 with 8 GB of RAM, I've just had a Pi500+ turn up at home with 16GB which I'll setup when I get back. So far I have used VS Code.

I would treat the hardware init stuff separately.  Those are OS things.  Generally speaking you define them somewhere and then if.up and if.down mechanisms give you control.  However I am only familiar with networking IFs.  I wouldn't worry about it. 

Make a batch script.  A common convention is "setEnv.sh" in the project folder to start things up if needed and set environment vars.

I found this:
https://forums.raspberrypi.com/viewtopic.php?t=141052

And a few others, seems like people just run the sudo command when they boot up.  As a blunt way of making that permenant you can add it to the /etc/rc.d/rc.local or equivalent boot script.

VSCode with Python plugins is not bad.  PyCharm is a dedicated Python IDE with a thousand dollar commerical offering.  Personally I use VSCode for quick and small things and only really use PyCharm for larger projects. 

[Simon]

I would treat the hardware init stuff separately.  Those are OS things.  Generally speaking you define them somewhere and then if.up and if.down mechanisms give you control.  However I am only familiar with networking IFs.  I wouldn't worry about it. 

Make a batch script.  A common convention is "setEnv.sh" in the project folder to start things up if needed and set environment vars.

So would a setEnv.sh file run automatically?

I found this:
https://forums.raspberrypi.com/viewtopic.php?t=141052

Yes I've seen that tutorial on the rpi forum. It spends a lot of time explaining how to modify a board so that you don't fry your RPi with 5V IO. These days the manufacturers have figured that a jumper to select the IO voltage is a good idea.

And a few others, seems like people just run the sudo command when they boot up.  As a blunt way of making that permenant you can add it to the /etc/rc.d/rc.local or equivalent boot script.

Yea well just to mess with my mind, this pi os has folders rc0.d to rc6.d with a bonus rcS.d but no simple rc.d and none have an rc.local file. Are the contents of any of these folders supposed to be run automatically on boot?

[Nominal Animal]

Yea well just to mess with my mind, this pi os has folders rc0.d to rc6.d with a bonus rcS.d but no simple rc.d and none have an rc.local file.

That is a traditional SysV init (as opposed to say systemd).  The N in rcN.d refers to "runlevel". If you run
    runlevel
it will display the current runlevel, typically "N 3" or "N 5".

Runlevel 0 is for poweroff.
Runlevel 1 is for rescue.
Runlevels 2, 3, 4 are normal non-GUI runlevels, and 5 is the normal GUI runlevel.
Runlevel 6 is for reboot.

Most init systems automatically also run /etc/rc.d/rc.local if it exists for runlevels 2, 3, 4, 5, but I suggest you write an actual script.
 

Are the contents of any of these folders supposed to be run automatically on boot?

For run levels 2, 3, 4, 5, yes.  So, if your system normally boots to say runlevel 3, then putting a startup script in
    /etc/rc3.d/SNNname
will cause your init to run when starting up, where NN is a two-digit order, as they are executed in ascending order.  In your case, I recommend 99 (run last).

Shutdown scripts are correspondingly
    /etc/rc0.d/KNNname
and those run before reboot
    /etc/rc6.d/KNNname

These are run as root with a single command line argument:

• start
• stop
• restart
• reload
• status

You should find existing startup scripts there, which you can adapt to your needs.  Or, you can do a looksee for "sysv init script" to find many guides and tutorials about these.

[Simon]

So having read this which is about as useful as a chocolate teapot: https://bash.cyberciti.biz/wiki/index.php?title=System_V_init_script&mobileaction=toggle_view_desktop

I learn from it only that this SysV "thing" is old hat and that I should probably keep up with the cool kids and use systemd ?

[paulca]

The raspberry pi interface config should allow you to have the OS adopt the CANBus.

Configure the Network Interface: Edit the /etc/network/interfaces file to automatically bring up the CAN interface at boot. Add the following configuration:
auto can0
iface can0 inet manual
    pre-up /sbin/ip link set can0 type can bitrate 500000 triple-sampling on restart-ms 100
    up /sbin/ifconfig can0 up
    down /sbin/ifconfig can0 down

AI Assisted.

Then you can do "if-up can0" and "if-down can0"

I believe "auto can0" line will try and "up" the interface at boot.