GUI with gauges in python

[eutectique]

You might also want to consider

Code: [Select]

ip link set can0 type can sample-point 0.875 restart-ms 25
ip link set can0 qlen 1000

At least J1939 requires this sample point.

[Nominal Animal]

I learn from it only that this SysV "thing" is old hat and that I should probably keep up with the cool kids and use systemd ?

No.  It wastes resources and is both a security risk and a single point of failure for the entire system.

[Simon]

Well apparently it's in use anyway so if it is going to be a security problem it's already there.

[Nominal Animal]

Well apparently it's in use anyway so if it is going to be a security problem it's already there.

Yep, and the CVE's for systemd prove the security problems.

It's similar to how Sendmail and Bind were extremely dominant around the turn of the century, even though they were full of bugs and security holes.  Only competition by other MTA's "fixed" the situation.  (They are still more buggy and less secure than the main competitors like exim and dovecot.)  This is nothing new, and regularly happens when socially more adept but technically less skilled people run a large software project, and have the backing of a large enough company.

I get often painted as a zealot and systemd-hater, but fact is, all my arguments have a technical basis I do try to explain (instead of convincing others of my righteousness), and involve very little emotion (except for rant-inducing frustration about lack of honest objectivity in these things).  I've seen all this before, and all this will happen with other software projects and packages in the future too, because we're only human.  It is not "hate" to point it out, just rational-logical evaluation.

[Simon]

Yes I understand there is some controversy, this post seems to explain it fairly well: https://www.reddit.com/r/HobbyDrama/comments/mljxue/software_developmentlinux_the_bitter_fight_over/

He says:

              "I also don't like it when it requires software to use it as a dependency to even work at all"

But surely this is not systemd requiring anything, it's because the people who wrote the software opted to use systemd in the first place as I am looking at myself now. It's peoples programs that require systemd or have I missed something?

[Simon]

So are multiple init systems used at once? pi os apparently uses systemd but the folders for the RC init system are also present.

[Nominal Animal]

So are multiple init systems used at once?

No.

pi os apparently uses systemd but the folders for the RC init system are also present.

Systemd can support SysV init scripts, yes.  Whether it does, depends on the distribution maintainers.

Raspbian/Raspberry Pi OS is a Debian derivative, and although Debian ostensibly supports multiple init systems, the discussions at the Debian Technical Committee mailing list shows that the systemd developers are actively disabling the support for other init systems (as most discussions involve a systemd-subsumed package and related compatibility issues).  The end result is that if a derivative wants to use anything besides systemd, they will need to not only replace the entire init system by themselves, but also recompile many/most userspace packages to remove the systemd dependencies.  Thus, at least current versions of Raspberry Pi OS actually do use systemd init.

(I did not know this before posting this post, because I do not use Raspberry Pi OS myself.  To see how much work is needed to switch Debian from systemd to other init systems, take a look at devuan.org, which is a proper attempt at supporting multiple init systems in a Debian derivative/fork.)

[Simon]

It feels like I have a morale dilemma / future proofing / portability issue and I just wanted to get this thing working.

[Simon]

I'll go back to the GUI / CAN bus threading stuff and worry about the self starting stuff later.

[paulca]

Yea take the linux thing separate.  You have working sudo commands, just run them when you want the interfaces to play with.

I gave you the RPI network/interface config file that will bring them up at boot.

I would work on the backend able to do what you want in a scrapy console app first.  Then write a GUI around it.

[Nominal Animal]

It feels like I have a morale dilemma / future proofing / portability issue and I just wanted to get this thing working.

Why?  You should not incorporate the service aspects to your application at all, just keep them separate and maintain them separately.

Generally, the application should not control the CAN interfaces at all.  (Does your browser control your network interfaces?  Mine does not.  If they try, I switch to a browser that doesn't.)

If you want to, like discussed before, you should have the application execute external scripts, say /usr/share/yourapp/can-up and /usr/share/yourapp/can-down that can be adapted to different environments and init systems.  Your application simply runs them, and uses the exit status (zero if success, 1..127 if error) to determine the result, perhaps collecting standard error to display to the user if it fails.

Similarly for the other OS/distribution-specific things.  Do not integrate the exact commands into your code.  Execute external scripts instead, so that those can be adapted to the circumstances without having to recompile your application.  Put them under /usr/share/yourapp/ with yourapp a global variable or constant.

When using Bash or POSIX shells, scripts that simply execute a fixed command should be written as

Code: [Select]

#!/bin/sh
export LANG=C LC_ALL=C
exec /path/to/command args...
where the export command ensures the expected locale (C being the default, non-localized locale), and exec replaces the shell process with /path/to/command.  Without exec, the shell process will stay resident in memory until the command exits.  In the exec statement, "$@" (including the double quotes) expands to all command line parameters.

Do not use system() or subprocess.run() to execute these scripts, because they block the parent process until the child process exits.  In Python, use subprocess.Popen() instead.  That way the parent and child processes run concurrently.  Use subprocess.DEVNULL for standard input, output, and/or error when you aren't interested in them, so that the kernel handles those streams, saving resources.

[paulca]

It's very common for me to develop and test python code on Windows10, even though the actual build->deploy process runs it on Linux.

I do have the advantage of only interfacing with the world via network.  Though that was a deliberate choice for good reasons.

That's just one of many reasons to not put system calls in the script but in the environment.

"How do you build that environment and maintain it as the system grows?" - is the age old question which software developers have to face.  When the fresh out of dev code hits reality.

Docker and containers is the modern way, however this will not save you here.  It will require that you can manage the can interfaces first.  Then configure your docker container to "map" them into the container for your python.  It adds extra steps initially which you don't need.

It goes back to the same arguments for "venv".  The want to stay away from having to reconfigure every OS your script touches when it's none of it's business.

For now you have a RaspberryPI which you can dedicate to this project?  Then just customise it to do as you bid, no matter how hacky.  Just make notes on what you did.   I find a "blog" platform which accepts copy and paste images and will retain the syntax highlighting of copy and pasted code and config.  So when you screw it up really badly and need to start again, you can accelerate the process.

The Docker way I mentioned...  we build the whole environment and OS in the build script for every "clean install".

[Simon]

So I'm back to getting the CAN interface to run with a GUI. Yes I could just kick out command line stuff but That's not really what I want.

[Simon]

So I wrote this little skit:

Code: [Select]

def main():
    counter = 0
    while True:
        with can.Bus(interface = "socketcan", channel = "can0", receive_own_messages=True) as bus0:
            print_listener = can.Printer()
            with can.Notifier(bus0, listeners=[print_listener]):
                # using Notifier as a context manager automatically calls `Notifier.stop()`
                # at the end of the `with` block
                print("Sending messages...")
                bus0.send(can.Message(arbitration_id=1, is_extended_id=False))
                print("sent 1")
                bus0.send(can.Message(arbitration_id=2, is_extended_id=False))
                print("sent 2")
                bus0.send(can.Message(arbitration_id=3, is_extended_id=False))
                print("sent 3")
                print("Messages sent, waiting to receive...")
                counter += 1
                print(f"Iteration {counter}")
                if counter == 10: return
                time.sleep(700e-6)


The result was:

Code: [Select]


(env) simon@pi500plus1:~ $ /home/simon/python/workspace/env/bin/python /home/simon/python/workspace/cantest.py
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 1
Timestamp: 1762719275.922895    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719275.923265    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719275.923546    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Timestamp: 1762719276.926164    ID:      001    S Tx                DL:  0                                Channel: can0
Iteration 2
Timestamp: 1762719276.926506    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719276.926774    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 3
Timestamp: 1762719277.928980    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719277.929258    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719277.929532    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 4
Timestamp: 1762719278.931764    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719278.932055    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719278.932330    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 5
Timestamp: 1762719278.933244    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719278.933533    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719278.933811    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 6
Timestamp: 1762719279.935992    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719279.936270    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719279.936537    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 7
Timestamp: 1762719280.939024    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719280.939336    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719280.939650    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 8
Timestamp: 1762719280.940880    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719280.941191    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719280.941514    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 9
Timestamp: 1762719280.942576    ID:      001    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719280.942881    ID:      002    S Tx                DL:  0                                Channel: can0
Timestamp: 1762719280.943203    ID:      003    S Tx                DL:  0                                Channel: can0
Sending messages...
sent 1
sent 2
sent 3
Messages sent, waiting to receive...
Iteration 10
Timestamp: 1762719280.944575    ID:      001    S Tx                DL:  0                                Channel: can0

So on iteration 2 you can see that the program has continued and is doing stuff when "interrupted" So presumably I could set the notifier up and go do stuff. In one error before I got this to work I saw mention of threading.

But this is on a while loop, most GUI systems have their own scheduling system so how do I get them to work together? can i call this code without the while loop in the  GUI's own scheduler?

[paulca]

Try:

with
with
   while

with is a resource initialiser.  If you run it each time in the loop, you create a new bus in the loop and a new listener.

If you are going to be using the bus / notifier a lot just create it in the top scope and then either just quit and it will be collected and released or close it out yourself before quiting.

"with" is a short cut util which is equivalent to:


get some resource, use it to the end of this block and destroy it.  Opening a file is a classic.  A "with" block will close the file etc once it leaves scope.

[paulca]

The pattern it seems to afford is:

Create one bus, one bus handler loop.
Wrap it in a module/class whatever.
Give it a method:  Add Listener which just proxies the bus method and "Send".

Now, your application becomes a set of decoupled listeners.  The listeners decide if the message is of interest and do whatever they want with it.  If they want to send they call the bus send method.
When a message arrives the bus sends it to each relevant listener.  You can probably extend can.Listener or one of the others like can.Printer() and reuse bits, override the bits specific to your message handling.

The "Main" then becomes:

Create bus.
Create listeners.
Attached listeners to bus.

while True:
   monitor and do nothing.

[Simon]

Yes so my dilemma is, how the hell do I get my hands on the message data!!???

My efforts so far below but I just can't figure out how to access the data.

Code: [Select]


bus0 = can.Bus(interface = "socketcan", channel = "can0", receive_own_messages=True)

def can_listen_handler(self):
    #msg = bus0.message_reader.get_message()
    #print(msg)
    can.Printer()

can_notifier = can.Notifier(bus0, [can_listen_handler])

#print_listener = can.Printer()

def main():
    counter = 0
    while True:
        #with can.Bus(interface = "socketcan", channel = "can0", receive_own_messages=True) as bus0:           
                # using Notifier as a context manager automatically calls `Notifier.stop()`
                # at the end of the `with` block
            can_notifier
            print("Sending messages...")
            bus0.send(can.Message(arbitration_id=1, data=[1,2,3,4,5,6,7,8], is_extended_id=True))
            counter += 1
            if counter >= 2:
                bus0.shutdown
                return
            time.sleep(2)



[Simon]

So you mean something like this:

Code: [Select]


def main():

    with can.Bus(interface = "socketcan", channel = "can0", receive_own_messages=True) as bus0:

        def can_listen_handler(self):   
            can.Printer()

        with can.Notifier(bus0, [can_listen_handler]):
            counter = 0
            while True:           
                print("Sending messages...")
                bus0.send(can.Message(arbitration_id=1, data=[1,2,3,4,5,6,7,8], is_extended_id=True))
                counter += 1
                if counter >= 2:
                    bus0.shutdown
                    return
                time.sleep(2)

This still does no collect the messages and it feels all very awkward. I mean lets say I set up the other CAN channel to do some proper transmit and receive. All of this is under the creation of the other bus and I am using tabs at such a rate that I'll need a very wide screen by the time I am done. I just wish python could have been "normal" and used braces, it's so much faster. Every time I want to edit I have to move all the code over a tab.

[jfiresto]

... I am using tabs at such a rate that I'll need a very wide screen by the time I am done. I just wish python could have been "normal" and used braces, it's so much faster. Every time I want to edit I have to move all the code over a tab.

The common solutions for that are to call functions that contain the overly indented code and to use an editor that is or can be tailored to the programming language. You can use the functions to help explain the code.

I just finished a transmogrified tokenizer for some mostly Python, testing code that helps recognize as statements, divider lines (e.g., ======), and comments that start with ##. It uses a callback and nested functions to sufficiently reduce the innermost statement indentation to column 16.

I still use EMACS (still works for me 40+ years later) which in Python mode lets you indent and unindent code blocks, four spaces, with Ctrl-C > and Ctrl-C <.

[paulca]

It looks awkward too

Get rid of the "with" entirely.  It's not really necessary, more dogmatic.

Just create the bus at the top of the file.

Code: [Select]

bus0 = can.Bus(interface = "socketcan", channel = "can0", receive_own_messages=True)

Outside of all functions.  Global scope.

The listeners is where you put your code.  By "create listeners" I didn't mean instanciate the default printer, I meant write one.

Something like:

Code: [Select]

class TestListener(can.Listener):
    def on_message_received(self, theMessage ):
         print("The timestamp: {0!s}".formatt( theMessage.timestamp ) )
         # rest of your code here

myListener = TestListener()

Then attach that to your bus0.

Code: [Select]

notifier = can.Notifier(bus0, myListener)

Then you can go and loop.

That's the gist.

You can put it all in one file, bare core is executed when it's "imported" or run.  No need for def main();.    When that starts to get old and messy, you can split it.  Don't worry about it yet, you can create dozens of classes is a single file.

[paulca]

Note, the next thing you will probably need is the way to send a message in response to another?

Pass the bus to the listener.  It may have a way to get it anyway, but you can pass one anyway.

Code: [Select]

class TestListener( can.Listener ):
   def __init__(self, theBus):
      self.bus = theBus

   def on_message_received(self, theMsg):
       pass # you code here

   def on_error(self):
      pass # etc

Then when in "on_message_recieved()" you want to respond you can call:

self.bus.send()

WARNING: In it's current state if you do this it will loop and cascade and crash the bus and the script.  Obviously calling send in a listener that receives all messages in cluding it's own is a message loop.

The idea would be to determine if the message is "for you" in the listener.  Just add  a function, "is_of_interest(theMsg)" which looks at the ID, sender, whatever parameters and handles one type of message.  Listeners can be single ID/topic or multiples.

def on_message_recieved(theMsg):
    if not is_of_interest(theMsg):
      return

Note that is the other way of getting around nested tabs/ifs "early return to simplify".

If your UI later wants to show a live updating value, then you could send that update to the UI in the listener thread.  Just be wary of that being the listener thread.

[paulca]

Not sure how closely this parallels your use case, but it's an example of my code.  It is a small module/service within an established system.

It's purpose is to collect data from the wider message bus and compact it down to a single short message for an MCU to display.  The MCU did not like interfacing with the full bus, it was just too much, too often, too verbose.  So I wrote this python service to "abridge" things into a summary topic.

https://gitlab.com/paulcam/home-auto-mqtt/-/blob/master/mqtt-heating/display_summary.py

I picked it for you because it's a bit "rough and ready" and not "professionally overdone", so might be comforting   No classes.  Just call back functions.  The imported classes are mostly also mine and available at the same link.  It is ultimately the same pattern as the can module.  Callbacks/Notifiers/Listeners/Observers/EventHandlers.  It's all async.

The "main" functional code is just creating a bunch of topics and attaching the call back functions.  Then using the clients "blocking_run" to hand it the main thread.  That's the way this client works, I'm still not sure on the CanBus module.  I think you will need a "while True" somewhere.

Code: [Select]

update_stats.pub_time = time.time()
temp_topic = "home/sensors/temperature/+"
power_topic = "home/sensors/mains/mainsPower"
solar_topic = "home/sensors/solar/+"
rh_topic = "home/sensors/humidity/+"

mqtt.subscribe(temp_topic)
mqtt.message_callback_add(temp_topic, on_temp_message)

mqtt.subscribe(power_topic)
mqtt.message_callback_add(power_topic, on_power_message)

mqtt.subscribe(solar_topic)
mqtt.message_callback_add(solar_topic, on_solar_message)

mqtt.subscribe(rh_topic)
mqtt.message_callback_add(rh_topic, on_rh_message)

mqtt.blocking_run()


[Simon]

It looks awkward too

Get rid of the "with" entirely.  It's not really necessary, more dogmatic.

Just create the bus at the top of the file.

Code: [Select]

bus0 = can.Bus(interface = "socketcan", channel = "can0", receive_own_messages=True)

Outside of all functions.  Global scope.

The listeners is where you put your code.  By "create listeners" I didn't mean instanciate the default printer, I meant write one.

Something like:

Code: [Select]

class TestListener(can.Listener):
    def on_message_received(self, theMessage ):
         print("The timestamp: {0!s}".formatt( theMessage.timestamp ) )
         # rest of your code here

myListener = TestListener()

Then attach that to your bus0.

Code: [Select]

notifier = can.Notifier(bus0, myListener)

Then you can go and loop.

That's the gist.

You can put it all in one file, bare core is executed when it's "imported" or run.  No need for def main();.    When that starts to get old and messy, you can split it.  Don't worry about it yet, you can create dozens of classes is a single file.

I used the built in message printer as for all the reading I have done, what the hell I call the message that was received eludes me. So I have nothing to take the message contents from, so I used the printer method just to test out the notifier.

[paulca]

The bus passes the message to the notifier which calls the on_message_recieved() of the listeners with the message.  So only the Listener sees the message. 

I know it can feel awkward that the plain code in front of you is not what is being executed behind the scenes.  Its all "inversion of control" based.  You create handlers that do the actual work, assign them to the bus and start the bus.  Your primary code path comes to a bizarre end and nothing appears to tie everything together.  You either trust the module you are using or read it's code to give you confidence.

Most of your code will exist in custom listeners.  Either full classes or just bare call back methods.  The can bus stuff will fade out for you and your message handlers (listeners) will be your bread and butter.

[Simon]

You bet it feels awkward!!!

The bus passes the message to the notifier which calls the on_message_recieved() of the listeners with the message.  So only the Listener sees the message

So the listeners must be a class that inherits from the listener class? I know that I should know more about python but even if I did I don't know if the garbled crap that passes for instructions on the python-can manual would make any sense even then....