RPi + custom micro-controller board + python

[Simon]

I'm trying to devise hardware and software for a HMI, yes, for real this time 

So given how the overall system may develop I want to use a single board computer that communicates with a custom board with a microcontroller that talks over some sort of simple serial connection UART or SPI seem to be the options. And I'm looking at python as I'm now satisfied that it's the fastest way to get things off the ground.

From what I see SPI could be faster than UART. I'm not sure where to start either way. On a microcontroller I would deal with serial comms in terms of strings of bytes and knowing exactly where to find them. With a serial port in python I'm not sure how I identify my start and end of data.

[radiolistener]

You don’t need to manually detect the start or end of data when using a standard UART interface — this is handled by the hardware. Each byte is framed by a start bit and one or more stop bits, so the receiver automatically reconstructs complete bytes. At the output, you receive a fully formed byte stream, ready for processing.

Typically, commands are organized as packets with a standard structure that includes fields such as packet length and a checksum. This way, you simply read the incoming byte stream as packets in FSM: if a packet validates correctly, you process it, if there’s an error or a timeout, you reset the receive buffer and start over. This process repeats continuously in a loop.

[Simon]

Yea I'm not worried about the bits no more than I have to worry on a microcontroller. It's when I get data from the port it will be a string of bytes that needs to represent the message from start to end.

I see that the serial library has methods to sit and wait for data up to a certain amount of characters so in theory I can get one packet at a time.

[voltsandjolts]

You could connect a display directly to the RPi, so why the mcu?

[Simon]

Because the display will be connected to the RPi. The MCU acts as an interface, any analogue channels, encoders, buttons and the CAN bus. Importantly the MCU is real time, so I have the best of both worlds.

[voltsandjolts]

Ah, yes, got it.

If you do go the uart route for comms between the mcu and RPi, you might consider using COBS for framing.
https://www.eevblog.com/forum/microcontrollers/implementing-uart-data-packets-with-consistent-overhead-byte-stuffing-(cobs)/

[abeyer]

I see that the serial library has methods to sit and wait for data up to a certain amount of characters so in theory I can get one packet at a time.

That works for simple stuff, and may be enough. Another built-in option if you use a plain text(ish) format is to use readline with an appropriate EOL character to read line-wise, which can allow variable length packets without extra work on your part. You'll likely want/need to set and handle timeouts for robustness, too.

As you get into more complexity it can be very beneficial to decouple the IO from the logic... At the simplest this could just be a ring buffer where your serial IO code just drops bytes at the tail as it receives them, while the protocol handler logic watches the head of the buffer for complete packets that it can process. A step up in complexity (but also, imho, a very nice decomposition model) is to define your protocol as a "pure" definition of the communications rules in a state machine that defines packet structure and any higher level statefulness. The protocol just consumes bytes as they arrive and produces some sort of "domain events" as the state machine dictates. Then your business logic gets built as a set of "handlers" that can respond to those events as needed.

[rteodor]

HDLC (High Level Data Link Control) framing. Its simple, can be implemented in a few lines of code.

Classic async HDLC use 0x7E to mark start and end of a frame or separation between frames. If 0x7E or 0x7D appear in the data, then data is "stuffed" a.k.a prefixed with 0x7D and the byte is sent with byte 5 inverted. That is all.

But it does not have to be exactly that: I use accolades for start and end (0x7B and 0x7D) because its easy to spot a frame during debug. For stuff char I use 0x7E and no inverting of bits.

[Simon]

Most of what I transmit is numbers, leading zero's are going to be very likely so converting them to something else will be a pain. As I transmit numbers there is no combination that is off limits. Literally the only way to avoid that is to use ASCII only, not impossible I suppose but it would more than double the bandwidth.

As I am coming from CAN bus in the system (CAN open) maybe I am overthinking this and should just use a CAN bus shield or adapter. The reason I was looking to make my own adapter was that I don't want to tie myself to a particular SBC or CAN bus adapter and be forever sorting out drivers. SBC's all seem to have inbuilt serial ports making it a good universal boundary between things that will be there on any SBC.

[Simon]

Well I got a CAN bus hat to work fairly easily. I thought it would be harder. It seems that dtoverlay is everyone's friend. I came across many discussions about the hardware pin drivers that seem to have to be updated with every new board and there is not one for the Pi5 as it's more than some address changes as the IO is now on a dedicated IO chip rather than the CPU.

If I understand correctly once I setup the dtoverlay parameters I can use any board I like without messing about with drivers.

So I guess next job is to try and make something graphical that will send and receive messages putting rxed messages on screen and with some sort of input to trigger a txed message. I suspect it's time to lean about concurrency and threading.

CAN bus will have the advantage that I won't need to actually transmit things like the message ID and time stamp over the serial bus, they are just there. I am assuming that CAN bus is easier on the system than UART as the controller handles the buffering and verification of the data integrity. 8 bytes of payload is generally enough for me as that is how I have to roll anyway with the rest of the system.

[rteodor]

The CAN signals concept is like having thousands of virtual wires each one dedicated for transmitting a single "signal".

Serializing data over a CAN frame works too, there is a standard now in cars (UDS I think) for transmitting data that is longer than 8 bytes. That is for diagnosis or RDS text strings between ECU's for example.

[Simon]

Well I tend to use the CAN open system as I work with motor drivers and they tend to use it. Having created the required code to run a CAN open system off a microcontroller I find it sufficient to deal with handing data around all my devices as they either sit on the same network or are selectively passing on data that may have been reprocessed that was originally gained over a CAN bus running CAN open so it's all the same. CAN open also has means of transmitting data that is longer than 8 bytes but I have not needed to resort to that.

What I will have fun with next is working out how to run program sections in different threads as from what I can see receiving CAN data is a blocking operation where the thread just halts waiting for a message. I don't want to hold up the entire program like that.

[HwAoRrDk]

Serializing data over a CAN frame works too, there is a standard now in cars (UDS I think) for transmitting data that is longer than 8 bytes.

You're thinking of ISO-TP. While UDS uses ISO-TP, it's a higher-level diagnostic protocol; ISO-TP is the transport layer that splits longer message payloads across multiple CAN frames.

[Simon]

So my modern Tkinter book says that I should not used asyncio with tkinter as the two schedulers will just screw each other. I should put tkinter and the blocking io stuff in another thread.

The can module has functions to listen to the bus that are blocking. But then there is the notifier system that has something to do with asyncio which should be fine as long as the CAN bus stuff is in it's own thread.

am I on the right path?

[abeyer]

I'd say that no path including tkinter is the "right" one  ... but not sure that's what you're looking for.

[Simon]

I'm looking for a solution to the problem, and every possible path with have people saying it is terrible whilst also having people saying it's terrific.

Is there a better solution ? tkinter is just the default GUI library. I'll use something else if it's actually a better solution.

[abeyer]

I haven't done much desktop gui from python in a while... but I think current versions of both gtk and qt have asyncio support.

IMHO tkinter being a "default" is just a bit of legacy cruft... tk was the lowest common denominator of what was likely to be available on a linux system early in python's life. However gui frameworks (outside a monolithic platform like Microsoft where you can control it all) really don't belong in a language's stdlib and in hindsight it was not a good idea. Some platforms no longer include tkinter in their default python install at all.

[Simon]

Yea I was trying to use qt in it's own IDE with C++ but the documentation is just a nightmare and I never really got anywhere. I don't know how much easier it becomes in Python.

Is GTK any better documented?

I bought this book "Modern TKinter for busy python developers" by Mark Roseman, he reckons it's had a bit of a rebirth and is not the awful thing it used to be but I don't know having not tried every offering.

[SiliconWizard]

I believe Nominal did post examples of using Qt with Python, and it may even have been in one of your threads?
If you prefer using Python for this, it certainly is more approachable.

[Simon]

Well python seems to be all the rage. I scarcely know C++, I have been using it for some 8 months now on a microcontroller but that is completely different from on an operating system so either way I start from scratch and python seems simpler as you say.

I'll see if I can find the examples, searching this forum is hopeless but maybe I can coerce google into it.

[SiliconWizard]

I think this was this one: https://www.eevblog.com/forum/programming/gui-with-gauges-in-python/msg5709465/#msg5709465

[Simon]

Ah yes, thank you. It has indeed been a year, I then went off to play with the QT IDE before loosing the plot and coming back to python.