programming a single board computer as a machine controller (HMI)

[Simon]

What I am looking at is a single board computer - no specific one so only using standard features.
Minimize the OS stuff to speed up boot and keep the user out of the system.
send and receive serial port messages
run some sort of application to display data, menu's etc
save data to files.

Python seems to be the universal thing these days on single board computers. Is it the whole solution, or part of it?

[SiliconWizard]

For RPi SBCs, Ultibo is an interesting solution. This is pretty much a baremetal environment with good support for almost all RPi hardware and the most complete I've run into. It requires programming in Pascal.

https://ultibo.org/

I don't know of any similar baremetal (no OS per se) framework for any SBC that would host a Python interpreter.
Baremetal frameworks for SBCs are rare enough already.
Apart from Ultibo, there's also a C++ library (Circle) for the RPi: https://circle-rpi.readthedocs.io/en
(being C++ might be plus for many compared to Pascal, but it's a lot less complete than Ultibo).

For anything other than RPi's, I have never seen anything like that. Doesn't mean it doesn't exist.

[Simon]

I can't tie myself to RPi, I'm not sure bare metal is what we want, the whole point was that something with an OS would have support for what we want.

[SiliconWizard]

I can't tie myself to RPi, I'm not sure bare metal is what we want, the whole point was that something with an OS would have support for what we want.

I see, in that case you can absolutely configure a supported Linux distro on the SBC to boot up with the minimum, enable only the strictly required services, and no desktop environment (this is often what takes the most additional boot up time and resources). Boot times in the order of just a few seconds are possible without much trouble. As this point having a working Python environment should be trivial as well.
So, that will require configuring a number of things manually if you start with the usual "customer" Linux distros that ship with these SBCs, but once it's done you can of course clone it as is in production.

I've used ArchLinux ARM ( https://archlinuxarm.org/ ) on SBCs (supports quite a few), because I'm used to ArchLinux and it's well adapted to preparing minimum Linux systems, but you should be able to do it with Ubuntu or Debian-based distros without much problem. You'll have again to strip them off significantly.

The one thing to really figure out is for the display part of it - install the minimum that can be used via a Python library (if Python is what you'll use) to implement GUIs and do not require a fully-fledged desktop environment.
Nominal had suggested using Qt for this kind of use case, then there's PyQt5 for Python. If he reads this, possibly he can give further guidance on how to setup Qt without a desktop environment, and ideally without X11 either.

[Nominal Animal]

What I am looking at is a single board computer - no specific one so only using standard features.
Minimize the OS stuff to speed up boot and keep the user out of the system.
send and receive serial port messages
run some sort of application to display data, menu's etc
save data to files.

There are three major choices:

• Debian, Arch, or some other Linux distribution with wide hardware architecture support
• Yocto Project
• Your own minimal distribution; see e.g. OpenEmbedded and Linux from Scratch

The Debian/Arch/Distro approach is to install a minimal system that provides the services your application needs.  You'll probably want to rip out systemd, and use either one of the existing init systems, or write your own, to speed up boot and the system robust enough as an appliance.  See Devuan for non-systemd alternatives like non-systemd udev.

Yocto and OpenEmbedded are designed to help with constructing your own distribution, whereas Linux from Scratch and Beyond shows all the bits and pieces needed to get a working system together.

The next choice is whether you'll run X11, Wayland, or on pure framebuffer (actually DRI with OpenGL ES acceleration, and not a traditional framebuffer).  X will let you run more than one application, and write the UI or entire application in whatever programming language you prefer.  I don't know how well Wayland is supported on armhf or arm64, especially Mali hardware.  Qt supports bare framebuffer (EGLFS plugin for Qt5 or Qt6), and you can use it in a commercial proprietary appliance relying on the GPL or LGPL license; in that case, using Yocto and Python3 for at least the user interface is probably a good idea (as that makes the licensing boundary extremely clear, although dynamic linking is considered a derivation boundary nowadays).  You can always put any performance-sensitive sekret sauce in a dynamically linked native library, and access via Python built-in ctypes module.

Hardware-wise, optimize for I/O ops on small files, and not maximum streaming read/write.  PCIe or SATA SSDs will be your friends here, even for low-power SBCs.  MicroSD cards are dead slow in comparison.

Python seems to be the universal thing these days on single board computers. Is it the whole solution, or part of it?

If you have any performance-sensitive stuff, you'll want to do it in native machine code, and using C or C++ for these is quite common.

Using Python for the user interface makes development cycle easier and faster, because you won't need to build the software before running it.  The Qt project has a somewhat straightforward Yocto + Qt Embedded toolchain, where your development machine uses a standard X+Qt environment, but the target on plain OpenGL ES with no X.  If you do run an X server, you can also use GTK+ via gobject introspection bindings, i.e. gir, with Python.  Basically, both GTK+ and Qt are very 'native-like' in Python.  I even prefer to use their UI builder interfaces at runtime, constructing the user interface widgets from an XML file instead of generating the code to do so, but this can slow the UI coming up initially by a second or so, mostly because of the large number of I/O ops.  See Glade for GTK+ for a GUI editor that can generate such XML .ui files, and Designer for Qt.  (In case someone claims you need a commercial Qt license, skip their mistaken beliefs, and go read e.g. this instead.)

Also, I personally love the idea of the UI being something knowledgeable end users could tinker with, using only an SSH client and a text editor, without risking the entire device, as long as they keep a backup copy of the original Python code and UI files, of course.  While Tux is my mascot, it is not my idol, and I've done a lot of proprietary work too; the dynamic linking when using open Python UI code and any proprietary processing in native-only dynamic libraries is also a valid copyright license/derivation boundary, so it is an interesting option even for proprietary appliances.

The appliance is always run using a dedicated user account, not as root, unless one is an idiot.  Compared to a standard desktop Linux system, the GUI application replaces the 'greeter', the application that displays the standard login form.  You do not necessarily need a window or session manager at all. Isolating the UI under a single user account also means that if you set up your init system right, you can simply kill all processes running as that user, and restart X and the GUI application, without having to reboot the system if something hangs up –– for example, if your main GUI process crashes.

Using X and multiple windows means that you can extend your UI with external programs, which can help keep individual parts simple, and ensure you utilize multi-core SoCs to their fullest.  Even though these SBCs don't have desktop-like amounts of CPU power, they can still shift a lot of data around, so do not be too wary about using pipes and Unix domain socket pairs to move data between processes.

As to serial stuff, you'll want to use termios instead of any serial library: the libraries are all crappy, and the termios interface already exposes everything you need.  If you have dedicated devices, like USB-Serial dongles, use udev and udev rules to generate symlinks in /dev, and use those symlinks names directly without any lookup/serial port search nonsense.  With Python, you'll want to use a separate thread for each serial port, or even two threads if using asynchronous messaging instead of a conversational request-response interface.

I seriously believe the same regarding USB bulk transfers and usbfs /dev/usb/nnn/mmm interface –– avoiding libusb altogether ––, but saying that out aloud will probably start a flamewar.

[ejeffrey]

One big question is when you say "save data" -- to where?  Local filesystem for local browsing only?  Save to removable media?  Local filesystem and run a samba or ftp server on the embedded device to allow remote access?  Automatically save to a user configured network server of some kind?  Save to a cloud service?  Is that a cloud service of your choice or the customers choice?  Do you want to include a user operated network client like an ftp client or browser to let them upload somewhere  manually?

Also does configuration data need to be loaded from any or all of the above?

This is the area where I usually see the most pain both for development and for your users.  Any network access is an external dependency that you must be able to deal with transient or permanent failure, and the user experience for setting up a lot of these is not great.  Local storage is easy and dependable but only delays the problem until the user wants to get the data.  Performance and reliability of local and removable storage is also an issue depending on your hardware choice.

Anyway I think this is an important part of the architecture to think about from the very beginning because it can have a lot of downstream effects.

[Nominal Animal]

Nominal had suggested using Qt for this kind of use case, then there's PyQt5 for Python. If he reads this, possibly he can give further guidance on how to setup Qt without a desktop environment, and ideally without X11 either.

One compiles the Qt core with support for the desired plugin, typically EGLFS.  This is well outlined here for Qt5, and here for Qt6.

Both Qt and Gtk+ can be run on plain X without a window manager or session management of any kind, but it can be quite weird.  Window positioning and sizing has to be done 'by hand', and unless you do a fullscreen window, you'll probably want a separate process to display a background image.  I recommend playing with that on a suitable VM first, to understand the limitations.  The four-pixel checkerboard you get familiar with, is what a plain X server displays with no window management or root window process running.

If the SBC has 2 GiB RAM or more, I would prefer using X myself.  The reason is that then you can split the UI into multiple processes –– one fullscreen one, and one or more floating windows on top ––, which especially with current Python interpreter only running one thread of Python code in a process, means you can get more work out of current multi-core SoCs.  Depending on the storage used, X startup can be a significant boot 'delay', however.  With an SSD instead of SD card or EMMC, it isn't an issue, though.  And in Linux, even el-cheapo JMS578-based USB3-SATA adapters should work well.

With X, stuff like network interface configuration could be their own process, instead of having to be a part of the main UI application.
Adding a configuration menu, populated from .desktop files in a specific directory, would make maintenance and adding config widgets later on much easier.  Python built-in configparser module makes reading the desktop entries really simple, too.

[MarkF]

I don't have any experience with any of these...

What about one of those Mini PCs?  I believe they can run Windows or Linux.
For instance, the Minisforum EliteMini N40.
Or, do you want something smaller?

[Simon]

I don't have any experience with any of these...

What about one of those Mini PCs?  I believe they can run Windows or Linux.
For instance, the Minisforum EliteMini N40.
Or, do you want something smaller?

Something that powerful is hopefully not necessary.

My intention is that the code can be run from a PC with a serial port, that is how universal I want to be. Screens are a bit of a problem as we only need 4-5inch and these days that means DSI and potential compatibility issues versus HDMI.

I have no understanding of Linux under the bonnet, I know the desktop can be disabled and that this may limit what I can do but that is my limit or knowledge. I have read a few pages of a python book and got a simple drawing to appear by copying out the program. What I am trying to do is establish a high level understanding of what is require so that I can pick the best way and learn it or direct someone else, without them going down a rabbit hole that is hard to back out of later.

An EMMC is the initial choice, I guess SSD's can be considered later if needed, files would be stared locally for removal by memory stick. A nice option might be to transmit logs automatically by email when the internet is connected. We will need to produce a particular format of PDF but I'm not too worried about that right now and that is ultimately why a SBC is being looked at as you need some processing grunt and space for the libraries.

[Simon]

What sensible options are there off the shelf to buy a unit with "easy" to use software? If I google there are a ton of them lots talking about factory environments and networking which is not quite what we want.

[MarkF]

If you don't buy into the RPi, I'm at a loss as to what other hardware choices out there.

With a RPi running Ubuntu or Debian with Python and PyQt, you would have:
 - a small SBC that you can add a display
 - serial I/O to talk to your system
 - Qt to create a display window with dropdown menus

I'm a C/C++ guy who has used Qt on Redhat Linux and Windows 7.  So, no experience with Python.
You will need Qt if you want typical dropdown menus in a window. 
However, Qt on a PC has a very steep learning curve.

  Unless I totally misunderstand your requirements, there is no easy solution and will probably require a pretty skilled programming person.  And from what others have said here, there doesn't appear to be any easy off the shelf tool out there.

[MarkF]

Come to think of it...

Take a look at Node-Red for the RPi / BeagleBone / Arduino.
It might do what you want

[Simon]

I'm thinking of stuff like this: https://www.unitronicsplc.com/

[Simon]

But yes nodered looks interesting