EEVblog Electronics Community Forum
Products => Computers => Programming => Topic started by: rx8pilot on March 22, 2020, 02:10:26 am
-
My Python skills: Medium
My PyQt5 skills: Beginner
Project: Skill building
Description: Taking my crude console applications and building a QT5 GUI for them.
I have written a handful of programs that do various things - gather data from the network, process data, read/write to files, send/receive serial data, etc, etc. They are all console apps and require a LOT of typing. Additionally, I have been learning how to build PyQt5 based GUI's. The goal is to take the existing backend code and develop a GUI to make it easier for me use.
Now that I have been able to build a starter GUI [using Qt Designer], I am trying to figure out where to turn next to get data in/out of the GUI from my existing code. Hoping that I don't need to zipper them together.
It appears that since the GUI and the background applications all need to be running simultaneously, should I be looking at multi-threaded solution? Perhaps someone knows of a learning resource that covers this [at a beginner level]?
The PyQt5 docs are a bit hard to follow for me on this topic, although they were much better getting the framework of my GUI working.
-
I haven't found any good resources for this, so I've kinda-sorta worked on my own guide for this, but I have nothing in publishable form yet. (My focus was/is creating Python3 - Qt5 graphical user interfaces for USB or USB Serial connected microcontrollers; ones that would actually work, instead of occasionally locking up or becoming unresponsive if the microcontroller took a couple of seconds to respond.)
First, you must separate tasks that need to occur synchronously, and tasks that should happen asynchronously. I am using synchronous in the sense that the user interface is not otherwise usable during the task; and asynchronous in the sense that the user interface stays fully functional during the task.
For example, loading and saving a document should be synchronous. Things like requesting and obtaining data from a remote source (say, for displaying a dash of your web services statuses) should be asynchronous.
Synchronous tasks are done in the user interface thread in your process.
Asynchronous tasks can be done in threads or processes. Because the CPython (standard Python interpreter) only interprets Python code in one thread per process, you'll want to run heavy computation in slave processes, connected to your process via pipes or sockets. To avoid deadlock (both ends expecting to read a response, or to write a response, first), you want to put the communicator or communicators in your Python code into a separate thread, that passes the results/objects to the GUI thread using a Queue, for anything that uses a request/response protocol.
If you can give me an actual (simple?) example case, I might be able to sketch out more details.
-
Ok, excellent.... my test setup is intended to do a couple of things.
[edit] The learning project is to do some automation at the Factory400 - multi-head air compressor, vacuum system, lighting, inventory control with RFID and barcodes, etc. Eventually, it will be handled with Rasberry Pi type SBC's with small touch displays. A fairly useless endeavor, but a great way to learn a LOT.
Module A:
Communicate with serial connected microcontrollers. Gathering data, sending configuration data, etc. Timing is not critical, and the data bursts are unpredictable. This is likely to be expanded to about 6-8 COM ports.
Module B:
Connected to a MongoDB or similar. It will use the data coming from Module A and process it with the data from the database to make make decisions, output the appropriate actions. Save the processed results to the database.
Module C:
The GUI that will display all the data from the other modules and allow me to input control information with sliders or direct input.
I would expect Module A and B to remain fairly busy, hoping the GUI will update graphically at a refresh rate suitable for a human (slow)
SIDE NOTE:
I found a resource called https://www.learnpyqt.com/ (https://www.learnpyqt.com/) which seems to be a good start. The random pieces and parts of tutorials and documentation have been challenging to punch through for me. I have been able to get some useful results, but I still don't feel like a 'get it', the topic has not sunk in yet.
Thank you for taking a moment to assist. :-+
-
a great way to learn a LOT
Fully agreed!
Note that I will try to write this in a way that might help others reading this thread too; I may be repeating things you know very well already.
As you probably already know, you can save your efforts and limit the sources of bugs, by concentrating on each module at a time. (Or, multiple modules at the same time, just completely separately, not combining them into the same program yet.) I personally write really simple main() function for testing the module I am working on, so that I can verify each chunk of code as soon as I write it. This way, I limit the possible sources of bugs to the code just added, or rarely to code that interacts with the just added code (but it is almost always the code just added). I am lazy, and this saves me tons of efforts.
Module A: Communicate with serial connected microcontrollers.
There are two completely distinct sub-tasks here: enumeration and operation.
Enumeration means you probe and list connected devices. This is usually synchronous, but is sometimes done asynchronously or continously, so that new devices are detected and listed as soon as they become available. In practice, you want to split enumeration off from the device operation module; you'll want to e.g. have a single supported serial device enumeration class. It may feel odd initially, but most devices on the same bus (here, using USB serial or standard serial ports) are detected the same way, so this actually does make sense.
In Linux, I tend to use a script as a helper process, for enumerating devices on the USB bus. It can examine the sysfs pseudo-file hierarchy, specifically idVendor and idProduct files in /sys/bus/usb/devices/*/, or use Bash to run for D in /dev/tty* ; do udevadm info "$D" ; done and process that output using e.g. awk (outputting the "E: DEVNAME=" field when the "E: ID_*=" fields match, with an empty line or a "P: *" line as a record separator).
For Linux SBCs, it can make more sense to use udev rules to create suitably named symlinks. Then, it is sufficient to use one or more glob (https://docs.python.org/2/library/glob.html).glob()s to detect devices when they come and go, say once every couple of seconds for example. (/dev is typically an udev filesystem, completely in RAM, so this should not cause any wear on your storage devices.)
For operation, there are two options: direct communication from Python using pyserial (portable) or termios (Linux/Mac/*BSD), or via a helper process. The helper process makes sense if the devices are timing-sensitive, or you support many different devices and protocols providing the same information. In that case, the helper process acts as a buffer and translator. If the device is not too sensitive wrt. timing (say, can deal with communication latencies of say 100 ms = 0.1 second), and no translation is needed, there is no reason to use a helper process.
For the Python operational code, there are two types of devices: those that use a request-response protocol, and those that output a stream of data, and accept/respond to commands asynchronously.
For the request-response devices I use a single thread per device. The thread runs a loop, that begins with waiting on the queue for a new command to apply. Then, it acts on the received command, by sending whatever command is needed to the actual device, and waits for the response. After parsing the response, the data is posted to the data queue, and the response (or command completed) is posted to the response queue.
For completely asynchronous devices, I use two threads per device. One thread only sends commands: it waits for commands to apply on the command queue, updates internal state shared with its sibling thread, and writes the command(s) to the device. The other thread only receives responses and data. Note that command responses are posted to the response queue, and data to the data queue.
As you can see, there are actually quite a few queues needed. Each queue is one-directional, and may contain any number of objects in flight. Each device has their own command queue. Only one thread gets items from the device command queue, but any thread can put commands to any device command queues. You will want to have a identifier (unique counter used by the GUI thread), so that a single queue can be used to report command responses (success/failure state). A separate queue is used for posting data objects.
You will want the GUI thread to be able to generate commands, and to process the command responses, because it is very much up to a human to decide what to do when stuff goes pear-shaped; we, the programmer, cannot know that beforehand. I like the idea of a status bar, where such communications errors could be shown. When designing the queue structure, and what kind of objects you put/get, this is worth thinking about.
Module B: Connected to a MongoDB or similar.
Sounds like a data arbiter to me.
Perhaps a multiplexer/arbiter that is the only one who gets objects from the data queue (where device threads put data objects to), forwarding summaries or changes to the UI thread using a separate queue?
The reason I tend to call this a multiplexer, is that a simple multiplexer would multiplex each data object, storing one copy in the database, and forwarding it to the UI thread. This way the UI thread would not need to worry about the database at all.
The data queue could also be used by the UI thread to tell the multiplexer/arbiter which data it is interested in. In this case, device threads would be puting data objects to the data queue, and the UI thread would put data selection objects, directing the arbiter as to what data to pull from the database, and whether the UI is interested in the latest data or not.
The most important thing for this module, however, is to abstract any database internals away. You will make changes to the tables at some point, so keep it easily modified/configured; and do spend a thought or two as to how to import and export old databases to newer versions. (It is much better to do a reimport than to try and keep backwards compatibility. Besides, you might wish to do periodic backups anyway; exporting the database tends to work better than snapshotting.)
Module C: The GUI that will display all the data from the other modules and allow me to input control information with sliders or direct input.
In general, it is crucial to understand that the user application runs under Qt, not the other way around.
Like most other toolkits, Qt is event-driven (https://en.wikipedia.org/wiki/Event-driven_programming). Instead of writing code that runs and occasionally calls Qt stuff, it is Qt that runs, and calls the user code as needed, whenever something interesting happens. There are ways to ensure user code gets run periodically, for example using the QTimer class. The main thing is that while the user code runs, the user interface is unresponsive. (No, events are not "lost" or missed; they just are not acted upon before the user code returns.) Because of this, we use worker threads or processes to do any heavy computation, or anything that takes appreciable time during which the user interface should stay responsive.
The event methods for most buttons or sliders should be basically very simple, just putting a suitable control or command object to the corresponding queue. The programmer is essentially trusting that everything will work right, and when the device threads respond with the result, the display is updated to reflect the change. (You'll want to let Qt update numeric values immediately; I just mean that the change of value in e.g. a slider should only result in a command object being put to a correct queue, and not much else. If it is e.g. a scroll event on the graph, the graph can be cleared, with the correct data displayed when it arrives from the arbiter/multiplexer.)
For scrolling or realtime graphs, I recommend splitting it into chunks. If the graph is continuously updated, use smallish chunks, and only redefine the current chunk, and move the others. If the graph is usually of historical data, use bigger chunks, and move them instead of redrawing them. This technique allows drawing quite complex graphs easily in real time, without consuming too much CPU time even on SBCs. Usually, the currently changing chunk is a QImage, with the already fully drawn chunks being QPixmaps. For a simple graph (a single time-varying curve, or a couple of them), using QtSvg instead may produce better results (as it may be better accelerated/optimized).
As a practical example, consider a realtime graph of say temperature and humidity. The horizontal position of the graph chunks would depend on time, with the newest chunk being redrawn each time; and when a chunk exits the view, it gets copied from the newest chunk, inserted after the newest chunk, and the newest chunk cleared. (Well, you probably want to do some nifty redraw stuff, and overlap chunks by a pixel or so, to ensure you have antialiasing over chunk boundaries. But that's details.) Note that arriving data is redrawn to the newest chunk, and that the chunks travel (move) according to time. (This same approach works with Javascript on web pages, too; increasing smoothness or update rate by not having to redraw more than a small area, with the rest being just copied/moved as images/pixmaps.)
Sure, you can make the entire graph a single QtSvg for example, and redraw it for each frame, but that requires quite a bit of CPU, or hardware acceleration.
As to implementation, you can write a module that generates a combination of sine waves, and emits new points regularly. Then, you can experiment on how to implement a sliding graph efficiently on your SBCs. You can even implement the control mode for the sine wave generator, so that if the user scrolls away from the "current time", the module emits the data as if it had read them back from the database; and only reports new data when in "tracking" mode.
I would be happy to try and help further, but since this is such a wide topic, you probably should delve a bit more into the details.
As I mentioned, I really haven't found any good resources for this stuff, possibly because it is a bit esoteric -- and because many of those who can do this stuff, get paid to do it, and don't want to divulge their secrets or work product... Me, I just want to see better tools out there.
I am pretty sure that the best possible thing here would be a sequence of increasingly complex tested-and-verified-to-work examples, but as I said, I haven't done that (yet).