a great way to learn a LOT
Fully agreed!
Note that I will try to write this in a way that might help others reading this thread too; I may be repeating things you know very well already.
As you probably already know, you can save your efforts and limit the sources of bugs, by concentrating on each module at a time. (Or, multiple modules at the same time, just completely separately, not combining them into the same program yet.) I personally write really simple
main() function for testing the module I am working on, so that I can verify each chunk of code as soon as I write it. This way, I limit the possible sources of bugs to the code just added, or rarely to code that interacts with the just added code (but it is almost always the code just added). I am lazy, and this saves me tons of efforts.
Module A: Communicate with serial connected microcontrollers.
There are two completely distinct sub-tasks here:
enumeration and
operation.
Enumeration means you probe and list connected devices. This is usually synchronous, but is sometimes done asynchronously or continously, so that new devices are detected and listed as soon as they become available. In practice, you want to split enumeration off from the device operation module; you'll want to e.g. have a single supported serial device enumeration class. It may feel odd initially, but most devices on the same bus (here, using USB serial or standard serial ports) are detected the same way, so this actually does make sense.
In Linux, I tend to use a script as a helper process, for enumerating devices on the USB bus. It can examine the
sysfs pseudo-file hierarchy, specifically
idVendor and
idProduct files in
/sys/bus/usb/devices/*/, or use Bash to run
for D in /dev/tty* ; do udevadm info "$D" ; done and process that output using e.g.
awk (outputting the "E: DEVNAME=" field when the "E: ID_*=" fields match, with an empty line or a "P: *" line as a record separator).
For Linux SBCs, it can make more sense to use udev rules to create suitably named symlinks. Then, it is sufficient to use one or more
glob.glob()s to detect devices when they come and go, say once every couple of seconds for example. (
/dev is typically an udev filesystem, completely in RAM, so this should not cause any wear on your storage devices.)
For operation, there are two options: direct communication from Python using
pyserial (portable) or
termios (Linux/Mac/*BSD), or via a helper process. The helper process makes sense if the devices are timing-sensitive, or you support many different devices and protocols providing the same information. In that case, the helper process acts as a buffer and translator. If the device is not too sensitive wrt. timing (say, can deal with communication latencies of say 100 ms = 0.1 second), and no translation is needed, there is no reason to use a helper process.
For the Python operational code, there are two types of devices: those that use a request-response protocol, and those that output a stream of data, and accept/respond to commands asynchronously.
For the request-response devices I use a single thread per device. The thread runs a loop, that begins with waiting on the
queue for a new command to apply. Then, it acts on the received command, by sending whatever command is needed to the actual device, and waits for the response. After parsing the response, the data is posted to the data queue, and the response (or command completed) is posted to the response queue.
For completely asynchronous devices, I use two threads per device. One thread only sends commands: it waits for commands to apply on the command queue, updates internal state shared with its sibling thread, and writes the command(s) to the device. The other thread only receives responses and data. Note that command responses are posted to the response queue, and data to the data queue.
As you can see, there are actually quite a few queues needed. Each queue is one-directional, and may contain any number of objects in flight. Each device has their own command queue. Only one thread
gets items from the device command queue, but any thread can
put commands to any device command queues. You will want to have a identifier (unique counter used by the GUI thread), so that a single queue can be used to report command responses (success/failure state). A separate queue is used for posting data objects.
You will want the GUI thread to be able to generate commands, and to process the command responses, because it is very much up to a human to decide what to do when stuff goes pear-shaped; we, the programmer, cannot know that beforehand. I like the idea of a status bar, where such communications errors could be shown. When designing the queue structure, and what kind of objects you put/get, this is worth thinking about.
Module B: Connected to a MongoDB or similar.
Sounds like a data arbiter to me.
Perhaps a multiplexer/arbiter that is the only one who
gets objects from the data queue (where device threads
put data objects to), forwarding summaries or changes to the UI thread using a separate queue?
The reason I tend to call this a multiplexer, is that a simple multiplexer would multiplex each data object, storing one copy in the database, and forwarding it to the UI thread. This way the UI thread would not need to worry about the database at all.
The data queue could also be used by the UI thread to tell the multiplexer/arbiter which data it is interested in. In this case, device threads would be
puting data objects to the data queue, and the UI thread would
put data selection objects, directing the arbiter as to what data to pull from the database, and whether the UI is interested in the latest data or not.
The most important thing for this module, however, is to abstract any database internals away. You will make changes to the tables at some point, so keep it easily modified/configured; and do spend a thought or two as to how to import and export old databases to newer versions. (It is much better to do a reimport than to try and keep backwards compatibility. Besides, you might wish to do periodic backups anyway; exporting the database tends to work better than snapshotting.)
Module C: The GUI that will display all the data from the other modules and allow me to input control information with sliders or direct input.
In general, it is crucial to understand that the user application runs under Qt, not the other way around.
Like most other toolkits, Qt is
event-driven. Instead of writing code that runs and occasionally calls Qt stuff, it is Qt that runs, and calls the user code as needed, whenever something interesting happens. There are ways to ensure user code gets run periodically, for example using the QTimer class. The main thing is that while the user code runs, the user interface is unresponsive. (No, events are not "lost" or missed; they just are not acted upon before the user code returns.) Because of this, we use worker threads or processes to do any heavy computation, or anything that takes appreciable time during which the user interface should stay responsive.
The event methods for most buttons or sliders should be basically very simple, just
putting a suitable control or command object to the corresponding queue. The programmer is essentially trusting that everything will work right, and when the device threads respond with the result, the display is updated to reflect the change. (You'll want to let Qt update numeric values immediately; I just mean that the change of value in e.g. a slider should only result in a command object being put to a correct queue, and not much else. If it is e.g. a scroll event on the graph, the graph can be cleared, with the correct data displayed when it arrives from the arbiter/multiplexer.)
For scrolling or realtime graphs, I recommend splitting it into chunks. If the graph is continuously updated, use smallish chunks, and only redefine the current chunk, and move the others. If the graph is usually of historical data, use bigger chunks, and move them instead of redrawing them. This technique allows drawing quite complex graphs easily in real time, without consuming too much CPU time even on SBCs. Usually, the currently changing chunk is a QImage, with the already fully drawn chunks being QPixmaps. For a simple graph (a single time-varying curve, or a couple of them), using QtSvg instead may produce better results (as it may be better accelerated/optimized).
As a practical example, consider a realtime graph of say temperature and humidity. The horizontal position of the graph chunks would depend on time, with the newest chunk being redrawn each time; and when a chunk exits the view, it gets copied from the newest chunk, inserted after the newest chunk, and the newest chunk cleared. (Well, you probably want to do some nifty redraw stuff, and overlap chunks by a pixel or so, to ensure you have antialiasing over chunk boundaries. But that's details.) Note that arriving data is redrawn to the newest chunk, and that the chunks travel (move) according to time. (This same approach works with Javascript on web pages, too; increasing smoothness or update rate by not having to redraw more than a small area, with the rest being just copied/moved as images/pixmaps.)
Sure, you can make the entire graph a single QtSvg for example, and redraw it for each frame, but that requires quite a bit of CPU, or hardware acceleration.
As to implementation, you can write a module that generates a combination of sine waves, and emits new points regularly. Then, you can experiment on how to implement a sliding graph efficiently on your SBCs. You can even implement the control mode for the sine wave generator, so that if the user scrolls away from the "current time", the module emits the data as if it had read them back from the database; and only reports new data when in "tracking" mode.
I would be happy to try and help further, but since this is such a wide topic, you probably should delve a bit more into the details.
As I mentioned, I really haven't found any good resources for this stuff, possibly because it is a bit esoteric -- and because many of those who can do this stuff, get paid to do it, and don't want to divulge their secrets or work product... Me, I just want to see better tools out there.
I am pretty sure that the best possible thing here would be a sequence of increasingly complex tested-and-verified-to-work examples, but as I said, I haven't done that (yet).
Home
Help
Search
Login
Register
