Author Topic: Pythonic solution for accessing class properties  (Read 3206 times)

0 Members and 1 Guest are viewing this topic.

Offline rx8pilot

  • Super Contributor
  • ***
  • Posts: 3634
  • Country: us
  • If you want more money, be more valuable.
Pythonic solution for accessing class properties
« on: November 13, 2020, 08:31:32 pm »
[Coming from a C world, please forgive if my nomenclature is not perfect]

In the context of trying to build a GUI with PyQt5. I have developed a dynamic layout and need to update a variable number of properties.

Code: [Select]
def prgsUpdate(self, barNumber, minVal, maxVal, remainVal):
        ''' Update progress bar specified by barNumber 0-6 '''
        self.ui.cycleRemain0.setMinimum(minVal)
        self.ui.cycleRemain0.setMaximum(maxVal)
        self.ui.cycleRemain0.setProperty("value", remainVal)
        print (f"CHANGE: ", remainVal)   

The goal here is to use the input variable 'barNumber' to address the 'cycleRemain0' property which there can by any number of - 'cycleRemain1, cycleRemain2, cycleRemain3' etc, etc. The number is updated dynamically.

What is the most 'Pythonic' approach to keep a function like this simple so I can call this and update 'self.ui.cycleRemain4......'. 'barNumber = 4'
Code: [Select]
prgsUpdate(4, 0, 100, 50)
« Last Edit: November 13, 2020, 08:40:20 pm by rx8pilot »
Factory400 - the worlds smallest factory. https://www.youtube.com/c/Factory400
 

Offline gmb42

  • Frequent Contributor
  • **
  • Posts: 277
  • Country: gb
Re: Pythonic solution for accessing class properties
« Reply #1 on: November 14, 2020, 12:57:28 pm »
Why are your progress bar instances held as named classes other than some form of collection, e.g. in an array?
 

Offline rx8pilot

  • Super Contributor
  • ***
  • Posts: 3634
  • Country: us
  • If you want more money, be more valuable.
Re: Pythonic solution for accessing class properties
« Reply #2 on: November 14, 2020, 08:24:26 pm »
Why are your progress bar instances held as named classes other than some form of collection, e.g. in an array?

For this application - the class that defines the progress bars is automatically generated based on information in a config file or database. Not so sure that is a great way to accomplish the dynamic functionality where the application can deal with all sorts of GUI layouts that are chosen based on the data available or the user preference. The class is built just before the objects are instantiated. Seems clunky and wrong - which is why I am here. My brain is struggling to escape the clutches of C.

The broad goal here specifically is to have a GUI with any number of progress bars, limited by the display resolution available. After those progress bars are in place, having a clean and concise function that can update them based on the name of the progress bar object.

I hope this makes sense, struggling to explain myself  :scared:

Factory400 - the worlds smallest factory. https://www.youtube.com/c/Factory400
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #3 on: November 14, 2020, 10:33:08 pm »
You can use bar = self.ui.__getattribute__("cycleRemain%d" % index).  If there is no such member, it'll raise an AttributeError exception.

You can also define self.bars = [ self.ui.cycleRemain0, self.ui.cycleRemain1, self.ui.cycleRemain2 ] and then use bar = self.bars[index] to refer to one of the three properties.  The self.bars is then an array of references to those objects, it does not create a deep copy, or copy the objects themselves.

In Qt, use the name attributes (you already use cycleRemain0 etc.), and bar = self.ui.findChild(QtWidgets.QProgressBar, "cycleRemain%d" % index).  It evaluates to a reference to the named QtWidget.  All you need to know are the widget names (which is, unsurprisingly, provided by the .objectName() member function).  If you use logical widget names, you can discover them dynamically too; see QObject documentation for details.
 

Offline rx8pilot

  • Super Contributor
  • ***
  • Posts: 3634
  • Country: us
  • If you want more money, be more valuable.
Re: Pythonic solution for accessing class properties
« Reply #4 on: November 15, 2020, 05:50:52 am »
You can use bar = self.ui.__getattribute__("cycleRemain%d" % index).  If there is no such member, it'll raise an AttributeError exception.

You can also define self.bars = [ self.ui.cycleRemain0, self.ui.cycleRemain1, self.ui.cycleRemain2 ] and then use bar = self.bars[index] to refer to one of the three properties.  The self.bars is then an array of references to those objects, it does not create a deep copy, or copy the objects themselves.

In Qt, use the name attributes (you already use cycleRemain0 etc.), and bar = self.ui.findChild(QtWidgets.QProgressBar, "cycleRemain%d" % index).  It evaluates to a reference to the named QtWidget.  All you need to know are the widget names (which is, unsurprisingly, provided by the .objectName() member function).  If you use logical widget names, you can discover them dynamically too; see QObject documentation for details.

Interesting!

Just recently learning about the magic methods....trying this now  :-+


Factory400 - the worlds smallest factory. https://www.youtube.com/c/Factory400
 

Offline jfiresto

  • Frequent Contributor
  • **
  • Posts: 689
  • Country: de
Re: Pythonic solution for accessing class properties
« Reply #5 on: November 15, 2020, 07:25:41 am »
You can use bar = self.ui.__getattribute__("cycleRemain%d" % index).  If there is no such member, it'll raise an AttributeError exception.

I am just curious. Is there a reason why you get the instance attribute at such a low level? Does Qt sometimes do unhelpful things using __getattr__()?
-John
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #6 on: November 15, 2020, 11:08:13 am »
You can use bar = self.ui.__getattribute__("cycleRemain%d" % index).  If there is no such member, it'll raise an AttributeError exception.
I am just curious. Is there a reason why you get the instance attribute at such a low level? Does Qt sometimes do unhelpful things using __getattr__()?
No, I just listed the choices from lowest level to highest level.  I don't use the low level methods with Qt, only .findChild().
 

Offline bd139

  • Super Contributor
  • ***
  • Posts: 22932
  • Country: gb
Re: Pythonic solution for accessing class properties
« Reply #7 on: November 15, 2020, 11:20:17 am »
Code: [Select]
class Thing(object):
    def __init__(self):
        self.prop1 = 1
        self.prop2 = 2
        self.prop3 = 3
       
    def setProp(self, n, value):
        setattr(self, "prop" + str(n), value)
       
x = Thing()
print (x.prop2)
x.setProp(2, 5)
print (x.prop2)
 

Online janoc

  • Super Contributor
  • ***
  • Posts: 3633
  • Country: de
Re: Pythonic solution for accessing class properties
« Reply #8 on: November 15, 2020, 01:55:58 pm »
Why not just use Python's property support instead?

https://www.programiz.com/python-programming/property
(scroll down for the final code using the @property decorators).

Wrap your Qt UI element access in a pair of setters/getters and make them into a property. Much better than messing with the low level attribute access.
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #9 on: November 15, 2020, 04:24:25 pm »
To repeat: Use PyQt5 ui.findChild() to find the widget whose properties you are modifying, supplying it the same name (as a string) as you used in the UI (XML file if using Qt Designer, QML if using Qt Quick – I prefer the Qt Designer XML/loadUi() approach).

Do not try to do it Pythonically.  When using PyQt5, use Python Qt5 facilities.  I recommend against using pyuic to compile the UI files to Python, and instead build the interface dynamically at application startup from the XML or QML description directly.  If you do wish to create the layout in Python, write the code yourself, instead of using user interface generators, and make the layout adapt to different window sizes and font sizes gracefully.

Name the widgets systematically.  If they vary (because you have multiple UIs for the same engine, or the number of properties controlled by the UI varies for some other reason), traverse the Python UI object hierarchy: all widgets inherit from QObject, which provides both the traversal functions and the name properties.

(I'm particularly fond of allowing end users write their own variant .ui files, with the application having a menu to choose between existing UIs.  The "save all widget data properties then restore them to the new UI" is so annoying to implement I wouldn't bother; treat switching UIs as basically an app restart.)



PyQt5 and PySide2 interfaces differ minimally: enough to trip you if you aren't careful of how you import the Qt modules, but if you do it right, your code will work with either just fine.  The two do the same thing, exposing Qt5 to Python, but PyQt5 has a longer history, being developed by Riverbank Computing Ltd, whereas PySide2 is relatively recent but provided by the Qt project itself.

QtPy provides just such a compatibility layer – including uic, QComboBox, and QHeaderView compatibility objects (the only "real" differences between PyQt5 and PySide2) – but you can do your own.  A typical application (using uic to build UIs dynamically) only really needs
Code: [Select]
# -*- coding: utf-8 -*-
import sys
while True:

    # Prefer PySide2, if installed.
    try:
        from PySide2 import QtGui, QtWidgets, QtCore
        import uic.loadUi
        from PySide2.QtCore import Signal, Slot
        break
    except:
        pass

    # Fall back to PyQt5, if installed.
    try:
        from PyQt5 import QtGui, QtWidgets, QtCore
        from PyQt5.uic import loadUi
        from PyQt5.QtCore import pyqtSignal as Signal, pyqtSlot as Slot
        break
    except:
        pass

    # No Qt5 support. Fail.
    sys.stderr.write("No Qt5 support!\n")
    sys.exit(1)

noting that the decorators are then @Signal and @Slot, and the modules are prefixed by QtGui, QtWidgets,and QtCore; with loadUi() in the root namespace (no module prefix).

The fallback PySide2 uic.py module (in the same directory) is then (ripped off from QtPy uic.py),
Code: [Select]
# -*- coding: utf-8 -*-
__all__ = ['loadUi']

# In PySide, loadUi does not exist, so we define it using QUiLoader, and
# then make sure we expose that function. This is adapted from qt-helpers
# which was released under a 3-clause BSD license:
# qt-helpers - a common front-end to various Qt modules
#
# Copyright (c) 2015, Chris Beaumont and Thomas Robitaille
#
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
#  * Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
#  * Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the
#    distribution.
#  * Neither the name of the Glue project nor the names of its contributors
#    may be used to endorse or promote products derived from this software
#    without specific prior written permission.
#
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
# IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,
# THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
# LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
# NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
# SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
#
# Which itself was based on the solution at
#
# https://gist.github.com/cpbotha/1b42a20c8f3eb9bb7cb8
#
# which was released under the MIT license:
#
# Copyright (c) 2011 Sebastian Wiesner <lunaryorn@gmail.com>
# Modifications by Charl Botha <cpbotha@vxlabs.com>
#
# Permission is hereby granted, free of charge, to any person obtaining a
# copy of this software and associated documentation files (the "Software"),
# to deal in the Software without restriction, including without limitation
# the rights to use, copy, modify, merge, publish, distribute, sublicense,
# and/or sell copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
# THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
# DEALINGS IN THE SOFTWARE.

from PySide2.QtCore import QMetaObject
from PySide2.QtUiTools import QUiLoader

class UiLoader(QUiLoader):
    """
    Subclass of :class:`~PySide.QtUiTools.QUiLoader` to create the user
    interface in a base instance.

    Unlike :class:`~PySide.QtUiTools.QUiLoader` itself this class does not
    create a new instance of the top-level widget, but creates the user
    interface in an existing instance of the top-level class if needed.

    This mimics the behaviour of :func:`PyQt4.uic.loadUi`.
    """

    def __init__(self, baseinstance, customWidgets=None):
        """
        Create a loader for the given ``baseinstance``.

        The user interface is created in ``baseinstance``, which must be an
        instance of the top-level class in the user interface to load, or a
        subclass thereof.

        ``customWidgets`` is a dictionary mapping from class name to class
        object for custom widgets. Usually, this should be done by calling
        registerCustomWidget on the QUiLoader, but with PySide 1.1.2 on
        Ubuntu 12.04 x86_64 this causes a segfault.

        ``parent`` is the parent object of this loader.
        """

        QUiLoader.__init__(self, baseinstance)

        self.baseinstance = baseinstance

        if customWidgets is None:
            self.customWidgets = {}
        else:
            self.customWidgets = customWidgets

    def createWidget(self, class_name, parent=None, name=''):
        """
        Function that is called for each widget defined in ui file,
        overridden here to populate baseinstance instead.
        """

        if parent is None and self.baseinstance:
            # supposed to create the top-level widget, return the base
            # instance instead
            return self.baseinstance

        else:

            # For some reason, Line is not in the list of available
            # widgets, but works fine, so we have to special case it here.
            if class_name in self.availableWidgets() or class_name == 'Line':
                # create a new widget for child widgets
                widget = QUiLoader.createWidget(self, class_name, parent, name)

            else:
                # If not in the list of availableWidgets, must be a custom
                # widget. This will raise KeyError if the user has not
                # supplied the relevant class_name in the dictionary or if
                # customWidgets is empty.
                try:
                    widget = self.customWidgets[class_name](parent)
                except KeyError:
                    raise Exception('No custom widget ' + class_name + ' '
                                    'found in customWidgets')

            if self.baseinstance:
                # set an attribute for the new child widget on the base
                # instance, just like PyQt4.uic.loadUi does.
                setattr(self.baseinstance, name, widget)

            return widget

def _get_custom_widgets(ui_file):
    """
    This function is used to parse a ui file and look for the <customwidgets>
    section, then automatically load all the custom widget classes.
    """

    import sys
    import importlib
    from xml.etree.ElementTree import ElementTree

    # Parse the UI file
    etree = ElementTree()
    ui = etree.parse(ui_file)

    # Get the customwidgets section
    custom_widgets = ui.find('customwidgets')

    if custom_widgets is None:
        return {}

    custom_widget_classes = {}

    for custom_widget in custom_widgets.getchildren():

        cw_class = custom_widget.find('class').text
        cw_header = custom_widget.find('header').text

        module = importlib.import_module(cw_header)

        custom_widget_classes[cw_class] = getattr(module, cw_class)

    return custom_widget_classes

def loadUi(uifile, baseinstance=None, workingDirectory=None):
    """
    Dynamically load a user interface from the given ``uifile``.

    ``uifile`` is a string containing a file name of the UI file to load.

    If ``baseinstance`` is ``None``, the a new instance of the top-level
    widget will be created. Otherwise, the user interface is created within
    the given ``baseinstance``. In this case ``baseinstance`` must be an
    instance of the top-level widget class in the UI file to load, or a
    subclass thereof. In other words, if you've created a ``QMainWindow``
    interface in the designer, ``baseinstance`` must be a ``QMainWindow``
    or a subclass thereof, too. You cannot load a ``QMainWindow`` UI file
    with a plain :class:`~PySide.QtGui.QWidget` as ``baseinstance``.

    :method:`~PySide.QtCore.QMetaObject.connectSlotsByName()` is called on
    the created user interface, so you can implemented your slots according
    to its conventions in your widget class.

    Return ``baseinstance``, if ``baseinstance`` is not ``None``. Otherwise
    return the newly created instance of the user interface.
    """

    # We parse the UI file and import any required custom widgets
    customWidgets = _get_custom_widgets(uifile)

    loader = UiLoader(baseinstance, customWidgets)

    if workingDirectory is not None:
        loader.setWorkingDirectory(workingDirectory)

    widget = loader.load(uifile)
    QMetaObject.connectSlotsByName(widget)
    return widget

This way, your Python3 Qt5 application will Just Work™.  The QtPy is even better compatibility layer implementation than the above snippet, as it implements various version checks and QComboBox/QHeaderView compatibility objects, but it introduces Yet Another Library Dependency, so I tend to recommend this crude-and-simple method instead.

Do not do any intensive calculation or communication in the main thread.  If possible, use the multiprocessing module, so that multiple CPU cores can be used.  The standard Python interpreter only executes one thread of Python code at a time within each process.  If you do e.g. serial or socket communications, feel free to use threading for those; it is okay to have many Python threads blocking/waiting in IO (they won't hinder the Python-executing thread, and the interpreter will switch when something happens; essentially, think of it as running on a single-CPU-core hardware).  It is only intensive, heavy CPU computation that you want to offload to separate processes.  You often want to use a thread in the main Qt5 Python process, running either a Qt5 event queue, or a Python Queue to communicate with the UI thread in a safe, robust manner.  (Use Qt5 event queue if you want to pass work using Qt5 signals and slots, Python Queue if you want to pass Python objects or raw data.)

This ensures that your UI remains interactive and snappy at all times even on small SBCs.  The overhead of threads and multiprocessing.shared_memory (to pass data via shared memory across multiple CPU-intensive Python processes) is insignificant in context; the Python interpreter itself isn't "perfect".

For best results, write any really CPU-intensive work in C (I prefer POSIX C, with pthreads; this works in Linux, BSDs, and Mac OS – I don't use Windows, so for Windows you might need to add an abstraction layer).  The built-in ctypes Python module makes this interfacing rather simple.  Compile such C libraries statically, without dynamic dependencies (other than what are built in to standard C library itself), and your Python + C application is portable across all Linux distributions using that single library for that hardware architecture.  For other hardware architectures, and other OSes (including Windows), you only need to recompile the library.  If you have the time, cross-compile the library to all architectures you can think of.  Determine the library to be used from sys.platform (OS type) and if 'linux', the hardware architecture from os.uname().machine.  (Naming the C libraries foo-linux-x86_64.so or foo-win32.dll or foo-win64.dll will make the code particularly easy and straightforward.)

As I've mentioned before, it is not difficult to write completely portable applications with Python3 and Qt5 user interfaces.  (You can, but do not have to, even compile them into Windows etc. exececutables using PyInstaller.)  With the above suggestions, you can make a package/archive/CD/USB stick, that you can stick to basically any common OS (on any hardware architecture you have compiled the C library for), and execute the Python + C library application directly.  True MultiArch applications, yo.
« Last Edit: November 15, 2020, 04:30:05 pm by Nominal Animal »
 

Offline rx8pilot

  • Super Contributor
  • ***
  • Posts: 3634
  • Country: us
  • If you want more money, be more valuable.
Re: Pythonic solution for accessing class properties
« Reply #10 on: November 15, 2020, 07:20:41 pm »
So, my first lesson in this thread is that PyQt5 is where a full and complete understanding of classes and inheritance is a per-requisite .

The information here helps in so many more ways than just the immediate problem I am trying to solve.

This approach is what has been kicking around in my mind. Since my Python-Fu is still rather limited, I keep trying to figure out how to best mesh good software design with the strengths and weaknesses of Python.
Quote
(I'm particularly fond of allowing end users write their own variant .ui files, with the application having a menu to choose between existing UIs.  The "save all widget data properties then restore them to the new UI" is so annoying to implement I wouldn't bother; treat switching UIs as basically an app restart.)

Factory400 - the worlds smallest factory. https://www.youtube.com/c/Factory400
 

Offline rx8pilot

  • Super Contributor
  • ***
  • Posts: 3634
  • Country: us
  • If you want more money, be more valuable.
Re: Pythonic solution for accessing class properties
« Reply #11 on: November 15, 2020, 07:26:04 pm »
Do not do any intensive calculation or communication in the main thread.  If possible, use the multiprocessing module, so that multiple CPU cores can be used.  The standard Python interpreter only executes one thread of Python code at a time within each process.  If you do e.g. serial or socket communications, feel free to use threading for those; it is okay to have many Python threads blocking/waiting in IO (they won't hinder the Python-executing thread, and the interpreter will switch when something happens; essentially, think of it as running on a single-CPU-core hardware).  It is only intensive, heavy CPU computation that you want to offload to separate processes.  You often want to use a thread in the main Qt5 Python process, running either a Qt5 event queue, or a Python Queue to communicate with the UI thread in a safe, robust manner.  (Use Qt5 event queue if you want to pass work using Qt5 signals and slots, Python Queue if you want to pass Python objects or raw data.)


Yes - indeed. My current self-education efforts are focused on threading in QT. On paper it seems straightforward, but in practice I need to clunk my way through some experiments before it will sink in. My goal is to create UI's that are almost always waiting on some data to arrive from slow sources. Without threading, the UI would be locked up almost all the time.  :--
Factory400 - the worlds smallest factory. https://www.youtube.com/c/Factory400
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #12 on: November 15, 2020, 10:33:43 pm »
Indeedlydeedly.  If you have an Arduino microcontroller, you might find the following sketch useful:
Code: [Select]
/* Delayed echo.  This is in Public Domain.
 * 
 * Waits for a non-empty input message (terminating with any newline encoding),
 * which is echoed back after a fixed delay.
 * (Only the first MAX_MESSAGE_LEN bytes of the message
 *  are recorded and sent back, but the incoming message
 *  can be of any length.)
*/

// Maximum message length
#define  MAX_MESSAGE_LEN   255

// Message echo delay in milliseconds
#define  MESSAGE_DELAY_MS  750

// End-of-message character used in the responses
#define  MESSAGE_NEWLINE  '\n'

// LED pin to light while keeping a message; can leave undefined
#define  MESSAGE_LED_PIN 13

unsigned char  message[MAX_MESSAGE_LEN];

void setup() {
  Serial.begin(115200);

#ifdef MESSAGE_LED_PIN
  pinMode(MESSAGE_LED_PIN, OUTPUT);
#endif
}

void loop() {
  size_t n;
  int c;

#ifdef MESSAGE_LED_PIN
  digitalWrite(MESSAGE_LED_PIN, LOW);
#endif

  // For microcontrollers with native USB, wait for USB connection.
  while (!Serial) /* nothing */ ;

  // Receive message
  n = 0;
  while (1) {
    c = Serial.read();
    if (!Serial) return;

    // Newline or NUL?
    if (c == '\n' || c == '\r' || c == '\0') {
      if (n) {
        // Message terminator
        break;
      } else {
        // Garbage from a previous multi-character terminator, ignore
        continue;
      }
    }

    if (c > 0 && c < 256 && n < MAX_MESSAGE_LEN) {
      message[n++] = c;
#ifdef MESSAGE_LED_PIN
      digitalWrite(MESSAGE_LED_PIN, (n & 1) ? HIGH : LOW);
      delayMicroseconds(4);
#endif     
    }
  }
 
  // USB disconnect?
  if (!Serial) return;

#ifdef MESSAGE_LED_PIN
  digitalWrite(MESSAGE_LED_PIN, HIGH);
  delay(MESSAGE_DELAY_MS);
  digitalWrite(MESSAGE_LED_PIN, LOW);
#else
  delay(MESSAGE_DELAY_MS);
#endif

  if (!Serial) return;
  Serial.write(message, n);
  if (!Serial) return;
  Serial.write(MESSAGE_NEWLINE);
  if (!Serial) return;
  Serial.flush();
}
The idea is that you can experiment with slow serial-type communicators with this.  The above is configured for Teensy 3.x (which have the LED on pin 13), but should work on any Arduino, if you change the MESSAGE_LED pin (or omit it).  It echoes the message back after a delay, lighting the LED (if you define MESSAGE_LED_PIN) so you can see when it is active, making debugging much, much easier.  It seems to have an excess of !Serial checks, but those just ensure that for native-USB microcontrollers like Teensies, if your program closes the serial device early, it "aborts" the echo and won't send the trailing garbage to whatever process opens the serial device next.

You can test it in the Arduino serial monitor.  To test it on the command line, first find out the character device corresponding to the microcontroller (usually /dev/ttyACM0), and then in a Bash shell, run
    ( exec 3<>/dev/ttyACM0 ; stty raw <&3 ; printf 'your C-like string with escapes goes here\n' >&3 ; read -u 3 FOO ; echo "$FOO" )

If you use Linux or Mac, I recommend using the Python termios module.  (I haven't actually used QSerialPort, because it is a separately-installed module, and I'm not sure PySide2 support for it is complete yet.)  Remember to set the serial port to raw, or at minimum disable echo, because otherwise the microcontroller will get stuck in a loop processing its own output.  This seems to catch a lot of people playing with microcontrollers!

(Most of the Python serial libraries I have looked at are utter crap, and lack basic error checking; they contain stuff like assuming POSIX C write() never fails, or never returns a short count... hyurgh.  Leaves a really bad taste in my mouth.  At least with termios, you can do it correctly in plain Python.)
« Last Edit: November 15, 2020, 10:36:46 pm by Nominal Animal »
 

Offline jfiresto

  • Frequent Contributor
  • **
  • Posts: 689
  • Country: de
Re: Pythonic solution for accessing class properties
« Reply #13 on: November 16, 2020, 07:35:44 am »
(Most of the Python serial libraries I have looked at are utter crap, and lack basic error checking; they contain stuff like assuming POSIX C write() never fails, or never returns a short count... hyurgh.  Leaves a really bad taste in my mouth.  At least with termios, you can do it correctly in plain Python.)
Would you include pyserial? I ended up using that for a simple DMM logging utility that would also work under Windows. Here are the utility's bugs through no fault of its own:
Code: [Select]
BUGS
    This program ignores the DMM for the first 100 ms -- and the
      possible, stale serial data in the que when the program started.
    Pyserial 2.7 ignores OSX and Windows serial data parity errors.
    A Control-C under Windows may wait until the DMM sends a line.
-John
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #14 on: November 16, 2020, 05:40:02 pm »
(Most of the Python serial libraries I have looked at are utter crap, and lack basic error checking; they contain stuff like assuming POSIX C write() never fails, or never returns a short count... hyurgh.  Leaves a really bad taste in my mouth.  At least with termios, you can do it correctly in plain Python.)
Would you include pyserial?
Which version?  The issue list is pretty long..

pyserial tries to be everything for everybody.  It never ends well.

Partially the problem is that we still treat binary stream devices like they were terminals, and they're really not; terminals have all sorts of baggage from the last few decades...  The other problem is that the different OSes really implement different rules, and trying to shoehorn them all under one shim is just .. not going to work well.

As to stale garbage data, I habitually discard the kernel buffer when changing terminal settings (via termios.tcflush(descriptor, termios.TCIOFLUSH)) to avoid getting already received but unread stale data.  My Arduino sketches have those !Serial tests for the same reason: Serial turns true whenever an application has the tty device open, and when the last application closes the device, Serial turns false.

If you use Python termios, set PARMRK and clear IGNPAR in the first attribute.  Then, input byte 255 will be read as 255 255, and byte X that had a parity error will read as 255 0 X.  (You do still need to check each read byte in sequence, as the first 255 in a sequence of 255s and 0s is the key.)



Just for fun, I wrote a stupid (and probably buggy!) but simple example of how to use Qt5 QThread and Qt signals and slots to pass serial device commands, requests, and responses between the main thread and the serial worker thread.  Now, the serial worker itself is not very good, because it does not use nonblocking I/O with the interruption pipe (that allows a secondary helper thread to impose operation-specific timeouts that termios cannot – VTIME is interbyte timeout, not message timeout), but this way the example is only a bit over 300 lines long.  (It does expect the uic.py from my previous post, and is intended to be used with the Arduino delayed echo sketch.)
Code: [Select]
#!/usr/bin/python3
# -*- coding: utf-8 -*-
import os
import sys
import termios
try:
    from PySide2 import QtCore, QtWidgets
    from PySide2.QtCore import Slot, Signal
    from uic import loadUi
except:
    try:
        from PyQt5 import QtCore, QtWidgets
        from PyQt5.QtCore import pyqtSlot as Slot, pyqtSignal as Signal
        from PyQt5.uic import loadUi
    except:
        raise NameError("No Python Qt5 support found")


class SerialWorker(QtCore.QObject):

    success = Signal(int)
    failure = Signal(int, str)
    response = Signal(int, bytes)

    def __init__(self, parent=None):
        super().__init__(parent)

        self.fd = -1
        self.tty = None

    def __del__(self):
        if self.fd != -1:
            if self.tty is not None:
                try:
                    termios.tcflush(self.fd, TCIOFLUSH)
                    termios.tcsetattr(self.fd, TCSANOW, self.tty)
                except:
                    pass
                self.tty = None
            try:
                os.close(self.fd)
            except:
                pass
            self.fd = -1

    def _abort(self):
        if self.fd != -1:
            if self.tty is not None:
                try:
                    termios.tcflush(self.fd, TCIOFLUSH)
                    termios.tcsetattr(self.fd, TCSANOW, self.tty)
                except:
                    pass
                self.tty = None
            try:
                os.close(self.fd)
            except:
                pass
            self.fd = -1

    @Slot(int, str)
    def open(self, ident, path):
        if self.fd != -1:
            self.failure.emit(ident, "Already open")
            return

        # Open the named device
        try:
            fd = os.open(path, os.O_RDWR | os.O_NOCTTY | os.O_CLOEXEC)
        except OSError as err:
            self.failure.emit(ident, err.strerror)
            return

        # Obtain termios settings
        try:
            oldtty = termios.tcgetattr(fd)
            newtty = termios.tcgetattr(fd)
        except:
            os.close(fd)
            self.failure.emit(ident, "Not a serial device")
            return

        # Raw mode, blocking reads
        newtty[0] &= ~(termios.ISTRIP | termios.INLCR | termios.IGNCR | termios.ICRNL | termios.IUCLC)
        newtty[0] |= termios.IGNBRK
        newtty[1] &= ~(termios.OPOST | termios.ONLCR | termios.OCRNL | termios.ONOCR | termios.ONLRET)
        newtty[2] &= ~(termios.CBAUD | termios.CSIZE | termios.CSTOPB | termios.PARENB | termios.PARODD)
        newtty[2] |= termios.B115200 | termios.CS8 | termios.CREAD | termios.HUPCL
        newtty[3] &= ~(termios.ISIG | termios.ICANON | termios.ECHO | termios.IEXTEN)
        newtty[4] = termios.B115200
        newtty[5] = termios.B115200
        newtty[6][termios.VTIME] = 0
        newtty[6][termios.VMIN] = 1
        try:
            termios.tcflush(fd, termios.TCIOFLUSH)
            termios.tcsetattr(fd, termios.TCSANOW, newtty)
        except:
            os.close(fd)
            self.failure.emit(ident, "Not a serial device")

        # Opened successfully.
        self.fd = fd
        self.tty = oldtty
        self.success.emit(ident)


    @Slot(int)
    def close(self, ident):
        if self.fd == -1:
            self.failure.emit(ident, "Not opened yet")
            return
        if self.tty is not None:
            try:
                termios.tcflush(self.fd, termios.TCIOFLUSH)
                termios.tcsetattr(self.fd, termios.TCSANOW, self.tty)
            except:
                pass
            self.tty = None
        os.close(self.fd)
        self.success.emit(ident)


    @Slot(int, bytes)
    def request(self, ident, data):
        if self.fd == -1 or self.tty is None:
            self.failure.emit(ident, "Not open")
            return

        if len(data) < 1:
            self.failure.emit(ident, "Empty request")
            return

        # Append newline
        data += b'\n'

        # Write all of the data.
        while len(data) > 0:
            try:
                n = os.write(self.fd, data)
            except OSError as err:
                self._abort()
                self.failure.emit(ident, err.strerror)
                return

            if n > 0:
                data = data[n:]

        # Read reply, until a newline or NUL
        reply = b''
        while (b'\0' not in reply) and (b'\r' not in reply) and (b'\n' not in reply):
            try:
                more = os.read(self.fd, 4095)  # 4095 is termios buffer size
            except OSError as err:
                self._abort()
                self.failure.emit(ident, err.strerror)
                return

            reply += more
            reply.lstrip(b'\0\r\n')

        # Assume we got a single reply.  Remove any duplicate separators.
        data = reply.replace(b'\0', b'').replace(b'\r', b'').replace(b'\n', b'')

        self.response.emit(ident, data)
        return


class SerialDevice(QtCore.QObject):
    """Serial device management class.
       This uses a separate QThread to manage the actual serial port communications.
    """

    success = Signal(int)
    failure = Signal(int, str)
    response = Signal(int, bytes)
    _open = Signal(int, str)
    _close = Signal(int)
    _request = Signal(int, bytes)

    def __init__(self, parent=None):
        super().__init__(parent)

        self.ident = 0

        self.thread = QtCore.QThread()
        self.thread.setStackSize(131072)
        self.thread.setObjectName('serialThread')
        self.thread.setTerminationEnabled(True)

        self.worker = SerialWorker()
        self.worker.moveToThread(self.thread)

        self.thread.finished.connect(self.worker.deleteLater)

        self._open.connect(self.worker.open)
        self._close.connect(self.worker.close)
        self._request.connect(self.worker.request)

        self.worker.success.connect(self.success)
        self.worker.failure.connect(self.failure)
        self.worker.response.connect(self.response)

        self.thread.start()

    def __del__(self):
        self.thread.quit()
        if not self.thread.wait(5000):
            self.thread.terminate()
            self.thread.wait()
        del self.thread
        del self.worker

    def open(self, path: str) -> int:
        if not isinstance(path, str) or len(path) < 1:
            return 0

        try:
            self._open.emit(self.ident + 1, path)
        except:
            return 0

        self.ident += 1
        return self.ident


    def close(self) -> int:
        try:
            self._close.emit(self.ident + 1)
        except:
            return 0

        self.ident += 1
        return self.ident


    def request(self, data: bytes) -> int:
        if not isinstance(data, bytes) or len(data) < 1 or b'\0' in data or b'\r' in data or b'\n' in data:
            return 0

        try:
            self._request.emit(self.ident + 1, data)
        except:
            return 0

        self.ident += 1
        return self.ident


class Ui(QtCore.QObject):

    def __init__(self, uifile, parent=None):
        super().__init__(parent)

        self.serial = SerialDevice(self)
        self.serial.success.connect(self.got_success)
        self.serial.failure.connect(self.got_failure)
        self.serial.response.connect(self.got_response)

        self.ui = loadUi(uifile)
        self.ui.findChild(QtWidgets.QWidget, 'connectButton').clicked.connect(self.do_connect)
        self.ui.findChild(QtWidgets.QWidget, 'deviceEdit').returnPressed.connect(self.do_connect)
        self.ui.findChild(QtWidgets.QWidget, 'disconnectButton').clicked.connect(self.do_disconnect)
        self.ui.findChild(QtWidgets.QWidget, 'commandEdit').returnPressed.connect(self.do_submit)

        # Shorthand properties
        self.widget = {}
        for name in ('deviceEdit', 'commandEdit', 'responseText'):
            self.widget[name] = self.ui.findChild(QtWidgets.QWidget, name)


    def do_connect(self):
        ident = self.serial.open(self.widget['deviceEdit'].text())
        if ident:
            self.widget['responseText'].append("%d < Opening device .." % ident)

    def do_disconnect(self):
        ident = self.serial.close()
        if ident:
            self.widget['responseText'].append("%d < Closing device .." % ident)

    def do_submit(self):
        text = str(self.widget['commandEdit'].text())
        ident = self.serial.request(text.encode(encoding='utf-8', errors='replace'))
        if ident:
            self.widget['responseText'].append("%d < Sending '%s' .." % (ident, text))
            self.widget['commandEdit'].setText("")

    @Slot(int)
    def got_success(self, ident):
        self.widget['responseText'].append("%d > Done." % ident)

    @Slot(int, str)
    def got_failure(self, ident, message):
        self.widget['responseText'].append("%d > Failed: %s." % (ident, message))
        pass

    @Slot(int, bytes)
    def got_response(self, ident, data):
        self.widget['responseText'].append("%d > Received '%s' (%d bytes)." % (ident, data.decode(encoding='utf-8', errors='ignore'), len(data)))

    def show(self):
        self.ui.show()


if __name__ == '__main__':
    QtCore.QCoreApplication.setAttribute(QtCore.Qt.AA_ShareOpenGLContexts)
    app = QtWidgets.QApplication(sys.argv)
    win = Ui("main.ui")
    win.show()
    status = app.exec_()
    del win, app
    sys.exit(status)

and the main.ui I used is
Code: [Select]
<?xml version="1.0" encoding="UTF-8"?>
<ui version="4.0">
 <class>Ui</class>
 <widget class="QWidget" name="Ui">
  <property name="geometry">
   <rect>
    <x>0</x>
    <y>0</y>
    <width>368</width>
    <height>172</height>
   </rect>
  </property>
  <property name="windowTitle">
   <string>Serial Example</string>
  </property>
  <layout class="QGridLayout" name="layout" rowstretch="0,0,1" columnstretch="0,1,0,0">
   <item row="0" column="0" alignment="Qt::AlignRight">
    <widget class="QLabel" name="deviceLabel">
     <property name="text">
      <string>Device:</string>
     </property>
    </widget>
   </item>
   <item row="2" column="0" alignment="Qt::AlignRight|Qt::AlignTop">
    <widget class="QLabel" name="responseLabel">
     <property name="text">
      <string>Response:</string>
     </property>
    </widget>
   </item>
   <item row="0" column="1">
    <widget class="QLineEdit" name="deviceEdit">
     <property name="text">
      <string>/dev/ttyACM0</string>
     </property>
    </widget>
   </item>
   <item row="1" column="0" alignment="Qt::AlignRight">
    <widget class="QLabel" name="commandLabel">
     <property name="text">
      <string>Command:</string>
     </property>
    </widget>
   </item>
   <item row="0" column="2">
    <widget class="QPushButton" name="connectButton">
     <property name="text">
      <string>Connect</string>
     </property>
    </widget>
   </item>
   <item row="0" column="3">
    <widget class="QPushButton" name="disconnectButton">
     <property name="text">
      <string>Disconnect</string>
     </property>
    </widget>
   </item>
   <item row="1" column="1" colspan="3">
    <widget class="QLineEdit" name="commandEdit"/>
   </item>
   <item row="2" column="1" colspan="3">
    <widget class="QTextEdit" name="responseText">
     <property name="lineWrapMode">
      <enum>QTextEdit::NoWrap</enum>
     </property>
     <property name="readOnly">
      <bool>true</bool>
     </property>
    </widget>
   </item>
  </layout>
 </widget>
 <resources/>
 <connections/>
</ui>

which you can open in Qt Designer if you like.

If I use one of my Teensies, connect, and then type garbage fast, and finally disconnect, here's what it shows in the text area:
Code: [Select]
1 < Opening device ..
1 > Done.
2 < Sending 'a' ..
3 < Sending 's' ..
4 < Sending 'd' ..
5 < Sending 'f' ..
2 > Received 'a' (1 bytes).
6 < Sending 'e' ..
3 > Received 's' (1 bytes).
7 < Sending 'd' ..
4 > Received 'd' (1 bytes).
8 < Sending 'djsdkg' ..
5 > Received 'f' (1 bytes).
6 > Received 'e' (1 bytes).
9 < Sending 'dshsdgdud' ..
7 > Received 'd' (1 bytes).
8 > Received 'djsdkg' (6 bytes).
10 < Sending 'sldhduhsd' ..
9 > Received 'dshsdgdud' (9 bytes).
10 > Received 'sldhduhsd' (9 bytes).
11 < Sending 'sodhsdhd' ..
11 > Received 'sodhsdhd' (8 bytes).
12 < Sending 'a' ..
13 < Sending 's' ..
12 > Received 'a' (1 bytes).
14 < Sending 'dd' ..
15 < Sending 'e' ..
13 > Received 's' (1 bytes).
16 < Sending 'ef' ..
17 < Sending 'f' ..
14 > Received 'dd' (2 bytes).
15 > Received 'e' (1 bytes).
16 > Received 'ef' (2 bytes).
18 < Closing device ..
17 > Received 'f' (1 bytes).
18 > Done.

While the UI stays responsive at all times, the 0.75 second delay before the MCU responds is obvious. < are reported when the UI has submitted a command/request, and > are the responses from the MCU via the worker thread.  To keep which response/success/error is related to which request/command, the SerialDevice class keeps an autoincrementing ident number, which is passed to the worker, and returned to the caller (if the request/command was submitted); and the responses have the corresponding ident.

Note that the microcontroller and the worker thread are strictly half-duplex: there is only one request in flight at any time, and the response is received before the next request is submitted.  The reason for the garbled order above is that the above order reflects real time order (top was earliest event, bottom latest event), and the Qt signals are queued (between the main thread and the helper thread).

The four signals (pressing enter at the text boxes, or clicking either of the two buttons) are connected separately in the code, but it is quite possible to use "magic" method names in the Ui class to automagically connect named widget events to their handlers.  For example, if you like to use on_widgetName_signalName as the magic handler names, all you need to add in the Ui class constructor is something like
Code: [Select]
    for methodName in dir(self):
        if methodName.startswith('on_'):
            namePart = methodName.split('_')
            if len(namePart) < 3:
                continue
            signalName = namePart[-1]
            del namePart[0], namePart[-1]
            widgetName = '_'.join(namePart)
            widget = self.ui.findChild(QtWidgets.QWidget, widgetName)
            if widget is None:
                sys.stderr.write("%s: There is no widget '%s'!" % (uifile, widgetName))
                continue
            try:
                getattr(widget, signalName).connect(getattr(self, methodName))
            except:
                sys.stderr.write("%s: Cannot connect widget '%s' signal '%s' to method '%s'!" % (uifile, widgetName, signalName, methodName))
                continue

Normally uiLoader does that, but because the ui is not our root object (where the methods are), I think this additional mapping is needed.
 

Offline jfiresto

  • Frequent Contributor
  • **
  • Posts: 689
  • Country: de
Re: Pythonic solution for accessing class properties
« Reply #15 on: November 16, 2020, 06:10:23 pm »
(Most of the Python serial libraries I have looked at are utter crap, and lack basic error checking; they contain stuff like assuming POSIX C write() never fails, or never returns a short count... hyurgh.  Leaves a really bad taste in my mouth.  At least with termios, you can do it correctly in plain Python.)
Would you include pyserial?
Which version?  The issue list is pretty long..

pyserial tries to be everything for everybody.  It never ends well.

I lost track of the versions. I have been slowly polishing the utility over the last 6 1/2 years and periodically update pyserial. For a while, pyserial could flush a serial input buffer, but then it stopped working, at least under OSX.
-John
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #16 on: November 17, 2020, 07:22:54 pm »
Yup.  But consider this: What would be the optimum interface for your use case?  (My point is, it varies a LOT.)

For example, if the output from the serial device is a stream with reliable separators (the most typical case; with newline as the separator), then I would suggest that the best Python interface would be something like (as a static method)
    readResponse(device, timeout, maxbytes=4096)
where timeout is the number of seconds (or milliseconds) to getting a full response; and the result is a tuple, say (bytes, terminator, parity-errors), where terminator is either the end separator or empty sequence if the command timed out, and parity-errors is either a boolean indicator whether this response had parity errors, or a list or tuple of indices to the bytes which had parity errors.

I know that in Linux this can be implemented easily in POSIX C (select/poll for nonblocking data, and a per-process timer with a SIGEV_THREAD_ID signal for the timeout), and doesn't even need pthreads.  On BSDs and Macs one can use a helper thread in interruptible sleep (nanosleep()); which delivers a signal changing a volatile atomic flag (volatile sig_atomic_t) or writes a single byte to a pipe also monitored by the serial reading thread.  That also works in Linux, making it a portable approach, but like I said, in Linux-only there is even a simpler option.  For Windows, I dunno, because I don't use it or write any code for it.)

For a pure binary datastream, I'd prefer slightly different interface:
    readStream(device, timeout, minBytes, maxBytes)
which blocks until timeout expires, or at least minBytes is received.  If at that time there are more than minBytes received (which is common, because USB transfers blocks of data, and not individual bytes, and USB is the most common serial interface nowadays), the call returns up to maxBytes.

But note how non-ordinary this interface is: each call returns three different pieces of information, and even when it times out, the caller gets both the notification (of the call timing out), but also any data received thus far.  I bet such an interface would seriously weird out typical Python programmers, and make them wonder.. asking for a simpler interface: "I just want to read the serial data, and none of that other stuff.  Can't you make it simpler?"
You see, most humans are utterly stupid.  And most programmers are just human.  (Me, I'm an utter uncle bumblefuck, but at least I'm aware of my deficiencies.)
 
The following users thanked this post: bd139

Offline jfiresto

  • Frequent Contributor
  • **
  • Posts: 689
  • Country: de
Re: Pythonic solution for accessing class properties
« Reply #17 on: November 17, 2020, 08:12:47 pm »
Fortunately, the digital multimeter my utility is reading, outputs CRLF terminated data blocks that pyserial can iterate over. Beyond that, I try to ask as little functionality from pyserial as possible, starting with the very first calls (here, in simplified form):

Code: [Select]
import serial                # Download from http://pyserial.sourceforge.net/

def main(argv):
    ...
    # Flush any stale input.
    # BUG: pyserial 2.7 will not flush the receive buffer of the v2.2.18 FTDI
    #      Chip USB serial driver, meaning its flushInput() doesn't. Try to
    #      flush the port by opening it with a 100 ms timeout, reading and
    #      discarding any input, and closing the port. A read with a much
    #      shorter timeout or a non-blocking read may cause pyserial 2.7/3.1
    #      to either lose qued characters or defer reading them. :(
    # Then (re)open the port with no timeout and repeatedly read
    # measurement line-blocks from the DMM....
    #
   def open_port(timeout=None):
        return serial.Serial(path, baudrate=2400, bytesize=serial.SEVENBITS,
                             parity=serial.PARITY_ODD,
                             stopbits=serial.STOPBITS_ONE,
                             timeout=timeout)
    try:
        with open_port(timeout=0.1) as dmm:
            stale_data = dmm.read(32768//2)    # BUG: Using a large int or None
                                               #      fails or reads 1 byte.
        with open_port() as dmm:
            for block in dmm:
                ...
    except EnvironmentError as exc:
        abort(exc)
    ...

if __name__ == '__main__':
    main(sys.argv)
« Last Edit: November 17, 2020, 08:32:21 pm by jfiresto »
-John
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #18 on: November 17, 2020, 11:02:47 pm »
Well, how about using the following?  It should be portable across Linux, BSDs, and Macs:
Code: [Select]
# -*- coding: utf-8 -*-
# This is in Public Domain
import sys
import os
import termios
import time
import select
import errno

class SerialMM:
    """Serial multimeter

       mm = SerialMM(device_path, default_sendtimeout, default_receivetimeout)
           Open the serial device at 2400 baud, odd parity, 7 bits per byte.
           Parity checking is enabled, and receive() will set the most significant
           bit for any received bytes with parity errors.
    """

    def __init__(self, devpath, sendtimeout=1.0, receivetimeout=1.0):
        self.fd = -1
        self.tty = None

        if isinstance(sendtimeout, (int, float)) and sendtimeout >= 0:
            self.sendtimeout = sendtimeout
        else:
            self.sendtimeout = -1

        if isinstance(receivetimeout, (int, float)) and receivetimeout >= 0:
            self.receivetimeout = receivetimeout
        else:
            self.receivetimeout = -1

        fd = os.open(devpath, os.O_RDWR | os.O_NONBLOCK | os.O_NOCTTY | os.O_CLOEXEC)

        try:
            oldtty = termios.tcgetattr(fd)
            newtty = termios.tcgetattr(fd)
        except Exception:
            os.close(fd)
            raise

        # c_iflags: ignore BREAK, enable input parity checking, no CR/LF translation
        newtty[0] &= ~(termios.ISTRIP | termios.IGNPAR | termios.INLCR | termios.IGNCR | termios.ICRNL | termios.IUCLC)
        newtty[0] |= termios.IGNBRK | termios.PARMRK | termios.INPCK

        # c_oflag: no post processing, no CR/LF translation
        newtty[1] &= ~(termios.OPOST | termios.ONLCR | termios.OCRNL | termios.ONOCR | termios.ONLRET)

        # c_cflga: 2400o7: clear baud rate mask, odd parity, 2400 baud, 7 bits per character
        newtty[2] &= ~(termios.CBAUD | termios.CSIZE | termios.CSTOPB | termios.PARENB | termios.PARODD)
        newtty[2] |= termios.B2400 | termios.CS7 | termios.CREAD | termios.HUPCL

        # c_lflag: raw mode, no signals, disable echo
        newtty[3] &= ~(termios.ISIG | termios.ICANON | termios.ECHO | termios.IEXTEN)

        # 2400 baud rate
        newtty[4] = termios.B2400
        newtty[5] = termios.B2400

        # Never block
        newtty[6][termios.VTIME] = 0
        newtty[6][termios.VMIN] = 0

        try:
            termios.tcflush(fd, termios.TCIOFLUSH)
            termios.tcsetattr(fd, termios.TCSANOW, newtty)
        except Exception:
            os.close(fd)
            raise

        # Device opened successfully.
        self.fd = fd
        self.tty = oldtty
        self.path = devpath
        self.buffer = bytes()

    def __del__(self):
        """Close serial device, reverting to original termios settings."""

        fd = self.fd
        tty = self.tty
        self.fd = -1
        self.tty = None
        if fd != -1:
            if tty is not None:
                try:
                    termios.tcflush(fd, termios.TCIOFLUSH)
                except:
                    pass
                try:
                    termios.tcsetattr(fd, termios.TCSANOW, tty)
                except:
                    pass
            try:
                os.close(fd)
            except:
                pass

    def send(self, data, timeout=None):
        """Send data (bytes) to serial device, spending at most timeout.
           Returns the number of bytes sent.
           Zero timeout is allowed (nonblocking operation),
           negative timeouts blocks until all data is sent."""

        if isinstance(timeout, (int, float)):
            usetimeout = timeout
        else:
            usetimeout = self.sendtimeout

        if self.fd == -1:
            # Device not open.
            return 0

        data_len = len(data)
        data_sent = 0
        started = time.monotonic()

        while data_sent < data_len:

            # We have nonblocking I/O
            try:
                n = os.write(self.fd, data[data_sent:])
            except InterruptedError:
                # Interrupted by signal delivery; ignore
                continue
            except OSError as err:
                # In some cases, n==0, and in other cases we have EAGAIN or EWOULDBLOCK error.
                if err.errno == errno.EAGAIN or err.errno == errno.EWOULDBLOCK:
                    n = 0
                else:
                    raise

            if n > 0:
                data_sent += n
                continue

            # We need to wait until we can write again.
            if usetimeout > 0:
                maxwait = started + usetimeout - time.monotonic()
                if maxwait <= 0.0:
                    # Timeout.  Return the number of bytes already sent.
                    return data_sent
            elif usetimeout < 0:
                maxwait = 1.0
            else:
                # Nonblocking operation
                return data_sent

            readfds, writefds, exceptfds = select.select((), (self.fd,), (), maxwait)
            # Because we do nonblocking writes, we don't actually care about the result.

        return data_sent

    def receive(self, timeout=None):
        """Receive newline-delimited sata from serial device, spending at most
           timeout seconds for a complete response.  CR and LF characters are omitted,
           so empty lines will never be received.
           Returns an empty bytes object if the timeout is exceeded before a full response is received.
        """

        if isinstance(timeout, (int, float)):
            usetimeout = timeout
        else:
            usetimeout = self.receivetimeout

        started = time.monotonic()
        while True:

            # Check if buffer contains a complete response yet.
            cr_index = self.buffer.find(b'\r')
            lf_index = self.buffer.find(b'\n')
            if cr_index > 0 or lf_index > 0:
                index = min(cr_index, lf_index)
                chunk = self.buffer[0:index]
                self.buffer = (self.buffer[index:]).lstrip(b'\r\n')
                return chunk
            elif cr_index > 0:
                chunk = self.buffer[0:cr_index]
                self.buffer = (self.buffer[cr_index:]).lstrip(b'\r\n')
                return chunk
            elif lf_index > 0:
                chunk = self.buffer[0:lf_index]
                self.buffer = (self.buffer[lf_index:]).lstrip(b'\r\n')
                return chunk

            # Nonblocking?
            if usetimeout == 0:
                return bytes()

            # Wait for more data to become available.
            while True:
                if self.fd == -1:
                    # Connection to device lost.  Could raise an Exception instead.
                    return bytes()

                if usetimeout > 0:
                    maxwait = started + usetimeout - time.monotonic()
                    if maxwait <= 0:
                        # Timeout exceeded.  We could raise an exception instead.
                        return bytes()
                else:
                    maxwait = 1.0

                try:
                    readfds, writefds, exceptfds = select.select((self.fd,), (), (), maxwait)
                except InterruptedError:
                    # Interrupted by signal delivery; ignore
                    continue
                if self.fd in readfds:
                    break

            # Receive more data.
            try:
                more = os.read(self.fd, 4096)   # TTY buffer size = 4095
            except InterruptedError:
                # Interrupted by signal delivery; ignore
                continue

            # Parity checks. b'\377\0z' is byte z with a parity error,
            # and b'\377\377' is a non-error 8-bit byte value 255.
            if more is not None and len(more) > 0:
                self.buffer = (self.buffer + more).lstrip(b'\r\n')

                i = self.buffer.find(b'\377')
                while i >= 0 and i <= len(self.buffer) - 3:
                    if self.buffer[i+1] == 0:
                        # Parity error. Mark by setting highest bit.
                        self.buffer = self.buffer[:i] + bytes((self.buffer[i+2] | 128,)) + self.buffer[i+3:]
                        i = self.buffer.find(b'\377', i+3)
                    elif self.buffer[i+1] == 255:
                        # Not a parity error, just 8-bit value 255.
                        self.buffer = self.buffer[:i] + bytes((255,)) + self.buffer[i+2:]
                        i = self.buffer.find(b'\377', i+2)
                    else:
                        # This should never occur.  We just keep the 255 byte.
                        i = self.buffer.find(b'\377', i+1)

if __name__ == '__main__':
    if len(sys.argv) < 2 or sys.argv[1] in ('-h', '--help'):
        if len(sys.argv) > 0:
            this = sys.argv[0]
        else:
            this = '(this)'

        sys.stderr.write("\n")
        sys.stderr.write("Usage: %s [ -h | --help ]\n" % this)
        sys.stderr.write("       %s SERIAL-DEVICE\n" % this)
        sys.stderr.write("\n")
        sys.stderr.write("This program will echo everything received from serial device,\n");
        sys.stderr.write("using 2400 baud, odd parity, 7 bits per byte.  Parity errors\n")
        sys.stderr.write("are marked by setting the high bit.\n")
        sys.stderr.write("\n")
        sys.stderr.write("Press Ctrl+C to stop the output.\n")
        sys.stderr.write("\n")
        sys.exit(0)

    mm = SerialMM(sys.argv[1])

    # To send something to the multineter, use mm.send().

    while True:

        try:
            line = mm.receive()
        except KeyboardInterrupt:
            break

        if len(line) < 1:
            try:
                sys.stderr.write("(No output)\n")
                sys.stderr.flush()
            except KeyboardInterrupt:
                break
            continue

        try:
            sys.stdout.write('"%s" (%d bytes)\n' % (line.decode(encoding='utf-8', errors='ignore'), len(line)))
            sys.stdout.flush()
        except KeyboardInterrupt:
            break

    del mm

Since the data is 7-bit, I marked parity errors by setting the most significant bit. send() takes a bytes object, and receive() returns a bytes object (but with CR and LF omitted). receive() never returns an empty line, because it consumes newlines; it does return an empty bytes object if the timeout is exceeded.

(I don't particularly like that way of noting the parity errors, but it can be changed if you know how you'd like to deal with parity errors – ignore that line? Or something else? – but this was the simplest option, only 12 lines of Python code.)

As usual, the normal timeout modes are supported: positive for a timeout, zero for nonblocking operation, and negative for infinite timeout.
I wrote it just now, so it probably contains bugs, and I only tested it with the Arduino sketch earlier (adding some mm.send(b'Foofah!\n') to the above code, so it'll respond).  But the idea behind the code is sound.

When the program opens the serial port, any already received data (and buffered but not sent data) at the tty level is discarded.  The initial line may therefore be partial one, simply because the multimeter might be in the middle of sending a line when the program opens the port.

There is a way to work around that, however: instead of flushing the input, we could do a nonblocking read of 4096 bytes (termios buffers are only 4095 bytes long), and discard everything up to and including the final newline (CR, LF, CRLF, or LFCR), and if anything is left, putting that part in self.buffer.  Want me to add that into this public domain example code?  (It would then make sense to refactor the buffering mechanism, and make it more robust by doing the parity marking on the chunk returned by receive().)

Again, this is just a first write, so don't think this is "finished" or quality code in any way.  It is just the first version, and requires some testing and especially using (in an actual application) to see what works, and what needs changing, which usually results in big changes in such "library" code.  But it isn't a big deal at all, comfy slippers stuff for me... Happy to help, if this helps!
 

Offline rx8pilot

  • Super Contributor
  • ***
  • Posts: 3634
  • Country: us
  • If you want more money, be more valuable.
Re: Pythonic solution for accessing class properties
« Reply #19 on: November 18, 2020, 12:43:35 am »
Why not just use Python's property support instead?

https://www.programiz.com/python-programming/property
(scroll down for the final code using the @property decorators).

Wrap your Qt UI element access in a pair of setters/getters and make them into a property. Much better than messing with the low level attribute access.

Still trying to understand the setters/getters.......

Dimming the lights, taking a deep breath and waiting for the 'click' in my brain.
Factory400 - the worlds smallest factory. https://www.youtube.com/c/Factory400
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #20 on: November 18, 2020, 03:08:13 am »
Let's define a simple class for stand-ins for the Qt objects:
Code: [Select]
class Stuff:
    def __init__(self, **kwargs):
        for name, value in kwargs.items():
            setattr(self, name, value)
Basically, foo = Stuff(bar=1, baz="two") defines an object foo which has attributes foo.bar (== 1) and foo.baz (== "two").

If we have foo, and the name of an attribute as a string, say aname, we can get and set the attribute using
Code: [Select]
    value = getattr(foo, aname)
    setattr(foo, aname, value)

If we have foo, and a member function name in say fname, we can call that member function using
Code: [Select]
    (getattr(foo, fname))(parameters, to, the, function)
or, equivalently,
Code: [Select]
    memberfunc = getattr(foo, fname)
    memberfunc(parameters, to, the, function)



Let's say you create some widget in Qt, and name them.  If you define them in an .ui file, just make sure the widgets have a name (a <property name="text"><string>nameGoesHere</string></property> block).  If you define them in Python, call widget.setObjectName("nameGoesHere") .  If your root widget is say self.ui, you can find any named widget using
Code: [Select]
    widget = self.ui.findChild(QtWidgets.QWidget, "nameGoesHere")

With this, you can implement the function in the first message in this thread:
Code: [Select]
def prgsUpdate(self, barNumber, minVal, maxVal, remainVal):
        ''' Update progress bar specified by barNumber 0-6 '''
        widget = self.ui.findChild(QtWidgets.QWidget, "cycleRemain" + barNumber)
        widget.setMinimum(minVal)
        widget.setMaximum(maxVal)
        widget.setProperty("value", remainVal)



Let's consider something more interesting: lets assume we have a dictionary with Qt widget names, where the value is a dictionary of method names with values being method parameters.  This means that prgsUpdate(self, 0, 100, 200, 150) and prgsUpdate(self, 1, 25, 75, 50) are described using
Code: [Select]
    settings = { 'cycleRemain0': { 'minimum': 100, 'maximum': 200, '=value': 150 },
                 'cycleRemain1': { 'minimum': 25, 'maximum': 75, '=value': 50 } }
Note that properties that start with a '=' name QObject properties.  The function that applies such settings can then be written as
Code: [Select]
    def setAll(self, settings):
        for widgetName, widgetSettings in settings.items():
            widget = self.ui.findChild(QtWidgets.QWidget, widgetName)
            for itemName, value in widgetSettings.items():
                if itemName[0] == '=':
                    widget.setProperty(itemName[1:], value)
                else:
                    (getattr(widget, "set" + itemName[0].upper() + itemName[1:]))(value)
This (assuming the above has no typos) works for all Qt widgets and their properties.

The inverse, or updating such a dictionary structure with the current values, is something like
Code: [Select]
    def updateAll(self, settings):
        for widgetName, widgetSettings in settings.items():
            widget = self.ui.findChild(QtWidgets.QWidget, widgetName)
            for itemName, value in widgetSettings.items():
                if itemName[0] == '=':
                    newValue = widget.getProperty(itemName[1:])
                else:
                    newValue = (getattr(widget, itemName))()
                widgetSettings[itemName] = newValue
        return settings

This is suitable when you know exactly what you wish to save from an user interface.  When you don't, you instead need to traverse through the widget structure, and determine the methods to use from the widget type.  This is not hard, but because of the large number of widgets, involves a lot of testing.

So, the thing I mentioned earlier, allowing an user to switch UIs, means we'll want to store just one thing per widget, ending up with a plain unordered dictionary.  For numerical values, you'll want to store the normalized or logical value, not a physical value (say, pixel counts or pixel coordinates).  Such dictionaries can be trivially stored to .INI -like files using the built-in configparser module; meaning, you can even let the user "snapshot" their user interface, including all text content et cetera, saving to simple ini files.

(If your application has multiple windows, or configurable user interface, saving the widget geometries (instead of their "contents") in a similar way allows the user to "snapshot" and restore their UI to different work flows.)
« Last Edit: November 19, 2020, 04:53:21 am by Nominal Animal »
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #21 on: November 18, 2020, 03:31:05 am »
When creating widgets in Python, you end up calling many of its property functions to set them.  This can be shortened with e.g.
Code: [Select]
    def applyWidget(self, widgetName, **kwargs):
        widget = self.ui.findChild(QtWidgets.QWidget, widgetName)
        for name, value in kwargs.items():
            method = getattr(widget, name)
            if isinstance(value, tuple):
                method(*value)
            elif isinstance(value, dict):
                method(**value)
            else:
                method(value)

For example,
Code: [Select]
    label = QtWidgets.QLabel("Some text label:")
    label.setObjectName("someLabel")
    label.setWordWrap(False)
    label.setAlignment(QtCore.Qt.AlignRight | QtCore.Qt.AlignTop)
    label.setProperty("myOwn", 5)
simplifies to
Code: [Select]
    label = QtWidgets.QLabel("Some text label:")
    applyWidget(label, setObjectName="someLabel", setWordWrap=False, setAlignment=QtCore.Qt.AlignRight|QtCore.Qt.AlignTop, setProperty=('myOwn':5))

If you have many of them that share calls, save them in a dictionary:
Code: [Select]
    labelCommon = { 'setWordWrap': False, 'setAlignment': QtCore.Qt.AlignRight|QtCore.Qt.AlignTop }
and then insert the dictionary as named parameters:
Code: [Select]
    label = QtWidgets.QLabel("Some text label:")
    applyWidget(label, setObjectName="someLabel", **labelCommon)
    other = QtWidgets.QLabel("Some other label:")
    applyWidget(label, setObjectName="otherLabel", **labelCommon)
 

Offline jfiresto

  • Frequent Contributor
  • **
  • Posts: 689
  • Country: de
Re: Pythonic solution for accessing class properties
« Reply #22 on: November 18, 2020, 09:13:36 am »
Well, how about using the following?  It should be portable across Linux, BSDs, and Macs: ...

Well, thank you very much for that, although you really did not need to. That is a lot of typing! I have been using Python fairly intensely since 2007. Being the slow creature that I am, my code was at first pretty much C and friends, but became less and less so until I had fully unlearned them after roughly seven years. I can guess where you might be on that journey.

Just a couple quick suggestions – to use or ignore – that I feel obliged to offer in appreciation of your efforts.

I do not think I have ever trapped failed writes to sys.stderr. If it has stopped working, I expect a kernel panic will shortly follow.

Rather than

Code: [Select]
    if isinstance(sendtimeout, (int, float)) and sendtimeout >= 0:
        self.sendtimeout = sendtimeout
    else:
        self.sendtimeout = -1

I might write

Code: [Select]
    try:
        self.sendtimeout = sendtimeout if (sendtimeout + 0 >= 0) else -1
    except TypeError:
        self.sendtimeout = -1

with the "+ 0" if the code will run under Python 2. While isinstance() can sometimes make life much simpler, rewriting code without it is one of those useful exercises I always try.

EDIT: Or on further thought, might it be better to raise a ValueError if sendtimeout is neither None nor a number?

EDITx2: Fish think, but too slowly. On third thought, since the function is for developers and I gather self.sendtimeout need only be numeric, I would write:

Code: [Select]
    self.sendtimeout = -1 if (sendtimeout is None) else sendtimeout + 0

and let the addition and exception detect and identify the type error, before the error becomes costly.
« Last Edit: November 18, 2020, 03:39:17 pm by jfiresto »
-John
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 4462
  • Country: fi
    • My home page and email address
Re: Pythonic solution for accessing class properties
« Reply #23 on: November 18, 2020, 04:11:12 pm »
I do not think I have ever trapped failed writes to sys.stderr. If it has stopped working, I expect a kernel panic will shortly follow.
Oh no, there is no failure in sys.stderr itself; they are to just to trap the KeyboardInterrupt exception that occurs when the user presses Ctrl+C.
If you run the code, you'll see that pressing Ctrl+C does not cause the program to abort, it exits the main loop cleanly.

I might write
Code: [Select]
    try:
        self.sendtimeout = sendtimeout if (sendtimeout + 0 >= 0) else -1
    except TypeError:
        self.sendtimeout = -1
Sure, why not?  That works perfectly well too.

The reason I happen to prefer isinstance checks is that I like to subclass tuples (since they're immutable; otherwise I'd use plain objects) for various message types (used in e.g. queue.Queue).  For example:
Code: [Select]
class Message(tuple):
    """Abstract message parent class"""
    def __new__(cls):
        return tuple.__new__(cls, ())

class SuccessMessage(Message):
   def __new__(cls, status):
       return tuple.__new__(cls, (status,))
   @property
   def status(self):
       return self[0]

class FailureMessage(Message):
    def __new__(cls, cause):
        return tuple.__new__(cls, (cause,))
    @property
    def cause(self):
        return self[0]
Now, if you had a perfectly flat type hierarchy, you could just use a class or instance method returning a type identifier.  However, isinstance(msg, Message) is True for both SuccessMessage and FailureMessage instances.  This means that code maintenance (adding new message types) is robust, because you only need to subclass the relevant class, and add specific handling at the point – no need to worry about adding the message type to various checks in between.  Simples.

Or on further thought, might it be better to raise a ValueError if sendtimeout is neither None nor a number?
Now that I think of it, I agree: raising a ValueError makes most sense.
 

Offline jfiresto

  • Frequent Contributor
  • **
  • Posts: 689
  • Country: de
Re: Pythonic solution for accessing class properties
« Reply #24 on: November 18, 2020, 04:52:17 pm »
I do not think I have ever trapped failed writes to sys.stderr. If it has stopped working, I expect a kernel panic will shortly follow.
Oh no, there is no failure in sys.stderr itself; they are to just to trap the KeyboardInterrupt exception that occurs when the user presses Ctrl+C.
If you run the code, you'll see that pressing Ctrl+C does not cause the program to abort, it exits the main loop cleanly.

You are right, I scanned past the KeyboardInterrupt arguments, probably because I was not expecting them that late. I catch them more globally:

Code: [Select]
if __name__ == '__main__':
    try:
        main(sys.argv)
    except KeyboardInterrupt:
        debug(1, lambda: ['Exiting on KeyboardInterrupt (SIGINT)'])

It is harder to mess that one up, and of course it is less to type.

Quote
The reason I happen to prefer isinstance checks is that I like to subclass tuples (since they're immutable; otherwise I'd use plain objects) for various message types (used in e.g. queue.Queue).

Distinguishing different type messages by instantiating them from different subclasses is cool. Beyond that, I like to duck type: it means less for me to remember and compose.

Quote
Or on further thought, might it be better to raise a ValueError if sendtimeout is neither None nor a number?
Now that I think of it, I agree: raising a ValueError makes most sense.

Or provoking a illuminating TypeError with three more characters. (It was my 2nd edit while you were composing your message.)
-John
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf