Is a C struct flatten in memory (Python wrap for a C lib)?

[RoGeorge]

I've tried yesterday to add a Python3 wrapping to a function from a C library, and failed. 

It all went like this:

 



This is the C function I've tried to call from Python, and the struct for *info bamboozles me:

Code: [Select]

typedef struct
{
    void (*broadcast)(const char *address, const char *interface);
    void (*device)(const char *address, const char *id);
    void (*service)(const char *address, const char *id, const char *service, int port);
} lxi_info_t;

...

int lxi_discover(lxi_info_t *info, int timeout, lxi_discover_t type);

The code is from https://github.com/lxi-tools/liblxi/blob/master/src/lxi.h , a C library used to control SCPI instruments over LAN (LXI).  The lxi_discover is meant to discover SCPI instruments present in the LAN.

For the other functions present in the same lib, the author already added a Python wrap:
https://github.com/lxi-tools/python-liblxi/blob/master/lxi.py

My attempt to add the discover function is about learning.  My instruments are set with a fixed IP so no need to discover them with a function call, though maybe others might have a different setup and find the lxi_discover useful, IDK.

I'm not sure what I was doing wrong in Python, but no wrapping syntax worked.  I think I don't understand the lxi_info_t struct.

- Can somebody explain to me in words please "void (*broadcast)(const char *address, const char *interface);"?
- How is the lxi_info_t struct datatype represented in memory?  Is it (for first line only) "pointer to function broadcast", "pointer to IP string", "pointer to interface name string"?

- How do I wrap lxi_discover, more precisely how to deal with the *info argument in Python?

[Siwastaja]

They are three function pointers.

On memory level, function pointer is like any other pointer - just a memory address.

The arguments to the functions (const char *address, const char *interface) are not stored in the function pointer itself. These are provided so that compiler knows how the functions look like, and can generate correct code when you call the function, through that function pointer.

Depending on the machine, most likely the address is either 32 or 64 bits, so this would be 3 * 4 bytes or 3 * 8 bytes, in that order. I don't exactly remember the padding rules but AFAIK pointers should be definitely self-aligning and not need padding, so struct is just these three memory addresses back-to-back without gaps. Hence, it should Just Work, even if you fail to provide "packed" attritube to the struct.

Regarding Python, I don't know. The only thing that comes in my mind is, do the C compiler and Python agree about the ABI; i.e., the calling conventions? But as I have no idea about Python, maybe someone else can give a better guess.

[RoGeorge]

So, talking about C only and assuming a pointer length is 8 bytes, the line:

Code: [Select]

lxi_discover(info1, 5, 0)means info1 is 8 bytes containing the memory address of a data structure, and that memory zone pointed by info1 will look like this

Code: [Select]

- 8 bytes containing the start address of a function named "broadcast"
- 8 bytes containing the start address of a function named "device"
- 8 bytes containing the start address of a function named "service"
The other arguments are easy, timeout 5 is an int (let's say is stored on 2 bytes), and 0 is an enum, so another int stored in another 2 bytes.

If I got the *info right, then how does one finds out (in C) what instruments were discovered by the "lxi_discover(info1, 5, 0)"?

This is a usage example of calling lxi_discover( ... ), from the file https://github.com/lxi-tools/lxi-tools/blob/master/src/discover.c

Code: [Select]

static int device_count = 0;
static int service_count = 0;

static void broadcast(const char *address, const char *interface)
{
    UNUSED(address);
    printf("Broadcasting on interface %s\n", interface);
}

static void device(const char *address, const char *id)
{
    printf("  Found \"%s\" on address %s\n", id, address);
    device_count++;
}

static void service(const char *address, const char *id, const char *service, int port)
{
    printf("  Found \"%s\" on address %s\n    %s service on port %u\n", id, address, service, port);
    service_count++;
}



int discover(bool mdns, int timeout)
{
    lxi_info_t info;

    // Set up info callbacks
    info.broadcast = &broadcast;
    info.device = &device;
    info.service = &service;

    printf("Searching for LXI devices - please wait...\n\n");

    // Search for LXI devices / services
    if (mdns)
    {
        lxi_discover(&info, timeout, DISCOVER_MDNS);
        if (service_count == 0)
            printf("No services found\n");
        else
            printf("\nFound %d service%c\n", service_count, service_count > 1 ? 's' : ' ');
    }

... 
    return 0;
}

I don't understand the interaction with those 3 callback functions.

- Does this means "address" and "interface" are global variables used to return data (here the IP of the discovered instruments) back to the main program?
- How does lxi_discover (together with the 3 callback functions) returns the discovered data about the existing instruments?

[gf]

Regarding Python, I don't know. The only thing that comes in my mind is, do the C compiler and Python agree about the ABI; i.e., the calling conventions? But as I have no idea about Python, maybe someone else can give a better guess.

Two possibilities are:
1) Use the C API (implement python bindings to the library in C, i.e. implement an adapter module)
2) Use the CFFI (enables direct calling of C functions from Python)

[gmb42]

The lxi_discover function is passed the lxi_info_t structure containing the user specified notification callbacks and when necessary the code will call the callbacks to notify of something of interest.

It's up to the user of lxi_discover what those callbacks do, in the example you've posted they simply print something to the console and increment the count variables.  You could for example, accumulate devices in a list of some sort.

[Cerebus]

So, talking about C only and assuming a pointer length is 8 bytes, the line:

Code: [Select]

lxi_discover(info1, 5, 0)means info1 is 8 bytes containing the memory address of a data structure, and that memory zone pointed by info1 will look like this

Code: [Select]

- 8 bytes containing the start address of a function named "broadcast"
- 8 bytes containing the start address of a function named "device"
- 8 bytes containing the start address of a function named "service"
The other arguments are easy, timeout 5 is an int (let's say is stored on 2 bytes), and 0 is an enum, so another int stored in another 2 bytes.

If I got the *info right, then how does one finds out (in C) what instruments were discovered by the "lxi_discover(info1, 5, 0)"?

This is a usage example of calling lxi_discover( ... ), from the file https://github.com/lxi-tools/lxi-tools/blob/master/src/discover.c

Code: [Select]

static int device_count = 0;
static int service_count = 0;

static void broadcast(const char *address, const char *interface)
{
    UNUSED(address);
    printf("Broadcasting on interface %s\n", interface);
}

static void device(const char *address, const char *id)
{
    printf("  Found \"%s\" on address %s\n", id, address);
    device_count++;
}

static void service(const char *address, const char *id, const char *service, int port)
{
    printf("  Found \"%s\" on address %s\n    %s service on port %u\n", id, address, service, port);
    service_count++;
}



int discover(bool mdns, int timeout)
{
    lxi_info_t info;

    // Set up info callbacks
    info.broadcast = &broadcast;
    info.device = &device;
    info.service = &service;

    printf("Searching for LXI devices - please wait...\n\n");

    // Search for LXI devices / services
    if (mdns)
    {
        lxi_discover(&info, timeout, DISCOVER_MDNS);
        if (service_count == 0)
            printf("No services found\n");
        else
            printf("\nFound %d service%c\n", service_count, service_count > 1 ? 's' : ' ');
    }

... 
    return 0;
}

I don't understand the interaction with those 3 callback functions.

- Does this means "address" and "interface" are global variables used to return data (here the IP of the discovered instruments) back to the main program?
- How does lxi_discover (together with the 3 callback functions) returns the discovered data about the existing instruments?

As lxi_discover goes through the discovery process it calls the 'callback' functions to return information as and when it feels like it (i.e. you have no control over when the callbacks are made, you just know that they are a side-effect of calling lxi_discover). You are expected to write those callback functions and pass pointers to those callback functions (as a struct) at the start of the discovery process to lxi_discover. Those particular instances of "address" and "interface" are function parameters in the callback functions that you will have provided and are both of the type "a pointer to a null-terminated C character string".

Take a look here (https://reptate.readthedocs.io/developers/python_c_interface.html) for a short tutorial in C<->Python interfacing. The latter part has an example of how to write a simple callback in python that can be passed to C. Found, by the way, as the first hit on searching for "python interfacing to c callback functions" - sometimes you have to know the name of what you're trying to find ("callback functions") before you can start searching.

[RoGeorge]

If a callback can point anywhere, including in another program - here a function from Python - wouldn't that be a security issue?

I mean, I can write a malicious function in Python (let's say a reset, a jump to zero), and because of the callback exposed in that library, the call will appear like it is coming from a trusted library.

How does the kernel distinguish where from did a call came, and who is granted to make a reset and who is not?

[Nominal Animal]

If a callback can point anywhere, including in another program - here a function from Python - wouldn't that be a security issue?

A callback can only point at addresses in the same process.  The process here is the Python interpreter.  Its privileges are those of the user who executed the interpreter.  Because the executable is the Python interpreter, the privileges on the Python source or object files do not matter, except for readability: they are just data the Python interpreter reads to decide what it does internally.

The Python interpreter uses dlopen() (in Linux/Android/Mac) to load shared libraries into the process memory.  Python provides ctypes, a built-in facility for doing exactly that, plus converting from the OS-and-machine specific binary ABI (application binary interface) to Python and vice versa.  The documentation includes examples how to deal with callback functions, too.

[Nominal Animal]

If you are running on Linux, this example may help.

First, a trivial dynamic library, mycall.c:

Code: [Select]

struct callbacks {
    int (*first)(char *);
    int (*second)(int);
};

int mycall(struct callbacks *cbs)
{
    /* We require a non-NULL pointer to a structure, or we return -1. */
    if (!cbs) {
        return -1;
    }

    /* If the first callback is non-NULL, we'll call it. */
    if (cbs->first) {
        int  retval = cbs->first("Hello, world!");
        /* If it returns nonzero, we return that immediately. */
        if (retval)
            return retval;
    }

    /* If the second callback is non-NULL, we'll call it. */
    if (cbs->second) {
        int  retval = cbs->second(42);
        /* If it returns nonzero, we return that immediately. */
        if (retval)
            return retval;
    }

    /* All done. */
    return 0;
}Compile this using e.g.
    gcc -Wall -Wextra -O2 -fPIC mycall.c -shared -Wl,-soname,libmycall.so -o libmycall.so
into libmycall.so dynamic library.

Here is a Python 3 (3.5 or later) example, example.py, using the above:

Code: [Select]

# -*- encoding: UTF-8 -*-

from ctypes import CDLL, CFUNCTYPE, Structure, byref, c_char_p, c_int

# Example first callback function
def python_first(arg: c_char_p) -> c_int:
    print(b'python_first() called with "%b".' % arg)
    return 0

# Example second callback function
def python_second(arg: c_int) -> c_int:
    print(b'python_second() called with %d.' % arg)
    return 0

# This is the callback structure we need:
# struct {
#    int (*first)(char *);
#    int (*second)(int);
# };
class struct_cbs(Structure):
    _fields_ = [ ("first", CFUNCTYPE(c_int, c_char_p)),
                 ("second", CFUNCTYPE(c_int, c_int)) ]

# Let's create an instance of that structure, with the example callbacks.
cbs = struct_cbs( (CFUNCTYPE(c_int, c_char_p))(python_first),
                  (CFUNCTYPE(c_int, c_int))(python_second) )

# Load the dynamic library
libmycall = CDLL("./libmycall.so")

# Obtain the symbol (function in this case) from the library
mycall = libmycall.mycall
# Its return value is an int (default, so this is not necessary),
mycall.restype = c_int

# Do the C call
retval = mycall(byref(cbs))
print(b'mycall_func() returned %d.' % retval)
Run this example using e.g.
    python3 example.py
and the output will be
    b'python_first() called with "Hello, world!".'
    b'python_second() called with 42.'
    b'mycall_func() returned 0.'

If the structure contained basic ctypes types (c_int, c_char_p, etc.), they would not need the (CFUNCTYPE(rettype, argtypes...))(pythonfunc) construction when creating the structure instance; only the function type does.  Basic types can just be supplied as-is.

Again, everything you need for this is described in the Python 3 ctypes documentation.

When you get your mind around it (and the fact that the process is the python interpreter, which loads "copies" of the dynamic libraries needed into its memory –– instances of the same library in different processes cannot access each other), it really is very simple to do all the interfacing from the Python side, at least in Linux/Android/Macs.
If you use Windows, sorry; I do not, and I'd loathe trying to help without being able to verify my examples/suggestions first.

[RoGeorge]

Wow, the complete example came extremely useful, thank you!   

It finally makes sense.  I'll go read the doc tutorials, too.  So far I've only compiled and run your example, then put the .so and .py sources side by side in editor, read the comments, and I think this time I've got it how it works without any doubt.  The example clarified all.  Thanks a lot!



The OS, you didn't guess! 

Left Windows behind some years ago and never looked back, trying almost all Linux distros, from Ubuntu to Arch or Gentoo.  Settled to Kubuntu LTS, for comfort.  Then I wanted to have disk snapshots, preferably ZFS, and after yet another long detour of trying distros, including OpenSUSE with BTRFS and also FreeBSD with native ZFS, I've settled to a mixture of experimental Ubuntu on ZFS root + KDE Plasma from Kubuntu (Kubuntu doesn't support ZFS root like Ubuntu).

That combination worked pretty well, it even has a new tool, https://github.com/ubuntu/zsys, integrated in the OS (developed by Ubuntu), between other feature, it makes automated snapshots and lets one to either roll back from the Grub menus, or just boot into former snapshots without rollback.

Worked for a while just fine, then I clicked a link from the presumably Assange's dead man switch files on Wikileaks, and my nicely cobbled ZFS Kubuntu showed me the login screen out of nowhere, and never managed to recover it with all the ZFS and the automated snapshots.

Got quite scared, even made an SPI Flash reader from a Raspberry Pi to check the BIOS chips from the motherboard.  No idea what happened, but since I didn't repair the old OS yet, I kept using a micro SD card on which I've installed FreeBSD.

FreeBSD is not that user-friendly as Ubuntu, but the more I use it the more I like it.  It is closer to the old Unix style, no Ubuntu style bloatware, no systemd, and so far all the must have programs from Linux are available in one way or another in FreeBSD, too.

It's been a month since I use FreeBSD daily, and so far only VirtualBox had a missing part (no VirtualBox PUEL drivers for USB 2 or 3, so only USB 1 for VB machines).  There is a mode to run Linux binaries in FreeBSD, and that might be a workaround, but I don't know yet how that works.

At this point I don't even know if I want to go back to Ubuntu.  I like FreeBSD a lot, and their doc pages are better.

TL;DR - by serendipity I'm running FreeBSD, and your example was no problem for FreeBSD, the compile line was the same.  GCC was not found at first try, so I typed "pkg install gcc", then it worked. 

[Nominal Animal]

TL;DR - by serendipity I'm running FreeBSD, and your example was no problem for FreeBSD, the compile line was the same.  GCC was not found at first try, so I typed "pkg install gcc", then it worked. 

You can also replace gcc with clang, or even cc (the default C compiler), and it should work just fine on all systems using ELF dynamic libraries and a gcc/clang-like interface for specifying the library name: Linux, Android, FreeBSD, OpenBSD, most other BSD variants, and even MacOS.  (The -Wl,-soname,libname.so records the library name into the generated binary; as if one passed -soname libname.so to the ld linker.  It is not crucial, only recommended.)

In my Makefiles that generate dynamic libraries, I use the recipe
    libname.so: sources-or-object-files-used
        $(CC) $(CFLAGS) -fPIC -shared $^ -Wl,-soname,$@ $(LDFLAGS) -o $@
and it almost always suffices.  (If I recall correctly, symbol visibility may in some cases require some tricks; -rdynamic adds all symbols to the dynamic symbol table, and can be useful.)  Sometimes the target is libname.so.$(VERSION), though, although I like to set the name and symbolic links (from major and default versions) at install time instead.


The key point here is that in most cases, the original dynamic libraries with C bindings do not need any changes or any additions to be usable from Python, and the built-in ctypes interface is actually pretty easy to use to do all the interfacing from Python.

You should, of course, write a Python module (with minimal interface code, as shown in my example), to Pythonize the interface, preferably hiding any OS-specific quirks.  That way, one can later wrap the module (in if sys.platform == "platform": for each OS –– "aix", "linux", "freebsd", "win32", "cygwin", or "darwin" –– and then do the OS-specific stuff) if OS-specific quirks are needed.

The ctypes approach does mean that the Python code needs to describe the C interfaces it wants to use, because (on ELF-based systems) the dynamic libraries do not describe them, only the symbol name.  GObject introspection files (gir packages) do contain those, and the Python gi module can use an import them.  This is why you only need e.g.
    import gi
    gi.require_version("Gtk", "3.0")
    from gi.repository import Gtk
to use Gtk+3.0 in Python, without having to deal with ctypes directly.  (In Debian derivatives, the library itself is package libgtk-3-0, and the introspection files are in gir1.2-gtk-3.0, usually installed at /usr/lib/hwarch/girepository-1.0/library.typelib.  My system has 130 for them installed.)

Since lxi-tools already uses Gtk, I do believe it would make sense to just add GObject introspection to liblxi.  I do see you've already started on the Python bindings two weeks ago, but maintaining it separately from the C sources is more work than just letting the introspection tools do it automatically.  This does add one more build dependency, on gobject-introspection, but it is only a build-time dependency, not a run-time dependency.  How this is done in real life, is described at https://gi.readthedocs.io/en/latest/.  Essentially, when the dynamic liblxi library is compiled, g-ir-scanner is run to extract the type information from the C sources and header files, which is then compiled into the typelib file, Liblxi-1.13.typelib, using g-ir-compiler.  The typelib file is packaged separately (gir1.2-liblxi-1.13), providing the GObject introspection.  When that is installed, to use liblx1, one would just do
    import gi
    gi.require_version("Liblxi", "1.13")
    from gi.repository import Liblxi
The obvious benefit of this approach is automation.  The downside is that if the bindings needed Python logic (currently conversion to/from ASCII), those would still have to be implemented as a Python module... but changes and additions to the C library interface would be immediately reflected in the GI typelib files, and accessible from Python.

Now, I do not have any Test Equipment that could use LXI, so I cannot really tell if a separate Python interface module or GObject introspection files make more sense.
My own approach would be to test both first on my own machine (which I cannot do, lacking any LXI equipment!), and if the GI works without issues as-is, then contact the project maintainers/contributors and suggest the change in the form of a suggested patch and an example.

I do not normally do GI on libraries that have no connection to GObject (because, uh, no reason), but GTK+ is a GObject-based library, so it does kinda sorta make sense here.
GI is most useful when the library interface is still evolving, as it keeps the foreign language interface –– not just Python, mind you! –– in sync with the C implementation without explicit developer effort, only testing needed to make sure nothing breaks accidentally.  A simple way to think about it is to consider the GI Python module a way to automate what the ctypes module does, but using information gathered from the actual C sources.

[lundmar]

Hi George,

I'm a little late to the party but I see you have been trying to create a Python wrapper for liblxi and its lxi_discover() function - the one function I did not have time to create a wrapper for at the time because it required a bit more work.

Normally, creating Python wrappers for regular C functions using cdll is fairly straight forward. However, as you clearly have experienced, because lxi_discover() is working with callback function pointers passed in a structure, the cdll wrapper definitions become a bit more involved and a deeper understanding of c is required to make it work.

Anyway, I finally had some time to revisit the issue and I have now implemented the python wrapper for the discover feature.

I have also added a python discover example script: https://github.com/lxi-tools/liblxi-python/blob/master/examples/discover.py

Also updated the README instructions found here: https://github.com/lxi-tools/liblxi-python

If interested, you can see the specific changes I made here: https://github.com/lxi-tools/liblxi-python/commit/60345e344d0575cda503d327afe5229550465afb

If you have any questions regarding the implementation I'll be glad to answer.

/Martin

P.S. The GObject suggestion could have worked too but I prefer the clean cut cdll solution so we do not have any dependency on GObject etc.

[Nominal Animal]

Looks fine to me , if it matters any –– noting that I just looked at the code, not ran it, due to lack of LXI devices.

Suggestion:

LD_PRELOAD is automatically handled by the dynamic linker, and is a list of libraries (not library directories).  It is not really suitable here.

LD_LIBRARY_PATH environment variable is a colon-separated list of additional directories to look in, and is automatically applied by the underlying dlopen() call Python's ctypes.cdll.LoadLibrary() uses.

When given just the file name, ctypes.cdll.LoadLibrary() uses the exact same mechanisms native dynamic linking does, so the following would work better:

Code: [Select]

# License and description omitted for brevity

from ctypes import *

_lib = None
for libname in [ "liblxi.so.1", "liblxi.so.1.0.0" ]:
    try:
        _lib = cdll.LoadLibrary(libname)
        break
    except OSError:
        _lib = None
if _lib is None:
    print("LXI library (%s) not found" % (", ".join(libname)))
    exit()

# And so on.
The internal library reference is then _lib, the leading underscore being a common way to denote internal variables.
We prefer the liblxi.so.1 over liblxi.so.1.0.0, so that if a later compatible version of liblxi is released, we'll use that.
Most Python bindings only use the major version, though: a single try: _lib = cdll.LoadLibrary("liblxi.so.1"); except OSError: stanza.

In the documentation, if they encounter the error (they will not, if they install LXI from Linux distribution repositories), tell users to add the library directory (not path to library) to LD_LIBRARY_PATH. For the current directory,
    LD_LIBRARY_PATH=$CWD:$LD_LIBRARY_PATH python3 python-file-to-run
and any directory,
    LD_LIBRARY_PATH=/home/mystuff/libs:$LD_LIBRARY_PATH python3 python-file-to-run

P.S. The GObject suggestion could have worked too but I prefer the clean cut cdll solution so we do not have any dependency on GObject etc.

It makes a lot of sense if glib is not a required dependency already, sure.

[lundmar]

Looks fine to me , if it matters any –– noting that I just looked at the code, not ran it, due to lack of LXI devices.

Suggestion:

LD_PRELOAD is automatically handled by the dynamic linker, and is a list of libraries (not library directories).  It is not really suitable here.

LD_LIBRARY_PATH environment variable is a colon-separated list of additional directories to look in, and is automatically applied by the underlying dlopen() call Python's ctypes.cdll.LoadLibrary() uses.

When given just the file name, ctypes.cdll.LoadLibrary() uses the exact same mechanisms native dynamic linking does, so the following would work better:

Code: [Select]

# License and description omitted for brevity

from ctypes import *

_lib = None
for libname in [ "liblxi.so.1", "liblxi.so.1.0.0" ]:
    try:
        _lib = cdll.LoadLibrary(libname)
        break
    except OSError:
        _lib = None
if _lib is None:
    print("LXI library (%s) not found" % (", ".join(libname)))
    exit()

# And so on.
The internal library reference is then _lib, the leading underscore being a common way to denote internal variables.
We prefer the liblxi.so.1 over liblxi.so.1.0.0, so that if a later compatible version of liblxi is released, we'll use that.
Most Python bindings only use the major version, though: a single try: _lib = cdll.LoadLibrary("liblxi.so.1"); except OSError: stanza.

In the documentation, if they encounter the error (they will not, if they install LXI from Linux distribution repositories), tell users to add the library directory (not path to library) to LD_LIBRARY_PATH. For the current directory,
    LD_LIBRARY_PATH=$CWD:$LD_LIBRARY_PATH python3 python-file-to-run
and any directory,
    LD_LIBRARY_PATH=/home/mystuff/libs:$LD_LIBRARY_PATH python3 python-file-to-run

Thanks. Though, don't worry about how it is loaded. I'm very familiar with how the dynamic linker works and the details of its environment variables. The current loading mechanism is a quick hack stolen from elsewhere. When I get time I'll rewrite it all properly and at the same time take care of a few known loading issues specific to different platforms.

P.S. The GObject suggestion could have worked too but I prefer the clean cut cdll solution so we do not have any dependency on GObject etc.

It makes a lot of sense if glib is not a required dependency already, sure.

It is not a required dependency. lxi-tools is the one depending on GTK4/glib/gir etc. but liblxi is distributed independently and has no such dependency - it is as lightweight as possible.

[Nominal Animal]

If you want, I can write you the Python lxi module based on ctypes and liblxi using the following logic:

Class hierarchy:
    Instrument
        VXI11Instrument
        RawInstrument
        HiSlipInstrument (later)

Discover function (module method) returns a list of Instrument objects,
    import lxi
    instruments = lxi.Discover('VXI-11', timeout=10.0, broadcast=None, found=None)
where timeout, broadcast and found are optional callables; found returning False if the instrument is not to be included in the list.

Constructions like
    instruments = []
    instruments += lxi.discover('VXI-11')
    instruments += lxi.discover('MDNS')
would be perfectly acceptable.

This can be extended to lxi_discover_if(), limiting the discovery to specific network interfaces, via additional optional keyword parameter interface later on.

A specific instrument can be chosen from the above list,
    instrument = instruments[0]
constructing an Instrument instance,
    instrument = Instrument('VXI-11', address=b'10.42.0.42', port=111, timeout=10.0)
or directly an Instrument subclass instance,
    instrument = VXI11Instrument(address=b'10.42.0.42', timeout=10.0)

Default timeout and maximum default response length can be set and queried via
    instrument.timeout = 5.0
    instrument.maxlength = 65536
or used as a named (optional) parameter in the method calls.

(instrument.type would be a read-only string. instrument.address would be bytes and instrument.port int, modifiable while not connected, and readonly when connected.)

The unconnected instance only describes a possible connection.
Communication is only possible after connecting,
    instrument.connect()
and before disconnecting,
    instrument.disconnect()
and disconnecting is done automatically when the instance is destroyed; setting instrument=None before exiting the Python code would suffice for clean disconnect.

In addition to the SCPI send and receive commands, i.e.
    instrument.send(message)
    data = instrument.recv()
it might make sense to do a combined query, ie.
    data = instrument.query(message)

A wrapper module can then be used to extend this to GUI toolkits, with all lxi code run in a separate thread, and queries and responses (both connected and discovery) communicated via Queues.  That part is toolkit-independent; but usually you also want to insert an event in the event loop notifying a response is available from the Queue.

This way, your Qt/Gtk/wxWidgets/FLTK GUI won't stall during any LXI-related event, and on multicore machines, LXI doesn't have to share a CPU core with the Python code, making for much more useful Python applications.

However, it would be nice to hear from the users, if the above interface seems any nicer than the existing one.
(I do not have any LXI/SCPI/VXI-11 instruments myself, but I do often create all sorts of Python and C interfaces to my experimental microcontroller projects.)

[eutectique]

- Can somebody explain to me in words please "void (*broadcast)(const char *address, const char *interface);"?

Have a look at cdecl utility:

Code: [Select]

> cdecl explain "void (*broadcast)(const char *, const char *)"
declare broadcast as pointer to function (pointer to const char, pointer to const char) returning void

It works the other way round as well:

Code: [Select]

> cdecl declare "broadcast as pointer to function (pointer to const char, pointer to const char) returning void"
void (*broadcast)(const char *, const char *)


[lundmar]

If you want, I can write you the Python lxi module based on ctypes and liblxi using the following logic:

Class hierarchy:
    Instrument
        VXI11Instrument
        RawInstrument
        HiSlipInstrument (later)

Discover function (module method) returns a list of Instrument objects,
    import lxi
    instruments = lxi.Discover('VXI-11', timeout=10.0, broadcast=None, found=None)
where timeout, broadcast and found are optional callables; found returning False if the instrument is not to be included in the list.

Constructions like
    instruments = []
    instruments += lxi.discover('VXI-11')
    instruments += lxi.discover('MDNS')
would be perfectly acceptable.

This can be extended to lxi_discover_if(), limiting the discovery to specific network interfaces, via additional optional keyword parameter interface later on.

A specific instrument can be chosen from the above list,
    instrument = instruments[0]
constructing an Instrument instance,
    instrument = Instrument('VXI-11', address=b'10.42.0.42', port=111, timeout=10.0)
or directly an Instrument subclass instance,
    instrument = VXI11Instrument(address=b'10.42.0.42', timeout=10.0)

Default timeout and maximum default response length can be set and queried via
    instrument.timeout = 5.0
    instrument.maxlength = 65536
or used as a named (optional) parameter in the method calls.

(instrument.type would be a read-only string. instrument.address would be bytes and instrument.port int, modifiable while not connected, and readonly when connected.)

The unconnected instance only describes a possible connection.
Communication is only possible after connecting,
    instrument.connect()
and before disconnecting,
    instrument.disconnect()
and disconnecting is done automatically when the instance is destroyed; setting instrument=None before exiting the Python code would suffice for clean disconnect.

In addition to the SCPI send and receive commands, i.e.
    instrument.send(message)
    data = instrument.recv()
it might make sense to do a combined query, ie.
    data = instrument.query(message)

A wrapper module can then be used to extend this to GUI toolkits, with all lxi code run in a separate thread, and queries and responses (both connected and discovery) communicated via Queues.  That part is toolkit-independent; but usually you also want to insert an event in the event loop notifying a response is available from the Queue.

This way, your Qt/Gtk/wxWidgets/FLTK GUI won't stall during any LXI-related event, and on multicore machines, LXI doesn't have to share a CPU core with the Python code, making for much more useful Python applications.

However, it would be nice to hear from the users, if the above interface seems any nicer than the existing one.
(I do not have any LXI/SCPI/VXI-11 instruments myself, but I do often create all sorts of Python and C interfaces to my experimental microcontroller projects.)

I welcome any contributions that improve the liblxi python bindings and if you would like to add further object abstraction on top of the cdll/ctypes mappings feel to do so via pull request. I only did the cdll grunt work to map the minimum python bindings directly because a friend requested it and that is why it is not a further polished solution. Actually, someone contributed the original python bindings using the boost library but that is a terrible dependency to introduce for the use of such a simple library so I was compelled to replace it with cdll/ctypes. As maintainer of liblxi I try hard to avoid maintaining language bindings because that is a maintenance burden I simply can't afford among my other open source efforts so any help on that point is appreciated. By the end of the day, such bindings are best made and maintained by those that use them.

[Nominal Animal]

By the end of the day, such bindings are best made and maintained by those that use them.

Exactly.  The 'you' in my last post was meant equally as much to you as RoGeorge.

I did not originally do this exactly because I don't have the equipment, so I cannot even test the code; and without seeing your full bindings first, I didn't even know how they were to be used.  The above model is what I came up with after looking at the bindings and the examples, and mapping to the closest Python bindings that I've found to work well.  Even so, it is a guess, and without enthusiastic users willing to test and perhaps even maintain it, it is unnecessary effort.

If one looks at e.g. PyQt5 bindings by RiverBankComputing – a British firm not directly related to Qt –, it is obvious why it is crucial that those who maintain it, themselves also use it.  For one, they need to document the interface and common use cases well, just to save repeating the same efforts in the long run. PySide was also independently developed, and PySide2 folded into the Qt Project only in 2015.  The PySide folks ended up developing API extractor (for Qt-based projects) and Shiboken, somewhat similarly to GObject Introspection to GObject/glib based projects.

In any case, if you get asked about something like this later on or somewhere else, do feel free to throw me or tell them to throw me an email.  The address is shown on my home page, which is linked to from my profile.  I cannot promise anything, but I do love to help just to see/hear others do useful stuff and create new tools, and I always do this stuff for free, only asking to pass the help forwards.