Author Topic: Download speed from Rigol DS1054Z or similar oscilloscope to a PC  (Read 6248 times)

0 Members and 1 Guest are viewing this topic.

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6675
  • Country: ro
Asking because I've just tried, and it took more than 21 minutes to download all the 24 million data points.  ::)

The PC can easily download from Internet at 700-800 Mbps, so the bottleneck is not the computer's hardware.
    - Software:  Kubuntu -> VScode (Codium) -> Jupyter Notebooks -> my Python scripts -> python-ivi -> PyVISA -> PyVISA-py -> python-vxi11
    - Hardware:  i7 Desktop -> LAN cable -> 100Mbps router (switch) -> LAN cable -> Rigol DS1104Z oscilloscope

Anybody else tried to download all the data sampled by a Rigol oscilloscope?
How fast does that worked for you, and with what software/hardware, please?

Online Fungus

  • Super Contributor
  • ***
  • Posts: 17149
  • Country: 00
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #1 on: November 12, 2022, 09:27:57 am »
What format did you download it in?  CSV?

Bottom line: The rigol is an oscilloscope, not a high speed internet device.
 
The following users thanked this post: RoGeorge

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6675
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #2 on: November 12, 2022, 09:28:56 am »
With the DS1000Z series it takes 55 seconds to download 24Mpts via usbtmc. Via ethernet it is more or less the same.
The bottleneck is the scope, not the connection.
Data needs to be downloaded in chunks of max 250,000 bytes.
Between those chunks there's a long delay in the response of the scope's firmware.
Why? Only Rigol knows...

Edit: I did a new test with the latest firmware (the one Rigol removed) and now the download time is 33 seconds for 24Mpts.

That was 8+ years ago.  Back then I was experiencing very slow downloads, but only when using Windows.  From Linux it was OK.  I thought that issue was sorted out.  Now, many firmware updates and many OS versions later, I'm seeing about the same problem, but with Linux.  :-//

Anybody else tried downloading the data samples from a Rigol recently, please?

Online Fungus

  • Super Contributor
  • ***
  • Posts: 17149
  • Country: 00
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #3 on: November 12, 2022, 12:30:40 pm »
Anybody else tried downloading the data samples from a Rigol recently, please?
yes 30-60 seconds is the right figure to download raw 24Mpts data using visa driver from VisaDSO in Windows here.. if it takes 21 minutes for you, something is wrong.

We know nothing at all about how OP is "downloading"the data...
 

Offline Karel

  • Super Contributor
  • ***
  • Posts: 2259
  • Country: 00
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #4 on: November 12, 2022, 02:08:21 pm »
Using DSRemote it takes exactly 30 seconds (per channel) to download the whole 24Mpts.
 
The following users thanked this post: RoGeorge

Online RoGeorgeTopic starter

  • Super Contributor
  • ***
  • Posts: 6675
  • Country: ro
Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
« Reply #5 on: November 14, 2022, 10:19:03 am »
Thanks for mentioning DSremote, forgot about that one.  Installed DSremote in Kubuntu 22.04 LTS, and it takes only 30 seconds to download data in its Wave Inspector.  :o

At first I was trying from Python-IVI, which is a layer on top on PyVISA, as described in the OP.  It was way too slow, so this weekend tried sending SCPI commands from C, by calling into the 'liblxi.so' library (liblxi is available in most of the official repos, "sudo apt install liblxi-dev" or from sources at https://github.com/lxi-tools/liblxi ).  Then tried with DSremote, and DSremote is still 3 times faster than anything else I've tried.

Always used RAW mode and BYTES, so 24MB of data to be downloaded for all the 24MSa of ADC data.  No matter the size of each consecutive download batches, the duration benchmark is for a complete 24Msa download.

  • 23 minutes - Python scripts, IVI calls -> python-ivi -> PyVISA -> PyVISA-py -> python-vxi11 -> LAN -> DS1104Z
  • 134 seconds - C program, SCPI requesting batches of 125'000 bytes at once -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
  •   86 seconds - C program, SCPI requesting batches of 250'000 bytes at once (maximum size according to the DS1000Z programming guide) -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
  •   65 seconds - C program, SCPI requesting batches of 750'000 bytes at once -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
  •   55 seconds - C program, SCPI requesting batches of 1'000'000 bytes at once -> lib lxi -> lib tirpc  -> LAN -> DS1104Z
  •   30 seconds  :o - DSremote downloading all the samples to be saved or viewed in its waveform inspector

    It's interesting how DSremote is at least twice than any other setup.  I've looked at the LAN packets with Wireshark, and it seems DSremote is not using VXI11 protocol for download, it is using plain TCP on the port 5555.  The SCPI commands from DSremote are as expected:
    Code: [Select]
        STOP:
        *OPC?
        WAV:SOUR CHAN1
        *OPC?
        WAV:FORM BYTE
        *OPC?
        WAV:MODE:RAW
        *OPC?
        WAV:YINC?
        WAV:YREF?
        WAV>YOR?

        WAV:STAR 1
        WAV:OPC?
        WAV:STOP 250000
        WAV:OPC?
        WAV:DATA?
            // incoming first 250'000 ADC samples

        WAV:STAR 250001\0x0a
        *OPC?\0x0a
            // (expected response from *OPC is 1\0x0a)
        WAV:STOP 500000\0x0a
        *OPC?\0x0a
        WAV:DATA?\0x0a
            // (data response is #9000250000<250kB_of_data_over_many_TCP_packets_of_mostly_65535_TCP_bytes_each>\0x0a

        // and so on until all the wanted 24 million samples are fetched
        // ...

     - SCPI terminator is always the 0x0a char <LF> (beware printf("\n") in Windows is two cars, <CR><LF>, printf("%c", 10) might be safer)
     - OPC? (operation completed) is probed by DSremote after each command (which I didn't probe, maybe I should)
     - timeout in the DSremote settings was left as found, 50ms, but if I put the same timeout in my C code, the value is too small and leads to timeout errors sooner or later.  I had to use 2000ms instead of 50ms.  ???



    This is my fastest so far, with C and liblxi, which is still about 3 times slower than DSremote:
    Code: [Select]
    // liblxi git at https://github.com/lxi-tools/liblxi
    // to install it from the ubuntu repository (lxilib-dev is in most of the Linux distros repos and in FreeBSD):
    //      sudo apt install liblxi-dev

    // headers in   /usr/include/lxi.h
    // .so lib in   /usr/lib/x86_64-linux-gnu/liblxi.so
    // display all installed files with 'dpkg -L liblxi-dev' or
    //      sudo apt install apt-file
    //      sudo apt-file update
    //      (sudo???) apt-file list liblxi-dev
    // no need to append LIBRRY_PATH or LD_LIBRARY_PATH

    // man lxi_<TAB><TAB>
    //      lxi_connect  lxi_discover  lxi_init  lxi_send  lxi_disconnect  lxi_discover_if  lxi_receive

    // compile and run this 'demo.c' from a Linux terminal (!!! -l options _must_ be positioned after the .c sources, not before)
    //      gcc -Werror -Wfatal-errors ./demo.c -llxi -ltirpc && time ./a.out

    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <time.h>
    #include <lxi.h>

    int main()
    {
        // 12 bytes preamble + max of 24 mil bytes of ADC samples + 1 byte for the end of string terminator
        #define CHARS_BUFFER_MAX    24000000L // in specs max Rx in a single chunk is 250k chars for Rigol DS1054Z

        char *rx24MB = NULL;                                    // declare a pointer to char, and initialize it with NULL
        rx24MB = malloc(CHARS_BUFFER_MAX * sizeof *rx24MB);     // allocated memory on the heap for rx24MB[CHARS_BUFFER_MAX]
        if (!rx24MB) {                                          // !!! always check the memory was allocated successfully !!!
            fputs ("ERROR: allocation memory failed for the 'rx24MB' buffer, exiting.", stderr);
            exit (EXIT_FAILURE);
        }

        int device, length, timeout = 2000;     // timeout is in ms
        char response[1000];    // allocated on the stack, segmentation fault if too large, Linux max stack is 8MiB

        char *command = "*IDN?";
       
        lxi_init(); // Initialize LXI library
        device = lxi_connect("192.168.1.3", 0, "inst0", timeout, VXI11);    // Connect to LXI device
       
        lxi_send(device, command, strlen(command), timeout);                // Send SCPI command ("*IDN?")
        lxi_receive(device, response, sizeof(response), timeout);           // Rx until "\n", or sizeoff(response) chars, or timeout

        printf("%s\n", response);
        // exit if no DS1054Z
    // exit(0);


        // enable only ch1
        command = ":CHAN1:DISP 1";
        lxi_send(device, command, strlen(command), timeout);
        command = ":CHAN2:DISP 0";
        lxi_send(device, command, strlen(command), timeout);
        command = ":CHAN3:DISP 0";
        lxi_send(device, command, strlen(command), timeout);
        command = ":CHAN4:DISP 0";
        lxi_send(device, command, strlen(command), timeout);

    // ??? here to insert ":RUN""
        // set acquisition mode to max 24_000_000 points memory depth
        // "auto", 12K, 120K, 1.2M, 12M, 24M"
        long mdep = 24000000L;

        command = ":RUN";
        lxi_send(device, command, strlen(command), timeout);
       
        // sprintf(command, ":ACQ:MDEP %li", mdep);
        // sprintf(command, ":ACQ:MDEP %li%s", mdep, "\0");
        command = ":ACQ:MDEP 24000000";
        lxi_send(device, command, strlen(command), timeout);

        // wait for triggered
        do {
            command = ":TRIGger:STATus?";
            lxi_send(device, command, strlen(command), timeout);
            lxi_receive(device, response, sizeof(response), timeout);
        } while (response[0] != 'T');

        // then wait for ADC to aquire enough data
        clock_t stop_time;
        printf("wait %ld\n", clock());
        stop_time = clock() + 30000;             //30 seconds
        printf("beep %ld\n", stop_time);
        while (clock() < stop_time) ;
        printf("go %ld\n", clock());

        // go to stop mode and prepare to dl 24MSa
        command = ":STOP";
        lxi_send(device, command, strlen(command), timeout);
        command = ":WAV:SOUR CHAN1";
        lxi_send(device, command, strlen(command), timeout);
        command = ":WAV:MODE RAW";
        lxi_send(device, command, strlen(command), timeout);
        command = ":WAV:FORM BYTE";
        lxi_send(device, command, strlen(command), timeout);

    // char c2[80] = {0};
    char c2[80];
    // loop to dl all samples in chunks of (max) 250_000 samples
        // set start-stop indexes for next chunk
        long rx_len, chunk_len;
        // long chunk_start, chunk_stop, chunk_size =  125000L;    //dl 24Msa/134s
         long chunk_start, chunk_stop, chunk_size =  250000L;    //dl 24Msa/86s  max from specs
        // long chunk_start, chunk_stop, chunk_size =  750000L;    //dl 24Msa/65s
        // long chunk_start, chunk_stop, chunk_size = 1000000L;    //dl 24Msa/56s  preferred ???
        // long chunk_start, chunk_stop, chunk_size = 1175000L;    //dl 24Msa/54s
        long samples_to_dl = 24000000L;

        if (samples_to_dl > mdep) {
            samples_to_dl = mdep;
        }
        for(chunk_start = 1; chunk_start < samples_to_dl; chunk_start += chunk_size) {
            sprintf(c2, ":WAV:STAR %li", chunk_start);
            puts(c2);
            rx_len = lxi_send(device, c2, strlen(c2), timeout);
            printf("ack_len=%ld\n", rx_len);

            //at the last chunk, chunk_len might be shorter than chunk_size
            if (chunk_start + chunk_size > samples_to_dl) {
                chunk_len = samples_to_dl - chunk_start + 1;
                chunk_stop = samples_to_dl;
            } else {
                chunk_len = chunk_size;
                chunk_stop = chunk_start + chunk_size - 1;
            }

            sprintf(c2, ":WAV:STOP %li", chunk_stop);
            puts(c2);
            rx_len = lxi_send(device, c2, strlen(c2), timeout);
            printf("ack_len=%ld\n", rx_len);

            command = ":WAV:DATA?";
            puts(command);
            rx_len = lxi_send(device, command, strlen(command), timeout);
            printf("ack_len=%ld\n", rx_len);

            rx_len = lxi_receive(device, (char *)(&rx24MB[chunk_start-1]), chunk_len, timeout);
            printf("rx_data_bytes=%ld\n", rx_len);

            int err = 0;
            if(rx_len != chunk_len) {
                printf("ERROR - received too few bytes: ");
                err = 1;
            }
            printf("chunk_len=%li rx_len=%li\n", chunk_len, rx_len);
            printf("\n");
            if(err) exit(err);
        }

    // Disconnect
    lxi_disconnect(device);
    }


    Not sure if I am doing something wrong, or if I should just use TCP instead of VXI11?  :-//
    « Last Edit: November 14, 2022, 10:56:13 am by RoGeorge »
     
    The following users thanked this post: egonotto

    Online Fungus

    • Super Contributor
    • ***
    • Posts: 17149
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #6 on: November 14, 2022, 10:43:50 am »
    if I should just use TCP instead of VXI11?  :-//

    Yes.
     

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #7 on: November 14, 2022, 12:37:58 pm »
    At first I was trying from Python-IVI, which is a layer on top on PyVISA, as described in the OP.  It was way too slow
    Python-IVI is transport-agnostic, so I'd think that you can tell the IVI driver to connect using TCP instead of VXI-11 using a different VISA resource name. I don't see why you'd need to use C for a throughput of maybe 2 MB/s.

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #8 on: November 14, 2022, 01:22:07 pm »
    I used C because the SW stack from python was heaving too many layers and I didn't know which one was slowing down the transfers.  The simplest way I was aware of (other than making direct socket calls) was to use liblxi from C.

    I did met some similar issues (10-100 slower transfers than it should be) about 7-8 years ago, when I was using SCPI sent to a TCP socket, directly from Python with no drivers, or IVI/VISA/VXI or alike.  Back then it was happening in Windows only, because of some specific algorithm Windows was using when it was increasing the TCP packets size.  That was broken by design for Windows, but it was working fine from Linux.

    Now I am seeing those 20+ minutes long transfers in Linux.  Don't know yet if it's caused by the same issue from 8 years ago, or if it's something different.  Maybe the wrong by designed RWIN (TCP Receive WINdow) algorithm used by Windows back then was meanwhile added as a new Linux "feature", IDK.  :-//
    https://hackaday.io/project/5807-driverless-rigol-ds1054z-screen-capture-over-lan/log/18807-solved-a-painfully-slow-lan-on-windows-and-a-wasted-weekend
    « Last Edit: November 14, 2022, 01:24:54 pm by RoGeorge »
     

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #9 on: November 14, 2022, 01:53:32 pm »
    Now I am seeing those 20+ minutes long transfers in Linux.
    Is that over VXI-11 or over TCP sockets (TCPIP0::host address::port::SOCKET)? VXI-11 is probably not optimized for high throughput, but plain TCP sockets should be pretty straight-forward.

    Offline DiTBho

    • Super Contributor
    • ***
    • Posts: 4227
    • Country: gb
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #10 on: November 14, 2022, 04:21:23 pm »
    too confusing  :-//
    The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
     

    Offline DiTBho

    • Super Contributor
    • ***
    • Posts: 4227
    • Country: gb
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #11 on: November 14, 2022, 04:42:40 pm »
    I don't have a Rigol DS1054Z, but sigrok seems to have support, see here
    The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
     

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #12 on: November 14, 2022, 06:40:05 pm »
    Now I am seeing those 20+ minutes long transfers in Linux.
    Is that over VXI-11 or over TCP sockets (TCPIP0::host address::port::SOCKET)? VXI-11 is probably not optimized for high throughput, but plain TCP sockets should be pretty straight-forward.

    Wireshark was telling the LAN packets were of type VXI-11 protocol, most probably the slow transfer was for VXI.  In fact, it might be the only one supported, I don't know how to configure IVI to use SOCKET instead of VXI-11.

    I've just tried "TCPIP0::host address::port::SOCKET" and it breaks with the err message "unrecognized resource string format".  By looking inside the code, the regular expressions used to validate the resource string doesn't know about the word "SOCKET", can only end in "INSTR".  Same in the man page, all examples are of type "TCPIP................INSTR"
    https://www.systutorials.com/docs/linux/man/1-pythonivi



    It is still possible to use python-ivi for all the other commands, and write a C function to fetch the ADC data points only using SOCKET and forcing the TCP packets size, the oscilloscope allows multiple connections.  Or I can dig in my own python2 repos from 8 years ago and check if the WAVE:DATA? scripts from back then are still fast (in Linux).

    IVI is very appealing, but for some reason it didn't get enough traction, and now  it feels like abandonware, the last accepted pull in the main project was in 2016 or so.  For example, two years ago I had to merge from many forks and found no python-ivi driver for my signal generator.  Now, I had to search again through 108 forks and merge the most active (active in the sense that they appear to have extra python-ivi drivers), thought there is still no driver for my generator, but there is a generator driver from another instrument, and it happens that the main controls are the same, so at least I can change the frequency from python-ivi and do the rest from buttons.

    I really like python-ivi, but it sunk me way too much time.  After all, I don't think any body else will reuse my automated test software, not even myself, so I can just use SCPI.  It's been 10 days since I've started to fool around with python-ivi and still tinkering around it instead of working at the data processing, which was the actual goal of the project.
    « Last Edit: November 14, 2022, 06:43:39 pm by RoGeorge »
     

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #13 on: November 14, 2022, 07:16:45 pm »
    I've just tried "TCPIP0::host address::port::SOCKET" and it breaks with the err message "unrecognized resource string format".  By looking inside the code, the regular expressions used to validate the resource string doesn't know about the word "SOCKET", can only end in "INSTR".  Same in the man page, all examples are of type "TCPIP................INSTR"
    https://www.systutorials.com/docs/linux/man/1-pythonivi
    Reading the source code, this regular expression is for the resources python-ivi can handle directly without PyVisa, like VXI-11. Anything else should be passed to PyVisa, unless PyVisa is not imported, which sounds like is the case for you (the exception on line 1837).

    IVI is very appealing, but for some reason it didn't get enough traction, and now  it feels like abandonware, the last accepted pull in the main project was in 2016 or so.  For example, two years ago I had to merge from many forks and found no python-ivi driver for my signal generator.  Now, I had to search again through 108 forks and merge the most active (active in the sense that they appear to have extra python-ivi drivers), thought there is still no driver for my generator, but there is a generator driver from another instrument, and it happens that the main controls are the same, so at least I can change the frequency from python-ivi and do the rest from buttons.

    I really like python-ivi, but it sunk me way too much time.  After all, I don't think any body else will reuse my automated test software, not even myself, so I can just use SCPI.  It's been 10 days since I've started to fool around with python-ivi and still tinkering around it instead of working at the data processing, which was the actual goal of the project.
    For me python-ivi is still worth it because it provides a fairly sensible way to abstract instruments, so I can use the same code for a Tektronix scope and a Lecroy scope, for example, which as a hobbyist with a heterogeneous lab with instruments from different manufacturers is very helpful. Even though I've had to develop most of the drivers that I use myself, this was a one-time effort. I don't just have three identical instruments to throw at the same problem, so if I need multiple scopes, power supplies or multimeters at the same time, they'll probably of different brands and vintages. I have not found an abstraction I liked better than python-ivi yet, and I certainly don't feel like inventing my own.

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #14 on: November 16, 2022, 01:39:58 pm »
    It looks like either python-ivi or pyvisa will use the same python-vxi11 module (import vxi11) to connect to an instrument over LAN.  I've tested and they are both very slow, it takes 23 minutes to fetch 24MSa.
    Code: [Select]
    # pip install python-vxi11
    # Successfully installed python-vxi11-0.9
    import vxi11

    dso = vxi11.Instrument('192.168.1.3')
    # alternatives (transfer speed is the same):
    # dso = vxi11.Instrument("TCPIP::192.168.1.3::INSTR")

    print(dso.ask('*IDN?'))



    dso.write(':RUN')
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')
    dso.write(':ACQ:MDEP 24000000')     # !!! can only be set when in :RUN mode
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:MODE MAX')          # !!! 1200 points in :RUN mode, all MDEP points in STOP mode
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:FORM BYTE')
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')



    dso.write(':WAV:SOUR CHAN1')
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:STAR 1')
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':STOP')
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')
    dso.write(':WAV:STOP 250000')       # !!! will be ignored and always 1200 in :RUN mode
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    wvfm_pre = dso.ask('WAV:PRE?')
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')
    print(wvfm_pre)

    print(dso.ask('WAV:MODE?'), dso.ask('WAV:STAR?'), dso.ask('WAV:STOP?'))



    dso.write(':STOP')
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    import time
    t0 = time.time()
    wvfm = dso.ask_raw(b':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
    t1 = time.time() - t0
    print('Time to fetch the first waveform chunk: ' , t1)
    print(type(wvfm), len(wvfm))
    while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

    to_fetch = 1_000_000    # fetch only 1Msa of ADC samples
    MAX_CHUNK = 250_000
    M_DEPTH = int(dso.ask(':ACQ:MDEP?'))

    all_wvfm = bytes()
    t0 = time.time()
    for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):

        dso.write(':WAV:STAR ' + str(i+1))
        while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

        dso.write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)))
        while dso.ask('*OPC?') != '1': print('Waiting for command to finish')

        wvfm = dso.ask_raw(b':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
        all_wvfm += wvfm[12:]

        print(len(wvfm))

    t1 = time.time() - t0
    print('Time to fetch ' + str(to_fetch) + ' ADC samples using "python-vxi11": ' + str(t1))



    dso.close()



    If I open a connection with SOCKET, or if I use something like the 'socketscpi' module, it fetches 24MSa in 23 seconds. 
    Code: [Select]
    # socketscpi is using direct TCP socket instead of VXI-11
    # pip install socketscpi
    # Successfully installed socketscpi-2022.8.0
    #   might work as a backend for PyVISA ????? for LXI instruments
    #   https://github.com/morgan-at-keysight/socketscpi/blob/master/examples.py

    import socketscpi

    dso = socketscpi.SocketInstrument(ipAddress='192.168.1.3', port=5555, timeout=1)
    # dso = socketscpi.SocketInstrument(ipAddress='192.168.1.3', port=5555, globalErrCheck=True)

    print(dso.instId)



    dso.write(':RUN')
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')
    dso.write(':ACQ:MDEP 24000000')     # !!! can only be set when in :RUN mode
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:MODE MAX')          # !!! 1200 points in :RUN mode, all MDEP points in STOP mode
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:FORM BYTE')
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')



    dso.write(':WAV:SOUR CHAN1')
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':WAV:STAR 1')
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    dso.write(':STOP')
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')
    dso.write(':WAV:STOP 250000')       # !!! will be ignored and always 1200 in :RUN mode
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    wvfm_pre = dso.query('WAV:PRE?')
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')
    print(wvfm_pre)

    print(dso.query('WAV:MODE?'), dso.query('WAV:STAR?'), dso.query('WAV:STOP?'))



    dso.write(':STOP')
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    import time
    t0 = time.time()
    wvfm = dso.query(':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
    t1 = time.time() - t0
    print('Time to fetch the first waveform chunk: ' , t1)
    print(type(wvfm), len(wvfm))
    while dso.query('*OPC?') != '1': print('Waiting for command to finish')

    to_fetch = 1_000_000    # fetch only 1Msa of ADC samples
    MAX_CHUNK = 250_000
    M_DEPTH = int(dso.query(':ACQ:MDEP?'))

    all_wvfm = b''
    t0 = time.time()
    for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):

        dso.write(':WAV:STAR ' + str(i+1))
        while dso.query('*OPC?') != '1': print('Waiting for command to finish')

        dso.write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)))
        while dso.query('*OPC?') != '1': print('Waiting for command to finish')

        wvfm = dso.query(':WAV:DATA?')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode

        b = bytes(wvfm[12:], 'latin_1')
        all_wvfm += b

        print(len(wvfm))

    t1 = time.time() - t0
    print('Time to fetch ' + str(to_fetch) + ' ADC samples using "socketscpi": ' + str(t1))



    dso.close()
    The downside is if I send a wrong command I have to power cycle the oscilloscope.




    My oscilloscope is Rigol DS1104Z (in python-ivi its driver module is 'rigol.rigolDS1054'), it has the fixed IP 192.168.1.3, and the port for SOCKET access is 5555.

    Do you have a working code example, please, for connecting to the oscilloscope from python-ivi, or even from PyVISA, but using SOCKET instead of VXI11, and then print the result of the '*IDN?'

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #15 on: November 16, 2022, 02:25:53 pm »
    I don't have any Rigol scope to test on, so I used netcat to create a listening TCP socket: nc -l -p 1052. Then I installed python-ivi from your Github, removed all the Python-2 style print statements that would give errors in Python 3 (after from __future__ import print_function, print(...) with parentheses should work in any Python version released after 2008), installed pyvisa and pyvisa-py, and made a small change to ivi/interfaces/pyvisa.py lines 32-39 and replaced the word visa by pyvisa. Apparently the module changed names in the past six years. And then it seems to work fine for me:
    Code: [Select]
    % python
    >>> import ivi
    >>> rig = ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')
    # This was necessary because I'm using a Linux terminal and not a real DS1054 scope
    >>> rig._interface.instrument.read_termination = '\n'
    >>> rig._interface.instrument.write_termination = '\n'
    >>> rig.identity.instrument_model
    'DS1102E'

    In netcat I'd see:

    % nc -l -p 1052
    *CLSRIGOL TECHNOLOGIES,DS1102E,DS1EB104702974,00.02.01.01.00
    *IDN?
    Red is what the driver sent, and green is what I typed to simulate the response of a scope.

    *CLS is what the DS1054Z driver sends when you instantiate it, and RIGOL TECHNOLOGIES,... is what I typed as a pre-emptive response to the *IDN? query since I can't type fast enough to stay within the timeout. I'm pretty sure that the same Python code, probably without changing the termination, would work with a real scope.
    « Last Edit: November 16, 2022, 02:33:49 pm by alm »
     
    The following users thanked this post: RoGeorge

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #16 on: November 16, 2022, 05:09:43 pm »
    Sorry for the invalid Python2 prints, not mine, I've removed them only locally, now removed them from my git fork, too.
    The problem is that I get invalid resource string for something like:
    >>> rig = ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')
    Code: [Select]
    ...
    python-ivi_RoGeorge_fork/python-ivi/ivi/ivi.py", line 1817, in _initialize
        raise IOException('Invalid resource string')
    ivi.ivi.IOException: Invalid resource string
    >>>

    If that resource string worked for you, then you must be having one of the proprietary VISA libraries installed, something like NI-VISA, Keysight VISA, R&S VISA, Tektronix VISA, and python-IVI or maybe pyvisa detects the closed source library and sends the resource string to it.

    I don't have any of the proprietary VISA libraries, only the FOSS python modules:  python-ivi, python-vxi11, pyvisa and pyvisa-py.  IIRC pyvisa also has some options to configure its backend.

    Also I've noticed your prompt is % (% prompt is usually for csh).  What OS did you used to test the line "ivi.rigol.rigolDS1054Z('TCPIP0::localhost::1052::SOCKET')"?

    TL DR;
    'TCPIP0::localhost::1052::SOCKET' doesn't work on my Kubuntu 22.04 LTS, I have no proprietary NI-VISA or alike installed.  Any hints what am I missing, or mis-configured, or what should I install to make my python-IVI work with SOKET connection just like yours, please?
    « Last Edit: November 16, 2022, 05:12:12 pm by RoGeorge »
     

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #17 on: November 16, 2022, 05:35:47 pm »
    This was on Ubuntu 22.04.1 LTS without any proprietary VISA installed, just PyVISA and PyVISA-py. These are the only modules installed inside the virtual environment where I ran these commands:
    Code: [Select]
    python = "^3.10" (resolved to Python 3.10.6)
    PyVISA = "^1.12.0" (resolved to 1.12.0)
    PyVISA-py = "^0.5.3" (resolved to 0.5.3)
    numpy = "^1.23.4" (resolved to 1.23.4)
    python-ivi = {git = "https://github.com/RoGeorge/python-ivi.git"}
    and as indirect dependency of PyVISA typing-extensions 4.4.0

    PyVISA used the PyVISA-py backend without any configuration on my side. This is with using PyVISA directly:
    Code: [Select]
    >>> import pyvisa
    >>> rm = pyvisa.ResourceManager()
    >>> res = rm.open_resource('TCPIP0::localhost::1052::SOCKET')
    >>> res.read_termination = '\n'
    >>> res.write_termination = '\n'
    >>> res.write('*IDN?')
    7
    >>> res.read()
    'ME.' (this is just what I typed in netcat)

    If this doesn't work for you, then maybe try explicitly setting the PyVISA-py backend as described here. I did have to modify ivi/interface/pyvisa.py as I described above before python-ivi would detect pyvisa. Without that modification I also got the Invalid resource string error.
     
    The following users thanked this post: RoGeorge

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #18 on: November 16, 2022, 08:31:39 pm »
    Thanks for pointing to the visa -> pyvisa name changes, wouldn't have noticed that too soon, if ever!
    Now it can open a SOCKET connection from python-ivi.  :D

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #19 on: November 16, 2022, 08:46:02 pm »
    Yeah, I guess the module was called "visa" back in 2015 when that code was written. I've never run into it myself because all devices I've controlled so far have a GPIB interface that I access via VXI-11 using a LAN to GPIB bridge (HP/Agilent, Tektronix, etc). So I have no use for anything but VXI-11 for now.

    Nice, does it work with the DS1054Z now?

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #20 on: November 17, 2022, 04:15:37 pm »
    The connection, yes it works, the PythonIVI driver for DS1054Z, not really.  For example I was expecting to see the <LF> (0x0a) at the end of the SCPI commands, but the PC doesn't send 0x0a in the LAN packets.  The oscilloscope replies with the expected 0x0a terminator.  Will look closer these days.

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #21 on: November 17, 2022, 04:21:50 pm »
    Did you try playing with read and write termination attributes that I set in my samples? I'd expect setting write termination to '\n' to fix the problem of missing line feeds when sending to the scope.

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #22 on: November 17, 2022, 04:47:58 pm »
    Only took a brief look, it was very late yesterday.  The read or write_termination are from pyvisa, ivi doesn't seem to have them.  I guess I should configure the pyvisa backend before making a connection from ivi, not sure how to do it.

    There is not much documentation, and I'm not a programmer.  Digging through code is much more difficult then reading a manual.  Overriding or extending classes bamboozles me a lot, since I don't have yet a clean map of the objects used in PythonIVI. 

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #23 on: November 17, 2022, 05:51:23 pm »
    See my earlier posts on how to set termination through python-ivi.

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #24 on: November 17, 2022, 11:14:15 pm »
    I don't know how to apply read/write_termination before opening the instrument.

    If I open the instrument first, by default the driver sends '*CLS' without the 0x0a terminator, which hangs the oscilloscope communication, and can recover only by a power cycle.  By looking through sources, found out that there are optional parameters possible to send when opening a new connection, one of the parameters being to not emit a "*CLS" command by default. 
    Code: [Select]
    import ivi
    ivi.set_prefer_pyvisa()

    dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::5555::SOCKET', id_query=False, reset=False)
    dso._interface.instrument.read_termination = '\n'
    dso._interface.instrument.write_termination = '\n'

    dso.identity.instrument_model
    dso._ask('*IDN?\n')



    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')
    dso._write(':ACQ:MDEP 24000000\n')     # !!! can only be set when in :RUN mode
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    dso._write(':WAV:MODE MAX\n')          # !!! 1200 points in :RUN mode, all MDEP points in STOP mode
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    dso._write(':WAV:FORM BYTE\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    dso._write(':RUN\n')



    dso._write(':WAV:SOUR CHAN1\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    dso._write(':WAV:STAR 1\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    dso._write(':STOP\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')
    dso._write(':WAV:STOP 250000\n')       # !!! will be ignored and always 1200 in :RUN mode
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    wvfm_pre = dso._ask('WAV:PRE?\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')
    print(wvfm_pre)

    print(dso._ask('WAV:MODE?\n'), dso._ask('WAV:STAR?\n'), dso._ask('WAV:STOP?\n'))



    dso._write(':STOP\n')
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    import time
    t0 = time.time()
    wvfm = dso._ask_raw(b':WAV:DATA?\n')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode
    t1 = time.time() - t0
    print('Time to fetch the first waveform chunk: ' , t1)
    print(type(wvfm), len(wvfm))
    while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

    to_fetch = 24_000_000    # fetch only 1Msa of ADC samples
    MAX_CHUNK = 250_000
    M_DEPTH = int(dso._ask(':ACQ:MDEP?\n'))

    all_wvfm = b''
    t0 = time.time()
    for i in range(0, min(to_fetch, M_DEPTH), MAX_CHUNK):

        dso._write(':WAV:STAR ' + str(i+1) + '\n')
        while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

        dso._write(':WAV:STOP ' + str(i + min(MAX_CHUNK, M_DEPTH, to_fetch-i)) + '\n')
        while dso._ask('*OPC?\n') != '1': print('Waiting for command to finish')

        wvfm = dso._ask_raw(b':WAV:DATA?\n')      # get 1200 points in :RUN mode or :WAV:STOP? points in :STOP mode

        # b = bytes(wvfm[12:], 'latin_1')
        # all_wvfm += b
        all_wvfm += wvfm[12:]

        print(len(wvfm))

    t1 = time.time() - t0
    print('Time to fetch ' + str(to_fetch) + ' ADC samples using "socketscpi": ' + str(t1))

    dso.close()

    This makes possible to connect to an instrument without sending any (wrong terminated) commands, then configure read/write_termination, which worked.  However, the read/write_termination settings seems to apply only for ivi functions.  For example, if I use osc._ask_raw(':WAV:DATA?\n'), then I must add the 0x0a manually, as '\n'.

    I had to look more into the ivi sources.  Maybe there is a way to set the automatic addition of the 0x0a terminator in pyvisa-py, so I won't end with a mixture of functions where some requires 0x0a while others don't.

    The transfer speed is 65 seconds to fetch 24MSa.

    Not bad, but still 3 times slower than the 22 seconds I get when downloading with https://github.com/morgan-at-keysight/socketscpi
    I wonder if it would be possible to use 'socketscpi' instead of 'pyvisa', or maibe to configure 'pyvisa' to use 'socketscpi' instead of 'pyvisa-py'.

    I've uploaded my current PythonIVI changes on gitlab, https://gitlab.com/RoGeorge/python-ivi, because github got crazy and doesn't let me any more to remote update my own repositories, unless I got a security token from them, which I don't want.  Adios GitHub, hello GitLab.

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #25 on: November 17, 2022, 11:37:30 pm »
    I don't know how to apply read/write_termination before opening the instrument.

    If I open the instrument first, by default the driver sends '*CLS' without the 0x0a terminator, which hangs the oscilloscope communication, and can recover only by a power cycle.
    Maybe calling dso._write('\n') right after instantiating the DS1104Z object would remove the hang? This would send the termination character the scope is waiting for.

    However, the read/write_termination settings seems to apply only for ivi functions.  For example, if I use osc._ask_raw(':WAV:DATA?\n'), then I must add the 0x0a manually, as '\n'.
    Looking at the PyVISA source, write adds a termination character, and write_raw not. So it's not a matter of ivi vs non-ivi functions, but of write_raw() not adding termination, because you're saying you want to write verbatim what you send. Does it work if you use dso._write(':WAV:DATA?') and then output = dso._read_raw()? ask is nothing more than write + read.

    I had to look more into the ivi sources.  Maybe there is a way to set the automatic addition of the 0x0a terminator in pyvisa-py, so I won't end with a mixture of functions where some requires 0x0a while others don't.
    Setting dso._interface.instrument.write_termination already sets the terminator in PyVISA, not ivi. dso._interface.instrument is the PyVISA resource.

    The transfer speed is 65 seconds to fetch 24MSa.

    Not bad, but still 3 times slower than the 22 seconds I get when downloading with https://github.com/morgan-at-keysight/socketscpi
    I wonder if it would be possible to use 'socketscpi' instead of 'pyvisa', or maibe to configure 'pyvisa' to use 'socketscpi' instead of 'pyvisa-py'.
    Changing Python-ivi to use socketscpi would probably be a big change. What you could consider is just like python-ivi currently supports using python-vxi11 directly without going through PyVISA, you could also add support for calling the python socket library directly for ::SOCKET resources without going through PyVISA. This would involve writing an instrument class around the socket library like the PySerial interface, and some minor changes to ivi.py around hereto add a case where it instantiates the newly created PySocket instrument instead of the vxi11 of PyVISA instruments.
     
    The following users thanked this post: RoGeorge

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #26 on: November 18, 2022, 09:08:24 am »
    Talking about configuring PyVISA to use 'socketscpi' as its LAN back-end because of this recent commit into socketscpi, commit that changed the old names of the socketscpi functions in order to match the pyvisa syntax, https://github.com/morgan-at-keysight/socketscpi/commit/3d8bea9b012c03a29d6676b60a21ac7e5a655ae4.

    By the commit description, 'socketscpi' might be already usable as a backend for PyVISA, just like now it is possible to force PyVISA to use 'pyvisa-py' instead of proprietary ni-visa shared libs (when both ni-visa and pyvisa-py are installed) like in this example from https://pyvisa.readthedocs.io/projects/pyvisa-py/en/latest/index.html
    Quote
    You can select the PyVISA-py backend using @py when instantiating the visa Resource Manager:
    Code: [Select]
    # pyvisa usage example
    import pyvisa as visa

    # rm = visa.ResourceManager()       # will look for proprietary ni-visa or alike first, if installed
    rm = visa.ResourceManager('@py')    # will ignore any installed VISA libs, and use pyvisa-py instead

    inst = rm.open_resource('<VISA_format_resource_of_an_instrument_to_open_here>')

    print(inst.query("*IDN?"))

    I don't know how to tell PyVISA to use socketscpi instead of pyvisa-py, or if socketscpi is ready to use as a pyvisa backend or not.  Will try to learn that by digging through sources.  Does this look feasible as a fix (python-ivi with prefer pyvisa  -->  pyvisa with forced backend socketscpi  -->  socketscpi using python's SOCKET)?
    « Last Edit: November 18, 2022, 09:10:31 am by RoGeorge »
     

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #27 on: November 18, 2022, 01:34:32 pm »
    Got 24MSamples in 22 seconds, with python-ivi, pyvisa, pyvisa-py and SOCKET connection!!!   :scared:
    :-DMM

    Found what was making 'pyvisa-py' 3 times slower than 'socketscpi' by looking side by side at the opening socket parameters in 'pyvisa-py' vs 'socketscpi'.  It was the default value of the TCP_NODELAY.  The download becomes fast if I add this extra line
    Code: [Select]
                self._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)
    at the end of the '_connect()' definition, inside the file lib/python3.10/site-packages/pyvisa_py/tcpip.py.  It's the line number five hundred and something in the modules installed with pip, probably would have to be inserted after line 840 if the same change were to be added in the github source file:
    https://github.com/pyvisa/pyvisa-py/blob/main/pyvisa_py/tcpip.py#L840


    However, I suspect there might be a way to set the nodelay parameter as True, but I don't know enough Python and/or OOP to figure myself the syntax for setting TCPIP NODELAY from the end user code, without changing the sources inside pyvisa-py files.

    Any idea how to set the nodelay parameter using python-ivi?

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #28 on: November 18, 2022, 09:23:53 pm »
    Talking about configuring PyVISA to use 'socketscpi' as its LAN back-end because of this recent commit into socketscpi, commit that changed the old names of the socketscpi functions in order to match the pyvisa syntax, https://github.com/morgan-at-keysight/socketscpi/commit/3d8bea9b012c03a29d6676b60a21ac7e5a655ae4.

    I don't know how to tell PyVISA to use socketscpi instead of pyvisa-py, or if socketscpi is ready to use as a pyvisa backend or not.  Will try to learn that by digging through sources.  Does this look feasible as a fix (python-ivi with prefer pyvisa  -->  pyvisa with forced backend socketscpi  -->  socketscpi using python's SOCKET)?
    I know that this might be a red herring at this point, but when I said earlier that I thought integrating socketscpi with PyVISA, I was confusing socketscpi with another library, so my statement that it would be hard was wrong. It does not look super hard, but would require looking at how PyVISA-py registers itself with PyVISA, and doing the same with a class that wraps socketscpi.

    Found what was making 'pyvisa-py' 3 times slower than 'socketscpi' by looking side by side at the opening socket parameters in 'pyvisa-py' vs 'socketscpi'.  It was the default value of the TCP_NODELAY.  The download becomes fast if I add this extra line
    Code: [Select]
                self._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)
    at the end of the '_connect()' definition, inside the file lib/python3.10/site-packages/pyvisa_py/tcpip.py.  It's the line number five hundred and something in the modules installed with pip, probably would have to be inserted after line 840 if the same change were to be added in the github source file:
    https://github.com/pyvisa/pyvisa-py/blob/main/pyvisa_py/tcpip.py#L840
    Congratulations on finding this! It makes sense that it's a socket option that has such a performance impact. It suggests me that there's something sub-optimal in the Rigol scope about assembling the data into TCP packets, but this is a good workaround.

    Any idea how to set the nodelay parameter using python-ivi?
    It took a bit of digging, but I think this should work:
    dso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(...)
     
    The following users thanked this post: RoGeorge

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #29 on: November 19, 2022, 12:04:57 am »
    Any idea how to set the nodelay parameter using python-ivi?
    It took a bit of digging, but I think this should work:
    dso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(...)

    Wow, thanks, I've tried today a thousand combinations and couldn't find any that works.  Now it gets all the 24 million samples in 22 seconds, connected like this:
    Code: [Select]
    import ivi

    # 24Msa in 22 seconds
    ivi.set_prefer_pyvisa()

    dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::5555::SOCKET', id_query=False, reset=False)

    from pyvisa import constants
    dso._interface.instrument.visalib.sessions[dso._interface.instrument.session]._set_tcpip_nodelay(constants.VI_ATTR_TCPIP_NODELAY, True)

    # 24Msa in 16 minutes
    # dso = ivi.rigol.rigolDS1104Z(resource='TCPIP0::192.168.1.3::INSTR', id_query=False, reset=False)




    Don't celebrate yet, (re)found one more issue!   ;D

    - A (telnet) data transfer will drop if inside any data packets it receives a 0x00.  This is some RFC spec (for text Telnet IIRC).
    - The ADC samples are coming as bytes, which means the incoming data packets will be truncated if one of the bytes is zero, because in the Telnet RFC 0x00 is considered a terminator, and will drop any bytes coming after a 0x00.
    - This issue is only observed when the input signal in the ADC is less than 5 divisions on the screen (ADC outputs 0x7f for zero volts, and 0x00 for any input voltage that is -5*volts/div or under)
    - Both the pyvisa->pyvisa-py->socket and the socketscpi are affected

    Don't know how to overcome this, or if it is possible at all to open some other kind of socket connection, one that doesn't consider 0x00 as a terminator.  :-//




    From when I've written my own SCPI scripts, I had to modify the Python's telnet module such that it won't end a data transfer at the first 0x00 in a data stream.  Since the 0x00 as a terminator is part of the RFC854 spec, this modification can not be merged upstream, so I've just made a local copy of the telnet.py (module) file, and modified the local copy to disregard 0x00 as terminator (commented out those two "if c==" lines:  https://github.com/RoGeorge/DS1054Z_screen_capture/blob/master/telnetlib_receive_all.py
    Code: [Select]
            try:
                while self.rawq:
                    c = self.rawq_getchar()
                    if not self.iacseq:
                        #if c == theNULL:
                        #    continue
                        #if c == "\021":
                        #    continue
                        if c != IAC:
                            buf[self.sb] = buf[self.sb] + c
                            continue
                        else:
                            self.iacseq += c
                    elif len(self.iacseq) == 1:

    Not elegant, but it worked.  The plan was to remove the telnetlib.py module later, and use a SOCKET connection instead of a Telnet one, though never implemented that/




    TL;DR
    So far PythonIVI using SOCKET and NO_DELAY is the fastest one, but it only works if the oscilloscope trace is nicely contained inside the screen.

    Do you happen to know any SOCKET parameter, so to transfer a specified number of bytes until all bytes were transferred (or timeout has occurred), but for binary, such that 0x00 is not considered a data terminator?
    « Last Edit: November 19, 2022, 07:30:15 am by RoGeorge »
     

    Offline lundmar

    • Frequent Contributor
    • **
    • Posts: 441
    • Country: dk
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #30 on: November 19, 2022, 02:41:58 am »
    RoGeorge,

    FYI - liblxi also supports connecting to instruments using raw sockets. Simply use 'RAW' instead of 'VXI11' in the lxi_connect() call.

    RAW is faster because it does not suffer the overhead of the quite heavy VXI11 (SUN OneRPC) protocol.

    Soon liblxi will also support HiSLIP which basically brings the same speed as RAW but with the benefit of some instrument relevant protocol features.

    I plan to release the HiSLIP feature start next year.
    https://lxi-tools.github.io - Open source LXI tools
    https://tio.github.io - A simple serial device I/O tool
     
    The following users thanked this post: RoGeorge

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #31 on: November 19, 2022, 09:34:37 am »
    Tried liblxi with TCP RAW, and I don't know how to handle the transfers.

    - it puts consecutive lxi_send() strings in the same data packet, which the oscilloscope won't understand.  I don't know how to flush the transmit buffer after each lxi_send(), so as a workaround I'm adding an *OPC? at the end of each command.

    - even so, when the oscilloscope sends the first 250 000 data bytes, the lxi_receive() returns only the payload of the first TCP packet, which is 1460 bytes, while the oscilloscope continues to send more packets.  I was expecting for the lxi_receive() to concatenate the payload of all the incoming TCP packets, and to stop only when the whole 250000 requested bytes are received (or if a timeout occurs), but lxi_receive() returns after the first TCP packet.  Not sure if this is a bug or it's the NO_BLOCKING setting of a TCP RAW connection.

    I don't know how to send TCP_BLOCKING (or alike) to the socket, or if such a feature exists in a liblxi TCP RAW connection.

    Code: [Select]
    // liblxi git at https://github.com/lxi-tools/liblxi
    // to install it from the ubuntu repository (lxilib-dev is in most of the Linux distros repos and in FreeBSD):
    //      sudo apt install liblxi-dev

    // headers in   /usr/include/lxi.h
    // .so lib in   /usr/lib/x86_64-linux-gnu/liblxi.so
    // display all installed files with 'dpkg -L liblxi-dev' or
    //      sudo apt install apt-file
    //      sudo apt-file update
    //      (sudo???) apt-file list liblxi-dev
    // no need to append LIBRRY_PATH or LD_LIBRARY_PATH

    // man lxi_<TAB><TAB>
    //      lxi_connect  lxi_discover  lxi_init  lxi_send  lxi_disconnect  lxi_discover_if  lxi_receive

    // compile and run this 'demo.c' from a Linux terminal (!!! -l options _must_ be positioned after the .c sources, not before)
    //      cd /home/muuu/wd8TB/2019/Z/_hobby/__Pnnn/221103___VNA_DG4102_DS1054Z_python/sw/C
    //      gcc -Werror -Wfatal-errors ./dl_24M_ADC_samples_ds1054z.c -llxi -ltirpc && time ./a.out

    #include <stdio.h>
    #include <stdlib.h>
    #include <string.h>
    #include <time.h>
    #include <lxi.h>

    int main()
    {
        printf("liblxi w DS1054Z using TCP mode RAW");
        // 12 bytes preamble + max of 24 mil bytes of ADC samples + 1 byte for the end of string terminator
        #define CHARS_BUFFER_MAX    24000000L // in specs max Rx in a single chunk is 250k chars for Rigol DS1054Z

        char *rx24MB = NULL;                                    // declare a pointer to char, and initialize it with NULL
        rx24MB = malloc(CHARS_BUFFER_MAX * sizeof *rx24MB);     // allocated memory on the heap for rx24MB[CHARS_BUFFER_MAX]
        if (!rx24MB) {                                          // !!! always check the memory was allocated successfully !!!
            fputs ("ERROR: allocation memory failed for the 'rx24MB' buffer, exiting.", stderr);
            exit (EXIT_FAILURE);
        }

        int device, length, timeout = 2000;     // timeout is in ms
        char response[1000];    // allocated on the stack, segmentation fault if too large, Linux max stack is 8MiB

        //char *command = "*IDN?";   // with or without \n, the osc responded in a VXI11 connection
        char *command = "*IDN?\n";
       
        lxi_init(); // Initialize LXI library
        //device = lxi_connect("192.168.1.3", 0, "inst0", timeout, VXI11);        // instument responded to either *IDN? or *IDN?\n
        // device = lxi_connect("192.168.1.3", 0, NULL, timeout, VXI11);           // Connect to LXI device
        device = lxi_connect("192.168.1.3", 5555, "inst0", timeout, RAW);       // must end in \n, or else *IDN? timeout as incomplete
        // device = lxi_connect("192.168.1.3", 5555, "TCP", timeout, RAW);       // *IDN? Timeout err for timeout 2000
        // device = lxi_connect("192.168.1.3", 0, "inst0", timeout, RAW);       // NOPE
        // device = lxi_connect("192.168.1.3", 5555, "inst0", timeout, HISLIP);       // *IDN? garbled chars for timeout 2000
       
        lxi_send(device, command, strlen(command), timeout);                // Send SCPI command ("*IDN?")
        lxi_receive(device, response, sizeof(response), timeout);           // Rx until "\n", or sizeoff(response) chars, or timeout

        printf("%s\n", response);
        // exit if no DS1054Z
    // exit(0);


        // enable only ch1
        command = ":CHAN1:DISP 1;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);           // Rx until "\n"?, or sizeoff(response) chars, or timeout

        command = ":CHAN2:DISP 0;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

        command = ":CHAN3:DISP 0;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

        command = ":CHAN4:DISP 0;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

    // ??? here to insert ":RUN""
        // set acquisition mode to max 24_000_000 points memory depth
        // "auto", 12K, 120K, 1.2M, 12M, 24M"
        long mdep = 24000000L;

        command = ":RUN;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

        // sprintf(command, ":ACQ:MDEP %li", mdep);
        // sprintf(command, ":ACQ:MDEP %li%s", mdep, "\0");
        command = ":ACQ:MDEP 24000000;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

        // wait for triggered
        do {
            command = ":TRIGger:STATus?\n";
            lxi_send(device, command, strlen(command), timeout);
            lxi_receive(device, response, sizeof(response), timeout);
        } while (response[0] != 'T');

        // then wait for ADC to aquire enough data
        clock_t stop_time;
        printf("wait %ld\n", clock());
        stop_time = clock() + 30000;             //30 seconds
        printf("beep %ld\n", stop_time);
        while (clock() < stop_time) ;
        printf("go %ld\n", clock());

        // go to stop mode and prepare to dl 24MSa
        command = ":STOP;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

        command = ":WAV:SOUR CHAN1;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

        command = ":WAV:MODE RAW;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

        command = ":WAV:FORM BYTE;*OPC?\n";
        lxi_send(device, command, strlen(command), timeout);
        lxi_receive(device, response, sizeof(response), timeout);

    // char c2[80] = {0};
    char c2[80];
    // loop to dl all samples in chunks of (max) 250_000 samples
        // set start-stop indexes for next chunk
        long rx_len, chunk_len;
        // long chunk_start, chunk_stop, chunk_size =  125000L;    //dl 24Msa/134s
         long chunk_start, chunk_stop, chunk_size =  250000L;    //dl 24Msa/86s  max from specs
        // long chunk_start, chunk_stop, chunk_size =  750000L;    //dl 24Msa/65s
        // long chunk_start, chunk_stop, chunk_size = 1000000L;    //dl 24Msa/56s  preferred ???
        // long chunk_start, chunk_stop, chunk_size = 1175000L;    //dl 24Msa/54s
        long samples_to_dl = 24000000L;

        if (samples_to_dl > mdep) {
            samples_to_dl = mdep;
        }
        for(chunk_start = 1; chunk_start < samples_to_dl; chunk_start += chunk_size) {
            sprintf(c2, ":WAV:STAR %li;*OPC?\n", chunk_start);
            puts(c2);
            rx_len = lxi_send(device, c2, strlen(c2), timeout);
            printf("ack_len=%ld\n", rx_len);
            lxi_receive(device, response, sizeof(response), timeout);

            //at the last chunk, chunk_len might be shorter than chunk_size
            if (chunk_start + chunk_size > samples_to_dl) {
                chunk_len = samples_to_dl - chunk_start + 1;
                chunk_stop = samples_to_dl;
            } else {
                chunk_len = chunk_size;
                chunk_stop = chunk_start + chunk_size - 1;
            }

            sprintf(c2, ":WAV:STOP %li;*OPC?\n", chunk_stop);
            puts(c2);
            rx_len = lxi_send(device, c2, strlen(c2), timeout);
            printf("ack_len=%ld\n", rx_len);
            lxi_receive(device, response, sizeof(response), timeout);

            command = ":WAV:DATA?\n";
            puts(command);
            rx_len = lxi_send(device, command, strlen(command), timeout);
            printf("ack_len=%ld\n", rx_len);

            rx_len = lxi_receive(device, (char *)(&rx24MB[chunk_start-1]), chunk_len, timeout);
            printf("rx_data_bytes=%ld\n", rx_len);

            int err = 0;
            if(rx_len != chunk_len) {
                printf("ERROR - received too few bytes: ");
                err = 1;
            }
            printf("chunk_len=%li rx_len=%li\n", chunk_len, rx_len);
            printf("\n");
            if(err) exit(err);
        }

    // Disconnect
    lxi_disconnect(device);
    }



    will output is this:
    Code: [Select]
    liblxi w DS1054Z using TCP mode RAWRIGOL TECHNOLOGIES,DS1104Z,DS1ZA164658712,00.04.05.SP2

    wait 15947
    beep 45970
    go 45970
    :WAV:STAR 1;*OPC?

    ack_len=18
    :WAV:STOP 250000;*OPC?

    ack_len=23
    :WAV:DATA?

    ack_len=11
    rx_data_bytes=1460
    ERROR - received too few bytes: chunk_len=250000 rx_len=1460
    « Last Edit: November 19, 2022, 09:37:13 am by RoGeorge »
     

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #32 on: November 20, 2022, 12:17:00 pm »
    To learn what SOCKET parameters are optimal for a connection with the instrument, I had to read about TCP sockets, and realized that for LXI instruments it is trivial to send SCPI commands (and read the response) directly from Python, using its included 'socket' module.  ;D

    Code: [Select]
    import socket
    socket.setdefaulttimeout(5000)

    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
        s.connect(('192.168.1.3', 5555))
        s.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)

        s.sendall(b'*IDN?\n')
        ans = s.recv(1024)

    print(ans)
    Will print
    Quote
    b'RIGOL TECHNOLOGIES,DS1104Z,DS1ZA123456789,00.04.05.SP2\n'

    Where:
        #192.168.1.3    # is the fixed IP of the instrument (a Rigol DS1054Z oscilloscope)
        # 5555              # PORT number (fixed by manufacturer) 5555 for Rigol, 5025 for Agilent, etc.
        # 5000              # the timeout, in ms
        # 1024              # the size of the receive buffer

    I only need to connect by LAN, using fixed IP, don't need any VXI11 or PyVISA or Telnet or any other dependency.  :-//
    « Last Edit: November 20, 2022, 12:31:02 pm by RoGeorge »
     

    Offline switchabl

    • Frequent Contributor
    • **
    • Posts: 445
    • Country: de
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #33 on: November 20, 2022, 01:37:12 pm »
    One thing to be aware of when using plain TCP sockets is that TCP is a stream-oriented protocol. Oh, of course it operates on packets at the implementation level. But it tries to hide all of that away and provide you with a pipe where you can put in bytes at one end (send) and take some bytes out again at the other end (recv). There is no concept of a "message" and one send on the scope might not correspond to a single recv on your side.

    In other words, your call to recv is allowed to return
    Code: [Select]
    b'RIGOL TECHNOLOGIES,DS1'
    or even just
    Code: [Select]
    b'R'
    The result will depend on the buffering strategies (and socket options) on either end as well as things like the underlying network packet size. So your example might not work on a different PC and it might not even work the next time you try it. You really need to keep calling recv until you have the expected number of bytes (or the termination character or hit a timeout).

    In order to minimize latency, the other socket option you might want to try is TCP_QUICKACK (in addition to or instead of TCP_NODELAY).
     
    The following users thanked this post: RoGeorge

    Offline lundmar

    • Frequent Contributor
    • **
    • Posts: 441
    • Country: dk
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #34 on: November 20, 2022, 06:24:51 pm »
    As it has been hinted, TCP is a stream oriented protocol so talking packets etc. does not make much sense.

    When you use RAW mode with liblxi the interface behaves a bit more low level and you basically have to do two things when sending commands and potentially receiving large amounts of response data.

    1. Send command: Append newline ('\n') to command string so that receiving instrument knows a command has been received and proceeds to process it
    2. Receive response: Keep calling lxi_receive() until it returns error (timeout) which will be the only indication that the instrument has no more response data to send.

    That is the downside of using TCP/RAW - one must rely on and wait for timeout because there is no protocol telling the client how much data to expect from the server.

    This is exactly what protocols like VXI11 and HiSLIP takes care of. VXI11 does it in a bad way because of its heavy protocol overhead but HiSLIP solves that.

    Now, I could move some of that complexity from the example to the liblxi implementation but I decided against it to give more control to the user so they can decide on buffering strategy etc.

    I've added some example code here which demonstrates how to send a SCPI command to request screenshot image data from an instrument and receive all the image data:

    https://github.com/lxi-tools/liblxi/blob/master/test/receive-image-data.c
    https://lxi-tools.github.io - Open source LXI tools
    https://tio.github.io - A simple serial device I/O tool
     
    The following users thanked this post: RoGeorge

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #35 on: November 21, 2022, 10:14:54 am »
    In order to minimize latency, the other socket option you might want to try is TCP_QUICKACK (in addition to or instead of TCP_NODELAY).

    Thanks for the hint.  Searched today about TCP_QUICKACK, and it seems to be Linux specific.  Other TCP stacks don't have the TCP_QUICKACK setting (e.g. the BSD one in FreeBSD or MacOS).
    Quote
           TCP_QUICKACK (since Linux 2.4.4)
                  Enable quickack mode if set or disable quickack mode if
                  cleared.  In quickack mode, acks are sent immediately,
                  rather than delayed if needed in accordance to normal TCP
                  operation.  This flag is not permanent, it only enables a
                  switch to or from quickack mode.  Subsequent operation of
                  the TCP protocol will once again enter/leave quickack mode
                  depending on internal protocol processing and factors such
                  as delayed ack timeouts occurring and data transfer.  This
                  option should not be used in code intended to be portable.
    Source:  https://man7.org/linux/man-pages/man7/tcp.7.html



    I've added some example code here which demonstrates how to send a SCPI command to request screenshot image data from an instrument and receive all the image data:

    https://github.com/lxi-tools/liblxi/blob/master/test/receive-image-data.c

    You are too kind, thank you.  I understand now the need for '\n' with lxilib in RAW mode, makes sense.  Also it clarifies my question about consolidating consecutive SCPI commands before sending them:
    Tried liblxi with TCP RAW, and I don't know how to handle the transfers.

    - it puts consecutive lxi_send() strings in the same data packet, which the oscilloscope won't understand.  I don't know how to flush the transmit buffer after each lxi_send(), so as a workaround I'm adding an *OPC? at the end of each command.
    'liblxi' works just fine, no need to flush.  The instrument will understand multiple commands from a single TCP packet, just that I was sending a wrong SCPI command, a bug in my code, sorry.
    « Last Edit: November 21, 2022, 10:20:26 am by RoGeorge »
     
    The following users thanked this post: lundmar

    Offline alm

    • Super Contributor
    • ***
    • Posts: 2903
    • Country: 00
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #36 on: November 21, 2022, 05:57:25 pm »
    - A (telnet) data transfer will drop if inside any data packets it receives a 0x00.  This is some RFC spec (for text Telnet IIRC).
    - The ADC samples are coming as bytes, which means the incoming data packets will be truncated if one of the bytes is zero, because in the Telnet RFC 0x00 is considered a terminator, and will drop any bytes coming after a 0x00.
    - This issue is only observed when the input signal in the ADC is less than 5 divisions on the screen (ADC outputs 0x7f for zero volts, and 0x00 for any input voltage that is -5*volts/div or under)
    - Both the pyvisa->pyvisa-py->socket and the socketscpi are affected
    I'm not sure why this is happening. As far as I know these libraries should use plain TCP sockets, and not implement the Telnet protocol. I would be careful with the word "raw TCP sockets", because raw sockets is generally understood to mean something else in network programming.

    I just tested it with PyVISA, and it works as expected for me:
    I used this as a mock server returning data containing null characters: echo '\0ff\0' | nc -vl -p 1052

    And then succesfully retrieved this data:
    Code: [Select]
    >>> import pyvisa
    >>> rm = pyvisa.ResourceManager()
    >>> res = rm.open_resource('TCPIP0::127.0.0.1::1052::SOCKET')
    >>> res.write_termination = '\n'
    >>> res.read_termination = '\n'
    >>> data = res.read()
    >>> data
    '\x00ff\x00'

    You don't happen to have the read termination set to \00, have you? Because then I could understand this happening.
     
    The following users thanked this post: RoGeorge

    Online RoGeorgeTopic starter

    • Super Contributor
    • ***
    • Posts: 6675
    • Country: ro
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #37 on: November 22, 2022, 06:00:29 pm »
    Might have been something I was doing wrong.  Can not reproduce that error any longer.  :-//

    Offline MiDi

    • Frequent Contributor
    • **
    • Posts: 615
    • Country: ua
    Re: Download speed from Rigol DS1054Z or similar oscilloscope to a PC
    « Reply #38 on: March 11, 2023, 04:29:14 pm »
    Had similar problems:
    On one PC the readout of 24MSamples took more than 40s and on another ~22s (Both Win10 latest updates, no TCP mods, but different python & module versions).
    Turning Nagle off in Win did not help - as was expected as PyVisa turns it off by default for the connection itself.
    Turning delayed ACK off in Win did improve the readout times a bit (~36s), but still slow.

    As it turned out changing the code did the trick to get ~22s on both PCs:

    instead single commands:
    Code: [Select]
    instrument.write(f":WAV:STAR {interval[0]}")		# start index of memory
    instrument.write(f":WAV:STOP {interval[1]}") # stop index of memory
    data += (instrument.query_binary_values(f":WAV:DATA?", datatype='B')) # get the data, B = unsigned char


    all commands at once did it:
    Code: [Select]
    data += (instrument.query_binary_values(f":WAV:STAR {interval[0]}\n:WAV:STOP {interval[1]}\n:WAV:DATA?", datatype='B')) # get the data, B = unsigned char


    For reference the full source code used:
    Code: [Select]
    #!/usr/bin/env python
    # -*- coding: utf-8 -*-
    # developed & tested with Python 3.6, pyvisa 1.11.3

    # Rigol DS1000Z data acquisition

    #from ctypes.wintypes import FLOAT
    #from unicodedata import decimal
    import pyvisa as visa # needs a backend e.g. NI-VISA or PyVISA-py
    import sys
    from time import time, sleep
    from matplotlib.ticker import EngFormatter
    from decimal import *

    def ef(value, places=None):
        return EngFormatter(sep="", places=places).format_eng(float(value)) #round(value, 4) - need to round in exp form

    def crange(start,end,step):
        i = start
        while i < end-step+1:
            yield i, i+step-1
            i += step
        yield i, end

    def wait_ready(instrument):
    #instrument.write("*OPC")
    instrument.write("*WAI")
    ready = instrument.query("*OPC?").strip()
    #print(ready)
    while ready != "1": # never occured, needed?
    ready = instrument.query("*OPC?").strip()
    print(f"\n-------------------not ready: {ready}-----------------------")
    #pass

    def is_error(instrument):
    wait_ready(instrument)
    status = instrument.query("*ESR?").strip()
    if status not in ('0', '1'):
    wait_ready(instrument)
    return instrument.query(":SYST:ERR?").strip() #:SYSTem:ERRor[:NEXT]?
    else:
    return False

    # connect to scope
    # initialize scope
    ##*RST                     p. 96 - Restore the instrument to the default state
    ##*WAI                     p. 97 - Wait for the operation to finish (maybe not helpful as it is only for parallel execution of commands)
    ##:ACQ:MDEP?               p. 21 - Set or query the memory depth of the oscilloscope (namely the number of waveform points that can be stored in a single trigger sample)
    ##:ACQ:TYPE?               p. 22 - Set or query the acquisition mode of the oscilloscope
    ##:ACQ:SRAT?               p. 23 - Query the current sample rate. The default unit is Sa/s.

    #def connect(timeout = 10000, chunk_size = 20*1024):
    def connect():
    try:
    # Get the USB device, e.g. 'USB0::0x1AB1::0x0588::DS1ED141904883'
    rm = visa.ResourceManager() # "@py" for PyVISA-py
    instruments = rm.list_resources() # rm.list_resources('USB?*')
    print(instruments)
    usb = list(filter(lambda x: 'USB' in x, instruments)) # TODO filter for ::DS1Z
    print(usb)
    #if len(usb) != 1:
    # print(f"No or multiple instruments found: {instruments}")
    # sys.exit(-1)
    #from pyvisa.highlevel import ascii, single, double
    #instrument = rm.open_resource(usb[0])

    # instrument = rm.open_resource(f"TCPIP0::192.168.1.26::5555::SOCKET", write_termination = '\n', read_termination = '\n', timeout = timeout, chunk_size = chunk_size)
    instrument = rm.open_resource(f"TCPIP0::192.168.1.26::5555::SOCKET", write_termination = '\n', read_termination = '\n')

    #instrument = rm.open_resource("TCPIP::192.168.1.26::INSTR", write_termination = '\n', read_termination = '\n')
    #instrument.write_termination = '\n'
    #instrument.read_termination = '\n'
    #instrument = rm.open_resource("USB0::0x1AB1::0x04CE::DS1ZA172316530::INSTR")

    #instrument.encoding='latin1' # hack - better do ask raw
    #rm.timeout = timeout
    #rm.chunk_size = chunk_size
    return instrument

    except Exception as e:
    print(f"Cannot open instrument: {str(e)}")
    sys.exit(-1)

    def setup(instrument, timebase_scale, ch1_scale, mem_depth):
    #timebase_scale = Decimal("50") # timebase scale in s
    #ch1_scale = Decimal("1e-3") # set channel scale in units (V, A, W)
    #mem_depth = "24000000" # set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000
    timebase_scale = Decimal(timebase_scale) # timebase scale in s
    ch1_scale = Decimal(ch1_scale) # set channel scale in units (V, A, W)
    mem_depth = mem_depth # set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000

    instrument.write(f"*CLS") # Clear event registers and error queue
    wait_ready(instrument)

    status = instrument.query(":TRIG:STAT?").strip() # Stop instrument: defined state
    while status != "STOP":
    instrument.write(":STOP")
    wait_ready(instrument)
    status = instrument.query(":TRIG:STAT?").strip()

    set_timebase_scale = Decimal(instrument.query(":TIM:SCAL?").strip()) # set timebase scale
    while set_timebase_scale != timebase_scale:
    print(f"set_timebase_scale: {set_timebase_scale}, timebase_scale: {timebase_scale}")
    instrument.write(f":TIM:SCAL {timebase_scale}")
    wait_ready(instrument)
    set_timebase_scale = Decimal(instrument.query(":TIM:SCAL?").strip())
    print(f"Timebase scale: {ef(float(set_timebase_scale))}s/div")

    set_ch1_scale = Decimal(instrument.query(":CHAN1:SCAL?").strip()) # query channel scale
    while set_ch1_scale != ch1_scale:
    print(f"set_ch1_scale: {set_ch1_scale}, ch1_scale: {ch1_scale}")
    instrument.write(f":CHAN1:SCAL {ch1_scale}")
    wait_ready(instrument)
    set_ch1_scale = Decimal(instrument.query(":CHAN1:SCAL?").strip())
    print(f"Ch1 scale: {ef(float(set_ch1_scale))}V/div")

    set_mem_depth = instrument.query(":ACQ:MDEP?") # memory depth = bytes to read
    if set_mem_depth != mem_depth:
    status = instrument.query(":TRIG:STAT?").strip() # Start instrument: needed for setting MDEP
    while status == ("STOP"):
    print(f"status: {status}")
    instrument.write(":RUN")
    wait_ready(instrument)
    status = instrument.query(":TRIG:STAT?").strip()

    while set_mem_depth != mem_depth:
    print(f"set_mem_depth: {set_mem_depth}, mem_depth: {mem_depth}")
    instrument.write(f":ACQ:MDEP {mem_depth}")
    wait_ready(instrument)
    set_mem_depth = instrument.query(":ACQ:MDEP?")
    print(f"Memory depth: {ef(float(set_mem_depth))}pts")

    trig_pos = set_timebase_scale * 6 # horizontal position of trigger on left edge (no pre-trigger)
    set_trig_pos = Decimal(instrument.query(":TIM:OFFS?").strip())
    while set_trig_pos != trig_pos:
    print(f"set_trig_pos: {set_trig_pos}, trig_pos: {trig_pos}")
    instrument.write(f":TIM:OFFS {trig_pos}")
    wait_ready(instrument)
    set_trig_pos = Decimal(instrument.query(":TIM:OFFS?").strip())
    print(f"Trigger position: {ef(float(set_trig_pos))}s")

    # read data (p. 241)
    #   :STOP - better use single and wait until finished (if that is possible)
    #   :WAV:FORM WORD - are there values beyond 255? Else BYTE would be sufficient
    ##S1.  :STOP               Set the instrument to STOP state (you can only read the waveform data in the internal memory when the oscilloscope is in STOP state)
    ##S2.  :WAV:SOUR CHAN1     Set the channel source to CH1
    ##S3.  :WAV:MODE RAW       Set the waveform reading mode to RAW
    ##S4.  :WAV:FORM WORD      Set the return format of the waveform data to WORD
    ##Perform the first reading operation
    ##S5.  :WAV:STAR 1         Set the start point of the first reading operation to the first waveform point
    ##S6.  :WAV:STOP 125000    Set the stop point of the first reading operation to the 125000th waveform point
    ##S7.  :WAV:DATA?          Read the data from the first waveform point to the 125000th waveform point
    ##Perform the second reading operation
    ##S8.  :WAV:STAR 125001    Set the start point of the second reading operation to the 125001th waveform point
    ##S9.  :WAV:STOP 250000    Set the stop point of the second reading operation to the 250000th waveform point
    ##S10. :WAV:DATA?          Read the data from the 125001th waveform point to the 250000th waveform point
    ##Perform the third reading operation
    ##S11. :WAV:STAR 250001    Set the start point of the third reading operation to the 250001th waveform point
    ##S12. :WAV:STOP 300000    Set the stop point of the third reading operation to the 300000th waveform point (the last point)
    ##S13. :WAV:DATA?          Read the data from the 250001th waveform point to the 300000th waveform point (the last point)

    #   improved read data
    ##S1.  :SING               p. 19 - Set the oscilloscope to the single trigger mode
    ##S2.  :TFOR               p. 19 - Generate a trigger signal forcefully. This command is only applicable to the normal and single trigger modes (see the :TRIGger:SWEep command) and is equivalent to pressing the FORCE key in the trigger control area on the front panel.
    ##S3.  wait for scope to finish
    ##S4.  :WAV:SOUR CHAN1     Set the channel source to CH1
    ##S5.  :WAV:MODE RAW       Set the waveform reading mode to RAW
    ##S6.  :WAV:FORM BYTE      p. 239 - Set the return format of the waveform data to WORD|BYTE|ASCii (WORD: higher byte is always 0) - default: BYTE

    def capture_data(instrument):
    force_trigger = True

    set_trig_swe = instrument.query(":TRIG:SWE?").strip() # query trigger sweep
    print(f"set_trig_swe: {set_trig_swe}")
    while set_trig_swe == "SING": # reset single trigger (defined state)
    instrument.write(":TRIG:SWE AUTO")
    wait_ready(instrument)
    set_trig_swe = instrument.query(":TRIG:SWE?").strip()

    while set_trig_swe != "SING": # single trigger
    instrument.write(":SING")
    wait_ready(instrument)
    set_trig_swe = instrument.query(":TRIG:SWE?").strip()
    print("measurement started")

    # TODO refactor
    # :TRIG:STAT? returns TD, WAIT, RUN, AUTO, or STOP
    # RUN (Pre-Trigger measurement) - WAIT (for trigger, may not be read out if triggered by signal) - TD (Post-Trigger) - STOP (measurement finished)

    #wait until measurement finished
    start_time = time()
    run_time = start_time
    current_time = start_time
    wait_ready(instrument) # even with wait_ready sometimes
    status = instrument.query(":TRIG:STAT?").strip() # get trigger status: TD, WAIT, RUN, AUTO or STOP
    prev_status = status
    wait_count = 0
    finished = False
    while not finished:
    if status != prev_status:
    print()
    run_time = time()
    current_time = time()
    if status in ("RUN", "TD"): # RUN/TD - measurement is runnning
    print(f"{status} {current_time - run_time:7.3f}s          ", end="\r")
    elif status == "WAIT": # WAIT - waiting for trigger
    print(f"{status} {current_time - run_time:.3f}s")
    if force_trigger:
    wait_ready(instrument)
    instrument.write(":TFOR") # force trigger (may need multiple tries to start)
    wait_count += 1
    print(f'trigger forced {wait_count}x')
    elif status == "STOP": # end on STOP (measurement finished)
    finished = True
    print(f"{status} {current_time - run_time:.3f}s")
    else: # AUTO - should not happen
    print(f"{status} {current_time - run_time:7.3f}s          ", end="\r")
    prev_status = status
    status = instrument.query(":TRIG:STAT?").strip() # get trigger status: TD, WAIT, RUN, AUTO or STOP
    #sleep(0.01)
    current_time = time()
    print(f"measurement finished in {current_time - start_time:.3f}s")

    def read_data(instrument, ch: int = 1):
    mem_depth = int(instrument.query(":ACQ:MDEP?")) # memory depth

    # :WAV needs to be en bloc, at least between STAR & STOP
    instrument.write(f":WAV:SOUR CHAN{int(ch)}") # data from channel 1-4
    wait_ready(instrument)
    instrument.write(":WAV:MODE RAW") # data from internal memory
    wait_ready(instrument)
    instrument.write(":WAV:FORM BYTE") # data as byte
    data = []
    start_time = time()
    for interval in crange(1, mem_depth, 250_000): # 250_000 is max chunk size for bytes
    #instrument.write(f":WAV:STAR {interval[0]}") # start index of memory
    #instrument.write(f":WAV:STOP {interval[1]}") # stop index of memory
    #data += (instrument.query_binary_values(f":WAV:DATA?", datatype='B')) # get the data, B = unsigned char
    data += (instrument.query_binary_values(f":WAV:STAR {interval[0]}\n:WAV:STOP {interval[1]}\n:WAV:DATA?", datatype='B')) # get the data, B = unsigned char
    print(f"{len(data)/mem_depth:3.0%} memory read in {time() - start_time:.3f}s ({len(data)}/{mem_depth}) reading: {interval[0]}:{interval[1]}", end="\r")
    print()
    # debug
    print(mem_depth)
    sps = float(instrument.query(":ACQ:SRAT?").strip()) # query the current sample rate in Sa/s
    print(f"SPS: {ef(sps)}")
    timebase_scale = float(instrument.query(":TIM:SCAL?").strip()) # query timebase scale
    print(f"Timebase scale: {ef(timebase_scale)}s/div")
    # debug end
    # debug
    #instrument.write(":WAV:FORM ASCii") # data as ASCII
    #ascii_data = instrument.query(":WAV:DATA?") # get the data
    #print("ASCII:")
    #print(ascii_data[11:1000])
    #print("ASCII END")
    # debug end

    return data

    # scale data
    # formula for converting in V (p. 242): scaled value = (value - YORigin - YREFerence) x YINCrement
    ##:WAV:YINC?  # p. 244, return in scientific notation, see details
    ##:WAV:YOR?   # p. 244, return as integer, see details
    ##:WAV:YREF?  # p. 245, returns 127 always?, see details
    ##:WAV:PRE?   # p. 246, returns 10 waveform parameters separated by ",", see details

    def scale_data(instrument, data):
    wait_ready(instrument)
    mem_depth = Decimal(instrument.query(":ACQ:MDEP?").strip()).normalize() # memory depth
    wait_ready(instrument)
    wf_parameters = instrument.query(":WAV:PRE?").strip().split(",") # get waveform parameters: <format>,<type>,<points>,<count>,<xincrement>,<xorigin>,<xreference>,<yincrement>,<yorigin>,<yreference>
    print(f"Waveform parameters: {wf_parameters}")
    while Decimal(wf_parameters[2]).normalize() not in (mem_depth, mem_depth/2, mem_depth/3, mem_depth/4):
    wait_ready(instrument)
    wf_parameters = instrument.query(":WAV:PRE?").strip().split(",")
    print(f"Waveform parameters: {wf_parameters}")
    yincrement = float(wf_parameters[7])
    yorigin = float(wf_parameters[8])
    yreference = float(wf_parameters[9])
    scaled_data = []
    for byte in data:
    scaled_data.append((float(byte) - yorigin - yreference) * yincrement)
    return scaled_data

    def save_data():
    pass

    def screenshot(instrument):
    sleep(2) # wait until all messages are gone (e.g. can operate now)
    start_time = time()
    screenshot = instrument.query_binary_values(":DISP:DATA? ON,OFF,PNG", datatype='B') #  ON,OFF,PNG - BMP is a bit faster
    stop_time = time()
    print(f"Screenshot took {stop_time - start_time:.3f}s")
    return screenshot


    def view_image(image):
    import matplotlib.pyplot as plt
    import matplotlib.image as mpimg
    img = mpimg.imread(image)
    imgplot = plt.imshow(img)
    plt.show()

    def main():
    #global measurement_filename
    timebase_scale = "1" # timebase scale in s
    ch1_scale = "1e-3" # set channel scale in units (V, A, W)
    mem_depth= "24000000" # set memory depth in points: 12000, 120000, 1200000, 12000000, 24000000

    data_path = r"D:\Dokumente\Software-Projekte\Rigol DS1000Z data acquisition"
    data_dir = ""
    data_ext = "csv"
    data_file = "test"

    scope = connect()
    setup(scope, timebase_scale, ch1_scale, mem_depth)

    # TODO get from capture_data and read_data as return
    probe_ratio = Decimal(scope.query(":CHAN1:PROB?").strip()).normalize()
    ch1_scale_ef = ef(Decimal(ch1_scale), places=0)
    sps_ef = ef(Decimal(scope.query(":ACQ:SRAT?").strip()).normalize(), places=0)
    acq_mode = scope.query(":ACQ:TYPE?").strip()
    mem_depth_ef = ef(scope.query(":ACQ:MDEP?").strip(), places=0)
    acq_time_ef = ef(Decimal(timebase_scale)*12, places=0)

    for n in range(1, 2):
    capture_data(scope)
    data = read_data(scope)

    print()
    print(f"data length: {len(data)}")
    #print(data[:100])

    scaled_data = scale_data(scope, data)
    print(f"scaled data length: {len(scaled_data)}")
    #print(scaled_data[:100])

    data_file_tail = f"x{probe_ratio} {ch1_scale_ef}V {sps_ef}SPS {acq_mode} {mem_depth_ef}pts {acq_time_ef}s"
    data_file_full = f"{data_path}\{data_dir}\{data_file} {n:02} - {data_file_tail}"
    import csv
    with open(f"{data_file_full}.{data_ext}", "w", newline="") as f:
    writer = csv.writer(f)
    for item in scaled_data:
    writer.writerow([format(item, ".4g")]) #to reduce the file size further, limit number to significant decimal digits / resolution (8bit/10div) depending on channel range

    #import numpy as np
    ##np_sd = np.asarray(scaled_data)
    #np.savetxt("D:\Dokumente\Software-Projekte\Rigol DS1000Z data acquisition\DS1000Z_DAQ.csv", scaled_data, fmt = "%.4g") #to reduce the file size further, limit number to significant decimal digits / resolution (8bit/10div) depending on channel range

    print (f"{data_file_full}.{data_ext} written")

    image = screenshot(scope)
    #view_image(image)

    with open(f"{data_file_full}.png", "wb") as f: # write screenshot to file
    f.write(bytearray(image))
    print (f"{data_file_full}.png written")

    if __name__ == "__main__":
    main()


    For the curious, the relevant parts of the wireshark logs are attached.
    « Last Edit: March 11, 2023, 04:32:15 pm by MiDi »
     
    The following users thanked this post: alm


    Share me

    Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
    Smf