Keysight 34465A PyVISA 50kHz Data Acquisition

[recon9330]

Hi,

I am trying to stream data from the Keysight 34465A using a python script at 50khz (hardware maximum according to model)  and am having issues keeping the internal buffer from overflowing (Maximum of 2mb). It seems the query command "query_binary_values"  which reads buffer data and also clear the data that has been read is too slow to keep up with the speed of data being populated into the buffer. Verified this by tracking the buffer size after each query and observed the buffer size slowly increasing each time a query is commanded.

I have played with chunk sizes of the query command and have been able to stream for ~5-6 minutes before overflowing the buffer. For reference I have no issues keeping up at lower sampling rates i.e. 20khz and below. Additionally, we have similar data acquisition systems developed in labview that are able to stream @ 50khz and keep the buffer from overflowing. I am not sure if it is an issue with how PyVISA implements the high level commands.

Has anyone with this DMM or similar been able to stream @ 50khz or faster with a python script?

For reference, so far I have been able to achieve 50khz streaming using the following snippet of configuration and commands for limited duration:

'''
#Setup Config
v34465A.write(':CONFigure:VOLTage:DC %G,%s' % (10.0,'MAXimum')) #General config, range, resolution
v34465A.write(':SENSe:VOLTage:DC:ZERO:AUTO OFF')  # Disable auto-zero for DC voltage measurements
v34465A.write(':SAMPle:COUNt %d' % int(self.sample_count)) # User input sample count (i.e. 1000000000)
v34465A.write(':SAMPle:SOURce %s' % 'TIMer')  # Use timer as the sample source
v34465A.write(':SAMPle:TIMer %f' % self.sample_interval)  # Set the sample interval user input (i.e. 0.00002s for 50khz)
v34465A.write(':TRIGger:SOURce %s' % 'IMMediate') # Trigger as fast a possible

# Configure digitization
v34465A.write(':FORMat:BORDer SWAP')  # Set byte order
v34465A.write(':FORMat:DATA REAL,64')  # Set data format

#Generate Data:
v34465A.write(':INITiate:IMMediate')  # Start wait-for-trigger mode

# While loop to get data
while self.running:
                    response = v34465A.query_binary_values(':R?', datatype='d', container=list, chunk_size=20000000)  # Fetch binary data
'''

Thanks

[Dr. Frank]

I have no experience with python, as I always used compiled Turbo Pascal on old machines.
But that should not be the problem here.

I have a lot of experience with fast digitzing and fast data transfer over GPIB on the 3458A, where I used many HW - near tricks on 20 and 40 MHz PCs to transfer 16Bit and 32bit @ 100 and 50 kSa/s, i.e. 200kByte/s into DMA.
The 3458A has dedicated hardware (Gate Array?) to make such fast transfers, trigger and arming from the A/D possible, and it has appropriate fast digitization commands and binary formats (word, dword, scaling parameter) available.

The 34410/411 as well allowed such fast transfers w/o internal memory, see datasheets (!), and those binary formats are available, if I remember correctly. But I don't own a 410/411, so no real experience, if that really works.

Anyhow, I got a 34465A with the MEM and the DIG options (important!), and also tried to achieve fast data transfer over GPIB directly into PC memory.. and also found that it is not really possible.
I have to review the programs I have written, but if I remember correctly, data is first transferred to DMM memory, which fills up and then slows down the transfer. To circumvent buffering in the memory is difficult or impossible...

Then, these binary formats word and dword, including scaling, are lacking (?), and there are some commands missing to configure the DMM over the bus for fast transfer (50kSa/s), with zero delay, etc. ... data transfer of REAL data requires 8 Bytes x 50kHz = 400KHz, that is difficult, even if you use LAN interface.
If you use the benchview software, there is as well always a mis-configuration for fast measurements, and that also slows down the sampling, after a few seconds.
You always have to manually configure sampling time and delay, to make at least 50kHz sampling, but as well not directly from A/D to PC memory..

I guess the 34465A/470As are lacking kind of state machine, like their predecessors, and data crunching is probably done instead in software, by the Windows CE 5.0, which would greatly slow down data transfer.
I guess, that a lot of hardware was taken over form the 34410/411A, but exactly this hi-speed digitization was not included.
Please study the operation manuals of the 34410/411 for further hints, what's lacking the 34465A, especially on this digitization topic.

Frank

Edit: Yes, I found an AN for the '411, which advertises:  'Up to 50 k readings/s, continuous to PC', as well: 'Retrieve readings up to 270,000/s from reading storage'.
On page 16 and 17 you'll already find a lot of hints, which commands or internal hardware you'd need on the '465A, and then you can search in its crude manual if these are available, or not.

[coromonadalix]

dont forget the L4411a  meter  loll  a  34410 11a  brother minus  front panel keyboard

[bateau020]

Tried the same on a 34465A. My analysis is twofold:
1) use a lower chunk size
2) The DMM does indeed not seem to be keeping up at 50kSa/s (it might keep up when writing to file, read that somewhere, cannot find it now).

(and it may be better to also use v34465A.write("DISP OFF"), as that lowers the DMM's CPU load)

About chunk size:

When using the chunk size you gave, the communication breaks with a IO error within the minute in my setup.
When I lower the `chunk_size` (20480 is default), the communication holds out pretty well (I stopped testing after 30 minutes).
What query_binary_values() does, is send repeated VXI DEVICE_READ commands with length = `chunk_size`. However, VXI DEVICE_READ length cannot be above 65536. So 20000000 as `chunk_size` seems wrong to me for starters, although it does not complain and does work for some time.

Please note that `query_binary_values()` does not have a predetermined processing time: It returns only after the DMM's buffer is emptied. So at 50kSa/s that is rather irregular at start, and can get very long later on, as you found. But at lower sample speeds it is still very irregular.
You cannot use `data_points` (it is ignored), because you use the ieee header (so does the DMM, so don't change that). As a result, it just keeps on sending VXI DEVICE_READ commands until it decides that all is well. And that can be after a very long time: over a minute. Hence: you can get large response buffers, creating spikes in PC CPU time, thus creating irregular intervals between the end of the read and the next "R?" command, thus making the DMM's buffer load irregular and increasing the risk of data loss. If you want to do a more streaming approach to spread out the load, you might try going lower level than `query_binary_values()`, like `read_bytes()`.

About the "keeping up at 50kSa/s":

The average time between samples varies a lot, when set to 20usecs, and a lot of samples get lost. It starts to become stable at around 50usecs in my setup, and that is with almost no processing between calls to `query_binary_values()`. Check with `time.monotonic_ns()` for example. So yes, Python + this DMM over LAN is not fast enough for 50kSa/s.


Other remark: Please note that you tell the DMM to make 1M samples repeatedly. I might be wrong here, but that seems to me that you will miss out on some samples every 1M samples, when the DMM restarts the loop.


EDIT:

see https://download.tek.com/document/1KW-61357-0_DMM6500_Datalogging_Application_Note_080318.pdf, appendix A for a streaming approach. It is another device, uses TSP script and lua (instead of SCPI), but the idea is identical to what could be achieved with ":R?" and lower level communication on the KS34465A. (DMM6500 does not have ":R?" by the way)

There is also an interesting phrase in there: " You can expect about 60,000 readings/second with this script, but this speed is limited by the Ethernet
cable of the LAN connection. Higher streaming speeds are possible by using a faster data bus such as a USB connection with PyVISA."
That device has 10/100BT and USB2.0 full speed, as does the KS34465A so that limitation is likely to apply to the KS as well. (rant: why no 1000?)

To be tested.....