EEVblog Electronics Community Forum
Products => Test Equipment => Topic started by: JonM on November 29, 2016, 04:30:52 pm
-
Hi Everyone - Back in August, while I was awaiting delivery of my new Keysight DSO-X 3024T, we discussed some of its' limitations including FFT length. I claimed that I planned to offload FFT processing when I needed longer FFTs. Here is what I have so far:
https://meekj.github.io/Rprogramming/scope_long_fft.html (https://meekj.github.io/Rprogramming/scope_long_fft.html)
https://meekj.github.io/Rprogramming/peak_fit_500MHz.html (https://meekj.github.io/Rprogramming/peak_fit_500MHz.html)
with code at:
https://github.com/meekj/oscilloscopeTools (https://github.com/meekj/oscilloscopeTools)
The examples illustrate using the scope as a RF spectrum analyzer using 500k time domain points. So far I have not figured out how to get the scope to acquire more points, but I have not tired very hard. Note that in the second example I am using a 200 MHZ scope to look at 500 MHz signals. Maybe I will win a higher bandwidth scope from Keysight this week and be able to make better measurements...
There was also a question about how fast the LAN interface is. For the 500k points in byte mode, plus the required meta-data, one data transfer cycle requires about 380 ms for already open connections.
Everything above was done in R with only a few add-on packages for the signal processing. The GitHub repository also includes a pure Perl utility to save waveform data to a file and a R program to plot data from the file. More on those later after I post some example plots.
So far testing has been done on Linux & MacOS. I'll attempt a Windows 10 test at some point.
Jon
-
I've written a similar (but simpler) long FFT in Octave for an MSOX3104A.
To get the maximum number of points from the DSO/MSO:
- You need the MEMUP license for 4Mpts, otherwise the max is 2Mpts.
- Use only one channel in each channel pair, Ch1/Ch2 and Ch3/Ch4. For example, if Ch1 is on, turn off Ch2.
- Turn off all digital channels.
- Math off.
- Normal acquisition mode.
- Single acquisition mode.
- Do the acquisition, force trigger if needed.
- When retrieving the waveform, only those points which are on the screen are returned, even in RAW mode. Change the horizontal sweep until all points are displayed (:WAVeform:POINts? reaches a maximum) before retrieving.
If you figure out a better method, especially for the last one, please post!
I get about 1.3 secs for 4000000 bytes + metadata with an already open connection.
-
Thanks Mark! That is very helpful information about getting the maximum number of data points. I had assumed that RAW mode did not require all points to be displayed, but I guess that I am not shocked. It's the sort of unexpected limitation that often pops up when you move away from the mainstream application of a device or system.
Jon
-
I'm not at all familiar with R and have not looked at your code but the 500K bytes transferred in 380ms seems really slow. The datasheet shows a 10/100Base-T and I assume you would be running at 100.
LabVIEW is not the fastest tool to be using but this clip was showing roughly the same amount of data before and after I changed the Ethernet card in one of my old LeCroy scopes.
https://youtu.be/GhVQGPra9DQ?t=236 (https://youtu.be/GhVQGPra9DQ?t=236)
I was trying some different ways to get higher transfer rates
https://www.youtube.com/watch?v=DlsBnumQaCA (https://www.youtube.com/watch?v=DlsBnumQaCA)
-
When I had ran the original tests, I was using my laptop with a dedicated line to the scope. I bought a new PC a few months ago and thought I would repeat the test with two 1G switches stuck between the PC and the scope, plus Windows 10.
Shown is transferring 10MS from the scope to the PC in 182ms. This is doing the memory copy technique I had stumbled onto. So I took a hit but not too bad. The memory copy only works with the faster data rates.
Switching to the built-in 100BT port and sending 500KS, it requires 62ms.
Are you connecting at 100Mb and sending binary data, one byte per sample, direct connection? If this is all true, I would load up Wireshark and see where the bottle neck is.
-
When I had ran the original tests, I was using my laptop with a dedicated line to the scope. I bought a new PC a few months ago and thought I would repeat the test with two 1G switches stuck between the PC and the scope, plus Windows 10.
Shown is transferring 10MS from the scope to the PC in 182ms. This is doing the memory copy technique I had stumbled onto. So I took a hit but not too bad. The memory copy only works with the faster data rates.
Switching to the built-in 100BT port and sending 500KS, it requires 62ms.
Are you connecting at 100Mb and sending binary data, one byte per sample, direct connection? If this is all true, I would load up Wireshark and see where the bottle neck is.
These smaller scopes the OP is working with have a small ARM chip running Windows CE, which is unlikely to have impressive transfer performance especially since it needs to pull the waveform data out of the acquisition memory. The windows based Lecroy scopes that store/manipulate their acquisition data in "normal" memory visible to the operating system should be much much faster at transferring the data, another advantage of having the data visible to the CPU as early as possible. But thats the architecture for an analysis platform contrasted to the realtime platforms.
-
When I had ran the original tests, I was using my laptop with a dedicated line to the scope. I bought a new PC a few months ago and thought I would repeat the test with two 1G switches stuck between the PC and the scope, plus Windows 10.
Shown is transferring 10MS from the scope to the PC in 182ms. This is doing the memory copy technique I had stumbled onto. So I took a hit but not too bad. The memory copy only works with the faster data rates.
Switching to the built-in 100BT port and sending 500KS, it requires 62ms.
Are you connecting at 100Mb and sending binary data, one byte per sample, direct connection? If this is all true, I would load up Wireshark and see where the bottle neck is.
Thanks for the comments!
Yes, I agree that the transfer rate does seem low, but don't forget about TCP slow-start (which might not be a factor here since the TCP session is kept open for multiple waveform transfers). The 520,491 byte waveforms require 309 packets and I need to do a more detailed analysis to see if slow-start might play a role in delaying the ramp up to maximum flow. Thanks to MarkL's suggestions on getting a Keysight scope to use all of it's (limited) waveform storage I will be transferring more data in the next tests as well.
I know how to tune the TCP stacks of two Linux servers separated by "long fat pipes" (1Gbps NYC to San Diego or 100 Mbps NYC to Tokyo) and fill the entire pipe with data for a bulk transfer (not necessarily a good thing to do if the pipe is shared!) but I won't be able to tune anything on the Windows CE of the Keysight scope, and will have to see if any Linux server side tuning will help.
In any case, for the current example, 380 ms is acceptable since I have a 2 second delay between each waveform so that the operator can observe the spectrum and signal averaging effect.
More to come. I captured the traffic a month ago but just started to look at it tonight.
Jon
-
The plotting in LabVIEW takes a fair amount of time. Typically I will bin the data and to a peak detect on it if I need faster update rates. On the plus side, putting the code together is fairly fast.
I added a second Ethernet board to the PC to set up dedicated connection with the scope. At least with 20Meg, it will run close to 600Mb. I've ran LabVIEW with sustained rates higher than this and suspect its the limit of the scope which is about 15 years old.
Attached showing the power spectrum using the direct link. The first is 400K. About 13ms to transfer the data and 42ms to run the FFT and plot the data. The test signal is to the right of the cursor at 1GHz.
Moving 10MBytes takes 175ms. A slight increase. Running the FFT and plotting that much data takes 1.444 seconds.
-
A bit of programming optimization, including setting mode, number of points, etc only once, has reduced the transfer time for 500k points to 150 ms. Looking at 10 ms averaged throughput from packet capture there are periods of 70 Mbps throughput, as well as much lower values. I have not tried any TCP tuning so far.
It will be sad if a 15 year old LeCroy beats the latest Keysight...
I will note that this is my third digital scope, the first was a LeCroy (probably their first model in 1984 or 1985), with a GPIB interface, of course. The second, an Agilent with RS-232 interface could require 44 minutes to get a long waveform!
Jon
-
It will be sad if a 15 year old LeCroy beats the latest Keysight...
This LeCroy is actually a PC running Windows 2000, so transfer rates we're seeing are between two PCs.
The scope hardware is nested inside the PC. I guess it is connected to the PC through PCI bus, which gives high transfer rates between the scope hardware and PC. But still much slower than real-time sampling, which is 20 GS/s for this scope.
Keysight MSOX3104A is not quite in the same class as LeCroy. But Keysight's data transfer speeds do beat lesser scopes too - about 3 times faster than Rigol DS1054Z.
-
My first DSO was my Hitachi analog scope with a home made digitizer and playback system. I had a serial port to get the data into a PC. Some CGA graphics to display the data. I should make a video of that thing.
My first LeCroy was the 7200 with the 68K VME chassis. 4GHz BW but RIS.
I picked up a 7200A from IOMEGA that I decked out. That was a 486 and used SCSI to get the data into the PC. You could use GPIB as well, or both. Both very nice systems. I still have the 7200 but gave the 7200A away over the summer with enough spare parts to almost build a second scope. I had the SCSI interface working but sadly I never did any benchmarking on it. It was way faster than the GPIB bus. :-DD
I upgraded this LeCroy from W2K to XP, added some RAM and installed the SSD. Not a bad scope for it's age.
-
I tried running the exact same program on my slightly newer LeCroy 64Xi going through the two switches. This scope has also been upgraded to XP, more RAM and SSD. It had a 1000BT and no way to upgrade it. When I got this scope I was not very impressed. The mechanics are poor. After some TLC it has proven to be very reliable. It's plenty fast for 99% of my work. LeCroy was very nice and tossed me a few options for it. It's now my main scope.
Transferring 500K samples in 12ms and 58ms to run the FFT and plot.
5M samples in 78ms. Note that there is no memory move. Using this technique slows this scope down.
Short clip of the the 64Xi sending 500K samples to the PC and showing the FFT with LabVIEW. Sweep generator attached to the scope so you can see something happening. This is with the scope sitting on my LAN.
https://www.youtube.com/watch?v=YaZ3uHgxues&feature=youtu.be (https://www.youtube.com/watch?v=YaZ3uHgxues&feature=youtu.be)
-
I am getting slightly better transfer speeds in Perl, 1 Million point waveforms in byte mode are taking 260 ms, or using about 30 Mbps. Everyone is probably correct that there is some data movement bottleneck in this "small" scope and it is unlikely that TCP tuning on the PC side is going to help. I do think that Keysight could have done better. The 100 Mbps LAN/VGA interface ($400!) is implemented in a Xilinx Spartan FPGA, as far as I can tell (not much else on the board besides some SRAM which may be the video memory (VGA is useless to me)). I have a tiny $150 Xilinx Zynq (Arm + FPGA) based board that can push data out at 480 Mbps. I think that Ethernet is implemented in the FPGA on that board. So, a PCI bus is certainly not a requirement to get data out quickly. And, there is nothing fundamentally limiting about ARM (as Someone mentioned that the 3000T series may be ARM based) with 100 Mbps Ethernet.
I'm going to write a simulator that sends a similar amount of data as the scope and run it on a Raspberry Pi. If the results are interesting, I'll post 10 ms resolution plots of the network data throughput for the scope and the RPi. Maybe that will nudge Keysight to have a look...
I have not determined what model LeCroy I used to have, but it was from the same era as their 9109 AWG. I know it was not a VME based scope, but I had the scope connected to a VME system and used it until I got an array processor (four VME boards) for which I built a DMA interface for a 12 bit 5 MSa/s ADC. The goal was to do 128k (real-time) and 512k (off-line from saved waveforms) FFTs as quickly as possible (chemical compounds were flowing out of a chromatograph on the order of one second at a time). That array processor, in a $100k or so, computer system took 3540 ms to compute a 512k point power spectrum while my current desktop does it in 27 ms. Part of what I am doing is seeing how well cheap modern components could do the job. The scope is not part of that (too expensive for one thing) but it's a parallel effort for comparison.
Jon
-
The goal was to do 128k (real-time) and 512k (off-line from saved waveforms) FFTs as quickly as possible (chemical compounds were flowing out of a chromatograph on the order of one second at a time). That array processor, in a $100k or so, computer system took 3540 ms to compute a 512k point power spectrum while my current desktop does it in 27 ms.
Nice! That old 7200 I have actually has support for TOF built in. The oldest Lecroy catalog I found shows the 9000s along with the 7200. May have been the same time frame. LC or GC?
-
The goal was to do 128k (real-time) and 512k (off-line from saved waveforms) FFTs as quickly as possible (chemical compounds were flowing out of a chromatograph on the order of one second at a time). That array processor, in a $100k or so, computer system took 3540 ms to compute a 512k point power spectrum while my current desktop does it in 27 ms.
Nice! That old 7200 I have actually has support for TOF built in. The oldest Lecroy catalog I found shows the 9000s along with the 7200. May have been the same time frame. LC or GC?
I will find a photo of the LeCroy scope at some point. It was not a 7200. It was lower bandwidth, and had a "very blue" panel as I remember.
Both GC & LC. It was the "the first high resolution LCMS" and "the world's most accurate mass spectrometer", at least at the time, thanks to the long FFTs and peak fitting similar to what I posted at the start of this thread. along with a few other features.
And, on more LeCroy history, the first device I remember was a 10 step attenuator box in 1973. In 1980 a LeCroy TDC (Time to Digital Converter) with my "poor man's CAMAC - Apple II interface" was a key part of my thesis work.
-
The catalog I have is from 1990. It has a fair amount of historical data going back to 1964 when they started. The 9400 series is covered going back to 1986 where the 7200 dates back to 1989.
Can you provide any further information on the product (mfg, product name, papers, pictures). I am interested in seeing what you were working on if it is the public domain. Not meaning to go off on a tangent but curious now that you bought it up. Is what you are doing now still tied to this work?
-
The instrument was a Fourier Transform Ion Cyclotron Resonance Mass Spectrometer. The overview is: ions are created in a vacuum chamber and then transported through an intermediate vacuum chamber and finally end up in a third ultra-high vacuum chamber. That third vacuum chamber is in a 7 Tesla superconducting magnet and the ions are trapped in box with six sides that can have voltages applied.
The ions have a "cyclotron frequency" that depends on their mass and they can be accelerated to significant size orbits by applying a frequency sweep across a pair of the cell sides. After the ions are accelerated they can be detected with a differential amplifier on the other pair of side plates.
In order to get good frequency / mass measurements you need ultra-high vacuum (1e-10 torr, yes it's higher when liquids are being dumped into the first vacuum chamber) so that the ions orbit coherently for as long as possible, a uniform high magnetic field, and a way to record, store, and process (FFT) long waveform records.
We used a LeCroy 9109 AWG for the frequency sweeps, but it provided a cool capability. Suppose you did not want to accelerate all masses but just look at certain mass ranges. We could define a frequency domain shape (usually 1 for accelerate range, 0 for no signal) and do an inverse FFT to get the associated acceleration waveform.
Our device was never sold commercially, although it was considered, however the patent was licensed to at least one of the major mass spectrometer manufacturers. Here is the patent (complete with typos):
https://www.google.com/patents/US4686365] [url]https://www.google.com/patents/US4686365 (http://[url) [/url]
I note that six Agilent patents reference mine.
There were papers, book chapters, and conference presentations on the device but none of them seem to be publicly available. The patent was written early in the project but it's a reasonable qualitative description. At some point I will share more on my Web site.
Add yes, the scope was a LeCroy 9400. Even after the ADC / array processor became the production data acquisition system the 9400 was used to view the time domain signal in real-time. Sometimes you could see a beautiful decaying signal, more often you just saw noise plus buried signal and the "magic of the FFT" pulled the signals (and some radio stations) out of the noise.
I have not worked actively in this field since 1996 although it was part of the "single job" I had for 32 years (sometimes an advantage of large companies).
Thanks for the interest!
Jon
-
Are you the John Meeks then and changed to Jon? I read the patent. That's some pretty intense effort. Once you had the spectral data, did you have other processing to try and figure out what the stuff was you were analyzing? I can't imagine pushing all this data to a 500MB mag tape and trying to work with the data.
-
I have always been "Jon Meek", the patent has some typos, including "cyclothon" in the title. I think that the metal plaque I got actually has my last name mangled and some more typos.
For chromatography data we did 128k FFTs in real-time to determine which waveforms needed to be saved to one of the 689 MB ESMD disks (only 480 MB after formatting and they required two people to lift them out of the shipping box!). The post processing pass did 512k point FFTs, found the peaks, fit the peaks (using a technique similar to that noted in the first post above), and computed the accurate mass and error bar (typically 0.3 ppm) for each peak. There was often a known reference peak used to compensate for "space-charge" frequency shifts when a lot of ions were present.
Chemists usually had some idea of what elements could be present in the molecules, like there could be 10-40 C, 0-6 N, 0-5 O, 0-9 F, and "unlimited" H. So, the masses of all possible permutations are calculated and if the mass of a combination falls within the error bar it is listed. For this example only two elemental combinations fit within the error (other high resolution mass spectrometers of the era had about 5 ppm error and 18 compositions would be possible. Note that only C has integer mass, for example N is 14.0030740209 mass units. And of course there are other isotopes that can help to identify the actual composition.
-
Funny that your name would have been wrong. So many checks along the way. You would think they would at least get your plaque right.
I figured the human would come into play for decoding the data. Thanks for all the info. It's a very interesting topic. And again, sorry to derail your Ethernet thread.
-
Thanks for the opportunity to discuss a project that took up almost all of multiple years of my life. It was a good experience. Next week I will be seeing several of the upper management people who made the project possible at a holiday event. They will be happy to know that it was discussed recently. In the end, the instrument I built was donated to the the Idaho National Engineering Laboratory and replaced with a commercial version from a company that licensed our patent. Unfortunately that commercial version had a much lower field magnet which reduced its performance (a function of magnetic field squared). On top of that, a power supply malfunctioned while the commercial company was moving from one country to another and there was a many month delay before a replacement could be supplied. There had always been a concern (valid, I agree) over what would happen if "something broke" and I was the only one who could fix it, but assuming that a well known commercial supplier would be more reliable did not work out in this case.
I'll start a new thread when I have new network performance data for the Keysight 3000T series.