EEVblog Electronics Community Forum
Products => Test Equipment => Topic started by: ivan747 on November 30, 2014, 04:22:01 pm
-
Hi guys,
Recently I've observed a problem with waveforms. You know, one can usually store wavefroms from an oscilloscope, data loggers, curtom projects, export them from simulators, create them using custom programs or suites like Matlab... but mostly in proprietary formats, non standard CSV files, binary dumps and oddly formatted text files. That means, waveforms are rarely in the same file format, and in the case of CSV and binary, they are not presented in a standard way either.
That is a really bad problem when you try to get them into Excel, or get them into an arbitrary waveform generator, or plug them into LTspice, or some other program.
What do you guys do about this? |O
-Ivan
-
With logic analyzers occurs the same, almost all apps can export to CSV, but almost none of them can import.
Coincidence, I doubt it!
-
I have the same frustration. Sounds like a business opportunity.
Create some glue-apps to 'normalize' the data from various sources to a single format. I have seen this done in my industry of high-end imaging. The various cameras used spit out their own version of data that is only useful after some proprietary processing. A group got together to develop a suite of tools that process all camera data into a central format while normalizing the data to fit in a certain pipeline. Kinda nice to be able to pick a tool without getting stuck with that manufactures proprietary way of working.
-
Often the use of one family/brand of equipment can minimise these problems as manufacturers optimise inter-connectivity through their range.
-
Often the use of one family/brand of equipment can minimise these problems as manufacturers optimise inter-connectivity through their range.
Well... yeah. Seems like instrument manufacturers see arbitrary waveform gens as oscilloscope accessories or something... :palm:
-
I have the same frustration. Sounds like a business opportunity.
Create some glue-apps to 'normalize' the data from various sources to a single format. I have seen this done in my industry of high-end imaging. The various cameras used spit out their own version of data that is only useful after some proprietary processing. A group got together to develop a suite of tools that process all camera data into a central format while normalizing the data to fit in a certain pipeline. Kinda nice to be able to pick a tool without getting stuck with that manufactures proprietary way of working.
100%. This is a business opportunity, and more.
As you point out various other technologies have undergone the transition from proprietary to open architectures based on standards.
One of the key transition points for the telecommunications industry (and the entire IT industry including the computer hardware and software industries) was the move from circuit switching to packet switching; packet switching in turn provided the path to TCP/IP – which was fundamental to nothing less than the development and adoption of the Internet.
In the early days of packet switching each network device vendor supported one or more protocols, from Async to IBM’s Bisync and SDLC, to HDLC, and many others. The packet switch industry developed a standard called X.25. Packet switches could talk to each other via X.25 but on the edges the switches required Packet Assemblers Disassemblers (PADs) that could convert a protocol such as SDLC into X.25; on the other end a switch with another PAD converted X.25 back to SDLC (for example). Eventually the entire architecture gave way to routers and swithches managing packets with TCP/IP – but for a period of time a translation architecture was required to deal with the propensity for proprietary formats.
More recently medical images were managed in proprietary formats. The header formats that contained patient data and the pixel formats were unique to each manufacturer. A GE MRI image layout was distinct from a Siemens MRI image layout. Likewise for CT, Ultrasound, and Nuclear Medicine images across all the manufacturers including GE, Siemens, Philips, Toshiba, and others. Then in the 1990s along came the global DICOM (Digital Imaging and COmmunications in Medicine) standard. Today, every digital medical image you get from any radiologist or any other doctor at any physician’s office or at any hospital is formatted according to the DICOM standard. The images can be moved between networked devices like any dataset and they can be displayed on any PC, regardless of which medical imaging device manufacturer’s equipment generated the image. And the images can be forwarded via disk or network back to any medical imaging device that can accept data input, regardless of the manufacturer. Virtually any device on the planet sold in the last decade or so that can accept a medical image can accept a DICOM image.
The precursor to DICOM was a standard called ACR-NEMA (American College of Radiology-National Electrical Manufacturers Association). In the early days the manufactures were reluctant to support an open standard because they feared it would inhibit sales of their proprietary equipment. During the transition period each manufacturer retained their images in proprietary formats and forwarded the images to a third party device called a Medical Imaging Gateway that converted the images from the proprietary format to the ARC-NEMA format; within a "MIG" the images could be converted from the ACR-NEMA format and outputted to another manufacturer’s device using the far end device’s proprietary format. For a while the manufacturers were happy to have ACR-NEMA as a buffer that protected them from other manufacturers being able to directly manipulate their proprietary formats. Eventually the user community, the academic community, and the manufacturers realized that more images could be generated, networked, stored, retrieved, displayed, and managed overall if each device supported DICOM natively as the industry standard – all of which resulted in more equipment (including hardware and software) being sold and purchased. Today, DICOM is the worldwide standard for medical images and virtually all medical images can be generated and viewed wherever and whenever they are needed regardless of which manufacturer’s equipment is used.
There is no doubt that the Test Equipment industry could spawn larger adoption of test equipment which would benefit commercial, academic, scientific, and social interests by moving to an industry standard for managing waveforms. Someday (probably not too many years from now) engineers will say, “I remember when it was easy to get the waveform from the generator to any oscilloscope but it was a clumsy process to get the waveform from the oscilloscope to any generator (except in some cases where the scope and the generator were made by the same manufacturer) - and it was also difficult to get the full fidelity of the waveform moved between generators from two different manufacturers. Now the field and record layouts have been standardized at a baseline level."
Ease of use is generally important to productivity and adoption.
Electro Fan
-
My Tek scope can save waveforms as CSV files directly. I guess that's the exception, not the rule?
Sent from my Tablet
-
What do you guys do about this? |O
Using better scopes? ;)
Even my 11yr old WaveRunner2 LT264M can save waveform data as standard csv file (no odd formatting here), and both of my newer LeCroy scopes (WavePro 7300A and WaveRUnner 64Xi) let me export waveform data as Excel file (or one of several other formats) directly, so getting data out of the scope isn't much of a problem.
-
Note that Scope waveforms need not be ordered as the display suggests; indeed, they are typically scrambled (suggestive of the random sampling architecture?) and multivalued (because multiple points are sampled for each timestep). It probably also saves the entire buffer, which you may or may not've wanted, or didn't adjust to an appropriate size for your measurement because you only view a small fraction of the total buffer.
It would be of academic interest to take such a data series, sort by time, and apply a filter spanning two or more time samples, producing a best fit line or curve (ideally a sinc curve, but I don't know if there's a statistical function handy for that), as well as error bars for your confidence interval. Additional options: cutting the entire array into one or two cycle samples, and folding them over to obtain sum (averaging) and difference (noise, interference?) signals. Or calculating the FFT, or the autocorrelation, etc.
Tim
-
Seems like it might be interesting to combine an open source instrument communication framework (e.g. python-ivi) with some sort of trace data management framework. That way you can import and export traces from flat files (CSV or proprietary) or the instruments directly. Python-ivi already contains some 'normalizing' code to get data in and out of standard python/numpy arrays when dealing with oscilloscopes, spectrum analyzers, and arbitrary waveform generators. Building an open source waveform viewer with waveform manipulation plugins (filtering, decoding, etc.) has been something I have been considering for a while, but have not had the time to dive in to.
-
Seems like it might be interesting to combine an open source instrument communication framework (e.g. python-ivi) with some sort of trace data management framework. That way you can import and export traces from flat files (CSV or proprietary) or the instruments directly. Python-ivi already contains some 'normalizing' code to get data in and out of standard python/numpy arrays when dealing with oscilloscopes, spectrum analyzers, and arbitrary waveform generators. Building an open source waveform viewer with waveform manipulation plugins (filtering, decoding, etc.) has been something I have been considering for a while, but have not had the time to dive in to.
That sounds like an open source alternative to labview, and I quite like it.
Well, now I have a project myself, I am actually collaborating in making a file format converter for waveforms:
https://www.eevblog.com/forum/testgear/wave2wave-a-waveform-format-converter-alpha-preview/ (https://www.eevblog.com/forum/testgear/wave2wave-a-waveform-format-converter-alpha-preview/)
The plan is to make it as powerful as possible for that task.