Rigol ds1000z WFM file reverse engineering

[Neuromodulator]

Hello,

I've been working a bit in reverse engineering the WFM file format so as to convert it to the more friendly CSV. So far the code I have looks like junk so I have no usable utility yet, but I've identified some key data fields in the file, like the time base, voltage range, active channels, and the time series. As soon as I write a program to do the WFM to CSV conversion I'll post it here. If there is interest I could post what I've found so far as well.

[bitseeker]

Sounds like a very handy utility, Neuro. Thanks for digging into it.

[BravoV]

Neuromodulator, thank you !  Subscribed.

.......  but I've identified some key data fields in the file, like the time base, voltage range, active channels, and the time series.

If its ok with you, please post the fields details you've discovered so far, tia.

[Neuromodulator]

Sure, here is what I've found so far (which I haven't double checked so may contain errors):

header =\
(
    ('az',    8, 'Model'),
    ('az',   28, 'Firmware'),
    ('i32',  88, 'Active Channels'),   # bit n = Ch(n-1)
    ('i32', 116, 'Points'),
    ('f32', 120, 'Time Base (s)'),
    ('ch',  124, 'Ch1'),
    ('ch',  152, 'Ch2'),
    ('ch',  180, 'Ch3'),
    ('ch',  208, 'Ch4'),
)

channel =\
(
    ('u8',   0, 'Active'),                 # 0 = Yes, 0 = No
    ('u8',   1, 'Coupling'),               # 0 = DC,  1 = DC
    ('u8',   2, 'Bandwidth Limit'),        # 0 = Enabled,  1 = Disabled
    ('i32',  4, 'Unknown'),                # Changes depending on if the channel is active and some other factor
    ('f32',  8, 'Voltage (v)'),
    ('f32', 12, 'Position (v)'),
    ('u8',  16, 'Inverted'),               # 0 = No, 1 = Yes
    ('u8',  17, 'Unit'),                   # 1 = A, 2 = V
)

az = ascii zero

The channel time series begins at position 3198, and is interleaved.

I'm also wondering if there is any tool to describe a binary file as a context free grammar and parse it automagically.

[Mechatrommer]

I'm also wondering if there is any tool to describe a binary file as a context free grammar and parse it automagically.

automagically? in our dream. not automagically? thats called hex editor. btw, maybe you'll find my log file helpfull (attached)

[Neuromodulator]

automagically? in our dream. not automagically? thats called hex editor. btw, maybe you'll find my log file helpfull (attached)

Thanks for the file, gonna take a look at it. When I talked about parsing, I ment parsing after the grammar has been defined, not detecting the grammar. There are tools that do that, but they are tailored to computer language parsing.

[Mechatrommer]

I ment parsing after the grammar has been defined, not detecting the grammar. There are tools that do that, but they are tailored to computer language parsing.

what grammar? its just content... binary file format. if you already know the grammar (i mean file chunk format) then you should be able to make a simple parser script. this is what i did when reverse enginerring the WFM on many files output to make content comparison.

[Neuromodulator]

what grammar? its just content... binary file format. if you already know the grammar (i mean file chunk format) then you should be able to make a simple parser script. this is what i did when reverse enginerring the WFM on many files output to make content comparison.

I know, I was just looking for a standard way to describe a binary file grammar and a tool that could read the file and parse the binary. Since that apparently doesn't exist I'll continue doing it manually with code.

[janoc]

I know, I was just looking for a standard way to describe a binary file grammar and a tool that could read the file and parse the binary. Since that apparently doesn't exist I'll continue doing it manually with code.

Uff, I think you are overthinking it a bit - the file format is not likely to change, so a grammar and a generic parser would be an enormous overkill for this.

[freman]

I've started doing this too, how far have we all gotten?

[TheoB]

Just curious. Why would you want to use this method? I just ask the scope for the raw sample data. You only need a USB or network connection. It's all well documented.

[freman]

It's not always convenient or possible to be hooked up to a network or computer at time of sampling

[HardWareMan]

Hi! Is there any update with new WFM file?

[preamp.org]

I'll do the Tread Digger^TM and revive this old thread.

I have recently acquired a used DS1104Z because I needed a 4-channel scope. And of course I immediately fell down the rabbit hole of getting the captured waveform data from the scope over to the PC for post-processing. As is widely known, saving CSV data to USB-drive takes ages and thus renders the scope unusable during that time. Saving to WFM is much quicker, but that file seems to be pretty much useless, unless a close approximation is sufficient.

I have not used any of the software that Rigol supplies for their scope, and I probably won't. Windows-only and ~500MB download is just not my cup of tea. But here's a question to anybody who did use that software: Is it actually possible to accurately convert a WFM file into a CSV file (or anything else, really) with Rigol tools, or won't even their own tools work on that files?

The reason I'm asking is that I have a strong suspicion that there's some vital information missing from the files altogether. From what I have figured out so far, the RAW values in the WFM file seem to be the exact RAW values that the ADC spits out. You can clearly see the range of that data change with the vertical scale setting and the applied input signal held constant, so it is not normalized to some calibrated value. To do this, you'd need to know the calibration data of the ADC, i.e. its internal range and divider settings, and apply this manually to the RAW data to obtain the correct voltage scale. Unfortunately, that calibration data seems to be missing from the WFM file!

A simple test was to make two almost-identical measurements, with only the voltage scale being a little different and thus leading to a different RAW data range. I have used 1V/Div for a resulting RAW value range of 29 to 141, and 0.9V/Div for a much bigger range of 32 to 211. The input signal was the same for both measurements, so you'd need to apply different conversion factors for both files to obtain the same resulting output voltage, obviously.
Next, I have compared the WFM files with a HEX editor to look for some differing data. Apart from the obvious things like the actual decimal scale factor voltage, instrument run time and a handful of bytes that must be some kind of CRC or checksum (as those bytes change for each individual WFM file, even if you immediately save the exact same data three times in a row, without even leaving the Storage menu or doing anything else, really), there are no other differences in the file! Nothing that would relate to any kind of selected mode, or calibration data. And there certainly is no huge table that would contain all the calibration values at once. The scale factor that needs to be applied to the RAW data seems to range from 1.00 to about 1.54, and that factor is not only different for each of the four channels, but also different for each setting of the vertical scale. And not just the coarse 1-2-5-settings, but for each and every vernier step: 0.9V needs a different factor than 0.91V, or 0.92V, or 0.93V... you get the idea. Such a table would be huge. Unless there was a linear relationship between those ranges, but I have'nt taken enough measurements to try and find one.

Honestly, at the moment this looks like a hopeless case. Short of manually determining the conversion factors for each channel/range-combo of interest, I don't see how this could work out. Except I'm missing something obvious... but others have tried this before, and did not succeed, either, AFAIK.

[Mechatrommer]

The reason I'm asking is that I have a strong suspicion that there's some vital information missing from the files altogether. From what I have figured out so far, the RAW values in the WFM file seem to be the exact RAW values that the ADC spits out. You can clearly see the range of that data change with the vertical scale setting and the applied input signal held constant, so it is not normalized to some calibrated value. To do this, you'd need to know the calibration data of the ADC, i.e. its internal range and divider settings, and apply this manually to the RAW data to obtain the correct voltage scale. Unfortunately, that calibration data seems to be missing from the WFM file!

VisaDSO did it, rev.eng of WFM format is far from 100% but good enough to extract some data out of it, attached is my VB translated code to decode WFM, if you still interested in this old (i would say obsolete or nonsensical) format... i would just connect my ds1000z to usb cable, much less work to do than switching usb stick back and forth... ymmv...

[preamp.org]

Well, VisaDSO refused my files as unsupported  .
I'll have a look at your code though, thanks for posting it here.

And how would you control the scope with a USB cable, using what software? That's not how I'd intend to use it, because I don't have a PC at the workspace where the scope is set up. I am more interested in using the roll mode to capture several successive minutes of data, with no real-time-PC-access necessary. No problem transferring the data with a thumb drive, but waiting half an hour for it being written as a CSV file is a real show stopper.

[Mechatrommer]

Well, VisaDSO refused my files as unsupported  .

if you send me your wfm file, i may have a look when i have time to see whats unsupported. but its been a while since i look at the code, so no promise since i can forget things.

And how would you control the scope with a USB cable, using what software? That's not how I'd intend to use it

VisaDSO is not meant to control your dso, its more on downloading data to PC/laptop. but it provides limited but related control such as setting memory capture to either screen or memory. ymmv.

[preamp.org]

Turns out that VisaDSO refused the files from the first folder that I tried (and then I didn't bother to try some more). Those files were recorded with 6M pts. It does accept some others, though, which were recorded with only 3k pts. Unfortunately, the voltage range is way off.

Please have a look at the attached example. Input signal is the same for all 4 Channels, about 6.2Vpp. They're all the same in the CSV file that the scope put out, just as you'd expect.
With VisaDSO I get 4 different results, namely:
5.15Vpp for CH1,
5.85Vpp for CH2,
3.95Vpp for CH3 and
4.05Vpp for CH4.

That's not exactly helpful, not even for relative comparisons between the channels, even if the absolute value was not spot-on.

[Mechatrommer]

Unfortunately, the voltage range is way off...
With VisaDSO I get 4 different results, namely:

since as said WFM has not been fully hacked, scaling can be way off, full scale correction setting in WFM has not been found. hence VisaDSO provides manual scaling, please see attached images in order to get to manual translation (scaling and offset) setting dialog box. i've quickly calculated the scaling and offset needed for NewFile1.wfm to get as close as your screen capture and manually scaled CH1 (clicking green button CH1) please refer to manual_scale.png, i've not scaled CH2-4 (offset left at 0 scale at 1) you can do it yourself if needed by clicking individually CH2-4 green button and ticking checkbox above them. i only provide CH1 as example. cheers.

as for unsupported NewFile3.wfm in doesntwork.7z i'll look into it later if its solvable...
i've traced file incompatibility in NewFile3.wfm is due to trigger channel reported by the file as channel 6, since VisaDSO only expects maximum trigger channel as CH4, then it will stop reading the file and spit "unsuppported file" to avoid any further unexpected values in the file. may i ask what trigger setting and channel your scope was set at while doing the capture file NewFile3.wfm? i cant possibly tried all the dso setting when developing the WFM reader code in VisaDSO.

[preamp.org]

may i ask what trigger setting and channel your scope was set at while doing the capture file NewFile3.wfm?

Trigger was set to AC. That would explain CH6 on a 4-Channel scope. I've attached the settings file that was generated together with the PNG.

[Mechatrommer]

may i ask what trigger setting and channel your scope was set at while doing the capture file NewFile3.wfm?

Trigger was set to AC. That would explain CH6 on a 4-Channel scope. I've attached the settings file that was generated together with the PNG.

i've added quick fix for ac sourced trigger, you can patch VisaDSO.exe with the attached file, and retry to open your NewFile3.wfm. consider this "unofficial nightly build release v1.1.4" just for you to fix just one thing. cheers.

[preamp.org]

Well thanks for your effort. Unfortunately it won't run under WINE on my PC at home, so I can only test it out tomorrow at work on the Windows-machine.

I don't know how or where you apply the manual conversion factor, but if you apply it directly to the RAW data, and only then do the other stuff, you'll end up with the correct voltage offset automagically. The factors for my sample file are 1.19 - 1.055 - 1.545 - 1.52 for CH1..4, respectively.

I have converted the data into the LTspice RAW format. You can open the file with LTspice, go to Plot Settings > Visible Traces and add all 4 channels there. The waveform viewer lends itself nice for this purpose, since it can apply lots of math to the traces (like factors and offsets...), integrate, has two cursors, zoom, ...

[preamp.org]

I have briefly tried Rigol's Ultra Scope software today. Might as well be that I missed something, but I was not able to load a WFM file with it. Looks like only CSV or something similar is supported. Which would reinforce my suspicion that not even Rigol has a tool to make sense of their own file format 

EDIT: It is possible to re-load the WFM files on the scope, and convert them to CSV after the fact - while having to wait a little eternity everytime, though. Unfortunately, the scope will refuse to load a WFM file that was recorded with another scope; further strengthening my suspicion, that the necessary conversion factors never actually leave the scope.