Author Topic: Rigol DSXXXX .GEL firmware file format  (Read 63128 times)

0 Members and 1 Guest are viewing this topic.

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Rigol DSXXXX .GEL firmware file format
« on: July 13, 2016, 05:40:13 pm »
After looking inside Rigol Digital Oscilloscope firmware file (DS1000ZUpdate.GEL)
I found this not to be very complicated. There was index with file names and
all binary data in one row. This can be extracted and may be modified if we
find out all tricks of it.
We don't like to break it - we like to make it better!

There are similarities between DS1000Z to DS6000. Only DS1000Z has index
with file names, the others have similar index structure.

We had some discussion in other threads from here:
https://www.eevblog.com/forum/testgear/new-rigol-ds1054z-oscilloscope/msg980793/#msg980793
but I make here new starting point and let's continue it here.
Who knows where we'll end up with this...

----------------------------------------------------------------------------------------------------------
After little pause we managed to solve GEL footer and made progress with new scope hacks.

04.08.2016 - Latest RigolPacker

15.04.2018 - More detailed updated GEL file format

15.04.2018 - GEL file modifications guide

15.04.2018 - Disassembly information, Functions, IDA scripts

21.04.2018 - Bootloader versions
« Last Edit: April 22, 2018, 02:26:23 pm by janekivi »
 
The following users thanked this post: BravoV, bitwelder, Circlotron, ProBang2, Gabri74, tooki, bitseeker, Wirehead, Marcos, simas1017, JimKnopf

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Re: Rigol DSXXXX .GEL firmware file format
« Reply #1 on: July 13, 2016, 05:42:06 pm »
For unpacking DS1000Z firmware 00.04.03.02.03 I made very noprofessional script.
It extracting all files and striping header if head = 24. You can set it to 0.

But if we like to pack someting back together we must add then this header.
So what is there. Header has 24 bytes where:
4 is file crc32
4 is some file type? Packed app has 03 00 00 00, packed gui stuff have 01, other have 00 00 00 00
4 is file lenght
4 is AA 55 55 AA (something spezial ?)
4 is Firmware version FB 7E 3D 00 -> 4030203
4 is buffer 00 00 00 00 (or for future use...?)
 
The following users thanked this post: BravoV, ProBang2, Marcos

Offline Danielw

  • Contributor
  • Posts: 11
Re: Rigol DSXXXX .GEL firmware file format
« Reply #2 on: July 14, 2016, 05:27:01 am »
Nice work!

Maybe we could start a collaborative IDA session, I know it exists plugins for that, but I've never tried any myself.
I looked through the code and found some functions rather quick. And a lot of variables are easy to identify from printf strings.

//Daniel
 

Offline BravoV

  • Super Contributor
  • ***
  • Posts: 7547
  • Country: 00
  • +++ ATH1
Re: Rigol DSXXXX .GEL firmware file format
« Reply #3 on: July 14, 2016, 05:29:55 am »
Thank you janekivi  :-+, subscribed.

Offline RhymeMess

  • Newbie
  • Posts: 7
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #4 on: July 14, 2016, 10:40:58 am »
I created a small python script that is a little bit less hacky. It properly decodes the header information and extracts the offset to create the correct files. Thus this works with all the DS1000Z Firmware files I tried.
DS2000 has an entirely different structure though.

After the files, there is always an 280 bytes footer I didn't yet look into...

*edit* I didn't remove the 24 bytes file header yet...

*edit2* I updated the script to strip the 24 bytes header and decompress the file if possible. AFAIK python > 3.3 is needed for decompression, it gracefully fails if decompression is not possible.

*edit3* Update here
« Last Edit: July 15, 2016, 06:38:36 am by RhymeMess »
 
The following users thanked this post: Marcos

Offline smithnerd

  • Regular Contributor
  • *
  • Posts: 120
  • Country: gb
Re: Rigol DSXXXX .GEL firmware file format
« Reply #5 on: July 14, 2016, 04:08:36 pm »
I did some quick analysis of the footer last night.

Comparing the final 280 bytes with those of previous firmwares, some patterns emerge:

There are 5 x 32 bit integers (hex):

80 00 00 00
01 00 00 00
80 00 00 00
01 00 00 00
04 00 00 00

Then 256 bytes of data.

then a final 32 bits trailing:

01 00 01 00

Visualising the 256 bytes of data as a bitmap clearly shows that there is more entropy in the first 128 bytes.
 
The following users thanked this post: Marcos

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Re: Rigol DSXXXX .GEL firmware file format
« Reply #6 on: July 14, 2016, 04:13:21 pm »
- When You strip 24 bytes header, do not cut 24 bytes from file end too.
- Actually the file without header is the right one and header making it to ... "SparrowAPP.out with header ..."
- My Python 3.4 didn't unpack anything and footer without crc is skipped as I see in script too.
  May be some plugin needed for all Pythons?
 
The following users thanked this post: Marcos

Offline RhymeMess

  • Newbie
  • Posts: 7
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #7 on: July 15, 2016, 06:36:59 am »
- When You strip 24 bytes header, do not cut 24 bytes from file end too.
The length is from the GEL Header, that includes those 24 bytes. I'm pretty certain those 24 bytes have to go as well. Its not "cut from the end" but "don't extend beyond the end".
Quote
- Actually the file without header is the right one and header making it to ... "SparrowAPP.out with header ..."
I changed it to write the file without header and a "file.header" file. The 24 bytes discussion should be a lot clearer as well.
Quote
- My Python 3.4 didn't unpack anything and footer without crc is skipped as I see in script too.
  May be some plugin needed for all Pythons?
I have pyliblzma installed which is probably needed. I also fixed a problem with the decompressor only decompressing the first file...

*edit* Small modification that also writes the header file, so it should be possible to reconstruct the entire firmware from the unpacked one.
Also this is a little decrappified.
« Last Edit: July 15, 2016, 07:26:57 am by RhymeMess »
 
The following users thanked this post: Marcos

Offline Shock

  • Super Contributor
  • ***
  • Posts: 4200
  • Country: au
Re: Rigol DSXXXX .GEL firmware file format
« Reply #8 on: July 15, 2016, 08:37:37 am »
If you have any missing firmware versions from this list (or one shown in red) please upload or send me a link in a PM etc, preferably the original Rigol zips.

Version 00.04.03.02.03 Date 2015-10-20 (Rigol Website)
Version 00.04.03.01.05 Date 2015-06-16 (Rigol Website)
Version 00.04.03.00.01 Date 2015-05-05 (Rigol Website)
Version 00.04.02.04.07 Date 2014-12-31  (Rigol Website)
Version 00.04.02.03.00 Date 2014-10-21
Version 00.04.01.02.00 Date 2014-07-28
Version 00.04.00.00.00 Date ? (Additionally contains boot code)
Version 00.02.03.05.00 Date ?
« Last Edit: July 15, 2016, 08:46:31 am by Shock »
Soldering/Rework: Pace ADS200, Pace MBT350
Multimeters: Fluke 189, 87V, 117, 112   >>> WANTED STUFF <<<
Oszilloskopen: Lecroy 9314, Phillips PM3065, Tektronix 2215a, 314
 

Offline bitwelder

  • Frequent Contributor
  • **
  • Posts: 964
  • Country: fi
Re: Rigol DSXXXX .GEL firmware file format
« Reply #9 on: July 15, 2016, 08:59:58 am »
Have you tried to have a look to the f/w upgrade files for other Rigol models, to find if there is a common pattern in the file structure and by looking at differences identify the fields related to models ID, etc. ?
 

Offline RhymeMess

  • Newbie
  • Posts: 7
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #10 on: July 15, 2016, 09:20:50 am »
I had a quick look into some files from here and they are all different from the DS1000Z. I looked into DS2000, DS4000, DS405X and DSA800 (which even doesn't contain a .GEL file).

DS2000 and DS4000 seem to have the same file structure...

*edit* Some more cleanups and the bytes 32-36 are 0x00070000 for every firmware, except for the boot update, where it is 0x000f0000. So I changed the default folder to $Device-$Version-$UpdateType...

*edit2* Probably last coffe break before the weekend; If you have PIL (or Pillow) installed, it will create a png like smithnerd did from the last file. Also did a few more sanity & crc checks, SparrowBootloader.sb should be written correctly now...
Also it always automaticall sets the output directory to $Device-$Firmware-$Type and now allows for multiple GEL Files. So "./unpack.py *.GEL" works properly now.
« Last Edit: July 15, 2016, 01:12:18 pm by RhymeMess »
 
The following users thanked this post: Marcos

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Re: Rigol DSXXXX .GEL firmware file format
« Reply #11 on: July 15, 2016, 02:51:46 pm »
- When You strip 24 bytes header, do not cut 24 bytes from file end too.
The length is from the GEL Header, that includes those 24 bytes. I'm pretty certain those 24 bytes have to go as well.
Its not "cut from the end" but "don't extend beyond the end".
I was looking the output and there was length difference 24+24 bytes. From Siglent experiment I think the file is
having the header in eeprom but for us to study the file we can strip it.

But latest scripts don't unpack my DS1000Z-00.04.03.02.03. I see only 6 files.
 
The following users thanked this post: Marcos

Offline RhymeMess

  • Newbie
  • Posts: 7
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #12 on: July 15, 2016, 03:14:20 pm »
No Error? Can you post a log of the script?
 
The following users thanked this post: Marcos

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Re: Rigol DSXXXX .GEL firmware file format
« Reply #13 on: July 15, 2016, 08:37:05 pm »
There is CRC errors with V7. If I skip crc check, the file is extracted correctly and crc is the same as in the header.
I have Python 3.4.4
 
The following users thanked this post: Marcos

Offline Userli

  • Regular Contributor
  • *
  • Posts: 72
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #14 on: July 15, 2016, 10:02:59 pm »
I was able to replace the start screen image (logo.hex) by the one attached.
To do so I wrote a small Windows application.
It replaces the RGB565 data and updates the CRCs.
The device accepted to load the same FW version including the new image.
« Last Edit: July 16, 2016, 07:52:25 pm by Userli »
 
The following users thanked this post: janekivi, Wirehead, Marcos

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Re: Rigol DSXXXX .GEL firmware file format
« Reply #15 on: July 16, 2016, 06:55:31 am »
And what You are thinking I was hoping to see here when started this thread : )

Only thing, Your program is very small... so there will be some dependencies.
It is written in Microsoft .NET? Not any of my computers get it to functioning.
It won't extract anything and crash, and logo button give error too: GDI+
« Last Edit: July 16, 2016, 08:20:15 am by janekivi »
 
The following users thanked this post: Marcos

Offline Userli

  • Regular Contributor
  • *
  • Posts: 72
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #16 on: July 16, 2016, 10:11:19 am »
The tool currently only reflects the knowledge you kindly shared and makes it easier accessible to Windows users. It requires .NET 4.5 and the most obvious bugs should be solved.
It uses the 7z SDK for decompression. I was unable to tune the parameters of the SDK such that I could compress a file again and get exactly the same as in the .GEL file.
Does anybody have an idea on how to determine the compression parameters from a compressed LZMA file?
Otherwise one could try a brute force approach by cycling over all possibilities.
Or maybe there is another LZMA implementation which does this?
I found that SparrowCalFile.hex has the wrong file sizes in the LZMA header. They are much smaller than the real sizes and the decompression fails with either of them.
« Last Edit: July 16, 2016, 07:53:26 pm by Userli »
 
The following users thanked this post: Marcos

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Re: Rigol DSXXXX .GEL firmware file format
« Reply #17 on: July 16, 2016, 12:04:22 pm »
I'm the dumb here but how to You use this program.
I can open firmware file.
I can now see original logo. I can see the image, save it and change (to view another but ...?)
Save image is saving original and what next? No any action here, I can only view images, I close the window?
Now the import logo button is active. Pressing id does...(nothing)?
Extract files button crash the app.
Save firmware file is saving original one.
 

Offline Userli

  • Regular Contributor
  • *
  • Posts: 72
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #18 on: July 16, 2016, 01:56:38 pm »
Looking a bit further into the 24 bytes headers of the individual files, it looks to me as if the uint32 in bytes 4 to 7 tell if the file is compressed. For all those having a valid LZMA header, bit 0 is 1 here.
The first file SparrowAPP.out has also bit 1 set. No idea why.

To change the image do as follows:
1) open firmware file
2) show logo
3) in logo window save image (as BMP)
4) modify BMP without changing it's size
5) click change image and select the changed BMP
6) on main panel import logo (this loads the image currently in the logo window into the main file data structure)
7) save firmware file
Now you should have a new .GEL file with the modified logo.

Clicking on extract files should ask you to select the directory for storing the files. Does this work?
« Last Edit: July 16, 2016, 07:54:14 pm by Userli »
 

Offline et328

  • Contributor
  • Posts: 42
  • Country: fi
Re: Rigol DSXXXX .GEL firmware file format
« Reply #19 on: July 16, 2016, 02:09:48 pm »
Clicking on extract files should ask you to select the directory for storing the files. Does this work?

Doesn't work, it asks for the extract directory and crashes after you press OK.
 

Offline Userli

  • Regular Contributor
  • *
  • Posts: 72
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #20 on: July 16, 2016, 02:33:01 pm »
There was indeed a dependency on the LZMA executable, which I missed. I'll try to integrate it in the main code. For the time being just put the LZMA#.exe into the same directory as the main rigolpacker.exe.
« Last Edit: July 16, 2016, 07:54:44 pm by Userli »
 
The following users thanked this post: Marcos

Offline janekiviTopic starter

  • Frequent Contributor
  • **
  • Posts: 368
  • Country: ee
Re: Rigol DSXXXX .GEL firmware file format
« Reply #21 on: July 16, 2016, 04:01:14 pm »
This is better now!
Let's step through this logo stuff too. I know the BMP and can edit it without changing its size.
800 x 480 x 3 + 54 bytes header is always 1152054 bytes. But there is no BMP, you have PNG.
Teach us too to edit this without changing its size.
 

Offline Userli

  • Regular Contributor
  • *
  • Posts: 72
  • Country: de
Re: Rigol DSXXXX .GEL firmware file format
« Reply #22 on: July 16, 2016, 04:24:49 pm »
Where do you get the .png ?
In step 3 you click the button "Save image" to save it as .bmp.

« Last Edit: July 16, 2016, 07:55:21 pm by Userli »
 

Offline et328

  • Contributor
  • Posts: 42
  • Country: fi
Re: Rigol DSXXXX .GEL firmware file format
« Reply #23 on: July 16, 2016, 04:43:32 pm »
File has .bmp extension but the actual fileformat seems to be .png
 

Offline smithnerd

  • Regular Contributor
  • *
  • Posts: 120
  • Country: gb
Re: Rigol DSXXXX .GEL firmware file format
« Reply #24 on: July 16, 2016, 05:11:55 pm »
The LZMA implementation used deviates from the spec slightly, looking at the header:

Code: [Select]
5d 00 00 80 00 cc 20 3c 00 ef a7 10 00

5d is packed lc, lp, pb bits, and 00 00 80 00 (8 MiB) is the dictionary size, as expected.

The next 8 bytes should be uncompressed file size as an unsigned little-endian 64 bit integer, but it isn't. Instead it is two unsigned 32 bit little-endian integers representing:

cc 20 3c 00 (3940556 bytes) uncompressed file size
ef a7 10 00 (1091567 bytes) compressed file size

Zero out the last 4 bytes and it decompresses without issue:

Code: [Select]
5d 00 00 80 00 cc 20 3c 00 00 00 00 00






 
The following users thanked this post: Marcos


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf