-
Rigol DSXXXX .GEL firmware file format
Posted by
janekivi
on 13 Jul, 2016 17:40
-
-
#1 Reply
Posted by
janekivi
on 13 Jul, 2016 17:42
-
For unpacking DS1000Z firmware 00.04.03.02.03 I made very noprofessional script.
It extracting all files and striping header if head = 24. You can set it to 0.
But if we like to pack someting back together we must add then this header.
So what is there. Header has 24 bytes where:
4 is file crc32
4 is some file type? Packed app has 03 00 00 00, packed gui stuff have 01, other have 00 00 00 00
4 is file lenght
4 is AA 55 55 AA (something spezial ?)
4 is Firmware version FB 7E 3D 00 -> 4030203
4 is buffer 00 00 00 00 (or for future use...?)
-
#2 Reply
Posted by
Danielw
on 14 Jul, 2016 05:27
-
Nice work!
Maybe we could start a collaborative IDA session, I know it exists plugins for that, but I've never tried any myself.
I looked through the code and found some functions rather quick. And a lot of variables are easy to identify from printf strings.
//Daniel
-
#3 Reply
Posted by
BravoV
on 14 Jul, 2016 05:29
-
Thank you janekivi
, subscribed.
-
#4 Reply
Posted by
RhymeMess
on 14 Jul, 2016 10:40
-
I created a small python script that is a little bit less hacky. It properly decodes the header information and extracts the offset to create the correct files. Thus this works with all the DS1000Z Firmware files I tried.
DS2000 has an entirely different structure though.
After the files, there is always an 280 bytes footer I didn't yet look into...
*edit* I didn't remove the 24 bytes file header yet...
*edit2* I updated the script to strip the 24 bytes header and decompress the file if possible. AFAIK python > 3.3 is needed for decompression, it gracefully fails if decompression is not possible.
*edit3* Update
here
-
#5 Reply
Posted by
smithnerd
on 14 Jul, 2016 16:08
-
I did some quick analysis of the footer last night.
Comparing the final 280 bytes with those of previous firmwares, some patterns emerge:
There are 5 x 32 bit integers (hex):
80 00 00 00
01 00 00 00
80 00 00 00
01 00 00 00
04 00 00 00
Then 256 bytes of data.
then a final 32 bits trailing:
01 00 01 00
Visualising the 256 bytes of data as a bitmap clearly shows that there is more entropy in the first 128 bytes.
-
#6 Reply
Posted by
janekivi
on 14 Jul, 2016 16:13
-
- When You strip 24 bytes header, do not cut 24 bytes from file end too.
- Actually the file without header is the right one and header making it to ... "SparrowAPP.out with header ..."
- My Python 3.4 didn't unpack anything and footer without crc is skipped as I see in script too.
May be some plugin needed for all Pythons?
-
#7 Reply
Posted by
RhymeMess
on 15 Jul, 2016 06:36
-
- When You strip 24 bytes header, do not cut 24 bytes from file end too.
The length is from the GEL Header, that includes those 24 bytes. I'm pretty certain those 24 bytes have to go as well. Its not "cut from the end" but "don't extend beyond the end".
- Actually the file without header is the right one and header making it to ... "SparrowAPP.out with header ..."
I changed it to write the file without header and a "file.header" file. The 24 bytes discussion should be a lot clearer as well.
- My Python 3.4 didn't unpack anything and footer without crc is skipped as I see in script too.
May be some plugin needed for all Pythons?
I have pyliblzma installed which is probably needed. I also fixed a problem with the decompressor only decompressing the first file...
*edit* Small modification that also writes the header file, so it should be possible to reconstruct the entire firmware from the unpacked one.
Also this is a little decrappified.
-
#8 Reply
Posted by
Shock
on 15 Jul, 2016 08:37
-
If you have any missing firmware versions from this list (or one shown in red) please upload or send me a link in a PM etc, preferably the original Rigol zips.
Version 00.04.03.02.03 Date 2015-10-20 (Rigol Website)
Version 00.04.03.01.05 Date 2015-06-16 (Rigol Website)
Version 00.04.03.00.01 Date 2015-05-05 (Rigol Website)
Version 00.04.02.04.07 Date 2014-12-31 (Rigol Website)
Version 00.04.02.03.00 Date 2014-10-21
Version 00.04.01.02.00 Date 2014-07-28
Version 00.04.00.00.00 Date ? (Additionally contains boot code)
Version 00.02.03.05.00 Date ?
-
#9 Reply
Posted by
bitwelder
on 15 Jul, 2016 08:59
-
Have you tried to have a look to the f/w upgrade files for other Rigol models, to find if there is a common pattern in the file structure and by looking at differences identify the fields related to models ID, etc. ?
-
#10 Reply
Posted by
RhymeMess
on 15 Jul, 2016 09:20
-
I had a quick look into some files from
here and they are all different from the DS1000Z. I looked into DS2000, DS4000, DS405X and DSA800 (which even doesn't contain a .GEL file).
DS2000 and DS4000 seem to have the same file structure...
*edit* Some more cleanups and the bytes 32-36 are 0x00070000 for every firmware, except for the boot update, where it is 0x000f0000. So I changed the default folder to $Device-$Version-$UpdateType...
*edit2* Probably last coffe break before the weekend; If you have PIL (or Pillow) installed, it will create a png like smithnerd did from the last file. Also did a few more sanity & crc checks, SparrowBootloader.sb should be written correctly now...
Also it always automaticall sets the output directory to $Device-$Firmware-$Type and now allows for multiple GEL Files. So "./unpack.py *.GEL" works properly now.
-
#11 Reply
Posted by
janekivi
on 15 Jul, 2016 14:51
-
- When You strip 24 bytes header, do not cut 24 bytes from file end too.
The length is from the GEL Header, that includes those 24 bytes. I'm pretty certain those 24 bytes have to go as well.
Its not "cut from the end" but "don't extend beyond the end".
I was looking the output and there was length difference 24+24 bytes. From Siglent experiment I think the file is
having the header in eeprom but for us to study the file we can strip it.
But latest scripts don't unpack my DS1000Z-00.04.03.02.03. I see only 6 files.
-
#12 Reply
Posted by
RhymeMess
on 15 Jul, 2016 15:14
-
No Error? Can you post a log of the script?
-
#13 Reply
Posted by
janekivi
on 15 Jul, 2016 20:37
-
There is CRC errors with V7. If I skip crc check, the file is extracted correctly and crc is the same as in the header.
I have Python 3.4.4
-
#14 Reply
Posted by
Userli
on 15 Jul, 2016 22:02
-
I was able to replace the start screen image (logo.hex) by the one attached.
To do so I wrote a small Windows application.
It replaces the RGB565 data and updates the CRCs.
The device accepted to load the same FW version including the new image.
-
#15 Reply
Posted by
janekivi
on 16 Jul, 2016 06:55
-
And what You are thinking I was hoping to see here when started this thread : )
Only thing, Your program is very small... so there will be some dependencies.
It is written in Microsoft .NET? Not any of my computers get it to functioning.
It won't extract anything and crash, and logo button give error too: GDI+
-
#16 Reply
Posted by
Userli
on 16 Jul, 2016 10:11
-
The tool currently only reflects the knowledge you kindly shared and makes it easier accessible to Windows users. It requires .NET 4.5 and the most obvious bugs should be solved.
It uses the 7z SDK for decompression. I was unable to tune the parameters of the SDK such that I could compress a file again and get exactly the same as in the .GEL file.
Does anybody have an idea on how to determine the compression parameters from a compressed LZMA file?
Otherwise one could try a brute force approach by cycling over all possibilities.
Or maybe there is another LZMA implementation which does this?
I found that SparrowCalFile.hex has the wrong file sizes in the LZMA header. They are much smaller than the real sizes and the decompression fails with either of them.
-
#17 Reply
Posted by
janekivi
on 16 Jul, 2016 12:04
-
I'm the dumb here but how to You use this program.
I can open firmware file.
I can now see original logo. I can see the image, save it and change (to view another but ...?)
Save image is saving original and what next? No any action here, I can only view images, I close the window?
Now the import logo button is active. Pressing id does...(nothing)?
Extract files button crash the app.
Save firmware file is saving original one.
-
#18 Reply
Posted by
Userli
on 16 Jul, 2016 13:56
-
Looking a bit further into the 24 bytes headers of the individual files, it looks to me as if the uint32 in bytes 4 to 7 tell if the file is compressed. For all those having a valid LZMA header, bit 0 is 1 here.
The first file SparrowAPP.out has also bit 1 set. No idea why.
To change the image do as follows:
1) open firmware file
2) show logo
3) in logo window save image (as BMP)
4) modify BMP without changing it's size
5) click change image and select the changed BMP
6) on main panel import logo (this loads the image currently in the logo window into the main file data structure)
7) save firmware file
Now you should have a new .GEL file with the modified logo.
Clicking on extract files should ask you to select the directory for storing the files. Does this work?
-
#19 Reply
Posted by
et328
on 16 Jul, 2016 14:09
-
Clicking on extract files should ask you to select the directory for storing the files. Does this work?
Doesn't work, it asks for the extract directory and crashes after you press OK.
-
#20 Reply
Posted by
Userli
on 16 Jul, 2016 14:33
-
There was indeed a dependency on the LZMA executable, which I missed. I'll try to integrate it in the main code. For the time being just put the LZMA#.exe into the same directory as the main rigolpacker.exe.
-
#21 Reply
Posted by
janekivi
on 16 Jul, 2016 16:01
-
This is better now!
Let's step through this logo stuff too. I know the BMP and can edit it without changing its size.
800 x 480 x 3 + 54 bytes header is always 1152054 bytes. But there is no BMP, you have PNG.
Teach us too to edit this without changing its size.
-
#22 Reply
Posted by
Userli
on 16 Jul, 2016 16:24
-
Where do you get the .png ?
In step 3 you click the button "Save image" to save it as .bmp.
-
#23 Reply
Posted by
et328
on 16 Jul, 2016 16:43
-
File has .bmp extension but the actual fileformat seems to be .png
-
#24 Reply
Posted by
smithnerd
on 16 Jul, 2016 17:11
-
The LZMA implementation used deviates from the spec slightly, looking at the header:
5d 00 00 80 00 cc 20 3c 00 ef a7 10 00
5d is packed lc, lp, pb bits, and 00 00 80 00 (8 MiB) is the dictionary size, as expected.
The next 8 bytes should be uncompressed file size as an unsigned little-endian 64 bit integer, but it isn't. Instead it is two unsigned 32 bit little-endian integers representing:
cc 20 3c 00 (3940556 bytes) uncompressed file size
ef a7 10 00 (1091567 bytes) compressed file size
Zero out the last 4 bytes and it decompresses without issue:
5d 00 00 80 00 cc 20 3c 00 00 00 00 00