Computing CRC-16 from .hex file

[fearless]

Context: After bootloading a .hex file into our microcontroller, I want to use a CRC to verify that the program memory was properly written.  Therefore, I need to compute the CRC on the host computer prior to downloading the .hex file into the microcontroller.

The goal: given a program on the host computer that takes as inputs:
* a well-defined CRC algorithm (the actual choice of CRC is not important for this question)
* a .hex file that contains binary data to be loaded into the microcontroller, with possibly discontinuous address segments
* a starting address and number of bytes over which to run the CRC algo
... compute the CRC for that memory range.

The question: Are there any techniques or utilities that will help me compute the CRC over that address range from the .hex file?

I initially thought this would be easy: just decode each data byte in the .hex file, if it is within the address range, pass it to the CRC algo.

But then I realized that .hex files can have discontinuous sections, and since the CRC algorithms are sensitive to the order in which bytes are fed to them, out-of-order sections would spoil the computation.

The best idea I've come up with is to allocate an array on the host computer, parse the .hex file and write only those bytes within the target address range into the array.  After the entire .hex file is read, then run the CRC algo other the array. 

How would you handle this?

[ataradov]

"Unpacking" the whole file (or just the regions of interest) is the only reliable way. You will also need to synchronize the way you will be filling gaps in your programming algorithm  (likely just keep erased state 0xff).

But the best thing you can do is normalize the HEX file and make sure that it is linear and contiguous.  You can also add CRC in the binary itself at that point, so you don't have to transfer it separately from the file.

This is one of the reasons why I will never use HEX files for anything. They are annoying to parse securely and don't have a standard way to deal with gaps and out of order addresses. They also don't add anything useful, we are not in the 80s anymore and not transferring the files on paper printouts.

[Ian.M]

Microchip Technology have a 'Swiss Army Knife' command line utility for manipulating IntelHEX hex files.  It can do CRC16 and insert the result into the hex file. See https://developerhelp.microchip.com/xwiki/bin/view/software-tools/xc8/crc-checksum/
It can also normalise hex files.
The Hexmate utility is bundled with their XC8 compiler (and IIRC also with the MPLAB X IDE).

[ataradov]

Hexmate is not that universal. It has not received many updates and fails on some files generated by XC32, for example. I don't know if it will handle well arbitrary chopped up files with many sections.

And it is hard to use if your file includes sections that are not a part of the main program space (like what is described in the original post).

[S. Petrukhin]

one of the reasons why I will never use HEX files for anything

I've never studied hex/bin in detail, but I'm wondering now - can a firmware image with address gaps be saved in bin?

[ataradov]

Only with a custom format. If you are adding CRC, then you are likely using some custom tool/script anyway, so if you really need multiple sections, just use a custom binary format.

But chances are that real applications don't need multiple sections. Or if they do, then non-main sections will usually have fixed or predictable size (assuming fuses or similar), so you don't really need to add any header information, just concatenate the binaries.

[S. Petrukhin]

That is, you do not want to print hex on paper, but manually write down starting addresses of individual bin pieces of single firmware to paper?   

It seems to me that hex is not so bad.
It indicates which addresses to write and data.
This allows you to record without first erasing, without disturbing the data in other addresses.

After all, it allows you to transfer your firmware to be downloaded to your device by another person and nothing else.
You do not store libraries and modules in hex, they do not need to be loaded into arbitrary devices.
There is no urgent need for standardization and global compliance.

Or do you dislike creative freedom and want to live only by standards? 

[ataradov]

manually write down starting addresses of individual bin pieces of single firmware to paper?   

What are you talking about? Why would you need to write anything down? The use here is your own bootloader and update process. You know what format the data is in.

This allows you to record without first erasing, without disturbing the data in other addresses.

All flash devices have erase granularity and often minimum write size requirements. So, you will have to reassemble the binary before committing it into the flash. Having chopped up HEX file is the worst nightmare of any firmware update process.

[S. Petrukhin]

HEX is not always used for flash and not always for the operation of a standalone programmer.

I use it to transfer the firmware update image to the embedded programmer. More precisely, the transmitter.
I get HEX from the compilator and pass pairs of address:word. I couldn't process BIN like that. And you know why. 
At the same time, updates are recorded in the second half of the flash without touching the first half, where the default firmware is located, to which you can return.

Of course, it is necessary to take into account the alignment and distribution of flash blocks.

[ataradov]

HEX is not always used for flash

You should read the OP. It very specifically talks about bootloaders and flash programming.

I'm not here to talk philosophy. From the bootlaaoding point of view, HEX is one of the worst formats.

[DavidAlfa]

The HEX file already has a checksum for each line, isn't that enough?

Code: [Select]

:100000A59F2CC0C03C4039403640334030402D4045Sum all bytes in (100000A59F2CC0C03C4039403640334030402D401 = 0xBB) and make 2-complement: 0x45.
This should match last byte.

[ataradov]

It would be enough to verify that HEX is not damaged, but once the firmware is written into the flash, all those checksum are gone. You can do verification, but if you are writing segmented chunks and have to do read-modify-write operations, it becomes trickier. It is still possible, but making bootloader code more complicated is never a good idea.

[HwAoRrDk]

The SRecord utilities might be able to do what you want.

The srec_cat utility can do things like read in a hex file, fill holes (i.e. gaps in addresses) in the data with certain byte value, calculate a CRC (various algorithms supported) and insert that checksum into the data, then output again as hex format.

[fearless]

I'm conversant with srecord suite, including how to compute a CRC and insert it into a specific address in the .hex file -- for example see

   https://community.nxp.com/t5/Kinetis-Design-Studio/Generating-CRC32-from-srec-files/m-p/709801#M9264

What I HAD forgotten about was normalizing the .hex file -- that does just what I need.   

(In the meantime, I wrote code that allocated a virtual memory buffer, filled it with 0xff bytes, then parsed the .hex file and wrote into the virtual memory buffer, then ran the checksum.  That works also...)

Case solved!