Encrypted hex files and firmware protection on PIC

[subunit]

  Hi folks,
 I am a biologist working substantially outside my area of competency so please bear that in mind.

 I am building a set of benchtop bioreactors to control conditions for a particular culture. Budget is tight, and typical off-the-shelf gear is inappropriate for my application, so I decided to roll my own using as much open-source, cheap hardware as possible. I went with a lower-cost manufacturer of sensors, transmitters, etc. as it looked like hobbyists were getting good results from their kit and the manufacturer appears to also be targeting the low end of industrial production (ie. people who can't afford to have Honeywell come and build them their control system).

 Things looked ok with my first reactor so I recently ordered a second set of sensors and PIC-based sensor circuits from the manufacturer. Between-sensor error seemed too high even after calibrating a couple of times, and there looked like some weird temp-dependent drift in the amount of between-sensor error, so I started to look at the circuit firmware. I noted that the mfr. had recently added a command allowing the exporting/importing of a calibration string from one circuit to another, which seemed like a possible solution, but I realised that the first set of circuits had older firmware without said command.

 Having recently learned how to flash an atmega8 with an AVR programmer (reprogramming some Chinese ESCs to convert some drone rotors into low-RPM magnetic mixers), I thought this should be a snap- hook up a PICkit to the circuit and flash the thing with the newest firmware. I was very taken aback when the mfr.'s rep insisted that I either had to disassemble the circuits and send them back to them (internationally) for a firmware update, or buy new hardware, even after I explained that the circuits are in use in a continuous culture system that does not readily permit me taking a reactor offline for days/weeks any time I want to update the firmware. This seemed to me to utterly defeat the point of firmware updates in an embedded system.

 After the rep mentioned that a "custom programmer" was required to flash the PIC (18F25K22- normal 8bit PIC), I pressed a bit and they mentioned that the PICs are programmed with encrypted hex files and that their programmer does some of this decryption. I couldn't understand what the point of this could possibly be until I searched around here a bit and found mention of "lock bits"- looking at the 18F25K22 datasheet it does seem that there are program memory protection bits. My guess is that these are enabled, and in conjunction with the custom decrypting PIC programmer, Shenzhen is prevented from directly knocking off what would otherwise be an easily cloned PCB.

The circuits that I have are smaller, Arduino-compatible ones that look to be aimed at the hobbyist market- ie. they're great for the kind of prototype benchtop reactor system I'm building, but they're probably not going to be adequate for scale-up once we get to that point. I mentioned above that the mfr. has some hardware that is apparently aimed at the industrial market- eg. sensor transmitters with DIN mounts. When I asked whether these also needed to be sent back for firmware updates, they indicated that this is indeed the case. I'm not an industrial control engineer, but I find the idea that someone should dismantle a control cabinet for an active production line in a commercial context, and mail the components back to the mfr, to get a firmware update, mind boggling. I've looked at other control manufacturers and the ones that I saw mostly seemed to have credentialled logins to firmware sections on their public websites where the user could go grab the latest firmware and deal with the problem themselves.

 I have questions following from this that I hope I could get some input on:

1) I have a PICkit clone in the mail that I ordered before understanding the situation. Although I believe the mfr. when they say I can't just copy the newer firmware from the new circuits to the old ones, I would like to hook up the PICkit anyway just to have a look before I send anything back to them. Is it possible that there are any "security features" above and beyond the program memory protection bits that could cause me some trouble here?

2) A simple google search turned up a fairly detailed paper describing a method for breaking PIC18F copy protection. As annoyed as I am, it looks like more work than it's worth to avoid freezing down a culture and paying a couple of courier bills to get a firmware update  . I assume, however, that industrious young engineers in low-income countries are aware of this kind of thing- is this type of security more a deterrent to eat into the profit margin of the would-be cloner just enough to make it not worth the trouble? Or is this harder than it sounds?

3) Is there any way of securely delivering PIC firmware to an end user that would allow them to flash the PIC themselves with a standard programmer, without risking every mook being able to decompile it?

4) The mfr. mentioned that they could not let users update firmware themselves without changing the CPU on the circuit. Does this imply that there are MCUs that are significantly more secure than a PIC?

5) Related to 3&4, how is it that most automation/industrial process control mfrs seem to be able to offer firmware downloads online without worrying about clones popping up on ebay, and without needing this "Encrypted Firmware Must Be Uploaded By Custom Programmer In Secure Facility" technique? Or am I wrong and this firmware-upgrade-by-snailmail thing is actually more common than I imagine?

[andersm]

If it's such a critical component you probably have a set of spares you can send off to get updated, cycle in and then send the other set to get updated as well? Or, arrange with the manufacturer to send you a replacement before you send yours in for updates.

[subunit]

If it's such a critical component you probably have a set of spares you can send off to get updated, cycle in and then send the other set to get updated as well? Or, arrange with the manufacturer to send you a replacement before you send yours in for updates.

Spares are an option, yes. For the industrial kit, at least, it would eliminate most of the cost savings vs. a more expensive manufacturer that offers in situ upgrades, though, which is one of the reasons I found it so weird. I think the company is too small to float replacements, unfortunately. This may be paranoia, but for sensor gear I would also prefer to keep the exact same hardware from experiment to experiment.

[woody]

3) can be achieved with a combination of on-chip protection and a boot loader that handles encrypted files. You can't copy the bootloader due to the protection bits. You can't decompile the firmware files due to the encryption so they can be freely distributed.

[NivagSwerdna]

After the rep mentioned that a "custom programmer" was required to flash the PIC (18F25K22- normal 8bit PIC), I pressed a bit and they mentioned that the PICs are programmed with encrypted hex files and that their programmer does some of this decryption.

Snake oil? Sounds hackable.  The decryption needs to be inside the PIC in the form of a decrypting boot loader to have any chance of being secure.

[Jeroen3]

You must evaluate if your effors are worth it compared to the 1000 dollar "read protected chip" services available.
Besides, any decryption running on a PIC isn't that complicated.

You might still want to sign the image, so that the bootloader doesn't flash images not from you.

[subunit]

3) can be achieved with a combination of on-chip protection and a boot loader that handles encrypted files. You can't copy the bootloader due to the protection bits. You can't decompile the firmware files due to the encryption so they can be freely distributed.

After the rep mentioned that a "custom programmer" was required to flash the PIC (18F25K22- normal 8bit PIC), I pressed a bit and they mentioned that the PICs are programmed with encrypted hex files and that their programmer does some of this decryption.

Snake oil? Sounds hackable.  The decryption needs to be inside the PIC in the form of a decrypting boot loader to have any chance of being secure.

If I'm understanding this correctly, the manufacturer could have used (or may currently be using) a decrypting boot loader, which would have enabled them to distribute the encrypted hex files without fear. I wonder if the "custom programmer" is just FUD to discourage customers from poking around? If the combination of protection bits and the decrypting bootloader offers good security on a PIC, it seems like a strange additional measure to take, particularly when it results in this kind of issue when you need to push firmware updates. My support thread seems to have been escalated to the CEO (small company), so I may press this point a bit to see what they say.

[Ian.M]

If they didn't design in a decrypting bootloader early in the design process they may no longer have enough memory to add one without replacing the MCU with a more powerful one. 

Also their custom 'programmer' may not be entirely FUD.  The PIC18F25K22's USART2 shares pins with ICSP, so its actually quite easy to implement a serial bootloader over the PGC and PGD ICSP pins.   Even if it didn't have a USART on the ICSP port, a bit-banged bootloader could be written to communicate via those pins.

[subunit]

If they didn't design in a decrypting bootloader early in the design process they may no longer have enough memory to add one without replacing the MCU with a more powerful one. 

Also their custom 'programmer' may not be entirely FUD.  The PIC18F25K22's USART2 shares pins with ICSP, so its actually quite easy to implement a serial bootloader over the PGC and PGD ICSP pins.   Even if it didn't have a USART on the ICSP port, a bit-banged bootloader could be written to communicate via those pins.

I think you may be right. As I'm reading up on bootloader encryption handling on PICs, it looks like the size of the bootloader in memory is a significant constraint, particularly if you want strong encryption. This is making more sense to me now- the company started out, I believe, making very low-cost circuits to handle sensor data for robots and other hobbyist/prototyping automation projects. It may have made sense to them not to have user-updateable firmware when those circuits were <$10. They may not have wanted to redevelop the product as they broached less disposable markets, but by the time you get to industrial transmitters in the hundreds of dollars it stops making as much sense.

[NorthGuy]

What are they selling? A PCB? What is on it? Can you post a good picture? Perhaps they provide schematics? In many cases, figuring out what is on PCB and writing your own firmware would be the best solution.

Hacking may not go as well as you expect. Old PIC18s had security holes, but I don't think K22 is one of these. So you will have to find holes in their security. At best, this requires way more skills and more work than re-writing the firmware. And even if you hack, you'll get the old firmware, not the updated one. What's the point?

Not to mention that writing your own software is perfectly legal while hacking is not.

[rstofer]

This may be paranoia, but for sensor gear I would also prefer to keep the exact same hardware from experiment to experiment.

Then you certainly don't want to upgrade the firmware and start from scratch.

What does the controller actually do?  It is far easier to duplicate the functionality than it is to try to extract the code.

[CJay]

Being a nasty, cynical type of person, I suspect that what the rep is telling you is circumstantial rumour and poppycock (CRAP) and he's trying to keep the firmware out of the hands of an end user because they think someone is going to rip off their IP or clone their product.

Would make sense to have a few spares and use them while the old FW versions are off for updating until you have a full set of up to date firmware.

It might be an idea to point them at http://www.flexipanel.com/TEAclipper.htm if they want to be able to do secure firmware updates

[cv007]

It sounds to me like they simply have the code protection bits set, and keep their firmware to themselves.

If one wants to protect firmware, this is the simple option- set the code protection bits (which requires a flash erase before programming), and don't let the firmware out in any way. Which of course requires also not letting anyone else program the pic.  Send in the pic, pic gets programmed, pic sent back. User never gets their hands on firmware in any way.

Any talk of encryption on their part makes no sense (sounds good, though). The only reason to go through the trouble of having an encryption scheme of any kind, is to be able to let someone else flash the firmware (via bootloader) without divulging the code. Since they require the pic sent in to reprogram, they have no need for encryption of any kind. I'm not sure what the purpose would be to hide behind 'encryption' talk.

[Bruce Abbott]

I find the idea that someone should dismantle a control cabinet for an active production line in a commercial context, and mail the components back to the mfr, to get a firmware update, mind boggling.

What I find mind-boggling is a manufacturer exposing their product to every hacker out there in the name of 'convenience'.

Indestructible malware

The chip’s program is called a firmware and a hard drive vendor may want to update it, thus correcting discovered errors or improving performance.

This mechanism got abused by the Equation group, which was able to download its own firmware to the hard drive of 12 different “categories” (vendors/variations). Functions of this modified firmware remain unknown, but malware on the computer obtains the ability to write and read data to/from the dedicated hard drive area... The data in this area may survive hard drive reformatting, plus firmware is theoretically able to reinfect hard drive’s boot area, infecting a newly installed operating system from the very beginning. To complicate things further, firmware checks and reprogramming rely on firmware itself, so it’s not possible to verify firmware integrity or reliably reupload firmware on a computer. In other words, once infected, hard drive firmware is indetectable and almost indestructible.

[subunit]

What are they selling? A PCB? What is on it? Can you post a good picture? Perhaps they provide schematics? In many cases, figuring out what is on PCB and writing your own firmware would be the best solution.

Hacking may not go as well as you expect. Old PIC18s had security holes, but I don't think K22 is one of these. So you will have to find holes in their security. At best, this requires way more skills and more work than re-writing the firmware. And even if you hack, you'll get the old firmware, not the updated one. What's the point?

Not to mention that writing your own software is perfectly legal while hacking is not.

Circuits are fairly simple PCBs with not much more than the PIC and what appear to be a couple of voltage regulators and some SMD resistors and diodes. They read probe voltages, compare to some calibrated function, and write measurement strings for I2C/serial transmission. I would post a pic but I don't really want to single out the mfr. since I still have to deal with them for the time being. It's almost certainly possible to write some firmware to replicate the functionality. If I was going to go the open-source/low-cost route in the future I would probably do so- this experience is certainly making me think carefully about hardware choices for scale-up. By the time we get to the point of using 4-20mA transmitters, I would like to understand more clearly what a $1000 honeywell transmitter is doing between the probe and the 4-20mA line that a simple homebrewed MCU unit generally wouldn't. The more I do this stuff the more I realise that "I can probably do that cheaper.." is often a siren song that is inviting me to spend hundreds of hours realising why commercial manufacturers do things the way they do and why they charge what they charge 

To be clear, I definitely do not intend on hacking anything- I was just interested to see that it was possible and wondering whether the on-chip security is really effective in this day and age. It seems like, generally, it is.

Being a nasty, cynical type of person, I suspect that what the rep is telling you is circumstantial rumour and poppycock (CRAP) and he's trying to keep the firmware out of the hands of an end user because they think someone is going to rip off their IP or clone their product.

Any talk of encryption on their part makes no sense (sounds good, though). The only reason to go through the trouble of having an encryption scheme of any kind, is to be able to let someone else flash the firmware (via bootloader) without divulging the code. Since they require the pic sent in to reprogram, they have no need for encryption of any kind. I'm not sure what the purpose would be to hide behind 'encryption' talk.

My feeling is that the manufacturer is in a precarious position- they are selling MCU-based sensor hardware that significantly undercuts typical scientific & industrial manufacturers, but which could easily and cheaply be knocked off. I don't think they manufacture the probes, so the main thing they have going for them are their cheap circuits which mean people like me don't have to shell out hundreds for rackmount devices to run sensors in prototypes and benchtop applications. Perhaps the paranoia is justified.

It might be an idea to point them at http://www.flexipanel.com/TEAclipper.htm if they want to be able to do secure firmware updates

This is useful, thanks, I will do.

What I find mind-boggling is a manufacturer exposing their product to every hacker out there in the name of 'convenience'.

Indestructible malware

Is the link you provided something that's possible on typical MCUs? In the case of a PIC, does the "firmware... reprogramming rely on the firmware itself", or the programmer? I think the SCADA system is a much more likely hacking target than individual pieces of control hardware. In any case, I don't think it's merely 'convenience' to require in-situ upgrades to hardware in a production context- from my limited research it does seem like this type of service is much more common industrially, though I am open to correction here.

[subunit]

Circuits are fairly simple PCBs with not much more than the PIC and what appear to be a couple of voltage regulators and some SMD resistors and diodes. They read probe voltages, compare to some calibrated function, and write measurement strings for I2C/serial transmission... It's almost certainly possible to write some firmware to replicate the functionality.

And already I have learned something- I am wrong about the simplicity of the circuit. Mfr. has a patented method to use a DAC, ADC, and some amplifiers to generate a signal from the probe that an MCU can easily read without introducing a lot of error from an analog frontend. I guess typical circuits are much more complex than what these guys were able to do, in order to deal with the typical ~-200 to +200 mV output range of the probe. I would need to learn a lot more about signal processing before I could write any firmware that does anything like this.

[NorthGuy]

Circuits are fairly simple PCBs with not much more than the PIC and what appear to be a couple of voltage regulators and some SMD resistors and diodes. They read probe voltages, compare to some calibrated function, and write measurement strings for I2C/serial transmission. I would post a pic but I don't really want to single out the mfr. since I still have to deal with them for the time being. It's almost certainly possible to write some firmware to replicate the functionality. If I was going to go the open-source/low-cost route in the future I would probably do so- this experience is certainly making me think carefully about hardware choices for scale-up. By the time we get to the point of using 4-20mA transmitters, I would like to understand more clearly what a $1000 honeywell transmitter is doing between the probe and the 4-20mA line that a simple homebrewed MCU unit generally wouldn't. The more I do this stuff the more I realise that "I can probably do that cheaper.." is often a siren song that is inviting me to spend hundreds of hours realising why commercial manufacturers do things the way they do and why they charge what they charge 

This certainly doesn't sound hard to replicate in PIC, Arduino, or alike. Especially if you have problem with their firmware giving unstable results. If you understand the physical processes behind the sensor, there shouldn't be hard to fix. If you find it difficult to do it by yourself, find a university math or EE student. He'll help you. You publish a paper together. He'll be happy.

As to the security. Most likely the PIC memory is protected. In this state, you can reprogram it with a new program, but you cannot read the existing one. When you ship it to them, they just re-program the new firmware and protect it again. Very simple. No encryption used or needed. That's the reason they must keep the software in-house and cannot give it to you. They could build in a secure bootloader to let you re-program it by yourself, but this would be less secure and much more hassle for them.

[cv007]

Perhaps the paranoia is justified.

I'm sure it is, and not letting the firmware out in any form is the most secure way to do it.

they mentioned that the PICs are programmed with encrypted hex files

If that is actually true, most likely it is a way to protect the firmware inside the company- whoever writes the firmware only lets an encrypted version out (to be programmed)- any copies floating around will be encrypted. When Billy Bob quits and starts his own company, that encrypted hex file he has stashed away will be worthless (unless he also has the bootloader code).  If the person writing the firmware quits, well then you ... I don't know.

This is probably not necessary to write (you already know), but-
the pic (or most other micros) has to have valid code in its flash to run (not encrypted)
to program a pic (or others), the normal programming methods have no way of 'hiding' what is being programmed (valid code)
if someone wants to protect/hide what is being programmed, it has to get inside the pic (or others) in its encrypted form- which means a program has to already be inside (bootloader) which is capable of decrypting, then programming itself- valid/decrypted code is the end result, but no one can ever see the valid code as the decryption takes place inside the pic

The end result in any case is valid code in the pic, whether it was encrypted at one time or not.

[amyk]

You must evaluate if your effors are worth it compared to the 1000 dollar "read protected chip" services available.

It was around US$1k a few years ago for most common PICs when I last checked, but it seems the price has gone up a bit for the newer MCUs... $3349.09 as of this post.

[subunit]

they mentioned that the PICs are programmed with encrypted hex files

If that is actually true, most likely it is a way to protect the firmware inside the company- whoever writes the firmware only lets an encrypted version out (to be programmed)- any copies floating around will be encrypted. When Billy Bob quits and starts his own company, that encrypted hex file he has stashed away will be worthless (unless he also has the bootloader code).  If the person writing the firmware quits, well then you ... I don't know.

I hadn't considered internal security. The company is only a few people. I wonder if this says something about the "internal culture". In any case, this thread has got me thinking much more from the engineer's perspective with regard to all of this, and feeling much more charitable about what seemed to be an incomprehensible design decision, so thank you all for that.

You must evaluate if your effors are worth it compared to the 1000 dollar "read protected chip" services available.

It was around US$1k a few years ago for most common PICs when I last checked, but it seems the price has gone up a bit for the newer MCUs... $3349.09 as of this post.

This is interesting. I didn't understand what Jeroen's post was talking about, but now I see that there are overseas services that will read a protected chip. I wonder what the legal status of these services are- an EE friend once told me that some teardowns are conducted aboard ships in international waters, I don't know how serious he was, though. My Chinese is not very good, I can't understand what they're saying they'll do- I see an interesting-looking light microscope rig, so I am guessing they physically open the package and probe it with some very finely drawn wires?

[amyk]

This is interesting. I didn't understand what Jeroen's post was talking about, but now I see that there are overseas services that will read a protected chip. I wonder what the legal status of these services are- an EE friend once told me that some teardowns are conducted aboard ships in international waters, I don't know how serious he was, though. My Chinese is not very good, I can't understand what they're saying they'll do- I see an interesting-looking light microscope rig, so I am guessing they physically open the package and probe it with some very finely drawn wires?

FIB = Focused Ion Beam. They actually patch the circuitry on the die using that.

[hans]

1) If the lock bits are set, the only thing you can do is erase the complete chip, including any EEPROM settings, bootloader etc.
Bootloaders only partially program the part, so if you have the firmware image you also need to know which parts need to be reprogrammed. This could be called 'custom programmer' (e.g. settings in a PICKIT tool) or handled internally by a bootloader.


2 to 5: if I needed to do secure firmware upgrades, I would write a bootloader with a symmetric-key cipher like AES CBC (which a PIC18F can do - just), keep a secret key in the bootloader, and decrypt any incoming firmware data with that key/cipher. Of course the controller is read protected.
The firmware images can be put up on the website without protections, because they are worthless without the right key which is kept private to the engineers and the bootloader. Customers can upgrade firmware via the bootloader and e.g. a 10$ USB serial cable. This is in favour of both parties - customers can upgrade without shipments and long downtimes, vendor has protected IP.

But even then, I would not always go to so much effort. If your product IP is primarily in the hardware (e.g. you're doing some fancy analog measurements at 24-bits), and the PIC is only banging out a SPI protocol for an ADC, doing some simple calibratoin math, and spewing out the results to a PC.. what's the point in investing so many man-hours into an encrypted bootloader which can be undone for a few k$?

Nevertheless, not offering on-site programming is a PITA and is a major overthought in any modern product though.

[subunit]

1) If the lock bits are set, the only thing you can do is erase the complete chip, including any EEPROM settings, bootloader etc.
Bootloaders only partially program the part, so if you have the firmware image you also need to know which parts need to be reprogrammed. This could be called 'custom programmer' (e.g. settings in a PICKIT tool) or handled internally by a bootloader.


2 to 5: if I needed to do secure firmware upgrades, I would write a bootloader with a symmetric-key cipher like AES CBC (which a PIC18F can do - just), keep a secret key in the bootloader, and decrypt any incoming firmware data with that key/cipher. Of course the controller is read protected.
The firmware images can be put up on the website without protections, because they are worthless without the right key which is kept private to the engineers and the bootloader. Customers can upgrade firmware via the bootloader and e.g. a 10$ USB serial cable. This is in favour of both parties - customers can upgrade without shipments and long downtimes, vendor has protected IP.

But even then, I would not always go to so much effort. If your product IP is primarily in the hardware (e.g. you're doing some fancy analog measurements at 24-bits), and the PIC is only banging out a SPI protocol for an ADC, doing some simple calibratoin math, and spewing out the results to a PC.. what's the point in investing so many man-hours into an encrypted bootloader which can be undone for a few k$?

Nevertheless, not offering on-site programming is a PITA and is a major overthought in any modern product though.

Very concise answer to my questions, thanks. The point about the IP being primarily in the hardware, so why invest so much in an encrypted bootloader is a good one!

FIB = Focused Ion Beam. They actually patch the circuitry on the die using that.

FIB works on old protection technology where protection fuse is on top or second from top metal layer where they can just remove some passivation and deposit gallium metal to rebuild the fuse in order to read data.

This is neat stuff, thanks. I have done some TEM work and thought one of those setups looked kind of like an SEM rig- did not know there was such a thing as a gallium ion scope. Looks like it even have some biological imaging applications!! This thread has been very educational