A general Q on certificates for an embedded HTTPS server

[peter-h]

I know it is generally dumb to run a server on an IOT box but I am documenting something and want to get it right

AIUI each box needs an individual certificate, so in practice these would be machine-generated (at factory test) and self-signed.

Increasingly, browsers complain about self signed certificates, and there is a view out there that one day the big ones e.g. chrome will refuse altogether.

I am told one can use a "proper" certificate but only if all servers shipped are on the same subdomain.

Is this correct, and/or is there some way around this?

[Nominal Animal]

Create a certificate authority (CA) key and certificate, and use that to sign individual device certificates.
Tell users to install the CA key by downloading the certificate (which is public, does not contain the private key) using their browser.  (The CA certificate MIME type does have to be application/x-x509-ca-cert, assuming the standard x509 format is used.)
The browsers then trust all certificates signed using that CA key (for the duration for which both the server and CA certificates are valid), regardless of the domain.
(This is how you buy valid certificates also, from any CA.)

You can also just install a self-signed certificate (by downloading it as a file, MIME type application/x-x509-user-cert, before it is used).

Another option is to use Let's Encrypt -signed certificates.  Let's Encrypt is a nonprofit Certificate Authority supported by all major (and minor) browsers and OSes.  (For example, my own "front page" is signed using Let's Encrypt -signed certificate.)  Your stack then has to basically include an ACME agent; see How It Works - Let's Encrypt.

[ataradov]

Certificate may include multiple domains (in the Subject Alternative Name field), but it is a limited list. No real CA would issue a broad certificate.

But I don't understand how would it work. Generally by IOT we understand something that connects to the home network and gets an IP from the router or something like this. There won't be a DNS entry unless you also configure it.

And there is no way to get a cert for just an IP. You can add an IP address to SAN field, but the primary name must be a domain. But that still does not help because IP is variable and local, and it is not possible to issue a certificate for a local IP.

Another issue that will come up is that there is a push to reduce certificate validity times. So, fixed certificates are going to be outdated by the time product gets to the customer.

For devices that can rely on external infrastructure a way to solve this is custom dynamic DNS and certificate issued to the domain of the DDNS. But this required you to maintain DDNS infrastructure, which is obviously a pain.

Things like asking users to install certificates and using Let's Encrypt create a bit of friction for very simple devices. If it is a smart light bulb, you would not expect a user to do this.

If it is an actual server (like a media server or something like this), then it is a viable option, as users generally should expect more maintenance.

But even then, Let's encrypt needs a domain name you control, which is not realistic for simple devices.

Here is a good description of the DDNS approach https://words.filippo.io/how-plex-is-doing-https-for-all-its-users/  It only works if you are willing to maintain that DDNS infrastructure, which makes sense for Plex, but not for small IOT devices.

[peter-h]

At a basic level, it would work same as any server works.

You rent a virtual server and you get an IP with it.

You then set up DNS pointing to that IP (usually you get a DNS control panel at the outfit from which you bought the domain).

If that server is your IOT box, and you have an ADSL service with a fixed IP (easy; I have this at home and at work) you open port 80 or 443 in your ADSL router for inbound traffic, and hey presto your IOT box is now on the internet. A friend has just done this with a little board running linux. Insecure as hell for many reasons e.g. every port 80 or 443 packet will be travelling along your home LAN, unless the router supports traffic separation between its multiple RJ45s, but that's a different debate.

But I am sure you knew all that.

I think you confirm that this proposal is not viable because a cert is only for a domain (not an IP).

However, I think whoever said to me that the subdomain method might work (to enable a CA-signed cert to be used with say 1000 IOT boxes) may have been right. Probably it would be done because in a typical DNS control panel you can specify subdomains like

fred001.yourcompany.com 12.13.14.15
fred002.yourcompany.com 123.213.44.15
fred003.yourcompany.com 124.213.54.151
fred004.yourcompany.com 129.113.74.215

and in fact I have this in my DNS control panel already.

Then if you buy a CA cert for yourcompany.com it should work, no?

Whether you can stuff 1000 entries into a DNS control panel is a different thing. I had a dig around and one company limits it at 50, while Cloudflare limits it to 3500 (!) on their higher paid plans.

DDNS I used many years ago and it was a total PITA.

[ataradov]

There is a huge distinction between a typical "consumer" IoT stuff, like light bulbs and smart tea kettles, and actual servers, even if small. If the appliance is installed in an environment where it gets assigned a fixed IP, which is also resolved in a DNS, then it is pretty easy. But in that case this all you need to do is provide a way to upload a certificate and it would be on the user to obtain it.

And yes, you can have wildcard certificate (*.yourcompany.com) valid for all hosts within the same domain. It may be better to have a sub-domain for all those devices though (*.gizmo.yourcompany.com).

[dobsonr741]

You can have a DNS service for any IP by: https://nip.io

Not recommended for production, only for experiments. In this day and age IPs not expected to be static, either.

[ataradov]

You can have a DNS service for any IP by: https://nip.io

This is basically what Plex did, but with their own server. It is not exactly DDNS, it is a DNS that resolved into IP in the name.

It is a cool idea, but it will not work for the purpose of this topic with public server. No CA will issue you a certificate for nip.io.

Although they mention that it somehow can work with Let's Encrypt. I guess LE would work for individual sub-domain of nip.io.

And their claim that you can stop editing /etc/hosts is strange. If you are fine with domain that still look like IPs, then sure, but I actually want readable names for things.

[dobsonr741]

Let’s encrypt requires very frequent renewals, not suitable for an embedded device.

I can see an internal CA issued cart can work, but again, I see this as a match only for a lab environment/experiment, nothing else.

[ataradov]

Frequent renewals are a reality for most CAs, so the device must support ACME protocol and request the certificates as they expire. Google pushing for universal 90 day expiry. They are not successful yet, but they are persistent.

[golden_labels]

Unless this is a closed network in an organisation,⁽¹⁾ never ask users to install a custom CA. The request is identical to asking for dropping TLS for entire browser traffic. Even if you do not care about security and your users, you are still putting yourself at risk of major browser or antivirus software simply blacklisting your CA as soon as your actions become widespread enough to be noticed.

What are solutions?

To use TLS as described, each device must have its own, unique domain name. That is because certificates authenticate against the domain. Are you willing to ship each your device with a second-level domain registered? Will you pay for renewal each year? If you do not and all devices share the same domain name, you may as well ship them with the same certificate. Having different certificates for a single domain spread across different devices provides no additional security.⁽²⁾

Alternatively, are the users expected to deliver their own domain names? If yes, Let’s Encrypt may be used. On-device automated system does that over ACME, without user’s interaction. The user must just configure the domain name and their email address.

If you are planning to use a centralized system with a third-level domain and unique certificates for it, device’s operation is bound to your service anyway. In such a situation you may as well make users connect to your central server, which will manage the devices without using public certificates (or not using HTTPS at all).

In either case, if this is a commercial product, you may buy certificates in bulk from major CAs. Wholesale prices for business are a story different from single-certificate the offer.

Finally, consider if the target audience needs to use a browser. Perhaps SSH is a better alternative? Then it becomes TOFU. While TOFU itself in general offers poor security,⁽³⁾ it is perfectly fine if the initial use is done as a configuration step within a trusted network.

Also, if having a hardcoded certificate in the device crossed your mind, remember the user must have the ability to force a certificate renewal in some way from the device. This may be necessary for many reasons; examples: periodic renewal went wrong, a secrets-leaking vulnerability in your firmware, a bit flip in storage, a certificate up in the chain getting retired/banned, or dropping an old algorithm by the browsers.

⁽¹⁾ In that scenario the organisation is in the control of the infrastructure and trust policies are their own choices. Though that also means the organisation should install certificates without user interaction.
⁽²⁾ If you want to issue a certificate against a particular local IP address, if any browser still supports that (no idea), it is the same situation.
⁽³⁾ Better than zero security, but but poor relatively to what is provided by things done properly.

[dobsonr741]

Even Unifi/Ubiquity is using self signed certs, for the IP address the Unifi controller was getting at installation time. It's a viable tradeoff.

An alternative you create a management cloud portal, where the users are going with their browsers. Then some communication protocol, between the IOT device and the portal handles what needs to be done on the device.

[Nominal Animal]

Unless this is a closed network in an organisation,⁽¹⁾ never ask users to install a custom CA. The request is identical to asking for dropping TLS for entire browser traffic. Even if you do not care about security and your users, you are still putting yourself at risk of major browser or antivirus software simply blacklisting your CA as soon as your actions become widespread enough to be noticed.

In reality, it is no different than trusting the 150+ CA organizations (and the tens of thousands of signing certificates issued by those) your browsers trust by default.  No, browsers won't blacklist such CA certificates unless they are misused.  After all, certificate mechanisms are completely based on trust in organizations keeping their CA keys securely secret and only signing certificates other CA authorities are willing to honor.

If you intend to sell a product intended to store other peoples stuff in a secure manner, you better be trustworthy anyway.  If being a trustworthy CA with associated obligations is too much of a burden, I do not think you should be selling storage appliances.


Thinking about the use cases a bit more, I do believe self-signed certificates that your appliance provides using a plain HTTP connection, say from http://appliance/certificate.crt, are most appropriate.

When served unsecured as application/x-x509-ca-cert (DER encoded), browsers will ask the user whether they want to install that certificate.  It is completely okay to serve certificates unsecured: if they are modified during transfer by a third party, they will no longer match the secret key anymore.

When a browser has installed a self-signed certificate, it will accept it as valid, and not complain about it being self-signed. This is the standard web appliance pattern.

If such an appliance has a second secret, say a salted hash of a password required to update the certificate (and associated secret key), it can provide a web form (over a secure connection, using a default/random certificate) with JavaScript that generates a new key and certificate, based on details supplied by the user.  (You can find such JS code online.)
The RSA PK calculations needed to generate a new self-signed certificate involve several orders of magnitude more computation than a single TLS handshake, so it is typically not feasible to do that on the IoT appliance itself.

Similarly, possibly even on the same form, you could define/generate a binary secret that is used to encrypt the actual storage.  That way, the end user would be in full control of their data and security.

[peter-h]

Frequent renewals are a reality for most CAs, so the device must support ACME protocol and request the certificates as they expire. Google pushing for universal 90 day expiry. They are not successful yet, but they are persistent.

This frequently fails even on "proper" servers. Repeatedly fixing that CRON job represents a fair chunk of the cost of the "IT man" who costs you a few k a year even if there is nothing to do

But this is a different argument to expecting future browsers to reject old certificates. Some of what is in cacert.pem is ~ 20 years old.

An alternative you create a management cloud portal, where the users are going with their browsers. Then some communication protocol, between the IOT device and the portal handles what needs to be done on the device.

Hmmm, yes.

This whole scheme is hard to implements unless you - the manufacturer - is deploying the hypothetical 1000 boxes in the field. You load the certificates into them in production, with some script incrementing the subdomain string. Then you install them yourself. Then you collect their (fixed) IPs, batch-load them into your DNS control panel (you aren't going to give your customer(s) access to that panel).

Actually each of them could upload the (fixed) IP it is on to a server you own, after first power-up, or periodically, and then you could just sell them freely. You would need to have the subdomain on a sticker on the box. And set up the DNS before the customer can access it. Various other details.

But clearly a self-signed cert can be avoided.

Anything above this level of complexity is above my pay grade

[dobsonr741]

An alternative you create a management cloud portal, where the users are going with their browsers. Then some communication protocol, between the IOT device and the portal handles what needs to be done on the device.

My point was different - maybe I need to clarify. The cloud hosted portal is not to plant certificates. It it to frontend all user interactions with the IOT device. The "needs to be done" scenarios are configuring, settings, readouts, etc - the day to day use cases, all done from your browser to the cloud proxy and then down to the device over MQTT/SSH/HTTP whatever you can better control than the on device certificate. As an added benefit you enjoy a better managed tracking.

One examples of this model: Ecobee

[peter-h]

If I understand you right, what you are describing is the fairly standard IOT usage scenario.

The IOT box is a client, not a server, and all public interaction is via a server which the mfg owns, and which can be properly secured. This also provides a revenue stream for the mfg - you charge $2/month for the server

That is how it should be done, and it sidesteps practically all "IOT security" issues. The individual boxes are clients, dialling up this server (with TLS, and checking the server's private certificate).

But my original Q was re running an IOT box as a server. It's been alleged that the most common way - each box having the same self-signed cert - is going to stop working one day, at least with common browsers.

[dobsonr741]

Two reasons why I do not see the expiring self signed certificate a big issue:

1. Can be updated when a new update version of the firmware is installed. This can fly for a commercial application.
2. Even when expired, it won't stop the browser from working. It's one click away to trust an expired certificate. Good for a hobby project.

[peter-h]

It's one click away to trust an expired certificate. .

Currently, sure.

Good for a hobby project

For sure but commercially?

Some customers can be really anal. Most of them are clueless about security, too. They read stuff on the internet

[dobsonr741]

What is the concern - the expiration or the fact a dialog need to be clicked, even for trusting the self signed cert?

The expiration date can be 10 years out, beyond the life of today’s gadgets.

[peter-h]

The concern is the customer being worried about having to click on "site not trusted - sure you want to proceed". Or his customer, etc.

Most people have no clue what "https" is.

The expiration date can be 10 years out, beyond the life of today’s gadgets.

Cheap and nasty consumer gadgets, possibly. Industrially, nobody would accept that argument.

The route which seems to be taken today, for browser config etc purposes, is to use http on 192.168.0.1 or some such And has been for many years. I have not seen anything which actually solves this, and uses https.

[ataradov]

There is no good solution for stuff on a local network. All certificates have to expire, so you need to have a way to update them. If your target audience will be scared of that, then use HTTP and don't worry about it. Provide HTTPS as an option for people that can figure out how to get a certificate valid for their setup.

[dobsonr741]

An alternative to this is to develop a mobile app to setup the device, for android an iOS. It is the only way to set it up. So that you control the security aspects in the native app. MyQ did this.

[AndyBeez]

You may be familiar with "routerlogin.com"? This is the domain used on Netgear routers to access the web management page. Netgear routers are smart enough to recognise this domain as a request to their web server - and not do a forward. There is a certificate installed on the router to enable port 443 access, which has the necessary trust path to keep every web browser happy.

A solution for your rollout might be, purchase a dedicated domain name that describes your brand, say "login-acme.iot", purchase a server certificate for that domain name [from Let's Encrypt], and deploy the certificate with the firmware, or as a part of your licensing package. All you need is a 'static route' of some kind to divert "login-acme.iot" to which ever IP address the IoT device is assigned [not quite sure how that's done in your deployment though]
If you have more than one IoT on the same subnet, you might be able to deploy IoT devices with fixed 2LDs - ie "97e0a2d1f318.login-acme.iot" The certificate has a wild card. It's just the web browser security that you're worried about.

I might also suggest having a certificate update function built into the IoT devices management page/API.

 

[ejeffrey]

Create a certificate authority (CA) key and certificate, and use that to sign individual device certificates.
Tell users to install the CA key by downloading the certificate (which is public, does not contain the private key) using their browser.  (The CA certificate MIME type does have to be application/x-x509-ca-cert, assuming the standard x509 format is used.)
The browsers then trust all certificates signed using that CA key (for the duration for which both the server and CA certificates are valid), regardless of the domain.
(This is how you buy valid certificates also, from any CA.)

Don't do that. It will work but it's a massive, massive, massive security hole for your users.  trusted signing certificates are trusted for any domain not just the ones you want and I am not going to trust some random small IoT vendor to treat their custom CA certificate properly and  neither should anyone else.  The CA system is bad enough already, but asking users to install random certificates is like throwing gasoline and nuclear weapons on the fire.

[peter-h]

All certificates have to expire

Can't you set 99 years or some such?

a mobile app to setup the device, for android an iOS

That's a horrible job. You need an experienced coder to do it well, especially for android. And keep updating it. No; it has to be a web page, and a simple one. No CSS, just plain HTML.

"routerlogin.com"? This is the domain used on Netgear routers to access the web management page

I read some stuff about people having lots of problems with that method but don't recall the details.

I agree HTTP without any security should be the best way - because that is what almost everybody is using, and has been for years

[golden_labels]

Can't you set 99 years or some such?

It’s not about theoretical limits, but about what is going to be actually supported. According to CA/Browser Forum’s Basic Requirements section 6.3.2 the maximum validity period can’t exceed 398 days. Apple, Google and Mozilla do not like certs with outrageously long validity periods, and similar policies are expected from other vendors. The last time I checked manually added CAs were excempted in Firefox (organisations may need that for their internal chains), but no idea about the current status and other clients.

I read some stuff about people having lots of problems with that method but don't recall the details.

No idea, what the problem was. But the idea is broken and must fail, so an educated guess is possible. The method assumes the client uses DHCP-supplied DNS address, that it uses UDP port 53 DNS, and that DHCP is working correctly in the first place. Aside from that infringing user’s freedom, it is also a bunch of invalid assumptions from purely technical perspective. Starting with major browsers using DoH, not DNS on port 53.

I agree HTTP without any security should be the best way - because that is what almost everybody is using, and has been for years

People were also not washing food and accepting feudalism for thousands of years.

[Nominal Animal]

The last time I checked manually added CAs were excempted in Firefox (organisations may need that for their internal chains), but no idea about the current status and other clients.

Manually added CAs (for organizations), and manually added self-signed certificates (for internal appliances and IoT things; including Azure IoT things according to Microsoft) are so common that it is extremely unlikely any major browser vendor will blacklist or disable them.

Even in the case of an exploit adding them without user intervention, the actual proper use cases are so useful to end users and organizations, that the only really acceptable solution will be to make it impossible to "silently add" certificates; including warning/prompting the user if new additional certificates are detected at profile load time.