The cost of "security"

[peter-h]

I am working on what would today be fashionably called an "IOT" product.

After probably a man-year, plus loads of libs like ST ETH, LWIP, USB, etc (most of which were buggy as hell) the code size is 170k. Not bad for a 1MB FLASH CPU. The 128k RAM has 60k spare.

Now we add MbedTLS, so one can do HTTPS. Code goes up by another 150k and the 60k spare RAM is now 10k. The "Mbed" is a bit of a joke really

And what does it achieve? Very little, it seems. IOT boxes can't be on open ports, because these are discovered by sniffers almost immediately, due to back doors and vulnerabilities which cannot be fixed (a 100BN $ company like M$ is still patching back doors in Windoze after 30 years, so an embedded coder has absolutely zero chance, and then new ones will be discovered after the product is sold and installed somewhere) so their "internet access" can only be as a client, calling up some server, and hidden behind a NAT router.

But if you are calling up a server which you control, you don't need the whole TLS crap, with PK, session key negotiation, x509 authentication. You can just use a shared key, with AES256 or whatever. You have the key dist issue but with PK you still need 100% secure "access control" on each box because the private key has to be secure. You don't need auth because a fake server won't be able to read the data anyway.

And if the IOT box is to call up some "public service" server, the certificate for that will expire regularly, so you need the whole root certificate store, currently about 200k, and a means of periodically updating that, too In the context of embedded systems, it mostly can't be done practically, and even if it can be done, it will sooner or later break or be forgotten (even the biggest firms have had their website certificates expire because the person whose job that was has left, etc).

It seems to me that a whole industry has grown up around this "security, privacy, etc" stuff and most of it is wasted.

The other thing is that "IOT" is going to end up with a really dirty reputation, as companies go bust, or just stop maintaining the servers. So the remotely controlled irrigation system, where you have a nice app connecting to that server, or even just a website served by that server, goes dead one day and there is no way to fix it. You have to rip the whole lot out.

[tepalia02]

The last paragraph is really something to consider seriously. This is why all IoT devices should have the option of manual control also. Just like the sonoff IOT switches have manual on-off push buttons.

[peter-h]

Anyway one could also look at MQTT or COAP or YAML configuration files or RPC or TCP APIs or any number of other things.
Most test equipment speaks, say, SCPI, simple, usually works, does not require the cloud.

The thing is that no matter how you shake this, you cannot have IOT boxes on open ports.

Access has to be via a "cloud".

It has to be thus for vulnerability reasons (experience shows that one can keep a reasonably secure unix server running for years without any maintenance).

And it has to be thus for commercial reasons: selling boxes, you make money only once, so your market will saturate, and getting say $2/month for the "cloud" keeps the money coming in.

There already is a big business around the second bit, with e.g. AWS doing MQTT servers.

Which is actually beyond wrong

Not really; I was referring to the security overhead of the comms between IOT and the "cloud", where the security requirement isn't specially difficult.

[madires]

And what does it achieve? Very little, it seems. IOT boxes can't be on open ports, because these are discovered by sniffers almost immediately, due to back doors and vulnerabilities which cannot be fixed (a 100BN $ company like M$ is still patching back doors in Windoze after 30 years, so an embedded coder has absolutely zero chance, and then new ones will be discovered after the product is sold and installed somewhere) so their "internet access" can only be as a client, calling up some server, and hidden behind a NAT router.

Yep, exposing IoT devices directly to the public internet is a bad idea because of the many security issues they have. However, access within a separated local network is usually fine and often necessary for management reasons. If remote access via internet is needed a VPN connection to your router can help. I'd also like to add that NAT isn't a security feature - the stateful firefall is. It's a common misconception.

PS: I consider any IoT device which requires the cloud as e-junk.

[peter-h]

If remote access via internet is needed a VPN connection to your router can help

True, and then you can use plain old http, but in a commercial context expecting people to set up VPNs is a total loser. I have set up so many of these... it would be a customer support nightmare.

I'd also like to add that NAT isn't a security feature - the stateful firefall is. It's a common misconception.

Sure, but NAT is still very very good. The router normally closes the hole after 180 seconds.

I consider any IoT device which requires the cloud as e-junk.

How would you solve the hacking issue?

[madires]

5: If the IoT thingy is supposed to be a "client" that talks to a server, ok, well in that case I better see the open source "server" implementation, client-server API documentation, et. al. on the vendor's git hub or similar distribution so I can run a locally hosted and controlled server on whatever host I may want on whatever LAN I may want without any "internet" connectivity needed.

Absolutely! Or the device supports a well documented protocol.

[Kjelt]

If you have a small IOT device you should tweak the security library to your demands, which means deleting 75% of the library. Only keep what you need to support and yes you're task is to keep it safe, read all the new security vulnerabilities, update , etc. And indeed you might not want your device directly to the internet just as you probably also don't want your linux server there, unless you're a linux admin and love to keep that thing up to date etc.

[madires]

True, and then you can use plain old http, but in a commercial context expecting people to set up VPNs is a total loser. I have set up so many of these... it would be a customer support nightmare.

It has become easier with WireGuard, IMHO.

Sure, but NAT is still very very good. The router normally closes the hole after 180 seconds.

It's the connection timeout of the stateful firewall. Think IPv6!

I consider any IoT device which requires the cloud as e-junk.

How would you solve the hacking issue?

Could you please elaborate which hacking issue(s) you mean? When I run a device locally with local access only I don't have to fear any internet scanners. If the device connects to some cloud server additionally it's an outgoing connection (I'd assume the connection is protected, e.g. by TLS).

[madires]

It has to be thus for vulnerability reasons (experience shows that one can keep a reasonably secure unix server running for years without any maintenance).

This sounds more like wishful thinking. No matter what, as long as you have a server connected to the internet you have to spend some effort to keep it up-to-date (following security related mailing lists, updating ASAP and so on). You can automate some tasks, but it won't be auto-magical.

And it has to be thus for commercial reasons: selling boxes, you make money only once, so your market will saturate, and getting say $2/month for the "cloud" keeps the money coming in.

Exactly! This is the real motivation for the current state of our lovely cloud-based IoT devices. You'll own nothing and be happy.

[MadTux]

Microcontrollers are there to "control" things, not to connect every toaster to the internet!!!

Have fun patching all that unnecessary garbage and with having a hacked fridge that phones with FBI, NSA, CIA, FSB, Mosad, the Nigerian prince and god knows whom, I'll never buy such crap.

The only good thing that comes from that, that you can get quite fast microcontrollers with about endless memory/flash for very little cash nowadays. But all those resources are soon consumed with IOT garbage code.

[peter-h]

as long as you have a server connected to the internet you have to spend some effort to keep it up-to-date (following security related mailing lists, updating ASAP and so on). You can automate some tasks, but it won't be auto-magical.

Sure, but people do this all day long. The standard unix OS + server packages are way more robust than the stuff people use for embedded systems. This has to work otherwise the WWW would vanish

[janoc]

It has become easier with WireGuard, IMHO.

Completely missing the point, IMO. Try to get your Wireguard (or any other VPN solution) through some corporate IT. Good luck with that one.

Heck, even getting a single outgoing port authorized/opened on a corporate firewall can be a month-long fight of paperwork, meetings, more paperwork - and then the port will be randomly closed again whenever some IT grunt decides to restore/reload router configuration from a backup that has not been updated ... Been there, done that ...

And if the customer doesn't have obtuse/obstructing IT dept. they likely don't have the knowledge/skills in house to maintain a VPN neither. So it will be on your back to support them because it was you who has sold them the internet connected gizmo.

If the device is something intended for consumers then expecting someone who buys that proverbial internet-connected light bulb to know how to set up a VPN is just a complete non-starter.

It's the connection timeout of the stateful firewall. Think IPv6!

Not sure what IPv6 has with it at all  Also, IPv6 still doesn't work everywhere (or if it does then there is no guarantee of performance, throttling and random issues). In addition all sorts of networks are connected/tunneled in all sorts of weird and wonderful ways (such as carrier-grade NATs common on cell networks).

[peter-h]

Not my quote

[tellurium]

If my IoT refrigerator is talking to your "cloud" server (and whatever legion of absolutely unnecessary and undesired analytics, metrics, tracking, adware

For the big business, it is actually very necessary. It is actually a key. Many products got IoT capability just because of that, whilst user experience is not really important. Why? Because IoT gives a way to "spy" on the customer behavior after the sale. Collecting stats gives an ability to tune the functionality of the product to be most profitable. And that's what big businesses care about, and they can get creative with marketing convincing people that they need that shiny IoT stuff.

Big business drives big money, and the whole industry in the end. So whether we like it or not, that's what is going on.

[Doctorandus_P]

I don't understand what particular brand of crazy these people are that think it is remotely ethically or logically reasonable to sell people devices and yet give the OWNERs anything less than full control of them.

There are companies with an ethical conscience, but most don't, especially when they think about profits, the ethical section gets shoved under the carpet. This will only change when their customers demand it, but all too often those customers think about short time convenience and are not even interested in technical details. Back in the '80-ies computers were simple enough to be understood by a single person, but some 20+ years ago they became so complicated that there is simply nobody in this world who can understand how they work anymore. You can only know some parts in detail, the rest superficially, and then hope the parts work together properly.

1984 was supposed to be a warning, not an instruction manual.

Indeed.
I just do not have any of that iot stuff, and It won't enter my home either.

[tellurium]

But if you are calling up a server which you control, you don't need the whole TLS crap, with PK, session key negotiation, x509 authentication. You can just use a shared key, with AES256 or whatever. You have the key dist issue but with PK you still need 100% secure "access control" on each box because the private key has to be secure. You don't need auth because a fake server won't be able to read the data anyway.

Actually you do need the whole TLS crap.
Inventing your own security rather than using an industry standard (TLS) - is asking for trouble.
Running a server on a device, other than temporarily for setup - is asking for trouble.
Not using TLS for outgoing connection can be viable - depends on the use case. If that's for the firmware self-update, then it is asking for trouble. If that's for some sort of non-essential stats tracking - might be OK.

[Kjelt]

Actually you do need the whole TLS crap.

Not all, you can remove all the cipher suites you don't support, not only the broken old ones also the very heavy resource occupying ones.
You can even only support one light weight cipher suite, as long as it is safe.
That will safe you more than 50% of the TLS code, perhaps 80% if you play it right.

[nctnico]

But if you are calling up a server which you control, you don't need the whole TLS crap, with PK, session key negotiation, x509 authentication. You can just use a shared key, with AES256 or whatever. You have the key dist issue but with PK you still need 100% secure "access control" on each box because the private key has to be secure. You don't need auth because a fake server won't be able to read the data anyway.

Actually you do need the whole TLS crap.
Inventing your own security rather than using an industry standard (TLS) - is asking for trouble.

Yes and no. For a secure connection with pre-shared keys and using 1 encryption algorithm (some form of AES) you only need about 1% of the entire TLS stack. Been there, done that. Nobody is suggesting to come up with a new encryption method!

[james_s]

Some time back a colleague of mine quipped that "The S in IoT stands for Security"

[peter-h]

I would actually be quite interested in opinions on which cipher suites are worth supporting in a product launched say today but which has to be compatible, as an HTTPS client, with other old stuff.

As a client, probably old stuff doesn't matter much.

Much of the size of MbedTLS is cipher suites.

[madires]

It has become easier with WireGuard, IMHO.

Completely missing the point, IMO. Try to get your Wireguard (or any other VPN solution) through some corporate IT. Good luck with that one.

Do companies large enough to have an IT department deploy IoT devices which are controlled remotely via the internet? And how did they manage to deploy VPNs at large scale for employees working at home (COVID 19)? BTW, I didn't say that VPNs are easy, just that WireGuard makes things easier (vs. IPsec for example).

It's the connection timeout of the stateful firewall. Think IPv6!

Not sure what IPv6 has with it at all  Also, IPv6 still doesn't work everywhere (or if it does then there is no guarantee of performance, throttling and random issues). In addition all sorts of networks are connected/tunneled in all sorts of weird and wonderful ways (such as carrier-grade NATs common on cell networks).

IPv6 is meant to be used without NAT (It's available, and there's NAT64 as a transition method). This is what I was hinting at. I know, some telcos and companies are still living in the IP past. However, we have IPv6 for 20 years now and they all have to migrate to IPv6 sooner or later because of the lack of IPv4 address space.

[peter-h]

And how did they manage to deploy VPNs at large scale for employees working at home (COVID 19)? BTW, I didn't say that VPNs are easy, just that WireGuard makes things easier (vs. IPsec for example).

They did it by enforcing known and uniform hardware at both ends. Then you can deploy the same config to all. Same with OpenVPN - if you force everyone to use an Ipad then you can use the same config file for all.

because of the lack of IPv4 address space.

That was supposed to happen 20 years ago but since a way was found to have multiple domains on the same IP, the issue has gone away.

IPV6 also does not solve vulnerabilities because the same hacking can be done. For example every instance of win10 has its predefined IPV6 address but that just means hackers who know the ranges can target them, in the hope of discovering win10+ installations which are not behind NAT.

[SiliconWizard]

Anything connected to a non-local network is a huge mess to deal with. Stay away from that for as long as you can.

[capt bullshot]

Didn't read the other replies, so sorry if I repeat something:

IMNSHO, TLS on an embedded device is just pure humbug. It adds a lot of CPU load, and code / memory hogging to an otherwise rather simple device, just because "HTTPS everywhere" is kind of a hype pushed by "Internet security experts". They are right in many aspects, I have to admit, but not for a simple embedded device that isn't supposed to be exposed to the internet anyway. And if done so in spite of good advice, as long as there's no secret login credentials to protect, HTTPS is of no use here. If you need to protect your client from all these "man in the middle" and whatnot attacks that TLS / HTTPS is supposed to do, you need a lot more infrastructure than just a TLS lib on your device.
Remember, the original purpose of HTTPS was to protect the traffic from eavesdropping, not to securely identify and authenticate the server / client.
So my not so humble advice is: Drop TLS on that embedded device and put a proper reverse proxy in front of it that does all the TLS infrastructure stuff.

[peter-h]

If you need to protect your client from all these "man in the middle" and whatnot attacks that TLS / HTTPS is supposed to do, you need a lot more infrastructure than just a TLS lib on your device.

Ultimately, to authenticate, you need to have a physically secure device, because if somebody can install a fake DNS server on your LAN and get inside your box and change the certificates, everything will be just fine

Drop TLS on that embedded device and put a proper reverse proxy in front of it that does all the TLS infrastructure stuff.

What is a reverse proxy? Isn't that another server which somebody has to set up and admin, for ever?

[capt bullshot]

Drop TLS on that embedded device and put a proper reverse proxy in front of it that does all the TLS infrastructure stuff.

What is a reverse proxy? Isn't that another server which somebody has to set up and admin, for ever?

Yes, indeed.
I have to admit, it's kind of impracticable / impossible for a device intended to be used by "not how the internet works aware" end users.

[ve7xen]

And what does it achieve? Very little, it seems. IOT boxes can't be on open ports, because these are discovered by sniffers almost immediately, due to back doors and vulnerabilities which cannot be fixed (a 100BN $ company like M$ is still patching back doors in Windoze after 30 years, so an embedded coder has absolutely zero chance, and then new ones will be discovered after the product is sold and installed somewhere) so their "internet access" can only be as a client, calling up some server, and hidden behind a NAT router.

TLS wouldn't really help you here anyway, you're still talking about exposing a lightweight microcontroller to the Internet, and despite best intentions it's going to be too bespoke and too infrequently updated for that to be a good idea, so this isn't the angle.

It gets you a few things, whether you care or not about them, or whether they are the best solution to the problem is up to you and your product, and maybe your customers. As usual it's all tradeoffs.

• Server validation. You know you're talking to your server, signed by your CA. If you're a commercial product, the corollary is that users 'can't' reverse engineer your protocol and redirect your device's traffic to their own collector - helps you lock the devices to your service and prevent MITM
• Client identification/validation, if you choose to use it. You can generate a per-device keypair, signed with your CA, and store it in some secure enclave at manufacturing time. This can be used to securely identify the device
• The obvious benefits of encryption in flight against snooping.

I can see not caring about #1 and #3 (though your customers might), but #2 seems like a bigger problem to me. If this is a service at scale, you need some strong way to validate that client updates are associated with the correct account. You can't rely on some plaintext identifier that can be easily fuzzed or even that's printed on the box or easily predictable. Otherwise someone's probably going to attack it and start trying to spam bogus spoofed data into your client's instances.

But if you are calling up a server which you control, you don't need the whole TLS crap, with PK, session key negotiation, x509 authentication. You can just use a shared key, with AES256 or whatever. You have the key dist issue but with PK you still need 100% secure "access control" on each box because the private key has to be secure. You don't need auth because a fake server won't be able to read the data anyway.

Whether you control the server is orthogonal to whether you need to securely authenticate it or not. Shared keys ship with the box and will eventually get extracted, if history is any indication, and then you're hosed, so that's not really a strong solution.

And if the IOT box is to call up some "public service" server, the certificate for that will expire regularly, so you need the whole root certificate store, currently about 200k, and a means of periodically updating that, too In the context of embedded systems, it mostly can't be done practically, and even if it can be done, it will sooner or later break or be forgotten (even the biggest firms have had their website certificates expire because the person whose job that was has left, etc).

You probably wouldn't want to use the 'normal' PKI for such a thing, you'd spin up your own CA and issue long-lived certificates. You don't want to rely on the public trust store when you control both ends, and there's no reason for it here. Unless of course you want to connect to client-provided endpoints, in which case yeah you need to either include (and update) the normal trust store or allow users to install their own trust store (which might be beyond their abilities).

The other thing is that "IOT" is going to end up with a really dirty reputation, as companies go bust, or just stop maintaining the servers. So the remotely controlled irrigation system, where you have a nice app connecting to that server, or even just a website served by that server, goes dead one day and there is no way to fix it. You have to rip the whole lot out.

I'm not sure what this has to do with TLS, but I strongly agree. I'd like to see some industry consortium coalesce around something like MQTT + some service discovery mechanism to make interoperability mostly transparent for the standard home use case (a bunch of sensors and a 'base station' on the same network), and flexible enough to point elsewhere if desired.

Remember, the original purpose of HTTPS was to protect the traffic from eavesdropping, not to securely identify and authenticate the server / client.

I don't know where you get this from, secure identification of both the server and client is a key feature / raison d'etre of TLS and by extension HTTPS.

[madires]

because of the lack of IPv4 address space.

That was supposed to happen 20 years ago but since a way was found to have multiple domains on the same IP, the issue has gone away.

No, that happens when RIRs run out of IPv4 address space to assign to telcos/orgs. Please see https://en.wikipedia.org/wiki/IPv4_address_exhaustion for more details. And shared web hosting (running multiple websites on a single IP address) is supported since HTTP 1.1 and only helps with webservers. There are much more connected devices than just webservers.

IPV6 also does not solve vulnerabilities because the same hacking can be done. For example every instance of win10 has its predefined IPV6 address but that just means hackers who know the ranges can target them, in the hope of discovering win10+ installations which are not behind NAT.

Yes, IPv6 is more or less as secure or insecure as IPv4. But this isn't the point, it's the lack of IPv4 address space. Which predefined IPv6 address do you mean? Loopback, link-local , ULA, multicast or global unicast? And even public unicast doesn't mean that a criminal can simply access your device. Usually you would have a SOHO router with a stateful firewall in front of the device, besides the device coming with its own firewall/protection.

[capt bullshot]

Remember, the original purpose of HTTPS was to protect the traffic from eavesdropping, not to securely identify and authenticate the server / client.

I don't know where you get this from, secure identification of both the server and client is a key feature / raison d'etre of TLS and by extension HTTPS.

Hypertext Transfer Protocol Secure (HTTPS) is an extension of the Hypertext Transfer Protocol (HTTP). It is used for secure communication over a computer network, and is widely used on the Internet.[1][2] In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS) or, formerly, Secure Sockets Layer (SSL). The protocol is therefore also referred to as HTTP over TLS,[3] or HTTP over SSL.

The principal motivations for HTTPS are authentication of the accessed website, and protection of the privacy and integrity of the exchanged data while in transit. It protects against man-in-the-middle attacks, and the bidirectional encryption of communications between a client and server protects the communications against eavesdropping and tampering.[4][5] The authentication aspect of HTTPS requires a trusted third party to sign server-side digital certificates. This was historically an expensive operation, which meant fully authenticated HTTPS connections were usually found only on secured payment transaction services and other secured corporate information systems on the World Wide Web. In 2016, a campaign by the Electronic Frontier Foundation with the support of web browser developers led to the protocol becoming more prevalent.[6] HTTPS is now used more often by web users than the original non-secure HTTP, primarily to protect page authenticity on all types of websites; secure accounts; and to keep user communications, identity, and web browsing private.

So no client authentication and "expensive" server authentication requiring external infrastructure.
Yes, HTTPS (on SSL and later TLS) provided server authentication but due to the required infrastructure and because you'd have to trust a third party authority, it was (and is) often called "broken by design". So IMNSHO HTTPS is just good for preventing eavesdropping.
Inside a corporate network, as a stupid end user on a company provided computer you're usually not even protected from eavesdropping because the company proxy does a "man in the middle" on your HTTPS connections for "corporate security" reasons and normally you don't even notice that since the IT department silently supplies their "not trustworthy" certificates to your browser.

[ve7xen]

Hypertext Transfer Protocol Secure (HTTPS) is an extension of the Hypertext Transfer Protocol (HTTP). It is used for secure communication over a computer network, and is widely used on the Internet.[1][2] In HTTPS, the communication protocol is encrypted using Transport Layer Security (TLS) or, formerly, Secure Sockets Layer (SSL). The protocol is therefore also referred to as HTTP over TLS,[3] or HTTP over SSL.

Okay, agreed...

The principal motivations for HTTPS are authentication of the accessed website

When the first 'principal motivation' listed is exactly what I said, it's pretty clear your argument doesn't hold up here. Certificate validation has been done since day-0 of HTTPS support in browsers - it is a MUST in the RFC, and client authentication has likewise been supported for a looong time, though it's not widely used. This functionality is inherited from SSL/TLS.

Management of the trust is an entirely separate issue, and yes the typical setup is not that strong, but this isn't anything to do with HTTPS or TLS, but how we have decided to ship a default trust store with browsers / operating systems and allow any public entity to obtain a certificate. You can easily implement it in such a way that only explicit trust is allowed, which is what you would probably want to do in a closed system like an IoT platform, for several reasons.

[madires]

SSL client certs are usually not used for HTTPS connections. But it's a feature of SSL/TLS and some services are using this. It can be used as 2FA method, for example.

[capt bullshot]

The principal motivations for HTTPS are authentication of the accessed website

When the first 'principal motivation' listed is exactly what I said, it's pretty clear your argument doesn't hold up here. Certificate validation has been done since day-0 of HTTPS support in browsers - it is a MUST in the RFC, and client authentication has likewise been supported for a looong time, though it's not widely used. This functionality is inherited from SSL/TLS.

Management of the trust is an entirely separate issue, and yes the typical setup is not that strong, but this isn't anything to do with HTTPS or TLS, but how we have decided to ship a default trust store with browsers / operating systems and allow any public entity to obtain a certificate. You can easily implement it in such a way that only explicit trust is allowed, which is what you would probably want to do in a closed system like an IoT platform, for several reasons.

Remeber, early versions of most browsers didn't complain on invalid or self signed certificates. So yes, I agree, server authentication was a principal intention of HTTPS, but wasn't trustworthy at all in the past. And "marketing" of HTTPS in its early and middle age days didn't emphasize on authenticating the server but rather "protecting your secret password" and preventing "eavesdropping your secret email". Today, this has changed, but In my still not so humble opinion it's borked more than ever because you need all that fragile infrastructure to secure the trust.
Of course, you still can provide your own trusted certificates for your own servers and your own clients to protect them from the public.

[peter-h]

Shared keys ship with the box and will eventually get extracted, if history is any indication, and then you're hosed, so that's not really a strong solution.

However, isn't that the same as if your box contains the certificate of your private server an somebody can replace that certificate...

No physical security = no security

early versions of most browsers didn't complain on invalid or self signed certificates

I think this is another reason for "HTTPS everywhere". Browsers warn your customers that they are going to die. But if you have a box which is a server, the certificate in that box will have to be self-signed, won't it?

Currently, it is still possible to config routers over HTTP locally, and remotely over HTTPS but the certificates there are IME always self signed.

The box I am working on has an HTTP server for LAN config etc, which will enable "secure" remote config via a VPN, if somebody really wants it. But the proper marketable way to do remote config/control is via a server and this server has to be maintained for the life of the product, which is a very tall order. Hence the HTTP server option.

[capt bullshot]

I think this is another reason for "HTTPS everywhere". Browsers warn your customers that they are going to die. But if you have a box which is a server, the certificate in that box will have to be self-signed, won't it?

A self signed cert makes your server unusable for most customers for said reasons, browsers make accepting such certs more and more complicated. You can get free and valid certs e.g. from Let's Encrypt, by they expire fast, so you'll have to make your server able to automatically renew the certs, and they have to be unique for each server then. Pretty elaborate  for an embedded device based on a typical MCU of today.

Currently, it is still possible to config routers over HTTP locally, and remotely over HTTPS but the certificates there are IME always self signed.

No, e.g. the FritzBox that is very common in Germany optionally uses Let's Encrypt certs. But they run embedded linux, so it's not too elaborate to renew them etc.

The box I am working on has an HTTP server for LAN config etc, which will enable "secure" remote config via a VPN, if somebody really wants it. But the proper marketable way to do remote config/control is via a server and this server has to be maintained for the life of the product, which is a very tall order. Hence the HTTP server option.

Agreed.
VPN or said reverse proxy that can do all the authentication and encryption stuff for you.
Before all that "cloud" hype, specialized companies sold VPN and/or reverse proxy boxes to connect between your device and the customers internet. Maybe they still exist in some niches.

All that "security" stuff has to be maintained to follow the latest "standards" or just trends. Otherwise it will be rendered obsolete by the strong opinion of the "security experts". Nothing wrong with these people, they're right in many aspects, but often they just don't understand it's pretty much impossible to maintain an embedded "server" according to their latest advice. So one needs some external means to stay secure here anyway.

[SiliconWizard]

If you want each of your devices to act as a server directly accessible through HTTPS and you can't use a self-signed certificate, it's going to cost you uh....

[ve7xen]

However, isn't that the same as if your box contains the certificate of your private server an somebody can replace that certificate...

No physical security = no security

The difference being it's trivial to revoke the compromised certificate without affecting the rest of the products in the field. The other difference is that the client certificate uniquely identifies a device (in the use case I describe, anyway), so exposure of one certificate also doesn't mean they have carte blanche to write data with assumed identity.

early versions of most browsers didn't complain on invalid or self signed certificates

I'd kinda need to see a reference for this claim, I know the idea of shipping a trust store goes back at least as far as Mozilla 1.0, so 20+ years, and considering SSL itself came out of Netscape in the prior era, with all the features I'm talking about in place, I'm pretty sure this was their intent. Not really relevant today, anyway.

I think this is another reason for "HTTPS everywhere". Browsers warn your customers that they are going to die. But if you have a box which is a server, the certificate in that box will have to be self-signed, won't it?

Currently, it is still possible to config routers over HTTP locally, and remotely over HTTPS but the certificates there are IME always self signed.

Yes, if the box itself is a server, you'd either need it to be self-signed, or signed by a CA that you control. It's not going to be economical or sensible to ship certificates that allow default trust on common platforms, and doesn't really add any value, which is why you usually see them using self-signed certificates here.

In what I thought you were talking about - my bad if I misinterpreted - where the client is the device reaching out to a server on the Internet, you can establish your own CA, and then make that the (only) trusted CA in your IoT device's trust store. A browser connecting to your API service is going to throw a security warning due to lack of trust, but your device will trust this certificate and work fine.

The box I am working on has an HTTP server for LAN config etc, which will enable "secure" remote config via a VPN, if somebody really wants it. But the proper marketable way to do remote config/control is via a server and this server has to be maintained for the life of the product, which is a very tall order. Hence the HTTP server option.

Yeah in this case HTTPS for local configuration is not adding much value, at least as long as users aren't going to be expected to want to expose it publicly (which is a terrible idea, TLS or not).

[capt bullshot]

However, isn't that the same as if your box contains the certificate of your private server an somebody can replace that certificate...

No physical security = no security

The difference being it's trivial to revoke the compromised certificate without affecting the rest of the products in the field. The other difference is that the client certificate uniquely identifies a device (in the use case I describe, anyway), so exposure of one certificate also doesn't mean they have carte blanche to write data with assumed identity.

In theory, yes. But revoking an individual certificate relies on reaching the critical participants with the revoke requests. If your infrastructure doesn't provide for this in a reliable way, nothing gets revoked.

[ve7xen]

In theory, yes. But revoking an individual certificate relies on reaching all parties with the revoke requests. If your infrastructure doesn't provide for this in a reliable way, nothing gets revoked.

My comments here are context-aware. We are talking about a closed system with a self-managed PKI for an IoT platform. This is trivial to accomplish in that setting.

[capt bullshot]

early versions of most browsers didn't complain on invalid or self signed certificates

I'd kinda need to see a reference for this claim, I know the idea of shipping a trust store goes back at least as far as Mozilla 1.0, so 20+ years, and considering SSL itself came out of Netscape in the prior era, with all the features I'm talking about in place, I'm pretty sure this was their intent. Not really relevant today, anyway.

OK, you've got me on this one. Afair, early browers warned once when they encountered a self-signed or otherwise bad cert (e.g. wrong domain), but with one click you usually accepted this forever with no visible indication from this point on. Many old versions of command line tools didn't complain at all with their default settings.

In theory, yes. But revoking an individual certificate relies on reaching all parties with the revoke requests. If your infrastructure doesn't provide for this in a reliable way, nothing gets revoked.

My comments here are context-aware. We are talking about a closed system with a self-managed PKI for an IoT platform. This is trivial to accomplish in that setting.

Yep, missed that one too. If do it all on your own, you're in charge to track all issued certs, enter and propagate the revokes and of course you never forget to do so and your system always works flawlessly.

[peter-h]

A self signed cert makes your server unusable for most customers for said reasons, browsers make accepting such certs more and more complicated. You can get free and valid certs e.g. from Let's Encrypt, by they expire fast, so you'll have to make your server able to automatically renew the certs, and they have to be unique for each server then. Pretty elaborate  for an embedded device based on a typical MCU of today.

IME, looking after a few websites, this is just a road to hell. These cheap or free rapidly-expiring certs just keep breaking your system, because you have to set up a cron job or whatever to keep renewing them, and then the issuer disappears so you have to change... This alone is a good reason to front your server with Cloudflare (which is free for noncommercial users).

the FritzBox that is very common in Germany optionally uses Let's Encrypt certs. But they run embedded linux, so it's not too elaborate to renew them etc.

They probably do the above, which works until it stops working

In what I thought you were talking about - my bad if I misinterpreted - where the client is the device reaching out to a server on the Internet, you can establish your own CA, and then make that the (only) trusted CA in your IoT device's trust store. A browser connecting to your API service is going to throw a security warning due to lack of trust, but your device will trust this certificate and work fine.

Yes, I think exactly that. My box doesn't have enough RAM for both HTTPS client and server anyway So it has HTTP server, and an NTTPS client whose sole purpose will be going out to some specific servers. If server authentication is needed (read: if the customer demands it) then the box can store that server's self signed cert, which will have a 100 year life.

If you want each of your devices to act as a server directly accessible through HTTPS and you can't use a self-signed certificate, it's going to cost you uh....

Yes and this, together with the continually worsening browser acceptance of self signed certs, is another reason why IOT is rarely going to be running exposed. I thought there was some way around this (e.g. getting Verisign to somehow sign 100000 certificates) but clearly there isn't.

[capt bullshot]

A self signed cert makes your server unusable for most customers for said reasons, browsers make accepting such certs more and more complicated. You can get free and valid certs e.g. from Let's Encrypt, by they expire fast, so you'll have to make your server able to automatically renew the certs, and they have to be unique for each server then. Pretty elaborate  for an embedded device based on a typical MCU of today.

IME, looking after a few websites, this is just a road to hell. These cheap or free rapidly-expiring certs just keep breaking your system, because you have to set up a cron job or whatever to keep renewing them, and then the issuer disappears so you have to change... This alone is a good reason to front your server with Cloudflare (which is free for noncommercial users).

the FritzBox that is very common in Germany optionally uses Let's Encrypt certs. But they run embedded linux, so it's not too elaborate to renew them etc.

They probably do the above, which works until it stops working

Fully agree to your points

In what I thought you were talking about - my bad if I misinterpreted - where the client is the device reaching out to a server on the Internet, you can establish your own CA, and then make that the (only) trusted CA in your IoT device's trust store. A browser connecting to your API service is going to throw a security warning due to lack of trust, but your device will trust this certificate and work fine.

Yes, I think exactly that. My box doesn't have enough RAM for both HTTPS client and server anyway So it has HTTP server, and an NTTPS client whose sole purpose will be going out to some specific servers. If server authentication is needed (read: if the customer demands it) then the box can store that server's self signed cert, which will have a 100 year life.

This works until the security experts found a feasible way to attack and re-create your self-signed server cert

Security is a never ending rat race.

[ve7xen]

IME, looking after a few websites, this is just a road to hell. These cheap or free rapidly-expiring certs just keep breaking your system, because you have to set up a cron job or whatever to keep renewing them, and then the issuer disappears so you have to change... This alone is a good reason to front your server with Cloudflare (which is free for noncommercial users).

Let's Encrypt isn't likely to go anywhere soon, and it's been a pretty big boon for security in the long-tail of the Internet, though yeah I'd probably not bake a third-party service into my IoT product. But there's another, probably bigger, problem using these ACME issuers for IoT devices, which is that they are based on DNS names and a challenge to prove ownership. You can't assume your box is exposed to the Internet to accept a challenge, or even that it has Internet access at all, if this is about local configuration. It might work for a router, which does usually have a public IP and can register itself a DNS name and a way to respond to challenges, as well as usually a DNS server so you can access the device's internal interface with a public hostname, but the IoT environment is a lot more constrained.

It'd actually be kind of reasonable for router vendors to start including an MQTT broker, and doing this certificate magic to make it accessible from your smartphone or whatever in a federated and fairly secure way, without relying on anyone's cloud service. But good luck, the industry is way too fragmented for something like that.

[gmb42]

I would actually be quite interested in opinions on which cipher suites are worth supporting in a product launched say today but which has to be compatible, as an HTTPS client, with other old stuff.

As a client, probably old stuff doesn't matter much.

Much of the size of MbedTLS is cipher suites.

If you restrict yourself to TLS 1.3 then the range of mandatory cipher suites is much reduced, see RFC 8846 Sect 9.1.

[gmb42]

Shared keys ship with the box and will eventually get extracted, if history is any indication, and then you're hosed, so that's not really a strong solution.

However, isn't that the same as if your box contains the certificate of your private server an somebody can replace that certificate...

No physical security = no security

To be pedantic the certificate is public knowledge, it's sent to the client.  It's the associated private key you have to protect, hopefully inside a secure enclave of some description.

[peter-h]

I'd probably not bake a third-party service into my IoT product.

Exactly. This is the main challenge: how to get a long product life.

I think one needs a route for doing firmware upgrades, and having the box "check in" to a server periodically is a good way to do it. And that has to be fairly secure otherwise, accidentally, or deliberately, somebody could brick all your boxes

If you restrict yourself to TLS 1.3 then the range of mandatory cipher suites is much reduced, see RFC 8846 Sect 9.1.

Doesn't that restrict compatibility a lot?

It's the associated private key you have to protect, hopefully inside a secure enclave of some description.

That's very difficult. I think a smartcard chip is the only way.

[westfw]

IOT boxes can't be on open ports, because these are discovered by sniffers almost immediately, due to back doors and vulnerabilities which cannot be fixed ... so their "internet access" can only be as a client, calling up some server, and hidden behind a NAT router.

I am not convinced of this argument.  For an IoT box, you ought to be able to limit the "attack surface" to a manageable size, with or without "full TLS support."
To some extent, I think it's a myth spread by the "IoT Server/Service" vendors (for obvious reasons.)
And laziness by IoT developers.  "We'll just run linux!  That'll make it all easy!"  :-(

(For instance, are their any known attacks for the WizNet embedded TCP chips?  Or the default ESP8266 AT-based TCP connections?  (DoS doesn't count.))

[peter-h]

I think it's a myth spread by the "IoT Server/Service" vendors

That is bound to happen

I have no knowledge of specifics of the hardware and software involved but having seen the poor state of the ETH code supplied by ST with the Cube IDE kit, I would expect loads of bugs, buffer overflow / stack overflow issues etc. The stuff seems to have been written by total amateurs. So I think crashing the box is probably not at all hard, but the consequences are minimal if you have a watchdog. If the 32F4 in my target crashes with say an illegal address or illegal instruction trap (the usual case after a stack corruption) the watchdog reboots it. In most applications the reboot will not even be noticed - about 3 seconds, of which 1.6 secs is the watchdog timeout.

Apart from crashing, what is the worst an attacker can do? With most server attack scenarios the attacker is after "fun" stuff like data theft, data deletion, or remote control of something. But most embedded boxes don't even have what one might call an OS. There is usually no file system and if there was, say FatFS, you won't get a console access to it, to run rm -r. As you say, the scope for these attacks will be limited in most cases.

But if somebody sold say 100k+ boxes, and an exploit became known, a thousands hackers would be in there even just to crash them. The obvious candidate is the Amazon Alexa - a fantastic "spy box", almost as good as a webcam in every house. But they are all behind NAT, and connecting only to Amazon's servers.

So this is especially a risk with linux based boxes, which is how a lot of consumer IT is done, and most chinese ripoffs are exactly that. Nearly all routers I have ever seen are linux boxes, so if you get root access you can do what you like. Old Drayteks were some custom OS... which ran fast. Their current models are linux and slow as a pig.

DOS will always be a threat because in domestic / small business cases it will be on ADSL which is dead easy to saturate, a firewall is of no help at all, but somebody must really hate you to set that up And the "cloud server" can go behind Cloudflare; this is their main business.

[tellurium]

(For instance, are their any known attacks for the WizNet embedded TCP chips?  Or the default ESP8266 AT-based TCP connections?  (DoS doesn't count.))

ESP32/ESP8266 - yes, CVE-2019-12586

[peter-h]

Great reading around above CVE

Somehow it doesn't surprise me that this is a WIFI attack.

But it seems to just crash the device, so presumably a watchdog would reboot it. So this is a DOS attack because you could park your car outside the factory and let it run all day. Not as good as installing a fake DNS server on their LAN, which has also been done from a car parked outside

[madires]

DOS will always be a threat because in domestic / small business cases it will be on ADSL which is dead easy to saturate, a firewall is of no help at all, but somebody must really hate you to set that up And the "cloud server" can go behind Cloudflare; this is their main business.

You can hire a DDoS service for just 25 bucks. It seems to be popular with online gamers to annoy competitors. And Cloudflare had a hiccup (websites unreachable) this morning - funny coincidence.

[madires]

[(For instance, are their any known attacks for the WizNet embedded TCP chips?  Or the default ESP8266 AT-based TCP connections?  (DoS doesn't count.))

Some examples for known issues in TCP/IP stacks for embedded devices:
- AMNESIA:33 https://www.forescout.com/blog/amnesia33-forescout-research-labs-finds-33-new-vulnerabilities-in-open-source-tcp-ip-stacks/
- NAME:WRECK https://therecord.media/namewreck-vulnerabilities-impact-millions-of-smart-and-industrial-devices/

A dedicated TCP/IP chip can limit the impact of a vulnerability to the chip itself. In case of an MCU with a software stack the vulnerability could be fixed by an update, but for a hardware stack it can't. However, this approach could help IoT devices to be more resilient (drawback: additional costs, larger PCB).

[peter-h]

And Cloudflare had a hiccup (websites unreachable) this morning - funny coincidence.

I use CF on a number of sites and on the whole it is superb. Everybody has some downtime; it's a matter of how bad. The hosting company I use was down for ~1hr this morning.

but for a hardware stack it can't

Can work both ways; you are stuck with the firmware. And lots of p1ssed off customers who, if they are much bigger than you (which they usually are, and the big ones get pretty aggressive these days) they can apply a lot of pressure. A lot of business goes into covering one's 6 o'clock. Not long ago I was lucky to have emails going back to 1995, which saved my skin in a big way. There is no such thing as a "hardware" stack; it has to be code, which somebody wrote. AFAIK one can implement more trivial functions like IP filtering in firewalls with an FPGA but a TCP/IP stack? No way, surely.

The reality of mfg business is that one always needs to have a damage limitation strategy. If you sell 1000 boxes to 1000 separate customers, due to widely varying applications perhaps only 50 will discover an issue, and you can fix them. And a customer who had a problem and had it promptly fixed is more loyal than one who never had a problem in the first place, etc, etc. But if you sell 1000 boxes to one customer and he finds a problem, you could be forced into bankrupcy, because a "modern large company" can hit you hard; very hard. I could give actual examples but won't because they might be recognised. So a remote firmware update could be a lifesaver.

AMNESIA:33 https://www.forescout.com/blog/amnesia33-forescout-research-labs-finds-33-new-vulnerabilities-in-open-source-tcp-ip-stacks/

"or inject malicious DNS records to point a device to an attacker-controlled domain"

I wonder what is the best strategy. Going to a fixed IP is not a good idea. Probably authentication is the only way. But if somebody has enough access to do this, they could install a fake DNS server on your LAN?

[janoc]

Do companies large enough to have an IT department deploy IoT devices which are controlled remotely via the internet? And how did they manage to deploy VPNs at large scale for employees working at home (COVID 19)? BTW, I didn't say that VPNs are easy, just that WireGuard makes things easier (vs. IPsec for example).

There is a big difference between an internal project like that - e.g. having to provide VPN for employees (likely some Cisco VPN gear or Fortinet and such, forget free stuff like Wireguard, that doesn't tie in with Active Domain and other enterprise systems) and an external vendor demanding a VPN to be put in place because of some gadget that some low level manager has ordered for a project. Good luck getting something like that authorized.

IPv6 is meant to be used without NAT (It's available, and there's NAT64 as a transition method). This is what I was hinting at. I know, some telcos and companies are still living in the IP past. However, we have IPv6 for 20 years now and they all have to migrate to IPv6 sooner or later because of the lack of IPv4 address space.

Yes but sadly you have to live with what the customer has available now, not what should happen sometime in the future. Also a lot of hardware simply doesn't support IPv6 properly, can't be upgraded/replaced for various reasons, etc.

And even when IPv6 is available, NATs are commonly used as a part of firewall for security reasons, as a supplementary line of defense, with the internal networks completely isolated on non-routable addresses.  There are also compliance/legal reasons because you can't allow any traffic to go out untracked/unmonitored. Forcing everything to go out through a centralized "chokepoint" helps with that.

Dealing with corporate/enterprise IT is a very complex beast and lots of these theoretical ideas how things "are supposed to work" don't apply because there are frequently non-technical reasons (commercial, legal, regulatory, departmental politics ...) involved.

[madires]

There is no such thing as a "hardware" stack; it has to be code, which somebody wrote. AFAIK one can implement more trivial functions like IP filtering in firewalls with an FPGA but a TCP/IP stack? No way, surely.

Of course it's a combination of hardware offloading plus software. Based on the NPU the offloading can go quite far. What I've meant is that a software-only stack can be patched, something with hardware offloading not so much.

I wonder what is the best strategy. Going to a fixed IP is not a good idea. Probably authentication is the only way. But if somebody has enough access to do this, they could install a fake DNS server on your LAN?

Perhaps DNSSEC, DoH or DoT?

[madires]

And even when IPv6 is available, NATs are commonly used as a part of firewall for security reasons, as a supplementary line of defense, with the internal networks completely isolated on non-routable addresses.

RFC 1918 addresses can be routed. You would be surprised how often ISPs/telcos/carriers see those addresses from customers (usually blocked by an ingress filter, but some still haven't heard of BCPs). BTW, you can also use private IPv6 addresses (ULA).

There are also compliance/legal reasons because you can't allow any traffic to go out untracked/unmonitored. Forcing everything to go out through a centralized "chokepoint" helps with that.

It doesn't matter if IPv4, IPv6 or both, all can be filtered or monitored without any problem to comply with whatever regulation. And again, NAT isn't a security feature, but a stateful firewall is. Also, there are many flavours of NAT, some would make your security staff go berserk. Have you ever met an 'expert' telling you that his network is totally safe because of NAT and when you check the firewall settings UPnP is enabled?

[ve7xen]

IOT boxes can't be on open ports, because these are discovered by sniffers almost immediately, due to back doors and vulnerabilities which cannot be fixed ... so their "internet access" can only be as a client, calling up some server, and hidden behind a NAT router.

I am not convinced of this argument.  For an IoT box, you ought to be able to limit the "attack surface" to a manageable size, with or without "full TLS support."
To some extent, I think it's a myth spread by the "IoT Server/Service" vendors (for obvious reasons.)
And laziness by IoT developers.  "We'll just run linux!  That'll make it all easy!"  :-(

(For instance, are their any known attacks for the WizNet embedded TCP chips?  Or the default ESP8266 AT-based TCP connections?  (DoS doesn't count.))

Sure, if you are making devices with NASA or Google levels of engineering, you can be fairly sure you have built a box that isn't vulnerable, at least to a trivial exploit. It's not laziness, it's the engineering tradeoff of having to build a working product you can sell for $30 on Amazon, there just isn't time or money available to hire high quality engineers and for them to put the time into security engineering or evaluation. So that's not the situation with cheap consumer electronics that are built out of half-assed example code stuck together with scotch tape. Chances are very high that some kind of vulnerability exists, since the engineering against them hasn't been done, and then because of the nature of the product, it's often unlikely to get fixed.

In some cases the issue might not even be immediately apparent to the device owner, if it ends up getting used as an amplification vector for DDoS or something like that, which is the most likely scenario for an embedded device. The only thing saving these devices is pretty much that they are almost never exposed to the Internet directly, and that there are so many different varieties of things to exploit. If there's a product that a) sells in large enough numbers and b) is widely exposed to the Internet, I can all but guarantee it gets exploited in short order to be used as a spam cannon or whatever. And is the vendor even incentiveized to care about such issues at all? Doesn't really seem like they care once the device is sold.

But even a basic DoS is dangerous. Maybe your unethical competitor finds out about this... Pretty trivial to scan the 'net for open boxes, DoS them constantly, and cause a lot of damage to your reputation.

Add on the issues with trying to help customers set up port forwarding or whatever to access their device(s), and the path towards having them be clients for an online service is pretty cemented. It avoids most of the security concerns, issues with having multiple devices on the same network, it punches through pretty much any NAT, it's easy for the user...

There are also likely to be vulnerabilities in the application, not the network stack, so I'm not convinced that using a TCP chip helps you much here, and if there *is* a problem it's more likely to be found due to ubiquity and then you can't do anything to fix it, so I dunno, seems like a bit of a wash.

If you restrict yourself to TLS 1.3 then the range of mandatory cipher suites is much reduced, see RFC 8846 Sect 9.1.

Should be able to get away with TLS_AES_128_GCM_SHA256 and TLS_RSA_WITH_AES_128_CBC_SHA which are all that is mandatory in TLS 1.2 + 1.3, if you only care about connecting to relatively modern servers. Some old old stuff might require RC4 or MD5 or something, but meh.

[madires]

If you want to support only TLS 1.3 be aware that in some countries, ASs or corporate networks it will be blocked. Latest tests for HTTP/3 (QUIC, TLS 1.3): https://ooni.org/post/2022-quick-look-quic-censorship/.

[tellurium]

But if you sell 1000 boxes to one customer and he finds a problem, you could be forced into bankrupcy, because a "modern large company" can hit you hard; very hard. I could give actual examples but won't because they might be recognised. So a remote firmware update could be a lifesaver.

That if of course if OTA mechanism is tested, robust and won't brick devices. Experience says that sometimes, a single bricked device out of a thousand successfully updated, is enough to get in trouble. A device could be bricked because of combinations of factors, e.g. specific configuration, environment, etc. Implementing a robust OTA firmware update is a non-trivial task on its own.

[peter-h]

Indeed; OTA needs careful testing, as well as protection from malicious "updates". In practice one would deploy updates by S/N, and only to where issues have been reported. Phone makers limit the risk by deploying country by country, etc.

There are also compliance/legal reasons because you can't allow any traffic to go out untracked/unmonitored

No chance of getting the PCI/DSS remote audit if they find other services on your IP. Eventually we gave up and went to Paypal for payment processing. Another story...

NAT isn't a security feature

NAT is an excellent security feature. It works out of the box, and most people don't need to open any ports (and on VPN routers the VPN terminator bypasses NAT so that still works, for RDP or whatever) whereas firewalls need expert config which is often done wrong. Another nail in the coffin of IOT on open ports.

exploited in short order to be used as a spam cannon or whatever.

That would need the box to be running Linux or some such.

[ve7xen]

NAT is an excellent security feature. It works out of the box, and most people don't need to open any ports (and on VPN routers the VPN terminator bypasses NAT so that still works, for RDP or whatever) whereas firewalls need expert config which is often done wrong. Another nail in the coffin of IOT on open ports.

NAT is only a security feature by accident, and in some configurations (though admittedly not ones you would find in a typical home network) it offers no security at all.

It is a 'security feature' because it requires stateful monitoring of outbound sessions, and only knows what to do with traffic related to them, so unassociated packets get dropped. This is exactly what a stateful firewall gets you in the typical default 'allow all outbound, allow nothing inbound' configuration. There's no significant difference in security posture or configuration effort here.

That would need the box to be running Linux or some such.

Definitely not, all it requires is RCE.

[JohanH]

NAT is an excellent security feature. It works out of the box, and most people don't need to open any ports (and on VPN routers the VPN terminator bypasses NAT so that still works, for RDP or whatever) whereas firewalls need expert config which is often done wrong. Another nail in the coffin of IOT on open ports.

You are confused by what NAT is and isn't. It's purpose isn't security. Use a stateful firewall and it doesn't matter if there is NAT or not. You use NAT when you don't have enough IPv4 address space.

https://www.f5.com/services/resources/white-papers/the-myth-of-network-address-translation-as-security

Btw, most mobile operators have already moved to IPv6. The US has already 50% IPv6 coverage. By this I'm not saying that IPv6 will take over tomorrow, but that it doesn't matter if you use IPv4 (with or without NAT) or IPv6 from a security point of view. The network is only as secure as you have configured it to be, and that means firewalls and other layers of security.

[peter-h]

Whether NAT is a "firewall" or not, if an IOT box is behind NAT, it can't be hacked, but it can go out to access some server. Or you can access it via a VPN terminating in the NAT router.

[nctnico]

Whether NAT is a "firewall" or not, if an IOT box is behind NAT, it can't be hacked, but it can go out to access some server. Or you can access it via a VPN terminating in the NAT router.

IMHO there are actually multiple things to consider that drive the requirements:
1) Malicious acts (making the device do something it is not supposed to do -including not working-)
2) Privacy requirements (make sure third parties can't collect data)
3) Tampering with data to get some (financial) gain

[tellurium]

I'd summarise the following strategy in making a device secure, in the order of complexity

1. It must not accept incoming connections - only outgoing
2. It must use TLS for all sensitive data exchange
3. (complex) It should keep credentials on a crypto element. Or/and, use flash encryption
4. (complex) For OTA, should use checksums and secure boot

The first two generally do not require much investment.

[westfw]

1. It must not accept incoming connections - only outgoing
  :
The first two generally do not require much investment.   

To support any meaningful extraNet usage, (1) requires a whole set of cloud based infrastructure to be the servers for both the IoT devices AND the people trying to use them.   That is a very significant investment (or choice of an existing services that might go away.)  And as they say:  "Now you have TWO security problems."  :-)  (Hmm.  Three, actually, if users also need a special app to access those servers.)

[madires]

Whether NAT is a "firewall" or not, if an IOT box is behind NAT, it can't be hacked, but it can go out to access some server.

Devices behind NAT can be hacked too, it's just a bit more complex. There's no 100% security! What you can do is to define how much security your product needs and trying to reach that goal.

[peter-h]

Devices behind NAT can be hacked too, it's just a bit more complex

I've read this loads of times, and asked each time how to do it, and never got an answer So I am asking again.

[gmb42]

If you want to support only TLS 1.3 be aware that in some countries, ASs or corporate networks it will be blocked. Latest tests for HTTP/3 (QUIC, TLS 1.3): https://ooni.org/post/2022-quick-look-quic-censorship/.

TLS 1.3 != QUIC or HTTP/3

The report you quote is all about HTTP/3, that's a whole chunk of (IMHO unnecessary) extra code for an IoT device that's simply looking for secure data transfer.

[gmb42]

Devices behind NAT can be hacked too, it's just a bit more complex

I've read this loads of times, and asked each time how to do it, and never got an answer So I am asking again.

This discussion is pointless, if you have the mindset that using NAT is "secure" you'll never be convinced that actually it isn't because the proper way to do it with a statefull firewall is "too hard". 

Any time this discussion every comes up on the interwebz there's always someone claiming they have no need for anything other than NAT, they've turned off IPv6 and they've never been hacked.

[peter-h]

... and the answer to the Q is?

Any time this discussion every comes up on the interwebz, some expert appears who says he knows how to do something, but never posts the details

The first two generally do not require much investment.

I'd say TLS is a huge investment, because the code needs a lot of integration, testing and a lot of bug fixing. It also probably forces a bigger CPU (more RAM etc). On my product, TLS doubled the code space, and soaked up 90% of the then remaining RAM.

No wonder people who have done this prefer to be a client, calling up some private server, because that doesn't expose their box to the wide world, and protects it from vulnerabilities which come to light in the future.

I often say this, and MbedTLS is a large enough project for this to not be true, but open source is wonderful until the coder gets himself a girlfriend

[JohanH]

... and the answer to the Q is?

You didn't read the article I posted, did you?

For your convenience:

"While it is true that stateful ingress IPv4 NAT will reject externally initiated TCP traffic, that does not mean that an external host cannot in certain situations send traffic to internal hosts or use other methods to circumvent the NAT. In fact, most network-based attacks assume this as a requirement of the compromise.

There are several ways to accomplish this circumvention, all of which can be prevented by a firewall. First, an attacker can either use a targeted or a sweep attack to send traffic to ports that are open in the NAT device’s state table. The purpose of this attack could be to create a denial-of-service (DoS) by invalidating an existing session on the host or NAT state table, to footprint an internal network, or to inject a malware payload into a third party’s existing session in an effort to compromise the internal host. Serious implications are seen in UDP traffic that is by design stateless; however, the same could be accomplished (given host susceptibility) in TCP or other protocols. In addition, NAT may not provide protocol conformance, sequence number checking, or any other layer 2 or layer 3 DoS security measures that firewalls or advanced security devices inherently provide. NAT also provides no tools to respond should security breaches occur."

[peter-h]

Thanks. I did wonder if an attack on NAT must involve getting into it during the 180 secs of the open channel.

First, an attacker can either use a targeted or a sweep attack to send traffic to ports that are open in the NAT device’s state table. The purpose of this attack could be to create a denial-of-service (DoS) by invalidating an existing session on the host or NAT state table, to footprint an internal network, or to inject a malware payload into a third party’s existing session in an effort to compromise the internal host

That is however quite a stretch; first you get past one box and then you need to exploit back doors in another box.

all of which can be prevented by a firewall.

Assuming the firewall has no back doors... A friend used to work at Cisco and even they used to regularly patch back doors in their 5 digit priced boxes.

[JOEBOBSICLE]

I don't think there's anything wrong with SSL or TLS, I just think it shouldn't be done on a MCU.

It's a solved problem on Linux. You need to implement NTP before you even start SSL handshakes.. It's such a huge amount of work to do on a micro when on Linux it takes five minutes

[tellurium]

I don't think there's anything wrong with SSL or TLS, I just think it shouldn't be done on a MCU.

I guess then, all those vendors who ship their MCUs with network support, are just wrong. They just don't realise that simple fact that networking is for linux.

[JohanH]

A friend pointed out DTLS.

Matthew Garrett, known kernel hacker and security expert, examined an Ikea IoT device and was impressed by their effort. Usually most devices he has examined have gone down in flames.

https://mjg59.dreamwidth.org/47803.html

[Kjelt]

That is however quite a stretch; first you get past one box and then you need to exploit back doors in another box.

The whole point IIRC is that none of your homenetwork equipment should be reachable from the internet.
Better still close all ports for outside traffic.
But a lot of people want to control some of their stuff (NAS, camera's, home automation) from outside, and there it goes wrong.
If one of those devices are compromised the hacker is on your home network.

To prevent that there is something called a DMZ, you actually can do this yourself when using two routers with their own DHCP server.
The first router is connected to the WAN and the network it provides is the DMZ, on that network (for instance 192.168.1.x)  you attach your 3rd party devices with the risk of them being hacked.
The second router on that first network will have its own DHCP server and the second network is your home network (for instance 192.168.2.x).
Ofcourse the whole security is based on the second router, if that is not updated and compromised your still vulnerable.

And no I have no pretentions, the real good hackers can get to you, one way or the other, but you have to make it too much trouble for them to do it and they will pick your neighbour, just as with real burglars

[ve7xen]

Thanks. I did wonder if an attack on NAT must involve getting into it during the 180 secs of the open channel.

It depends what you mean by 'attack'. Since NAT is not a security feature, it doesn't have a defined security intent, and what you consider an 'attack' can also be considered normal behaviour. You probably won't be able to perform a 3-way handshake and open your own TCP session to the target, but if you can predict most of the 5-tuple of an open connection (e.g. api.yourdomain.com port 443 destination and you already know the target's IP) and fuzz the rest you can probably get packets inside without too much trouble. At that point your 'security' comes down to the functional mechanisms of TCP (again, not intended to be security features), and how diligent the micro's implementation is about checking them and what it does when it receives invalid ones. It might be enough to simply get a packet to the exploitable stack, regardless of the headers, if you can trigger a buffer overflow or something, or maybe the sequence error handling code is the part that's a problem in the first place.

How 'secure' a NAT implementation is to this kind of shenanigans is down to the implementation, but the point is that NA(P)T is network address translation, it's not a security mechanism and provides no security guarantees - it's an address translation mechanism as the name implies. The fact that it provides a modicum of security is just a side effect of the fact that (in the home router context) it is many-to-one, so there's no sensible default behaviour when you get packets on the 'one' side other than to discard. In many cases your NAT box also includes a firewall, of course, and it might not even tell you this.

That is however quite a stretch; first you get past one box and then you need to exploit back doors in another box.

A stretch for whom? This is the kind of exploit that's too 'costly' to do en masse, but that's not much of a stretch at all in a targeted attack, it's pretty trivial to execute. Is it likely one of your customers gets attacked in this way? I guess it depends who your customers are and what the exploit offers an attacker. If you sell enough units and the exploit is juicy enough, that cost equation just might flip, and that's when someone adds it to a botnet drone and millions of devices get compromised over night.

Are people going to bother finding or attacking an exploit like this in a device that sold 10,000 units? Almost certainly not, but are we talking about actual security here, or security through obscurity? The idea that you're secure because nobody is attacking you says nothing about the security of your product at all.

Assuming the firewall has no back doors... A friend used to work at Cisco and even they used to regularly patch back doors in their 5 digit priced boxes.

Good security is layered. The failure of one layer shouldn't compromise the whole system, and if nothing else, it's one more layer to attack on the way in. Hence why you still need good internal security even if you have a strong perimeter. FWIW though Cisco's software quality is trash, I'd definitely not hold them up as a security poster child.

[ve7xen]

Whether NAT is a "firewall" or not, if an IOT box is behind NAT, it can't be hacked, but it can go out to access some server. Or you can access it via a VPN terminating in the NAT router.

IMHO there are actually multiple things to consider that drive the requirements:
1) Malicious acts (making the device do something it is not supposed to do -including not working-)
2) Privacy requirements (make sure third parties can't collect data)
3) Tampering with data to get some (financial) gain

The "CIA Triad" - confidentiality, integrity, and availability.

[madires]

TLS 1.3 != QUIC or HTTP/3

The report you quote is all about HTTP/3, that's a whole chunk of (IMHO unnecessary) extra code for an IoT device that's simply looking for secure data transfer.

Simply ignore HTTP/3 (uses QUIC and TLS 1.3) and focus on TLS 1.3. The message is that some countries/orgs don't like TLS 1.3.

[2N3055]

... and the answer to the Q is?

You didn't read the article I posted, did you?

For your convenience:

"While it is true that stateful ingress IPv4 NAT will reject externally initiated TCP traffic, that does not mean that an external host cannot in certain situations send traffic to internal hosts or use other methods to circumvent the NAT. In fact, most network-based attacks assume this as a requirement of the compromise.

There are several ways to accomplish this circumvention, all of which can be prevented by a firewall. First, an attacker can either use a targeted or a sweep attack to send traffic to ports that are open in the NAT device’s state table. The purpose of this attack could be to create a denial-of-service (DoS) by invalidating an existing session on the host or NAT state table, to footprint an internal network, or to inject a malware payload into a third party’s existing session in an effort to compromise the internal host. Serious implications are seen in UDP traffic that is by design stateless; however, the same could be accomplished (given host susceptibility) in TCP or other protocols. In addition, NAT may not provide protocol conformance, sequence number checking, or any other layer 2 or layer 3 DoS security measures that firewalls or advanced security devices inherently provide. NAT also provides no tools to respond should security breaches occur."

That article is about carrier grade use of NAT.. If you read it carefully (and RFCs linked in it) you will see that there are many things called NAT..

NAT we are talking about (one implemented on all Internet routers) is Masquerade NAT, a stateful Network Address Port Translation (NAPT), outbound direction type. With IOT devices that use only outbound SSL connections there is very low probability of compromise by outside actor. If you have internally compromised network that is not router or IOT device fault. If you have IOT device that is compromised  that is not router's fault.

There is this prevalently bad thinking in IT security industry (driving up the FUD to pump up profits, most likely) that ANYTHING is possible. There is also a lot of people that really don't understand how things really work, so if someone says "there was this exploit on this version of XYZ by ThatNetworkCompany because they bodged that particular implementation, in next version of security scanners ALL versions of XYZ older than newest patch are proclaimed unsafe, even for hundreds for manufacturers that didn't screw up implementation and are perfectly fine... Because it is easier to just forbid certain name/version number than to expect security experts to actually think and do their job...

NAT is not firewall in a sense it simply isn't. NAT does this specific stateful network traffic filtering. It doesn't do (deep) packet inspection, doesn't monitor traffic, you cannot create application layer filters..
But it will prevent you to connect from the outside to internal network..

For instance, there is this "NAT attack":

NAT Slipstreaming allows an attacker to remotely access any TCP/UDP service bound to any system behind a victim's NAT, bypassing the victim's NAT/firewall (remote arbitrary firewall pinhole control), just by the victim visiting a website.

So you read it and it has nothing to do with NAT, i.e. NAT was not compromised.  User from secure network behind the NAT connected to server that installed exploit to your PC(internal device) and now you have malware bot inside your network...
This is not NAT problem or attack. It is disingenuous by industry to proclaim it is.. And ANY and ALL malware Command-and-control attacks for many years are doing just that, how is that new thing in 2020 and specifically connected to NAT ?

Reaction to this by industry was supposed to be education campaign to educate people that NAT is not going to protect you from this kind of malware exploit because it specifically was not design to be firewall but simple stateful network filter.  Proper name for that type of exploit would be Malware slipstreaming, and recommendation would be to install antimalware software on critical nodes... It has nothing to do with NAT. Yes a good FW would also be able to maybe detect something wrong..

Also DDOS attacks are carrier problem and not a SECURITY problem, but availability problem. Unless you have janky router/fw that will core dump and reboot into telnet session with exposed root access when subject to DDOS...

[madires]

And there are attacks like CSRF which can impact a device inside a LAN protected by a firewall. Security is a process, and not just simply adding some TLS lib, placing a firewall in front or whatever. It starts with a lot of questions. What needs to be protected? How much security do I need (e.g. regulations)? How much I'm willing to pay (money and convenience)? What would be the damage in case of a security incident? What are possible attack vectors?

[madires]

Another typical security issue is a faulty implementation: MEGA: Malleable Encryption Goes Awry (https://mega-awry.io/). We have this problem especially with complex protocols, like IPsec. Over the years many implementation faults were found in a multitude of IPsec products, including super expensive carrier/enterprise class boxes. Or check the CVEs for OpenSSL. The sad reality is that many IoT devices never will see any updates.

[janoc]

And even when IPv6 is available, NATs are commonly used as a part of firewall for security reasons, as a supplementary line of defense, with the internal networks completely isolated on non-routable addresses.

RFC 1918 addresses can be routed. You would be surprised how often ISPs/telcos/carriers see those addresses from customers (usually blocked by an ingress filter, but some still haven't heard of BCPs). BTW, you can also use private IPv6 addresses (ULA).

There are also compliance/legal reasons because you can't allow any traffic to go out untracked/unmonitored. Forcing everything to go out through a centralized "chokepoint" helps with that.

It doesn't matter if IPv4, IPv6 or both, all can be filtered or monitored without any problem to comply with whatever regulation. And again, NAT isn't a security feature, but a stateful firewall is. Also, there are many flavours of NAT, some would make your security staff go berserk. Have you ever met an 'expert' telling you that his network is totally safe because of NAT and when you check the firewall settings UPnP is enabled?

You don't need to explain me that. I have been building and running corporate networks for a while as part of my job, so I believe I know quite well what is and what isn't possible or safe.

The problem is that you are talking about things from a typical engineering point of view - and totally ignoring the corporate logistics of putting anything of this in place. Which tells me you don't really have an experience trying to deploy anything like this in a large corporate network because these things are the 90% of the problem, not whether or not NAT is not a security feature or whether some address is or isn't routable.

[AndyBeez]

@janac I agree. We do not know the use case of the OP's device. A tiny SoC suggests a quasi domestic setting, such as a wifi lightbulb or thermostat.

As you say, it not the what but the where. I will jail any IoT device on their own subnet, VLAN, SSID and block WAN access. But I do this because I can. To most people a DMZ is something in Korea, not a setting on their vanilla router. In the corporate space, only tight control and enforcement keeps departments from establishing their own WAN behind the corporate firewall connection. Deployed IoT devices may also come with 4G built-in.

IoT devices have two nasty habits: calling home, often an endpoint in China. And keeping WAN facing ports open, providing a front door for hackers. Has the OP  pen' tested their device?

[madires]

The problem is that you are talking about things from a typical engineering point of view - and totally ignoring the corporate logistics of putting anything of this in place. Which tells me you don't really have an experience trying to deploy anything like this in a large corporate network because these things are the 90% of the problem, not whether or not NAT is not a security feature or whether some address is or isn't routable.

I thought we're discussing security issues of IoT devices and not corporate bureaucracy. BTW, I had plenty of it too. Can be very frustrating, and you need a thick skin and a lot of patience. Anyway, I just wanted to point out common misconceptions of network basics.

[TomS_]

For example every instance of win10 has its predefined IPV6 address but that just means hackers who know the ranges can target them, in the hope of discovering win10+ installations which are not behind NAT.

Incorrect. OSes dont get pre-defined IP addresses, that is just nonsense.

IP addresses are determined locally on the LAN using a variety of methods, the OS itself being none of them.

[madires]

What about ::1, ff01::1 or ff02::1?

[peter-h]

I am checking out the above. I was told otherwise... but it could be win11.

On a slight tangent but picking up a point above:

I am told that an HTTPS client does not need a public and private key pair pre-generated. That is done in the TLS negotiation. All it needs is the server certificate, and that is optional and is needed only if you want to protect yourself from a spoofed DNS pointing to a spoofed server.

[madires]

For HTTPS the client (webbrowser) doesn't need a certificate, but it may use one optionally. But the server (webserver) must have a certificate for identification (please see https://www.rfc-editor.org/rfc/rfc2818#section-3.1).

[peter-h]

The curious thing is that the client does not need to store anything in a secure manner, for an encrypted connection.

Only if it needs to authenticate the server then it needs to store the server's certificate in a secure manner.

[madires]

In case there's a misunderstanding, encryption and certificate are two different things. The certificate is used for identification (or authentication).

[Stray Electron]

The other thing is that "IOT" is going to end up with a really dirty reputation, as companies go bust, or just stop maintaining the servers. So the remotely controlled irrigation system, where you have a nice app connecting to that server, or even just a website served by that server, goes dead one day and there is no way to fix it. You have to rip the whole lot out.

   I didn't read the whole thread but what I'd really like to know is this: WHY do these manufacturers make all of these devices so that they CAN be reprogrammed?  Why don't they just put all of the OS in ROM and then NO ONE could hack, not even the OEM. Who, I will add, that I don't trust any further than I trust the hackers in China, Russia, etc, etc, etc.  The one thing I really worry about with all IOT devices is that the manufacturer can later decide to brick them or degrade their performance enough that they're inoperable (i.e. Apple).

[ve7xen]

The curious thing is that the client does not need to store anything in a secure manner, for an encrypted connection.

Everything required for the session is or can be ephemeral. Typically you also want to have trusted storage (it doesn't need to be secret, just trusted) for your trust store, so you can confidently identify the servers you talk to, but as you say, that's optional if you don't want protection against MITM, and depending on your application you may not even care about trusted storage here, if you allow the user to manipulate the trust store.

The most 'difficult' requirement on the client side for a proper TLS implementation is probably the need for a secure RNG.

   I didn't read the whole thread but what I'd really like to know is this: WHY do these manufacturers make all of these devices so that they CAN be reprogrammed?  Why don't they just put all of the OS in ROM and then NO ONE could hack, not even the OEM. Who, I will add, that I don't trust any further than I trust the hackers in China, Russia, etc, etc, etc.  The one thing I really worry about with all IOT devices is that the manufacturer can later decide to brick them or degrade their performance enough that they're inoperable (i.e. Apple).

It's kind of opposite of how you are presenting it. If ROM code has a severe vulnerability, it's out in the wild and nothing can be done about it. At least with programmable devices, there is the potential for field updates to occur.

[Bassman59]

A friend pointed out DTLS.

Matthew Garrett, known kernel hacker and security expert, examined an Ikea IoT device and was impressed by their effort. Usually most devices he has examined have gone down in flames.

https://mjg59.dreamwidth.org/47803.html

I've gotten rid of most of my other IoT crap (mainly because the shit died) and the IKEA stuff is not only much less expensive than the competition, it does not have any cloud service capabilities at all.

Of course you read reviews of the products: "PROs: Works well. CONs: They do not offer remote access to the devices, so you must be on the local network."

As if not having a cloud service for remote access is something bad and to be avoided ...

[tellurium]

Only if it needs to authenticate the server then it needs to store the server's certificate in a secure manner.

Not exactly. Server sends its certificate during the handshake. What client needs to store, is the CA cert to verify server's cert:

1. With TLS, any party can authenticate it's peer. If client authenticates server, client must have a CA (certificate authority) cert stored. During the handshake, server sends its certificate, and client verifies it against the CA cert.
2. Likewise, a server also may authenticate a client. Then, a server must store a CA cert. During the handshake, a client sends its certificate, and server verifies it against the CA cert.
3. If both client and server exchange certificates, it is usually called a "two-way TLS". For example, AWS IoT uses that method to authenticate MQTT clients.
4. Browsers usually only authenticate servers, but servers do not authenticate clients (browsers). So it is a so-called "one-way TLS". Therefore browsers are installed together with a big bundle of CA certs from various authorities. When a browser connects to a HTTPS server, one of the pre-bundled CA certificates is used for verification, so no extra action is required from the user.

[peter-h]

client must have a CA (certificate authority) cert stored

Can it be a self signed one? I don't think you can get a (e.g.) Verisign signed one which is valid for 100 years, which is exactly what one does want for a server one one's control.

[ve7xen]

client must have a CA (certificate authority) cert stored

Can it be a self signed one? I don't think you can get a (e.g.) Verisign signed one which is valid for 100 years, which is exactly what one does want for a server one one's control.

If it's a closed system, you don't need to talk to Verisign (or any other public CA). The only reason they are needed is to get browsers/OSes to trust your certificate by default, there's no technical requirement for them. You can make your own CA to sign certificates and use whatever expiry terms are appropriate, or as you suggest you can use self-signed certificates with explicit trust instead of transitive trust through a CA.

Certificate signing is meant to 'chain' up to a root trust store. So for example google.com's certificate might be signed by Google's CA, and that CA's private key might be signed by Verisign (for example), which the browser has in its trust store. Sometimes there are 3 or 4 levels of chaining to reach the root, but as long as there is a continuous chain of signatures that gets to the trust store, it theoretically can be any depth (I think).

Nothing stopping you from adding the server certificate directly to the trust store either (whether it is self-signed or CA-signed). In a constrained implementation, you might be able to omit the whole chaining thing from the code and just authenticate the server directly.

[tellurium]

client must have a CA (certificate authority) cert stored

Can it be a self signed one? I don't think you can get a (e.g.) Verisign signed one which is valid for 100 years, which is exactly what one does want for a server one one's control.

Sure.
A CA cert, a cert and a private key could be self-generated.

Usually the issue with the browsers, which use a pre-installed CA bundle from known authorities. None of the CA certs from the bundle would verify a self-signed cert, and a browser would raise a security error. To make a browser happy, as ve7xen has pointed out, the self-generated CA could be added to the browser's CA bundle manually.

An IoT TLS client also needs a CA bundle to verify server's cert. If it is only one domain, and a cert issuer is known, then a CA bundle can consist of only one CA certificate. That is important for mbedTLS, because mbedTLS keeps the CA bundle in memory all the time.

[AndyBeez]

What exactly is this IoT device and what is it's deployment scope? If it's a one off gadget used on a private network, then the CA can be self signed (using OpenSSL) and no one else will care. If it's deployed anywhere else, then you will need to spend money on a validated certificate.

Self signing example: https://devopscube.com/create-self-signed-certificates-openssl/

[peter-h]

Usually the issue with the browsers, which use a pre-installed CA bundle from known authorities

This is on a tangent, but I sometimes wonder how easy it may be for an attacker to simply replace these certificates in somebody's Chrome etc installation, install a fake DNS server on their PC, and then use that to capture their banking credentials. The eventual defence would be only that setting up a new payee usually involves answering a phone call...

And this is the problem: if you can't guarantee security of the client machine, any server certs stored on it are meaningless. And in most IOT scenarios this is difficult because getting onto the LAN is quite feasible, with an "inside job" attack (an email with a trojan, etc).

That is important for mbedTLS, because mbedTLS keeps the CA bundle in memory all the time.

There are some patches for that but you still have the problem of updating those certs securely. So being able to process what is today about 200 kbytes' worth of certificates in your IOT box is not worth a whole lot. I am pretty sure most IOT boxes which implement server auth are storing just one cert with a 100 year life

[TomS_]

What about ::1, ff01::1 or ff02::1?

Still not related to the OS, and good luck reaching it beyond the local host. 

[TomS_]

I didn't read the whole thread but what I'd really like to know is this: WHY do these manufacturers make all of these devices so that they CAN be reprogrammed?  Why don't they just put all of the OS in ROM and then NO ONE could hack, not even the OEM.

So when they find a bug, rather than being able to issue a firmware update to fix it, you just end up with ewaste instead? Or buggy devices that remain vulnerable because they cant be fixed? Yea... no.

[madires]

What about ::1, ff01::1 or ff02::1?

Still not related to the OS, and good luck reaching it beyond the local host. 

ff02::1 is link-local. 

[peter-h]

you just end up with ewaste instead?

Which results in more hardware sold, which is a good thing, isn't

Almost nobody in the IT sphere does firmware updates more than a year or two out.

There is also a risk of bricking a whole load of boxes in one go, which is commercially too great a risk, so firmware updates would in any case be made available only to users who complain, and you tell them to go it gradually.

The box I am working on will certainly be able to download a new image from a private server, validate it, and flash it. I already have that working, except for the download.

[gmb42]

Usually the issue with the browsers, which use a pre-installed CA bundle from known authorities

This is on a tangent, but I sometimes wonder how easy it may be for an attacker to simply replace these certificates in somebody's Chrome etc installation, install a fake DNS server on their PC, and then use that to capture their banking credentials. The eventual defence would be only that setting up a new payee usually involves answering a phone call...

Users can be persuaded to install "bad" certificates themselves.  IMHO, this is the major problem with self-signed certs that are not installed by a management system (e.g. Group Policy) and have workflows that ask users to manually install a cert from a link.  Teaches them really bad behaviour.

And this is the problem: if you can't guarantee security of the client machine, any server certs stored on it are meaningless. And in most IOT scenarios this is difficult because getting onto the LAN is quite feasible, with an "inside job" attack (an email with a trojan, etc).

Agreed, and a there's a big difference here between a general purpose PC which as all the tools to install random CA certs but is also likely to have management software (e.g. AV etc) that tries to eliminate bad certs, and a specific device (IoT thing) that actually makes it difficult to modify the CA certs.

[peter-h]

Let's say you are doing MQTT so you are an MQTT client, and you are using AWS as the MQTT broker.

I hope I have the required proportion of correct corporate lingo

Am I right in that the client cannot do authentication with just one certificate, and needs to store the whole certificate chain (currently around 200k)?

This appears to be the case for every application where one is accessing any kind of "public" server - not least because their certs expire so fast so your only hope is to get the current one and check its signatures. Only with a private server, under your own control, can one make do with a single certificate.

This is very relevant to MbedTLS which needs 200k of RAM to do the above.

How do people deal with this, given how much IOT stuff has small CPUs? Perhaps they don't authenticate the server.

[tellurium]

Am I right in that the client cannot do authentication with just one certificate, and needs to store the whole certificate chain (currently around 200k)?

No, that is not correct.
AWS IoT demands two-way SSL.
Meaning, a client verifies server against the CA, plus it sends its own cert to the server - so a server can auth client on its side.
Thus, an AWS IoT client needs to store:
  - a CA bundle, that's basically one cert. Maybe two if you want to get fancy: one RSA, one EC
  - its own cert
  - its own private key

[NorthGuy]

Am I right in that the client cannot do authentication with just one certificate, and needs to store the whole certificate chain (currently around 200k)?

This is not a chain, but rather a collection. A collection of trusted certificate authorities.

When a peer passes you a certificate, there's no way for you to know if the certificate is real or not. To verify, the certificate is signed by a trusted authority, one of the many. You look up your own collection of the trusted authorities and find if the authority who signed the peer's certificate is in your collection. If it is not, you won't trust the certificate.

You can roll out your own certificate authority. Then you only need one trusted certificate in your list - your own. With that, you will trust the certificates you have issued by yourself, but will not trust anything else - such as Verisign certificates. This is because Verisign is not on the list of your trusted authorities. If you want to trust Verisign certificates, you must put Verisign's root certificate into your collection of trusted sources. The more sources you want to trust, the bigger is you trusted sources collection.

The collections included with browsers is a list of certificates of authorities which are commonly trusted. Whether you want to trust these authorities, it's up to you. Any of these certificate authorities can be hacked, commit fraud, or otherwise produce bad certificates. Although chance of this is very small, it's a risk for you because you have chosen to trust the bad authority.

[Kjelt]

Whether you want to trust these authorities, it's up to you. Any of these certificate authorities can be hacked, commit fraud, or otherwise produce bad certificates. Although chance of this is very small, it's a risk for you because you have chosen to trust the bad authority.

Diginotar anyone ?

[madires]

And don't forget CRLs (specifically ARLs)!

[IDEngineer]

The "S" in IOT stands for "Security".

[tellurium]

The "S" in IOT stands for "Security".

Nice joke, however I don't get those people who rant about lack of security in IoT.
IoT is not a particular technology to rant about it.
Likewise, one can say "The S in Internet stands for Security". Or, "The S in Microcontroller stands for security" .

[IDEngineer]

The point is, people need to have realistic expectations. IOT as typically architected is, almost by definition, insecure. If that's acceptable to you, fine. But don't complain later.

[mikerj]

The "S" in IOT stands for "Security".

Nice joke, however I don't get those people who rant about lack of security in IoT.
IoT is not a particular technology to rant about it.
Likewise, one can say "The S in Internet stands for Security". Or, "The S in Microcontroller stands for security" .

Because security on your toaster is a non-issue until it's made "smart" enough to be connected to the internet.  Would you want to wake up and find you've been subject to a Denial of Toast?

[peter-h]

The overriding issue here is that the trade mags need catchy headlines, and the internet needs loads of clickbait.

And "security" is always a good topic, along with "privacy" and all the great spinoffs, like google downranking search results which are not https. I think these days they are mostly removed altogether... why?? Stupidity. Need 10000 bit crypto because some academic showed that 1000 bits can be cracked (in 1000 years and only if you have 10TB of RAM and 1TB of known plaintext) Most editors are clueless and even the best ones are "failed engineers". People are going crazy over privacy these days. GDPR this and GDPR that.

The world has filled up with BS and IOT is just another load of BS.

Even in applications where IOT has value (not fridges or toasters), IOT boxes will never be secure enough over time to avoid IOT getting a really crappy reputation.

What are the views on dyndns? Some years ago that was all the rage, for servers on a dynamic IP. You still have to get the domain hosted somehow.

[madires]

What are the views on dyndns? Some years ago that was all the rage, for servers on a dynamic IP. You still have to get the domain hosted somehow.

There are still many DynDNS services (free and paid) to choose from. But you need to ask your ISP if any ports are blocked by default since they often block TCP 25 (SMTP) and other common sources of trouble.

[SiliconWizard]

Yeah. Of course the problem with DynDNS is that the service you chose may shut off unexpectedly and you'd lose access. There are usually very few guarantees.

[IDEngineer]

Yeah. Of course the problem with DynDNS is that the service you chose may shut off unexpectedly and you'd lose access. There are usually very few guarantees.

That will always be the weakness of depending upon outside third parties for access to your stuff.

There's an additional problem if you trust your data to the "cloud": Ownership and privacy. US courts have ruled that unless you directly control the equipment hosting your data, you "lose expectation" of privacy and have far weaker claims for infringement or undesired access. This is why attorney firms generally don't use "cloud storage" and still run their own in-house hardware - they understand the legal risks of not retaining control of their equipment and the data hosted on it.

I personally try very hard to not store my data on any sort of "cloud" services, and my company does the same. The expectation around here is that if we store data outside our own equipment it's likely to be publicly readable forever. Some of our customers use "cloud" type services but we almost always treat them as read-only from our perspective. If THEY choose to put data we've created for them on the "cloud" that's THEIR assumption of risk.

Bottom line: If you let some third party get between you and your property, you lose a lot of control/rights and assume a LOT of risks. Caveat emptor... "free" is almost never worth more than you paid for it.

[peter-h]

Of course the problem with DynDNS is that the service you chose may shut off unexpectedly and you'd lose access

Sounds like all of IT then

Almost nothing in IT just keeps running for ever. One can get very close to it running some private services, but servers need patching and general cleanups when logfiles overflow, certificates expire, etc. You can remove one of these (e.g. fronting with Cloudflare takes care of certificates) but you can't remove them all. It's a constant cost.

All very relevant to the private servers (with a public https front end) which most IOT needs. Got to be funded somehow, and it can't be funded by a stream of hardware sales.

I have had stuff running on in-house hardware and it isn't cheaper or more reliable. You need a really big UPS for a start. The one thing you do get that way is vast storage which would be expensive on virtual hosting (this is due to the pricing structures of hosting companies, presumably because they don't want BW to be eaten by hosting of p0rn videos, but bandwidth-based billing is a lot harder). Probably at the high end of corporate stuff one can make it work.

[SiliconWizard]

Which is why I suggested to stay away from all that if you can.

[Nominal Animal]

I much prefer a model where each internet-enabled device has a fixed address to a local host, and a pair of keys and passphrases used for encrypting that connection using a known-good algorithm (say, AES256) and identifying the other end, set only with physical access to the device.

Physical access itself would also need a simple key, perhaps one based on factory-fused chip ID or something.

At bootup, reset, or loss of connection, the device makes an initial (TCP) connection to the local host.  The device initiates a handshake with the host, ensuring that both ends know the shared secrets without transferring any of the secrets themselves, and transferring initial vectors for use in the encryption in a secure (un-overheardable, un-replayable) manner.  After this, this (TCP) connection is the only one that the device supports.  (Actually, having more than one allowed local host + gateway pairs, for failover purposes, would be nicer; with TCP connection attempts coming from the failover addresses only answered if there is no current connection or the current connection is stale/timeout.)

The device only needs to expose an UART (four pins: VCC, TX, RX, GND), and one could define a simple, compact format for transferring the configuration data.  Heck, a five-pin SPI might work even better (since it has a separate clock line).  One could then configure such devices using an USB-UART/USB-SPI cable, or even make a dedicated configuration device for field work.  Heck, it ought to be possible to combine even firmware updates to the format...  As an open one, many different devices could use the same format (and therefore config devices and software).

For catering to current practices, have the devices support DHCP, and be individually preprogrammed to use some Cloud host.

Why not?  Because us humans tend to be rather stupid, and equate security with secrecy, even though best security is the kind that is secure even when completely public (like AES encryption standard).

[tellurium]

For catering to current practices, have the devices support DHCP, and be individually preprogrammed to use some Cloud host.

What protocol would they (a device and a cloud) talk?

[Nominal Animal]

For catering to current practices, have the devices support DHCP, and be individually preprogrammed to use some Cloud host.

What protocol would they (a device and a cloud) talk?

Irrelevant with respect to security, since the point is that it is the underlying TCP transport that is encrypted with a symmetric cipher.

In other words, any protocol you want.  The only limitation is that individual logical messages do need to be padded to the cipher block size, or else you need to implement a transport (blocking) protocol on top of the data stream, so that one can construct a full cipher block with only partial data used.

I do have my own: a transport protocol that supports multiplexing multiple independent binary streams, and a list-based format to replace XML with something easier to produce and consume with minimal memory resources.  But these are a matter of use case, since they are all a trade between computational efficiency, protocol overhead, and personal taste.  (Some love netstrings and protocol buffers, others prefer stream formats.  HTTP and HTML are neither, with lots of historical baggage, and require surprisingly large amounts of RAM to properly support.)

[tellurium]

Irrelevant with respect to security, since the point is that it is the underlying TCP transport that is encrypted with a symmetric cipher.

No it is not irrelevant.

You're talking about a simple serial config protocol that many devices/vendors could employ and standardise on. I guess the same applies for the cloud? Then, a standard, well known, simple protocol should be employed for communication. And it does exist : it is called MQTT. Simple, low-overhead, etc etc. And if we're talking about standard MQTT, all your reasoning about custom self-made AES based security is irrelevant, cause you'd need to use TLS.

Of course you can roll out your own auth/encryption mechanism based on AES or whatever, and use you custom TCP server for communication, but I expect the only company that is going to use that, will be your own.

[Nominal Animal]

Irrelevant with respect to security, since the point is that it is the underlying TCP transport that is encrypted with a symmetric cipher.

No it is not irrelevant.

From my point of view it is, because the entire point is to avoid the costly and slow TLS connection setup, by using already existing symmetric cipher key pair, one key per direction.  You know, the one that can take a couple of seconds on an average 32-bit ARM MCU?  When you already have a shared secret per direction, the connection setup can be made much more lightweight using a suitable cryptographic hash.  You do need reliable random nonce generator (I recommend incremental counter and a unit-specific key, put through that same hash function), so it isn't trivial to do it right, but it worked sufficiently well before we standardized on SSL and then TLS.

MQTT ... use TLS

Yes, we absolutely must make all our devices trivially DDOS'able.

The entire point is to achieve security and robustness with currently available microcontrollers and their very limited resources.
TLS has an extremely high connection cost, and requires quite a lot of resources, which is exactly why they are not the best option for current microcontrollers.

That said, I wouldn't mind using MQTT on top of a TCP/IP connection encrypted with a suitable symmetric cipher and pre-shared keys.

I expect the only company that is going to use that, will be your own.

Why should I care?  I only care about the devices I use and rely on myself.  I was just describing what I personally prefer, because I know it works better than the alternatives.

It is people like you who insist on slow, bloated implementations that only seem to work in reality, because on paper they look so hip and cool.  Just like network router manufacturers who seem to hire highschoolers to fork a random Linux installation for their devices, cobbled together with spit and bubblegum, full of security holes and inane choices.  Very, very few companies actually use the people who know how to do that stuff, too.

[Kjelt]

From my point of view it is, because the entire point is to avoid the costly and slow TLS connection setup, by using already existing symmetric cipher key pair, one key per direction.  You know, the one that can take a couple of seconds on an average 32-bit ARM MCU?  When you already have a shared secret per direction, the connection setup can be made much more lightweight using a suitable cryptographic hash.  You do need reliable random nonce generator (I recommend incremental counter and a unit-specific key, put through that same hash function), so it isn't trivial to do it right, but it worked sufficiently well before we standardized on SSL and then TLS.

I agree that you do not need TLS unless you are a public server who needs to communicate with unknown 3rd parties, therefore authenticating eachother over a TTP which bloats everything bigtime.

If you have your own closed system even over the internet the above can be enough when you keep in mind to always follow standardized security guidelines in the communication, which means a single version of the suites offered by TLS, never use the secret key directly but always indirectly with hash with a random number both parties generate in the challenge/response messages.
In the challenge/response both sides proof to eachother that they are who they are this is common security practice, can be found in every security design pattern book.

[tellurium]

It is people like you who insist on slow, bloated implementations that only seem to work in reality, because on paper they look so hip and cool.

I insist to use something standard. TLS is a standard - yes it is bloated, but it is standard. It is known, documented, and pen-tested by many security groups thus it is secure with high level of assurance.

If you're self-baking your own security mechanism to replace TLS, then there is no doubt you can make it significantly more lightweight than TLS. The problem with that it is proprietory, not known, not documented, there is no way to easily interoperate with it. And with high level of assurance, it is not secure - although you might think it is. "Don't invent security" slogan exists for a reason.

There may be an impression that I like TLS. I do not like it. Just as yourself, I think it is bloated and not really suitable for embedded systems. Said that, I still think that TLS is the way go - unless you're a security guru, or there is no way to use TLS in a given project but some degree of protection is still required. I am not a security guru.

When a security community comes together and works out a good, documented, lightweight security standard & implementation for embedded systems - I'll gladly switch to it and forget about TLS.

[Nominal Animal]

It is people like you who insist on slow, bloated implementations that only seem to work in reality, because on paper they look so hip and cool.

I insist to use something standard.

That is your prerogative, but why the heck would you even participate in a discussion that discusses non-standard solutions, then?
To try and force everyone to use the standards you prefer?

Like I said twice already, I'm only describing my preference.  I tried to describe the reasons why: it is a well known technique, whose only drawback is that the shared secrets must be provided separately.  It is the initial (connection-specific) key exchange in TLS that is slow and cumbersome; after the key exchange, TLS too uses a symmetric cipher.  Thus, in my view, the obvious solution is to avoid the key exchange, by prearranging the shared secrets.  This leaves just a lightweight authentication step for the handshake.

Just because something is standard, does not make it good or practical.  Just because a standard is popular, does not mean it is the best option for all the cases where it is popular.
TLS is best we have for transport layer (say, TCP/IP) security, but it isn't a magical bullet; and it isn't the best possible for all use cases.

The problem with that it is proprietory, not known, not documented, there is no way to easily interoperate with it. And with high level of assurance, it is not secure - although you might think it is. "Don't invent security" slogan exists for a reason.

First, it would be worth zero if it was proprietary.  The entire idea is to use the same mechanism with lots of Arduino-style Ethernet-connected devices (I happen to love Teensies, and Teensy 4.1 has both native Ethernet support, and a nice Arduino/PlatformIO development support package, Teensyduino).  Nothing about it is proprietary, and it is a well known technique.

To simplify, it is like TLS except that the costly public-key exchange (using either server public key, or Diffie-Hellman key exchange) is avoided by pre-transferred secrets.  The secrets themselves are never transferred encrypted or in plaintext; only the cryptographic hashes of (the/a) secret and random nonces are, to authenticate the other side.  (I prefer separate passphrases and symmetric keys.)  The authentication technique I'd use is the same as used for password hashing, although there might be even better, zero-knowledge ones, that would still be feasible with current MCUs.

Second, I've done server-side security stuff for a quarter of a century, and am well aware of the difficulty in implementing a "new" security solution.  I am only a hobbyist with embedded devices, and an utter Uncle Bumblefuck when it comes to EE, but I know Unix/SuS/POSIX/Linux software development and especially server side stuff.  That is, I know exactly how easy it is to do it badly by reinventing a square wheel; I would not trust myself to design a new security scheme either.   I am no cryptographer, I only know how to implement the schemes.  The point is, this is not a "new" security solution at all.  I'm old enough to have seen this pattern used before SSL and TLS became ubiquitous, and I happen to know that it still is used by parties who do not trust large integer factorization to be unfeasible, but do trust their physical site security.

(I do use TLS on the server side quite happily, but I do not use the default Apache/Nginx configuration, for example.  I can even point out the flaws in Apache SuExec mechanism that tend to bite people but are not mentioned in the official documentation, if you like.)

When a security community comes together and works out a good, documented, lightweight security standard & implementation for embedded systems - I'll gladly switch to it and forget about TLS.

That would be very nice indeed; I'd love that too.

I don't expect them to, though, because very few cryptography specialists understand the limitations of embedded systems, and how that affects the operations underlying the math.  That is, most of them tend to be mathematicians, with not so good understanding about real-world programming: they do not intuitively grasp the details in computational cost.  The few exceptions (like DJB, V. Rijmen, and J. Daemen) are rather well known, but getting them involved in something like this is a pipe dream.

[ve7xen]

Yes, we absolutely must make all our devices trivially DDOS'able.

What?

The entire point is to achieve security and robustness with currently available microcontrollers and their very limited resources.
TLS has an extremely high connection cost, and requires quite a lot of resources, which is exactly why they are not the best option for current microcontrollers.

It solves a lot of practical and security problems, and is well understood, properly implemented, and compatible with existing infrastructure. The fact that it takes 2s or whatever to establish a connection is practically irrelevant in the type of devices we're talking about that are either going to open a connection and stream data over it continuously, or open a connection occasionally with no real-time constraints.

It is people like you who insist on slow, bloated implementations that only seem to work in reality, because on paper they look so hip and cool.

It's people like you who insist on reinventing the wheel that lead to incredibly fragmented markets where nothing works together, and products go obsolete and have to be tossed in the bin. Standards are never ideal, but they are a far sight better than everyone implementing a shitty version themselves.

Just like network router manufacturers who seem to hire highschoolers to fork a random Linux installation for their devices, cobbled together with spit and bubblegum, full of security holes and inane choices.  Very, very few companies actually use the people who know how to do that stuff, too.

To be honest, your 'solution' sounds far more cobbled together than the mature protocol stack that has developed around standards, and in the real world would almost certainly be far more vulnerable to attack. As you say, few companies know how to 'do that stuff', so they shouldn't be implementing it. And those who think they know what they're doing are constantly adding themselves to the long list of security geniuses who think they know better but end up implementing something incredibly vulnerable to stupid well known attacks. Don't invent your own security.

That would be very nice indeed; I'd love that too.

DTLS + CoAP

[madires]

In case someone likes to learn some basics I recommend to get a copy of 'Network Security' by Kaufman, Perlman and Spencer (Prentice Hall). Not easy to read, but explains how the important concepts and protocols work.

[tellurium]

DTLS + CoAP

DTLS is not much different from TLS.

And CoAP is just an application protocol, like HTTP or MQTT - but over UDP. A pretty shitty protocol, shall I say.

[ve7xen]

DTLS is not much different from TLS.

Use it in one of the PSK modes and it is quite simple and avoids the expensive session setup and key agreement, though of course you could use this with normal TLS as well. DTLS allows avoiding the need to implement TCP, which saves a fair amount of code. It's about as simple as you will get for a properly designed encrypted network protocol.

And CoAP is just an application protocol, like HTTP or MQTT - but over UDP. A pretty shitty protocol, shall I say.

Yes? That's exactly what it is, and it's intended to be easily translatable directly to HTTP. For the basic request/response needs of an IoT device it's generally going to be fine. If you need more capability than that, you can probably afford to implement TCP...

[Nominal Animal]

It's people like you who insist on reinventing the wheel

Which part of "this is nothing new" did you miss?

If this was new, I'd agree with you too.  But it is not.

Don't invent your own security.

GOD DAMMIT, WHICH PART OF "THIS IS NOTHING NEW" IS TOO DIFFICULT FOR YOU TO UNDERSTAND?


That said, it turns out that some TLS libraries –– but by no means all –– do support PSK.

TLS 1.2 and 1.3 do define pre-shared key ciphersuites that would avoid the costly public key math completely (DH, ECDH, or RSA key exchange): TLS_PSK_WITH_AES_128_CBC_SHA256, TLS_PSK_WITH_AES_256_CBC_SHA384, TLS_PSK_WITH_AES_128_GCM_SHA256, TLS_PSK_WITH_AES_256_GCM_SHA384, and TLS_PSK_WITH_CHACHA20_POLY1305 cipher suites; the two first ones supported by MbedTLS, all by WolfSSL.  TLS does add a bit of comms overhead (having to parse the handshake messages to see if there is support for these cipher suites), but that is perfectly acceptable.

The last one, TLS_PSK_WITH_CHACHA20_POLY1305, pre-shared key with ChaCha20-Poly1305 is particularly interesting, as WolfSSL, OpenSSL, and GnuTLS already support it, and even mbedTLS might/should/could/ought to.  Thus, a possible approach would be to define a "TLS 1.3 application profile" (in the RFC 8446 sense) where the only cipher suite supported is TLS_PSK_WITH_CHACHA20_POLY1305, and then either find a configuration of e.g. WolfSSL or fork it (SPDX-License-Identifier: GPL 2.0+) to see how minimal an implementation would suffice.  On the server side, existing GnuTLS and OpenSSL support means developers just need to be damn careful with the pre-shared keys.  On the application side, the nonce generation would need some examination from cryptographers to verify the logic is sound; I'd like some kind of counter, device-specific ident, thrown through a cryptographic hash function, discarding half the data (to throw off attacks based on consecutive observations of the nonce), to obtain the nonce.  That counter must survive powerouts and reboots.

(The fact that DJB is behind ChaCha20 and Poly1305 is not the reason I find it so interesting; nor is it because ChaCha20-Poly1305 is being heavily used by Google and Microsoft (e.g. QUIC).  It's the fact that it looks like it is the most lightweight and performant on processors without specific encryption accelerators, and doesn't need huge buffers.)

Final note: Pre-shared keys, PSK, does have one security flaw, if you consider it one: it does not provide forward security.  On one hand, that sounds bad; but on the other hand, that –– providing the secrets "offline" to the endpoints –– is inevitable anyway, and is in my opinion acceptable tradeoff here.

[Kjelt]

Final note: Pre-shared keys, PSK, does have one security flaw, if you consider it one: it does not provide forward security. 

You mean forward secrecy?
Yes but you should provide an embedded device with multiple unique secret keys, one for the firmware update for instance another for communication etc.
With a firmware update that needs a seperate secret key you can replace or update the communication key(s) if needed 
I am also a big fan of device unique keys in such a way that if one device is cracked/hacked the other devices are not compromised and the hacked devices can be blacklisted on the companies server. A new firmware update round will then suffice to continue and block the compromised devices.

[Nominal Animal]

Final note: Pre-shared keys, PSK, does have one security flaw, if you consider it one: it does not provide forward security. 

You mean forward secrecy?

Yeah, that's the correct term.  Oops.

Yes but you should provide an embedded device with multiple unique secret keys, one for the firmware update for instance another for communication etc.
With a firmware update that needs a seperate secret key you can replace or update the communication key(s) if needed 

That would be even better, yes.  (Perhaps even one for data, one for config, and one for firmware? Perhaps with alternates for each, to make transitions easier?)

Also, if the device exposes an UART or similar local interface for configuration, that too should be possible to encrypt in the exact same way as the IP connection.  By default, I'd like it to be unencrypted, but it could add an extra hurdle for anyone wanting to compromise them.  (In particular, it would make it hard for a nefarious person to just replace my device with a similar-looking and behaving but already compromised one.)

I am also a big fan of device unique keys in such a way that if one device is cracked/hacked the other devices are not compromised and the hacked devices can be blacklisted on the companies server. A new firmware update round will then suffice to continue and block the compromised devices.

Right.

For home devices, I actually prefer devices that one cannot remotely update at all, only via local connection.  It isn't really viable for companies, but for a home user (like my own use cases), I do prefer local-only update/setup, because it makes it easier to write code that is not vulnerable to typical remote attacks.  Remember, I'm talking about "typical IOT devices", Arduino-like gadgets and sensors, not full embedded devices.

As an example, I like my own wireless router config interface to only be accessible on a dedicated ethernet port on it, not wirelessly nor from internet.  It does mean that I occasionally need to plug a laptop or something to that ethernet socket, and see what is going on... but aside from Linux IP stack bugs, it's darned hard to compromise something you cannot reach via IP.  (I do have packet filter rules dropping dedicated config ethernet port source IP addresses from all other interfaces, of course: spoofer, no spoofing.)

[ve7xen]

Which part of "this is nothing new" did you miss?

It's nothing new cryptographically, but you were talking about making your own nonstandardized implementation following a well-trodden path, to avoid perceived limitations of the standard way of doing things, rather than using TLS. This is practically the definition of reinventing the wheel. I suppose this is just because you didn't realize TLS had standardized PSK modes. Though even in the absence of those modes, I do think that in almost any case, the extra cost of the key agreement is not a great argument for making up your own system, even if it is based on sound principles and primitives (which is deceptively difficult).

Doing the PSK way to the Cloud also introduces some practical scaling and key association issues. It makes plenty sense for a locally-administered system, but you probably want PKI for something centrally managed at scale. Or you use the same PSK everywhere, but that sounds like a terrible idea. So I think the PSK model makes the most sense either where you are only interested in local clients, or where there is a local 'gateway' device that implements full TLS for the link to the cloud service. The need for that extra device might also tip the scales a bit on whether you want to include a full implementation directly on the device itself.

Final note: Pre-shared keys, PSK, does have one security flaw, if you consider it one: it does not provide forward security.  On one hand, that sounds bad; but on the other hand, that –– providing the secrets "offline" to the endpoints –– is inevitable anyway, and is in my opinion acceptable tradeoff here.

The middle ground of course is the DHE_PSK modes where you obviously do have to implement Diffie-Hellman, but you can still avoid the x509 mess needed for a complete implementation. I agree though that if you're going that far to get the features you need, you can probably afford to implement the whole shebang, and that PFS is probably not that critical for most IoT situations.

[Nominal Animal]

This is practically the definition of reinventing the wheel. I suppose this is just because you didn't realize TLS had standardized PSK modes.

No.  Using TLS for this is just one option, that's all.

The thing that worries me is whether the TLS libraries do implement it correctly; it is sort of a rare beast.  See e.g. node.js and MbedTLS support (history) for PSK, for example.

It may very well be that QUIC (RFCs 8999-9002) might be a better fit, if we look at standards.

Doing the PSK way to the Cloud also introduces some practical scaling and key association issues. It makes plenty sense for a locally-administered system, but you probably want PKI for something centrally managed at scale.

Like I said, DH and ECDH key exchange can take a couple of seconds of MCU computation.

If the server can contact the device, it is a simple matter of spoofing IP packets for an attacker to trigger costly operations, causing Denial-Of-Service of the device.

So I think the PSK model makes the most sense either where you are only interested in local clients

Which is exactly my intended use case.  The devices themselves are cheap Cortex-M4/M4+/M7 -class microcontrollers.  They have limited IP stacks and not a lot of memory.  If you want something more capable, it is a simple matter of notching up to a proper CPU with a gigabyte of RAM.  I don't want to; I have lots of Linux SBCs I can use for that stuff.  I want more tools in my toolbox, not one-spanner-fits-all-because-standards.

It is my intent that the host (or 'gateway') device(s) IP address(es) would be the only one(s) that the device even reacts to.  That way, direct access to these devices could be completely blocked at the nearest firewall.  The next step is making the device-host/gateway/server handshake lightweight enough that bad connection attempts are rejected in a fraction of a second, and do not consume all of the available computational resources for seconds at a time; while the device cannot communicate upstream, it can still do useful work.

[Kjelt]

For home devices, I actually prefer devices that one cannot remotely update at all, only via local connection. 
It isn't really viable for companies, but for a home user (like my own use cases), I do prefer local-only update/setup, because it makes it easier to write code that is not vulnerable to typical remote attacks.

Yes only one of the issues also seen with the attacks on NAS'ses at the moment, is that the typical home-user is not only not security aware, but also lacks to update the firmware often, to mitigate security vulnerabilities.
In that case you need to support multiple different versions of the firmware/protocol.

Lets discuss the big elephant in the embedded security room: the lack of a small secured storage space (locker/vault) in the uC itself.
Unreadable from the outside (jtag included) and only accessible from within the code, and after erasing the code/or write protection it should be erased instantly.
Manufacturers do become aware of the need but still not that many offer this in their affordable line of uC's.
Without it you need to obfuscate your keys yourself which is mediocre at best.

[tellurium]

Lets discuss the big elephant in the embedded security room: the lack of a small secured storage space (locker/vault) in the uC itself.
Unreadable from the outside (jtag included) and only accessible from within the code, and after erasing the code/or write protection it should be erased instantly.
Manufacturers do become aware of the need but still not that many offer this in their affordable line of uC's.
Without it you need to obfuscate your keys yourself which is mediocre at best.

Yes, keeping keys on an un-encrypted storage is a security hole.

Are you talking uCs providing functionality like secure elements do, e.g. https://www.microchip.com/en-us/product/atecc608a or https://www.st.com/en/secure-mcus/stsafe-a100.html?

As far as I understand, a secure storage is not enough, since if you can read protected data into RAM, you could read it from RAM. So not only private keys are stored inside the element, but crypto operations as well happen there. You give encrypted data to the element, read un-encrypted data back.

[Kjelt]

Agreed but such a device should be inside the uC since monitoring the traffic between external security devices can be logged and all the unencrypted data can be stored. Perhaps the secret key is safe the data is not.
That is why my previous company rejected these external devices.

[peter-h]

The basic problem with IOT security is that you can't ensure physical security.

The attacker is 100% sure to obtain a specimen of your box and attack it using all the usual means. The PCB can be copied; this is a totally casual service in china (I've been offered it for $250, when the PCB company misunderstood and thought I lost the gerber data ). The CPU can be attacked and probably read out. Then any private key can be read out. He can also rig it up and capture any data, which may not be worth anything, but you never know, but it will reveal which services the box is trying to connect to.

If you were making a product which networks wholly within a sphere under your control (clients talking to your server) then a shared key is as good as anything else, and you don't get the 2-3 sec TLS handshake which actually achieves nothing IF you assume that one of your boxes has been disassembled.

TLS is secure only if you assure a) physical security of the server and b) physical security of the client and especially its server certificate store. That's more or less ok for online banking.

The bottom line is that long term IOT product functionality is achievable only if calling up a server under your control.

It has to be under your control because some commercial service can change its API and then all the installed boxes are screwed. Lots of commercial "cloud" services have changed their APIs in recent years. It affected people using e.g. winXP, which could not update its client because the updated client checked the OS version, which you can laugh at, but lots of industrial applications still run on XP and for very good reasons. But this can be solved with a VM, if you are desperate enough. With an IOT box, you will be screwed no matter how desperate you are, unless you implemented "OTA" firmware update, and are willing to take the risk of wide deployment. And for how many years are you willing to support a product with OTA?

And if the server is under your control, you can just use AES256 with a shared key

[Nominal Animal]

Lets discuss the big elephant in the embedded security room: the lack of a small secured storage space (locker/vault) in the uC itself.
Unreadable from the outside (jtag included) and only accessible from within the code, and after erasing the code/or write protection it should be erased instantly.

Absolutely.

For device authentication purposes, it would be even better if one could just supply a salt (half the size of the hash), and after some time has passed, get back a say SHA256 hash of the salt and the built-in secret.  (Basically, a slow hash silicon module with one half of the input value an unreadable OTP.)  This would uniquely authenticate each device in a manner that would make it very, very hard to duplicate the device even if you had full access to the device itself (because finding out the unseen hash takes a lot of iterations; completely unfeasible for SHA256).  The OTP secret pepper would be completely unreadable once programmed, which is the way I like it: it's hard to divulge something you do not actually know!

Manufacturers do become aware of the need but still not that many offer this in their affordable line of uC's.
Without it you need to obfuscate your keys yourself which is mediocre at best.

I don't trust obfuscation.  I prefer schemes where the device end does not actually know the secret, but has a salted hash of the secret.
It is not perfect by any means: an attacker can still duplicate the device, but at least they have to use the same salt and hash.

In this respect, Public Key cryptography is superior, because knowing one of the keys in a key pair does not reveal the other, and what one key encrypts, the other key decrypts.  It's annoying it is so damned slow to do.

As far as I understand, a secure storage is not enough, since if you can read protected data into RAM, you could read it from RAM. So not only private keys are stored inside the element, but crypto operations as well happen there. You give encrypted data to the element, read un-encrypted data back.

Yes, exactly –– and dammit, I didn't know they're that affordable (ATECC608B less than an euro apiece in singles at Mouser, if they had any).

The problem with a separate chip is that an attacker might just swap it to the compromised device; you need to pot it all in epoxy and stuff.

The basic problem with IOT security is that you can't ensure physical security.

It's not with just IOT security, it completely applies to servers as well.  It is a public secret that the easiest way to compromise a company is to compromise someone with administrator access.  Usually the executives are much easier to compromise than the actual IT personnel.

Were you aware (wrt. your embedded HTTPS server task) that crypto chips like as ATECC608B (PDF datasheet) tellurium pointed out, are so cheap and easily available? (Well, not easily now because of chipageddon.)

I definitely wasn't, and am now looking for one to add to my Teensy 4.1 to handle the annoying parts of TLS support.  (The Teensy 4.1's I have do have both native Ethernet and a total of 17 Mbytes of directly accessible memory – 8 + 8 of it in PSRAM –, but still, those crypto chips look very interesting.  Only when bought off reputable western sources, though )

The bottom line is that long term IOT product functionality is achievable only if calling up a server under your control.

Agreed – if by your, you mean the user of the IOT product.  We all know how manufacturers will shut down the servers the instant the product stops making enough profit.

[tellurium]

Yes, ATECC608s are pretty cheap.

It is not a trivial piece of hardware to use, but offers a lot. mbedTLS can be patched to delegate crypto ops to it - see https://github.com/MicrochipTech/mbedtls-examples. Then, not only a key is kept secure, but TLS gets accelerated. Handshake time could be reduced to a sub-second numbers.

The Microchip team that developed ATECC608 did a significant job in promoting it and integrating in many places, especially those associated with AWS. There are modules like https://docs.espressif.com/projects/esp-idf/en/latest/esp32/api-reference/peripherals/secure_element.html with ATECC608 integrated, and many devboards like https://eu.mouser.com/new/microchip/microchip-ac164160-avr-iot-eval-board/.

Another functionality ATECC608 provides is the ability to mass-provision devices - the mechanism is not trivial but when it comes to large volumes, it becomes crucial to make it secure. AWS was mentioned - they have a mass-provision process based on ATECC608.

[Nominal Animal]

The Microchip team that develops ATECC608 did a significant job in promoting it

Perhaps to the hardware folks, but definitely not to the software development side; not even to Linux kernel folks (which is, no matter how you look at it, a huge part of the embedded networked device world).  Neither linux-kernel nor openwrt-devel mailing lists have a single mention of the chip, for example.

I do understand it is impossible to reach everyone, I'm just trying to convey my surprise at these chips existing at this price point; and therefore, I expect/suspect many other integrators and software developers to not know either.

(You don't happen to have a couple of these chips spare in your kit? Sorry, had to ask )

[peter-h]

The problem with a separate chip is that an attacker might just swap it to the compromised device; you need to pot it all in epoxy and stuff.

This is always an issue where an external chip contains something of value, and is traditionally addressed by copying the unique S/N (which these devices contain) to somewhere inside the CPU FLASH. Or write the unique CPU S/N (which some CPUs have) to some secure storage. And compare the two at startup. This works unless someone disassembles the firmware... Not sure if there is any basic way to solve this "transplant" issue.

Were you aware (wrt. your embedded HTTPS server task) that crypto chips like as ATECC608B (PDF datasheet) tellurium pointed out,

Yes, I was. These chips have been around for a long time, and before that you could just put a smart card chip on your board. I did a project 27 years ago with a Siemens sle44c200 which did on-chip crypto and was a few quid back then. The Microchip chips are just a lot cheaper and faster.

The CPU I am using, 32F417, has hardware crypto, but doesn't do the really useful ones like RSA and EC. It does DES, 3DES and AES but only AES is in common use now and that is used for the session, not the handshake.

We all know how manufacturers will shut down the servers the instant the product stops making enough profit.

That's true also.

[Nominal Animal]

The CPU I am using, 32F417, has hardware crypto, but doesn't do the really useful ones like RSA and EC. It does DES, 3DES and AES but only AES is in common use now and that is used for the session, not the handshake.

Seems like ATECC608B, if they were available, would have been a good addition; it'd have handled the critical weakness of 32F417 (handshake and certificate verification), while leaving the symmetric encryption to the 32F417.

(But I really know nothing about real EE design; I'm just a verbose hobbyist software dev/physicist/bumbler.)

[peter-h]

I do agree but it depends on whether

- you want to stick even more eggs into the one basket (which these days would have been a disaster; no production for some years)
- you actually need a fast TLS handshake (faster than 3 seconds); I'd say in most embedded applications you don't need it because TLS will be a client, thus hidden
- you need a device with a unique ID (useful for a lot of stuff but the 32F417 has one already, as has the Adesto SPI FLASH I am using, etc)
- your product is likely to attract counterfeiter attention (99% of stuff won't)

The issue I had with TLS messing up something else was due to poor software design; if done properly across RTOS tasks, the TLS delay will not screw up other stuff. The problem is that this stuff is so complex that almost nobody understands it, so people are reduced to integrating libs and spending months looking for bug patches for this "open source" (read: unsupported) code on the internet.

[Nominal Animal]

I'd say in most embedded applications you don't need [a fast TLS handshake] because TLS will be a client, thus hidden

The operations involved in the TLS handshake make it pretty difficult to portably co-operate with other stuff.  Being able to offload the computation-heavy part –– be this a client or the server –– to a separate chip is definitely easier.

Consider this from the library developers' point of view.

They can make everything 'monolithic' (in the thread/task sense), and leave any thread/task switching to others.  Easiest, and most portable approach, so this is the usual choice.

You can design the library API so that each call does a capped amount of CPU work.  That amount varies on every architecture, because it is defined at the source level; and the code complexity shoots through the roof.  Not worth it for a portable library.

Now, if you offload the most computationally intensive (and most difficult security-wise) parts to a separate chip, the second one becomes feasible, because typical "heaviest" operations are moving buffer contents or parsing through received headers (marking values seen into flags, because these clients don't really need the header values per se, just detect certain header values).  Essentially, by just designing what each call does sensibly, you get an acceptable implicit runtime caps, without there being any runtime capping in the code, by virtue of what each call needs to do.

So, I'm not saying these crypto chips are interesting because "wow! faster", but because "hmm; this could really simplify a callback-based/event-driven HTTPS library API".  Like I said, my Teensy 4.1's have 17 MiB (17,825,792 bytes) of directly accessible RAM, so I'm nowhere near as constrained as you are.  (But still, I'm purely a hobbyist, so the comparison is completely nonsensical.  Also, I constantly forget the business aspects of this for you. Sorry.)

The problem is that this stuff is so complex that almost nobody understands it

I suspect it is more that "nobody wants to spend the time and effort to learn this stuff", because there are so many libraries and easier ways to get seemingly similar results (by using existing libraries without contributing to them and fixing them first).  And that those who have done it, work as a developer/integrator to one of the embedded device manufacturers at quite good salaries...

When I first started with crypto stuff (again, mostly server-side back-end programming), the learning curve was steep.  Vertical, really, even when OpenSSL came along.  (Before that, I had used Blowfish, and hash-based stuff with MD5 and SHA, but all with pre-arranged keys and XOR-data-with-hash stuff –– output feedback or OFB mode; no key exchange or anything really complicated.  Even that took me time to get right, but then again, I could just be a bit slow.)

[kjpye]

The Microchip team that develops ATECC608 did a significant job in promoting it

Perhaps to the hardware folks, but definitely not to the software development side; not even to Linux kernel folks (which is, no matter how you look at it, a huge part of the embedded networked device world).  Neither linux-kernel nor openwrt-devel mailing lists have a single mention of the chip, for example.

Possibly because the datasheet seems to be only available under NDA.

[tellurium]

(You don't happen to have a couple of these chips spare in your kit? Sorry, had to ask )

I'm pretty sure I do have a couple of them.
My company was one of the first integrating them in our product and made it work with AWS IoT (then featured on AWS's reInvent 2016).

As a demo, we had a MQTTS client (which uses two-way TLS with AWS IoT) which listened for a specific MQTT topic. The data on that topic was expected to be a JS code. Once a device received it, it executed the code and sent a result to a response topic. That was just to demonstrate how MQTTS client and JS engine running on a small device (that was ESP8266 with 50K free RAM). At the time, both TLS library and JS engine were our own. Since then, we've switched to mbedTLS cause it was free since ARM bought it.

PM me your address, I'll send over.

[tellurium]

Possibly because the datasheet seems to be only available under NDA.

Yep. It is not an issue to get it though, but unfortunately all secure element vendors I know, require NDA.
Maybe that's a regulations thing.

[Nominal Animal]

The Microchip team that develops ATECC608 did a significant job in promoting it

Perhaps to the hardware folks, but definitely not to the software development side; not even to Linux kernel folks (which is, no matter how you look at it, a huge part of the embedded networked device world).  Neither linux-kernel nor openwrt-devel mailing lists have a single mention of the chip, for example.

Possibly because the datasheet seems to be only available under NDA.

Aw crap.  I missed that, even though it is highlighted in orange at the top of the datasheet.

On the other hand, the interface library to the chip is available at github, so I don't think Microchip is per se hostile to open source and developers like myself; it wouldn't be the first NDA I've signed (and scrupulously followed).

(As to the license stating "exclusively with Microchip products", it does not say "with exclusively Microchip products".  The library itself makes that clear, by stating that it "is structured to support portability to ... multiple environments including bare-metal, RTOS, and Windows/Linux/macOS".)

Yes, I too believe this NDA is some export control thing.

[peter-h]

The chinese always get these "confidential" data sheets immediately. See this thread for example
https://www.eevblog.com/forum/embedded-computing/sd-card-proprietary-modes-have-these-escaped-into-the-wild/msg3781682/#msg3781682