How would you do a heating controller, IOT client, securely?

[globoy]

The architecture would be interesting.

Not that interesting or perhaps helpful.  I'm sure I got bit because I am lazy and primarily counted on obscurity for protection (as was mentioned earlier in this thread).  I used a localhost forwarding service running on a Pi to make the internal webservers of some devices (cameras, temperature monitoring, etc) visible at a set of fixed domains presented by the localhost forwarding service provider to deal with changing ISP addresses.  Everything has login accounts and I hoped the fact that the domains were not registered anywhere (they were sub-domains to the service provider - one per service) would provide the obfuscation.  It worked for a while but was discovered and the Pi owned.  Everything went offline.  I had the neighbor login to the Pi locally and he found that they had somehow changed the ssh login so that it never provided a shell prompt.  Of course I kept putting off updating software on the Pi because that would be some work. So there's that too.  Fortunately nothing was control.  Only monitoring.

I'm going to figure out something new (a dedicated VPN between the houses perhaps), completely reload the Pi software from scratch and wipe and restore the router.  Also wipe and reload as much of the monitoring device's system firmware as possible (and maybe take any system I can't wipe permanently offline).

[Silenos]

I would add my story of being attacked: I was messing with pbx server on linux device, with SIP protocol (which immeasurably sucks, and it runs on plain text packets by default). Stuff was on my home network, with another pc generating test SIP traffic, all on fixed static ips.
Suddenly my logs exploded with the flood of declined INVITEs (sip packets to initiate the session). Checked the wireshark, saw the external traffic and run to kill the router, just in case.
It was a bot which likely was generating the variations of packets to either bruteforce the session or ddos or probe for no password session (I thankfully had only password ones). The bot got interested likely because it the server was correctly responding with 4xx and it never got any further in the session. And attack packets did not even bother to strip the open source pbx label used to generate the traffic or the users/uris string suggesting russian origins of this shit.
Still not sure how it got discovered, I did not have the prior traffic recorded. Though get soured once again against the internet-connected devices.

[peter-h]

SIP hacks are real and frequent. A friend lost 5 digits to this. Misconfigured (by him) Cisco gear

However one would not be running SIP on a heating controller

[globoy]

So peter-h, what are you going to do for this system?

[peter-h]

I am waiting to learn how you can port scan millions of internet IPs to locate say 100k heating controllers which are behind NAT routers

And without using obscure side attacks.

NAT has no open inbound ports, unless there is outgoing traffic. There are possible issues e.g. PnP routers having inbound ports open by some other products' installation code; I don't know how common this is. I've seen a few try it (one was a Vodafone SureSignal femtocell, and in general VOIP products might do it).

[globoy]

I'm wondering how the IOT part works.  How is it accessed/configured/updated remotely?

[peter-h]

See the earlier posts in this thread. The vendor provides a server which the IOT boxes periodically call up to collect config updates or whatever.

Providing that server also delivers a long term revenue stream to the vendor, which is important otherwise he will have no way to fund long term support of an "internet product" which will lose some big chunk of its functionality if something (beyond his control) changes (a simple example is the painful process of constantly developing an app for the constant stream of new versions of IOS or Android).

[globoy]

Thanks.  Yeah,  I saw those -  using https and talking to a web server.  I was wondering if you could share any detail about this mechanism.  How will you authenticate with the server and/or have the server authenticate with the device?  Other than encrypting updates will you do any additional authentication or validation of them?  I guess you'll have storage for at least 2 firmware images and a roll-back mechanism for failed updates.  Any other specific security related functionality?

I ask because I have a similar project for a client who a) doesn't want to spend money on security and b) has an existing web infrastructure to which he is adding the device I am working on for him so I have to follow an existing web API.  It's easy to say stuff like, "well this device will be turned off most of the time" and there's only going to be hundreds or thousands of them in the field so no-one will care so we don't have to worry about the attacks zilp has detailed but I'm still nervous, if only for my reputation, because I know how this client will react if the device is ever compromised (he'll blame everyone but himself). 

[nctnico]

@globoy: I'd walk away from that project if the customer doesn't want to be sensible. Do it right or don't do it at all.

I have a similar project on my hands (bigger amount of units though) and I took the initiative to get the security implemented properly. At some point you have to protect a customer against themselves.

[peter-h]

As I said I am not actually doing a heating controller but may one day do something which uses a similar setup for remote admin and OTA updates.

The obvious way is to use TLS (not because you have to but because it is a standard, and customers who are "internet security experts" (everybody these days) will demand it. So you would have a certificate on the server, preferably not self-signed. The client (a TLS client does not need to store any keys etc) will check this (by reference to cacert.pem or similar) and connect to the server periodically. So there is server authentication, plus session encryption. The good thing about this part is that you control both ends so no need to worry about the server suddenly not supporting say AES so all the clients need a firmware update, which could be disastrous if it goes wrong.

If you do it carefully you won't need firmware updates for many years - other than client functionality. Note that the vast majority of "retail IT" boxes are never enhanced, especially not heating controllers (I've seen remotely controlled products which are 15+ years old).

Obviously the server needs to be secure, otherwise somebody could do something malicious and trash the installed base (config damage, or firmware damage if OTA is possible). I don't see a way to 100% avoid this, other than not have the product features. There are many details you could do to tighten things up but ultimately it all hangs on things like the security of your DNS control panel. If that is compromised, you are buggered. You can't even DKIM-sign emails if your DNS is compromised. I don't know much detail on server config; I've paid others to do that.

As pointed out by others further back, the two other options which avoid this central vulnerability are

- access the IOT boxes over a VPN (needs the customer to have a VPN-capable router (too much to demand IMHO)
- have the IOT boxes as servers i.e. on an open port (and hope that none of the guaranteed flood of attacks will compromise them - way too much to hope for with an embedded system running on something like an arm32)

The 2nd option above also requires the customer to have either a PNP router (and your app installation will open a port) or if he has a half decent router he has to do this manually (and he may have an opinion on that; rightly too).

But as I've said this "central vulnerability" is also your long term revenue stream. Without this you are selling boxes, each one you sell just once, and you need to keep selling them for ever to fund long term support. It's a crap business model.

[nctnico]

Unfortunately something like TLS Interception exists:
https://english.ncsc.nl/binaries/ncsc-en/documenten/factsheets/2019/juni/01/factsheet-tls-interception/Factsheet+TLS-interceptie+1.1+EN.pdf

[zilp]

So you would have a certificate on the server, preferably not self-signed.

Can you explain what the advantage of having it not self-signed is?

[tellurium]

Thanks.  Yeah,  I saw those -  using https and talking to a web server.  I was wondering if you could share any detail about this mechanism.  How will you authenticate with the server and/or have the server authenticate with the device?  Other than encrypting updates will you do any additional authentication or validation of them?  I guess you'll have storage for at least 2 firmware images and a roll-back mechanism for failed updates.  Any other specific security related functionality?

I ask because I have a similar project for a client who a) doesn't want to spend money on security and b) has an existing web infrastructure to which he is adding the device I am working on for him so I have to follow an existing web API.  It's easy to say stuff like, "well this device will be turned off most of the time" and there's only going to be hundreds or thousands of them in the field so no-one will care so we don't have to worry about the attacks zilp has detailed but I'm still nervous, if only for my reputation, because I know how this client will react if the device is ever compromised (he'll blame everyone but himself).

One way I've described earlier:

    DEVICE  <---- Websocket over TLS --->   BACKEND <---- HTTPS ----->  OPERATOR

The backed would terminate connections from devices. Devices can be authenticated either:
  a) using TLS certificates - so-called two-way TLS. That's the way how AWS IoT authenticates devices
  b) using auth tokens, or password strings. Both device and backend would store those, and device can send either a token ot its hash in the WS handshake

For users / administrators, the backed would provide a usual HTTP REST API. Usual auth schemas are user/pass login, or Oauth.

Now, OPERATOR can authenticate with BACKEND and send REST API calls to list devices, see their status, change their config, update them, etc.

[tellurium]

I have a similar project on my hands (bigger amount of units though) and I took the initiative to get the security implemented properly. At some point you have to protect a customer against themselves.

Heh, yeah.
They pay for security either way.
One way is to pay early, and to pay less.
Another way is to pay late, and pay much more, with money and possible lost business.

[Siwastaja]

The client (a TLS client does not need to store any keys etc)

But of course the client does need to store their own private key. How otherwise would you authenticate the client?

[tru]

Unfortunately something like TLS Interception exists:
https://english.ncsc.nl/binaries/ncsc-en/documenten/factsheets/2019/juni/01/factsheet-tls-interception/Factsheet+TLS-interceptie+1.1+EN.pdf

Erm, that is so misleading.  You need client machine access to install and trust a root certificate on the client PC.  It's a known technique mainly used by web developers for security debugging purposes.  Fiddler classic is such a tool that does this.  A root certificate needs to be installed:
https://docs.telerik.com/fiddler-everywhere/installation-and-setup/trust-fiddler-ca

[peter-h]

But of course the client does need to store their own private key. How otherwise would you authenticate the client?

Obviously you know the answer

The client accesses the server by doing a DNS lookup. Now obviously the DNS could be poisoned. But the client can be disassembled anyway and any private key extracted from it. There is no perfect solution, unless you put in a secure crypto module.

However one can add additional security within the comms, and normally one would even if TLS is used. For example firmware updates would have a hash on them, although again this is not secure from client disassembly.

Ultimately any single client can be messed with, but that's OK. What you really do not want to happen is somebody trashing the installed base of say 100k of them, or any significant part of it. And that seems fairly easy to do so long as the server is not penetrated. The next thing to do is set up the server so it does not contain a database of the installed clients. I think that is easy enough because the ultimate mutual authentication could be between the client and the customer's app trying to access it. The server's purpose can be just to facilitate the clients being, ahem, clients, all behind various NAT routers so they can't all be casually discovered.

I am sure many people have been around this before (it is a very common application) but they won't be posting the details here But I also know that many don't bother; I recently stayed in accommodation where the heating was remotely controlled and the controller was a server and the router had port forwarding.

[zilp]

But of course the client does need to store their own private key. How otherwise would you authenticate the client?

Obviously you know the answer

No, obviously not, because you can't.

The client accesses the server by doing a DNS lookup. Now obviously the DNS could be poisoned. But the client can be disassembled anyway and any private key extracted from it. There is no perfect solution, unless you put in a secure crypto module.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

However one can add additional security within the comms, and normally one would even if TLS is used. For example firmware updates would have a hash on them, although again this is not secure from client disassembly.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

Ultimately any single client can be messed with, but that's OK. What you really do not want to happen is somebody trashing the installed base of say 100k of them, or any significant part of it. And that seems fairly easy to do so long as the server is not penetrated.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

The next thing to do is set up the server so it does not contain a database of the installed clients. I think that is easy enough because the ultimate mutual authentication could be between the client and the customer's app trying to access it. The server's purpose can be just to facilitate the clients being, ahem, clients, all behind various NAT routers so they can't all be casually discovered.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

I am sure many people have been around this before (it is a very common application) but they won't be posting the details here

Obviously not for someone who dismisses all advice anyhow because they feel superior in their cluelessness and prefer insecure solutions.

[peter-h]

It is important to be aware of one's weaknesses, even if they are pointed out without offering much useful detail

[zilp]

It is important to be aware of one's weaknesses, even if they are pointed out without offering much useful detail

The useful details have already been offered. You dismissed them all.

Also, I asked you a question some posts ago that maybe could get you to reflect on where you lack understanding. Apparently, you simply ignored that question.

Also, feel free to actually spell out the reasoning behind this previous post of yours, as to why you think that all those conclusions of yours make sense. Maybe, just maybe, that will make you realize that you have no idea.

It is just blatantly obvious that you are regurgitating these ideas that you have heard somewhere without any real understanding of how they relate or what their applicability is in a given scenario. That whole post is a series of non-sequiturs that maybe is held together in your head by some vague superficial conceptual connections, but just make no sense at all if you actually had any more than surface-level understanding of the subject.

Like, there is a context in which people (experts) will recommend that you should not be using self-signed certificates, because self-signed certificates in that context have security risks. But if you had any understanding of how any of this works and why people say that in that context, you would realize that the reasons for that recommendation makes no sense in your application, and that in actual fact, a self-signed certificate might actually be the more secure choice.

If you want to design a secure system, you have to get beyond the stage of repeating rules of thumb that you've heard somewhere, and actually understand the reasoning behind those rules of thumb. You have to understand the properties of cryptographic primitives (like hashes, signatures, MACs, symmetric and asymmetric ciphers, key agreement schemes) and how they are employed in crypto protocols like TLS, and you won't get there by smugly dismissing everything people who actually understand the subject are saying.

[Siwastaja]

But the client can be disassembled anyway and any private key extracted from it.

Easily, but that is why you use unique private key for each client, so that you can't buy a product and then pretend to be another customer. But yeah, if you break in or otherwise gain physical access to the heating controller, then you can read out the private key, and pretend to be that single heating controller. This is usually the level of risk we can accept, because one could also just replace the physical heating controller with any malicious device (which could connect to another server completely).

In my opinion, authenticating the client is important so that you can't pretend being another customer, but your mileage may vary.

[peter-h]

Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?

Then you are vulnerable to staff theft.

[Siwastaja]

Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?
Then you are vulnerable to staff theft.

Well, if you offer a paid service then quite obviously you can't avoid having a list of customers, and if you have that, then it can leak. Important consideration is to avoid putting any unnecessary sensitive data into that database. For example, I think, but I'm not a security expert, that the device private key should not be stored anywhere else except the device itself.

Think about leaking list of your customers with their billing info (i.e. home addresses) combined with eavesdropping of the realtime heater controller data, from which is easy to see if they are on vacation. Burglars are happy!

[zilp]

Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?

Then you are vulnerable to staff theft.

Staff theft ... of what?

[Nominal Animal]

The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.

The only way to combat this is to assume everyone is a potential exploiter.

This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.

Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.

The customer/payment database does have the above abstract client ID, and whenever the payment state changes, only that change is pushed from the customer database to IoT server, preferably via an intermediate which separately logs such pushes, isolating the customer database from the client/IoT database in an auditable way.