How would you do a heating controller, IOT client, securely?

[nctnico]

The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.

The only way to combat this is to assume everyone is a potential exploiter.

This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.

Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.

I don't think this will lead to a workable situation. If you have many devices in the field, you'll have a call center for customer service. People from this call center will need to be able to look at payment status and be able to change the state of the subscription quickly in order to keep customers happy. This is the area where the 'accounting' pillar of good security practises comes in. At some point the system will need to cross reference revenue in versus services rendered. When you break this down to an operator level, you can easely see if a call center employee is handing out too many freebees or not.

I'm not sure whether having seperated databases is a good idea. Lots of synchronisation issues. A long time ago I used stored procedures (which also did authentification) on a PostGresql database to keep an IoT device seperated from the actual database.

[Nominal Animal]

The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.

The only way to combat this is to assume everyone is a potential exploiter.

This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.

Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.

I don't think this will lead to a workable situation.

You mean, you think implementing sensible business practices requires actual experience and knowledge of how to do secure systems, and thus is too costly in your opinion?  That kind of thinking is exactly why so many peoples' personal details and credit card information has been "accidentally leaked".

People from this call center will need to be able to look at payment status and be able to change the state of the subscription quickly in order to keep customers happy.

Sure.  They have very limited access to the customer/payment database.  In that database, any change to the service status must trigger a related push to the IoT server within a guaranteed service interval, and here that could be seconds, say one minute maximum to see the server response change.

When you break this down to an operator level, you can easely see if a call center employee is handing out too many freebees or not.

That kind of thing is easily monitored automatically, and is not an actual problem in real life.

I'm not sure whether having seperated databases is a good idea. Lots of synchronisation issues.

There is no synchronization.  There is exactly one direction in which changes are pushed, and enforcing this is crucial for the security model.  If you have synchronization issues doing this you should not be doing anything with databases.

Even having the IoT server be technically capable of querying the customer/payment database for details instantly breaks the security model.  Separated databases are required for true privilege separation.  If the IoT server is technically able to access the customer information, then the customer information will be exfiltrated in the very first break of the IoT server.  In my opinion, any company that does that should get heavy penalties when that happens.

Security models that survive in the real world are those that do not assume every part in the model is secure.  They still provide security, albeit in degraded/reduced form, when invididual parts are compromised.  There are single point failure sources, like the customer/payment database implementation; minimizing both their number and access/exposure to them is at the core of proper security design.

[nctnico]

The point I'm making is that confining information to a few points is likely to end up with a system which isn't useable in a practical situation. You'd be unpleasantly surprised if you know how many people at a bank have access to your records. Same for the phone company. My assumption is that accesses are logged and audited though.

And there ain't no such thing as 'one way pushing' between systems. Synchronisation between distributed systems will fail at some point for some reason. So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information (whether this process is hidden from the application layer or not). And what to do when the backend database is not working? Ideally the call center operator needs to have a way to see the backend is malfunctioning so there has to be some access to the backend as well from the call center. Again, you'll need information at various places. Worse if transactions need to be corrected manually. The British Post Office has learned that the hard way (see Mr Bates vs The Post Office ).

If you look at the stacked layers of a typical system you can also draw a column spanning all layers next to it. This column is called 'logging, error and exception handling' and typically ends up asking for operator intervention. Another weak point.

[Siwastaja]

So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information

But the whole point is that the databases should have different information. If they have the same information, there is no security benefit from the separation at all and then indeed only extra complexity and syncronization issues ensue. But this wasn't at all what Nominal Animal suggested.

[nctnico]

So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information

But the whole point is that the databases should have different information. If they have the same information, there is no security benefit from the separation at all and then indeed only extra complexity and syncronization issues ensue. But this wasn't at all what Nominal Animal suggested.

Nobody is suggesting that both databases are or should be 100% identical. Only that there is shared piece of information (a subset of all information available) between the databases which needs to be kept in sync. In case of a system with IoT devices this shared information could be about the services enabled in the IoT device.

[Nominal Animal]

My phone, electricity, and insurance companies all have web portals for payments, bills, and service status.
These obviously have access to the payment database, but with very limited privileges.  They need that access to work.

However, an electricity meter (power meter) installed at a client site will not have direct access to the customer/payment database.
It will access a separate database containing only information relevant for electricity meter operation, typically with customers identified by an arbitrary ID included in the meter.  That database is synchronized periodically with the customer/payment database.

If the electricity company provides an interface for customers to monitor their power use, that will use a yet separate database, containing only electricity usage histories (typically at reduced resolution) and no payment history.

Many legal auditability requirements can only be fulfilled by separating different parts of the services into separate databases, and periodically checking for differences, and having an exact transaction record.  Many of them are used to monitor for both technical problems as well as human operator misbehaviour.

While it is technically easiest to use a single database for everything, it also means any problem anywhere in your system is potentially catastrophic.  Yes, there are companies that operate this way, but I condemn them to hell.  There is very little difference to setting up a company that really is just a front for its executives to sell the customer information on the black market.  Ineptitude should not be a legal defense at all.

And there ain't no such thing as 'one way pushing' between systems.

Bullshit.  You typically do that via an intermediary that has very limited access to the customer/payment database (typically read-only plus delete access to a single table containing pending changes), read-write access to the IoT client database, is strictly limited in its automated actions, and leaves an auditable trail of every single transaction it makes between the two databases.  Exactly how this is set up depends on the database engine used.

It is the intermediary, via its access permissions, that provides the privilege separation between the two databases.

I would expect this to include a full state update every N hours using the same mechanism but as if all customer states had changed, with the intermediary checking for differences between the IoT database and the information from the customer/payment database, as any differences would indicate some updates were lost which should never happen.

[nctnico]

And there ain't no such thing as 'one way pushing' between systems.

Bullshit.  You typically do that via an intermediary that has very limited access to the customer/payment database (typically read-only plus delete access to a single table containing pending changes), read-write access to the IoT client database, is strictly limited in its automated actions, and leaves an auditable trail of every single transaction it makes between the two databases.  Exactly how this is set up depends on the database engine used.

It is the intermediary, via its access permissions, that provides the privilege separation between the two databases.

I would expect this to include a full state update every N hours using the same mechanism but as if all customer states had changed, with the intermediary checking for differences between the IoT database and the information from the customer/payment database, as any differences would indicate some updates were lost which should never happen.

The latter is exactly what I wrote in the part you snipped.  Indeed you'll need to read the data back from the target database in order to verify it (and take action when the verification fails). That is a bit more involved than just pushing data (one way) onto the target database and call it a day. You can only say that from a very high level functional perspective.

Still, you can ask yourself the question whether you'd actually need the second database for the IoT devices. What functionality does it add to the system? Why can't they connect with the intermediary directly and fetch data from the primary database through the intermediary?

[Nominal Animal]

Still, you can ask yourself the question whether you'd actually need the second database for the IoT devices. What functionality does it add to the system? Why can't they connect with the intermediary directly and fetch data from the primary database through the intermediary?

It provides privilege separation (principle of least privilege) in the case the IoT server is penetrated.  The separate database accessible from the IoT server limits the data that is exfiltratable in the case of full penetration of the IoT server.  It also helps avoid single point of failure.

Consider the case of the intermediary requiring updates.  Let's say that takes 15 minutes.  In your suggestion, that would mean the IoT server being down for 15 minutes.  In my case, it would mean changes in the payment database take effect on the IoT server 15 minutes (plus a bit) later.

Furthermore, consider the case where the IoT servers are geographically distributed across the world (or just a continent, if you prefer).  I for sure would consider the distributed replication of the full customer/payment database a huge risk, because there is no need.  Instead, I would only distribute the IoT service databases, with the intermediaries managing the data updates.  In the case of all IoT servers being breached, customers' payment information would be absolutely safe; additional systems not accessible by the IoT servers at all would need to be compromised for that.

(To make it clear: the intermediary is an automated process, with its own limited access to each database, which also provides an auditable trail of the changes it has applied.)

[zilp]

https://blog.apnic.net/2024/06/18/off-path-tcp-hijacking-in-nat-enabled-wi-fi-networks/

Seems relevant to this. Forget about the WiFi part here, the interesting part is that many home router style devices don't even seem to check TCP sequence numbers, so getting packets to an IoT device behind it is easier than one might have thought. Which just goes to show that relying on NAT for security is not a good idea. It's not meant as a security mechanism, manufacturers don't consider it a security mechanism, and obviously design their devices accordingly.

[peter-h]

Interesting - many thanks.

However, the attacker still has to send his packet to the right IP. And these IPs will be

- in most cases, dynamic (though probably static for days or longer)
- fairly randomly distributed across the IP address space (but if you know the ISP, that narrows it down a bit)

And AIUI he has to do the attack when there is some outgoing NAT-crossing traffic. So a lone IOT box behind its own NAT router should be OK.

Interesting that Draytek are not in that list. And Linksys is a bought-in chinese outfit, not "really Cisco".

[zilp]

However, the attacker still has to send his packet to the right IP. And these IPs will be

- in most cases, dynamic (though probably static for days or longer)
- fairly randomly distributed across the IP address space (but if you know the ISP, that narrows it down a bit)

... or not. And even if, it's not all that relevant anyway.

If we are talking about targeted attacks, then those things might help with hiding a vulnerable device. But then, with targeted attacks, it's probably a bad assumption that the attacker doesn't know the address already anyway. There are so many ways to find out someone's IP address if they operate any of their end-user devices there, that you generally should assume that the attacker knows the IP address.

And if we are talking about non-targeted attacks, then it's a numbers game. If a million devices is hidden behing 10 million IP addresses, then it's not hard to hit most of them. And in any case, sending a packet to every single IPv4 address is perfectly doable, too, 4 billion isn't that much (and you can ignore multicast and RFC1918 and stuff anyway).

And AIUI he has to do the attack when there is some outgoing NAT-crossing traffic. So a lone IOT box behind its own NAT router should be OK.

Only if that lone IoT box isn't talking to some server "in the cloud", as that would be NAT-crossing traffic.

[Siwastaja]

A very realistic scenario:

Some IOT product, say a tv/media box, a party speaker, talking toaster, whatever, gets really popular, so that every other household (of those who would buy any IoT to begin with) has those. We have numerous examples of such products, many of which were supported by manufacturers only for a year or two and then possibly bricked due to lack of interest from the manufacturer.

Their server gets compromised - you can do nothing about it.

Attackers use that device to scan every network for IP addresses of anything connected to the same router. Very simple, they can scan the whole space (just 256 addresses) within the same subnet. (This is exactly what our box is intentionally doing to discover compatible devices to control, to avoid having users to configure IP addresses.)

Now they have discovered significant % (possibly double digit) of your heating controllers, and have access to every one of them (through the compromised toaster). The last and really only line of defense is the design of your product.

Don't stress about it too much; but be honest to yourself and do your best, instead of trying to figure out flawed excuses why security isn't important for you.

[peter-h]

If a million devices is hidden behing 10 million IP addresses

If I am selling a million boxes, I will get some specialist to design the system, not ask bland one-liner posters on EEVBLOG

that you generally should assume that the attacker knows the IP address.

If these boxes were sold to customers via some retail channel(s) then the above assumption is hugely invalid. How could you possibly know. Even if you could get into that retailer's database and get the user names and addresses, that won't tell you...

Their server gets compromised - you can do nothing about it.

Useful but only if that server is holding client IP data. It might not. It doesn't need to. Maybe in logfiles... The server does need to hold valid subscription data (IOT box S/N or some such).

do your best

Sure, but one needs to be realistic about the attack surface. That attack mentioned above is likely to be really laborious.

One bottom line which I find hard to escape from is that there are many types of remotely controlled heating controllers, marketed into the airb&b type of space where the hosts obviously need this (because few have 100% occupancy) which are not obviously being hacked and trashed all over the place. These are mostly plain servers sitting on an open port! How do they work? The firmware is mostly never updated, and anyway code like MbedTLS is hardly high grade server quality code, and never can be.

[zilp]

If a million devices is hidden behing 10 million IP addresses

If I am selling a million boxes, I will get some specialist to design the system, not ask bland one-liner posters on EEVBLOG

Sure. I mean, if you know it beforehand. But the point is that this is a sliding scale, as I have said before: If it's a one-off project, you probably don't have to worry too much, the more units you sell, the more risky it gets. But the risk doesn't start at a million.

that you generally should assume that the attacker knows the IP address.

If these boxes were sold to customers via some retail channel(s) then the above assumption is hugely invalid. How could you possibly know. Even if you could get into that retailer's database and get the user names and addresses, that won't tell you...

I think you are misunderstanding the distinction that IT security people make between targeted and non-targeted attacks.

Non-targeted attacks are when the attacker simply tries to compromise as many devices/systems as possible, but doesn't care whose devices/systems in particular those are. Be it to use the compromised systems to attack other systems or for distributing spam, or to install ransomware and extract money from whoever happens to be the victim, or to extract data to either blackmail whoever the victim happens to be with threats of publishing the data, or to sell the data or access to the compromised systems on the black market. The point is that an attacker doing a non-targeted attack doesn't have the goal to attack any particular company or person, but rather will just try to compromise as much as possible with as little effort as possible, so as to maximize profit. Which is why the difficulty of findind a particular device is irrelevant for non-targeted attacks: Any sufficiently popular device with a known vulnerability will be at risk of being targeted with non-targeted attacks.

Targeted attacks on the other hand are when an attacker starts with the goal of compromising a particular person, company, government, whatever, because they want to hurt or sabotage them or extract information from them, or whatever. And one of the first things that a targeted attacker will try to figure out is public IP addresses or their victim, and, as I said, there are plenty of ways to do this, so you absolutely should assume that a targeted attacker will know the IP address(es).

Their server gets compromised - you can do nothing about it.

Useful but only if that server is holding client IP data. It might not. It doesn't need to. Maybe in logfiles... The server does need to hold valid subscription data (IOT box S/N or some such).

No, you get the IP data by waiting for the devices to connect to you.

do your best

Sure, but one needs to be realistic about the attack surface. That attack mentioned above is likely to be really laborious.

You obviously have no idea how attackers work (and in particular what tools they use), so I would strongly recommend that you don't rely on your intuition as to what attacks are laborious. Just because you have no idea doesn't mean people who do this for a living don't either.

One bottom line which I find hard to escape from is that there are many types of remotely controlled heating controllers, marketed into the airb&b type of space where the hosts obviously need this (because few have 100% occupancy) which are not obviously being hacked and trashed all over the place.

Why does it matter whether the hacking is "obvious"? And obvious to who anyway? Obvious to the owner? Obvious to the victims of DDoS attacks that get knocked offline with traffic generated by all kinds of compromised systems? Or ... who else? And why would you possibly expect an attacker to trash a compromised device? How would a attacker profit from trashing devices?

All of this ist just you telling us that you have no idea what is going on in the IT security space. Just because you haven't heard of attacks doesn't mean that they aren't happening, and your gut is not a reliable source for finding out about how systems get compromised and abused and how often that happens.

Also ... possibly all of that is just because those particular devices happen to be secure? Which tells us nothing about your designs?

These are mostly plain servers sitting on an open port! How do they work? The firmware is mostly never updated, and anyway code like MbedTLS is hardly high grade server quality code, and never can be.

"high grade server quality code" isn't a thing.

[peter-h]

Then explain how one can have a website running Linux, Centos, NGINX, etc for 10 years and it hasn't been trashed.

My point is that embedded code has no hope of reaching this level of robustness.

[zilp]

Then explain how one can have a website running Linux, Centos, NGINX, etc for 10 years and it hasn't been trashed.

Why would anyone trash anything? That's just one huge non-sequitur.

My point is that embedded code has no hope of reaching this level of robustness.

Yeah, I understood your point just fine. What is missing is a justification for why you think that would be the case.

[Siwastaja]

Their server gets compromised - you can do nothing about it.

Useful but only if that server is holding client IP data.

Obviously it does; but it appears you did not understand the scenario I explained.

Obviously their server connects to their devices, that's the definition of IoT. Type and "direction" of the connection does not matter the slightest. Obviously their server can update the firmware of their own devices, I don't think there are many IoT devices which do not do that. Therefore, it is trivial to inject a backdoor into the compromised devices. Communication to these backdoors happen simply via the compromised server.

Using the backdoor you can just scan the whole subnet and get the local 192.168.x.y addresses of anything connected, including your heating controller. And that same backdoor can connect to those devices; just as easily as our "heating controller" scans the 256 addresses of the subnet to find a battery inverter, connects to it, and starts commanding it. Works fine on all 200+ customers having those battery inverters.

If they were doing their due diligence, they would notice the weird traffic patterns, but if it is another company where people like you decide, then it will be wide open for months, maybe years.

If I am selling a million boxes, I will get some specialist to design the system, not ask bland one-liner posters on EEVBLOG

It's not that obvious. Maybe your heating controller is the best thing since sliced bread and you get surprised by the sales growth. If it happens at a point where 2-3 years of development already went into the product, you could have a codebase which is large and mostly doing good things (i.e., important features). Such codebase is slow and expensive to rewrite; and while in sudden growth situation, you obviously can't say "sorry, we can't sell, we are redoing the project, please come back in 1.5 years"; no, you must keep selling. So any safety improvements must be something this "specialist" can glue on within a few man-months. But safety as an afterthought is a colossally bad idea.

That is why you should initially make good fundamental choices. I can assure you, from the experience we are getting right now, during fast growth even just fixing bugs (even of a relatively well-designed system) and serving customers takes all of your time. Similarly, if you hire a security expert, all of their time is spent fixing bugs and issues even in a well-designed systems (there is no such thing as "perfect").

If you initially decide "oh, we are just selling some tens of units, so no one is important, so safety is not important, so we don't even try to do it right", then that decision is pretty much set in stone. And I hope everyone involved truly understands this choice and its implications, basically that you can't ever grow as a business, and you can't sell it to others who would like to make it grow. Or -- you can, but then you will become one more horror story we can read in newspapers, and someone might end up in jail. No kidding. And investors specifically do not love being conned; you can't omit the fact hard-limiting decisions like this have been made; -- you can, but again, you might end up in jail.

OTOH, "doing your best" and "being honest", or in legal speak, due diligence, is surprisingly strong protection, even if things end up going horribly wrong.