How would you do a heating controller, IOT client, securely?

[peter-h]

The context is that an IOT box should always be a client, behind NAT, and never a server, especially not on an open port.

The problem is if you want it remotely accessible. There are many such products.

This has been done for decades (I used to do one in the 1980s which was accessed over a phone line modem, so the security was a) knowing the phone # and b) knowing the password) but how do people do it nowadays?

If the IOT box is a client, there is no way to connect to it freely (obviously). I suspect the existing products are either a server on an open port (with all the vulnerability issues) or the thing which the user's "heating config app" connects to is not the IOT box itself but a copy of its config residing on the public server which fronts the entire installed base of these boxes, and the IOT box retrieves this config periodically, say every 15 mins.

The obvious advantage of the public server is that the mfg gets a long term revenue stream And this means that when the mfg goes bust, you lose the remote config capability. And this has happened a number of times...

[Jeroen3]

The way it is done today: 
Both controller and IoT box are clients and connect to cloud server to relay the commands.

If you do not want to run your own server there are IoT platforms available to integrate. Eg: particle.io, balena.io
There also was an Amp Hour podcast with the founder one of those platforms iirc.

Also VPN solution if you products support it, eg: Zerotier, Tailscale...

[zilp]

The context is that an IOT box should always be a client, behind NAT, and never a server, especially not on an open port.

Why?

[AVI-crak]

Control of gas boilers, ventilation, lighting - must be fully autonomous. External control should be completely excluded.
The logic of the smart home should filter reports and warnings for the user, so as not to overload with unnecessary information.
An external server can change the address, owner, gender, or just send it on a pedestrian journey. The world is changing very quickly.

[selcuk]

Are you asking about a secure communication method over the network? If so, you may use MQTT with TLS. Your device can stay behind a firewall. It will have a continuous TCP connection to a MQTT server. The client initiates the connection and doesn't require a port forwarding on NAT. Your central application will have another connection to the same MQTT server. Then they can share messages instantly. There is no need to wait for the next poll time.

This is especially useful if you use GSM or similar paid networks. When the client doesn't have a static IP address and isn't accessible from the outside, random services trying to connect it cannot increase the used data and the bill will be limited.

[Siwastaja]

The context is that an IOT box should always be a client, behind NAT, and never a server, especially not on an open port.

Why?

One reason that comes into mind is that a TCP client taking a connection to a server on the Internet will always work without the user having to think about firewall/router/NAT configuration - if it didn't, pretty much nothing would work and they would notice it. It is also then irrelevant what IP address the device obtains, as long as the server address or hostname is correctly configured in the device, and this is possible to fix.

Having a local server accepting incoming TCP connections is possible, but more iffy to set up.

[zilp]

One reason that comes into mind is that a TCP client taking a connection to a server on the Internet will always work without the user having to think about firewall/router/NAT configuration - if it didn't, pretty much nothing would work and they would notice it. It is also then irrelevant what IP address the device obtains, as long as the server address or hostname is correctly configured in the device, and this is possible to fix.

Having a local server accepting incoming TCP connections is possible, but more iffy to set up.

Well, that is a reason to have a client on the IoT box. It's not a reason to not have a server. And also, it's not really a reason for it to always be/have a client, at best it's a reason to have a client if that is an application scenario that you need to support, which isn't universally the case.

The point being that I was not asking why one might want to have a client on the IoT box, but why that should always be the case, and why there should never be a server.

After all, you can have a client thingy that registers with the manufacturer's servers for remote discovery/connection establishment/whatever purposes, say, but also support, I dunno, local discovery via broadcast/multicast and incoming connections via other name resolution mechanisms (i.e., hard-coded addresses or DNS, say) to allow for operation without internet connectiviy and/or the manufacturer's infrastructure.

[peter-h]

I would have thought it was obvious why you don't want to have an embedded system as a server on an open port, but e.g.

- the IOT box will be detected by port sniffers within hours and attacked
- its software will never be up to date security-wise (due to constant R&D cost, and commercial risk of OTA)
- auto update download (client-triggered OTA basically) is possible but risky since the mfg publishing a bad update could brick the entire installed base!
- look at how many versions of say MbedTLS come out, and the recent ones (v3+) are a big job to integrate into a product which has v2
- only a proper heavy duty server (e.g. Centos, Linux and Apache, NGINX etc) should be exposed, and the admin and security patching of such is much easier
- the server can be protected against DOS with e.g. Cloudflare
- IOT box having both client and server is a whole load of RAM; MbedTLS client alone is ~60k
- the client mode means the entire "connection" which the IOT box has to support is private and hidden, which enables operation with no software updates for years or decades
- one would use TLS for data encryption, with the server certificate having a 100 year life (one can argue that one may as well just run aes256 with a shared key; there is no difference between a shared key and a long lived certificate, is there?)

Both controller and IoT box are clients and connect to cloud server to relay the commands.

I guess you mean a) the customer's app and b) the heating controller are both clients and connect to the "proper heavy duty server". a) is obvious and b) is what I am getting at.

If so, you may use MQTT with TLS

Yes; I have read up a bit on MQTT. But it doesn't directly address the main reason for the architecture I suggest. MQTT is just a more efficient protocol, and it is a separate debate on how exactly to do the protocols. The way I would do this is that the heating controller ("the IOT box") is an HTTPS client, and HTTPS is implemented with MbedTLS (my product does exactly all this already, although it isn't actually a heating controller).

The IOT box would call up the server say every 10 mins to collect any config changes, and since it is behind NAT it would be very hard to hack it. NAT can be penetrated using a couple of methods but AFAIK both involve - at a minimum - the attacker gaining control of the server which the IOT box periodically calls up, which is obviously possible but with good admin should be hard, otherwise the internet could never exist Then if the IOT box is hard to trash (a simple way would be to have no way to remotely re-flash it) you have pretty good security. My box does support OTA updates but the security around it is very good - basically as good as the 32F4 level 2+ protection (yes I know there are attacks on these chips, and the attacker will always be able to get hold of one of your products for disassembly).

It would be highly risky to build this around some commercial service. These have a habit of ditching protocols and needing a big redesign.

I don't get the advantage of a VPN for this.

Obviously the server will need careful firewalling. If say you have an installed base of 10k boxes, each one will need to store an https url for the server (not a fixed IP which is dodgy long-term) but publishing a DNS will immediately start chinese etc hacking. But this is no worse than attacking the server's customer side. Servers just have to be properly set up.

[AndyBeez]

Personally, I would take a path of creating a locked down IoT device. Regardless of the client only model, a locked down device should be the default in today's networked landscape. As well as considering data in transit   through TLS encryption, for example, the design should be robust enough to resist deconstruction by dumping the flash. This can be done with a secure element or hard key store - the hacker gets garbage. Collectively, all design features must guarantee that all iincoming malicious attacks will never reach the GPIO.

PDF IoT Security Foundation : IoTSF Secure Design Best Practice Guides
https://www.iotsecurityfoundation.org/wp-content/uploads/2019/03/Best-Practice-Guides-Release-1.2.1.pdf

[peter-h]

Interesting doc - thanks.

However, most of it is obvious, and some bits e.g.

All test access points on production units must be disabled or locked, for example by blowing on-chip fuses to disable JTAG

is a misunderstanding of what is actually possible e.g. the vast majority of embedded CPUs can and have been hacked via the old methods of glitching VCC, /RST, clock input, low/high temp, etc. The really secure chips e.g. smartcard chips, can't run these IOT applications, and/or tie you to some weird device, tools, etc. There is some level of reverse engineering risk which is really hard to avoid.

But does it matter? The really risky area is somebody hacking the installed base of devices, not just a few of them. This will always be possible, indeed trivial, if somebody hacks the server (whether or not the actual devices are affected, is irrelevant, because obtaining control of the server enables misconfiguring the entire installed base, and setting 10k heating controllers to +5C 24/7 is as good as wiping their flash). But server admin is a well established discipline...

[AndyBeez]

You need to define both your threat landscape and the cost of compromise. I can understand your server-less methodology, but is a base unit really the server backbone? Fundamental question, why does everything need a cloud connection? You could have  IoT devices 'meshed' together on some kind of VLAN - but when devices span networks, we enter a whole new house of pain.

Poor design, implementation, infrastructure and support, are a few areas where IoT has gained a bad reputation. Who cares if a kid in China can read everyone's home wifi key off of their spin dryer? Maybe the peado on the laptop parked outside does. As an IoT designer, building a black box should be the philosophy. Unfortunately, most IoT hardware is some ESP32 variant with a open source codebase. Secure elements and secure processors are the reserve of large companies who signed the NDA.

[peter-h]

why does everything need a cloud connection

A higher-end heating controller needs a remote config feature.

For example you stay in a ski resort. Most apartments are owned by remotely (no pun intended) located people, who get notifications from booking.com or airb&b that somebody booked a 5 day stay, arrival 1400, so they turn on the heating remotely.

Like I said, this requirement has been there since for ever. In the 1980s you used a phone line (which was normally shared with the house phones; the heating controller would go off-hook only after say 20 rings) and a modem chip or module. I even did one with an MT8870 DTMF decoder for turning on/off one of 8 relays

Today you do it over the internet, but the problem is that everything which you can remotely connect to is easily discoverable and therefore hackable. Port knocking is quite good. And with an embedded system you have poor defences because the software is of necessity rather primitive. I see the MbedTLS devs (I am on their mailing list) grind through N versions in a year or two, and address increasingly esoteric "vulnerabilities" like zeroing malloc'd buffers before doing a free() (which are worthless unless an attacker is practically inside your box already, but there is nothing to help with back doors in the low level code connecting e.g. LWIP to the ETH hardware). This IOT stuff is simply impossible to secure... the only way is to have it hidden behind NAT and have it run as a client.

I agree with your other comments. The problem is that everybody on the internet is a security expert So, DES, PPTP, FTP, MD5, etc are all "crap". Yet practically nobody is attacking crypto itself.

[zilp]

I would have thought it was obvious why you don't want to have an embedded system as a server on an open port, but e.g.

Yeah, that's what it seemed like. Which is why I am trying to get you to really think this through. This is widely believed, especially among people who don't know all that much about IT security, but really, it's pretty close to cargo cult and often leads to bad security.

- the IOT box will be detected by port sniffers within hours and attacked

How would it be attacked? And how does not having a server prevent that?

- its software will never be up to date security-wise (due to constant R&D cost, and commercial risk of OTA)

And how does not having a server help with that?

- auto update download (client-triggered OTA basically) is possible but risky since the mfg publishing a bad update could brick the entire installed base!

As for accidental bricking, there certainly are things one can do to minimize that risk (like two boot images in flash with some automatic or manual way to recover into an old version if the new version is broken).

But also note that automatic updates are a huge security risk, as a compromise of the update deployment system can be used to fully compromise and/or brick all installed devices.

- look at how many versions of say MbedTLS come out, and the recent ones (v3+) are a big job to integrate into a product which has v2

I am assuming that you are talking about versions due to security fixes here? If so: How does not having a server help with that?

- only a proper heavy duty server (e.g. Centos, Linux and Apache, NGINX etc) should be exposed, and the admin and security patching of such is much easier

It seems like you are suggesting that an IoT device that only acts as a client is not exposed? Why do you think that?

Really think about this.

If there is a buffer overflow in the IP layer, say, how does not having a server on the device prevent exploitation of that buffer overflow?

- the server can be protected against DOS with e.g. Cloudflare

Maybe. But be aware that that can also be a huge security risk, especially if this implies TLS termination by Cloudflare (Cloudflare is a very juicy target).

- IOT box having both client and server is a whole load of RAM; MbedTLS client alone is ~60k

Of per-connection state, you mean? Well, yeah, that's a thing to consider, but certainly not a reason to never have both.

- the client mode means the entire "connection" which the IOT box has to support is private and hidden, which enables operation with no software updates for years or decades

No, it most certainly does not mean that. If you want to estimate IT security risks, you have to think in terms of attack surface, i.e., what parts of the code in your system are reachable by an attacker, and how difficult it is to get into a position that would allow that attack to be performed.

Take my example above: The low-level IP stack does not care about "client" or "server" at all, all it is concerned with are IP packets. Just because you happen to only use one TCP socket to establish an outbound connection, doesn't mean that someone who can send IP packets to your device can't exploit a vulnerability in that code.

Now, you might be thinking: "But how would anyone possibly send IP packets to the device?" Well, for one, even from the public internet through a stateful firewall that blocks incoming connections, that is way easier than your wording above would suggest.

A firewall isn't a magic device that prevents packets from all unauthorized sources from reaching your device. All it does is that it matches a bunch of fields in a packet against fields it has seen in other packets to determine whether the packet might belong to an established connection. But the entropy of those fields isn't all that high. With TCP over IPv4, that's 32 bit each for the IP addresses of the two endpoints, 16 bits each for the port numbers of the two endpoints, and the 32 bit sequence number.

So ... 128 bits? That should be enough to prevent someone just blindly guessing the values to get a packet through the firewall, shouldn't it? Well, if these indeed were purely random 128 bits, sure. But of course, they aren't. In your IoT-device-connects-to-manufacturer's-server scenario, the IP address of the server is public knowledge, so trivial to "guess". The port number of the server is also public knowledge, so doesn't help either. Now, if the attacker has no idea where the devices that they are trying to attack are on the network, then you could consider the client IP address as sort-of random. But of course, that's not really necessarily true, as it might not necessarily be a secret that someone has one of your devices installed, and thus, if someone were to try and run a targeted attack against them, they might easily know the (public) IP address where the device is to be found (be it a business with static addresses, or you get the victim to load some page or image or whatever from your web server, or maybe you can get them to send you an email where the email provider puts their address into the email headers ...), so you really have to assume that the client IP address isn't exactly a secret either.

Well, then we at least have 32 bits of random sequence number, right? Well ... for one, you'd have to make sure that the initial sequence number is really random (especially on embedded systems with bad randomness sources, that can be a problem, though most implementations nowadays at least try to make the ISN unpredictable). But also, the firewall can not do an exact match, because it has to allow for reordering and packet loss within the transmission window. I would be surprised if any firewall let through less than 64 KiB of window, this reducing the remaining entropy of he sequence number fo 16 bits. And I would not be surprised if many firewalls didn't check sequence numbers at all.

So ... well, I guess the only somewhat reliable entropy is the client port number, then? That also depends on the client actually randomizing the port number, of course. And also, if your code establishes multiple connections in short order, chances are that the firewall takes a while to time them all out, so multiple port numbers might actually be let through.

Also, mind you that NAT is not a firewall, and chances are that NAT is less strict than a firewall when matching connections.

So ... you are relying on 16 bits of randomness to prevent an attacker exploiting a vulnerability in your IP stack. I.e., on average 32768 packets to compromise your device. So, with a Gbit/s fiber link, that would be matter of less than a milisecond.

And all of that assumes that the attacker can't eavesdrop the legitimate connection, which is also a risky assumption to make.

And that is just one of many possible attack vectors. Depending on the vulnerability, an attacker might also just be able to use other devices behind the firwall to attack your device, for example. If the user has a smartphone or PC on the same network (which would certainly be common in home setups), an attacker might be able to get the user to load some web page that then can use the numerous javascript APIs in the browser to send packets to your device without those packets ever even passing through a firewall that might exist between the public internet and your device.

- one would use TLS for data encryption, with the server certificate having a 100 year life (one can argue that one may as well just run aes256 with a shared key; there is no difference between a shared key and a long lived certificate, is there?)

That depends on the cipher suites that you are using with TLS. Best practice nowadays would be to only use cipher suites with forward secrecy, i.e., cipher suites that use ephmeral (elliptic curve) diffie hellman key agreement, so that a compromise of the server's secret key does not allow decryption of past connections.

Also, as PSK that's used directly for payload encryption is at high risk of IV reuse and/or difficult to reliably protect against replay attacks.

Or more generally: If you have to ask this question, do not even think about designing your own cryptographic protocol, including "just encrypt the data befrore transmitting it". You are almost guarenteed to fuck it up in a way that allows trivial compromise of the encryption.

The IOT box would call up the server say every 10 mins to collect any config changes, and since it is behind NAT it would be very hard to hack it. NAT can be penetrated using a couple of methods but AFAIK both involve - at a minimum - the attacker gaining control of the server which the IOT box periodically calls up,

No, that is not necessary. Actually, that wouldn't even count as "hacking the NAT", as the server that the client legitimately connects to obviously can send packets to the client device, that is just how a NAT is expected to be used. Of course, there will also be potential attack surfaces that are reachable only from that server, but that would just be using the NAT the way it is intended to be used, not in any way "hacking" it.

Really, the idea that NAT "prevents unsolicited connections" is just a myth. It is a very commonly believed myth, but it still is just a myth.

which is obviously possible but with good admin should be hard, otherwise the internet could never exist

Well, in any case, it is strongly recommended to have defense in depth. It is generally a very bad idea to have a central point of (security) failure, because that makes for a juicy target. While it might be true that it is easier to secure one server than thousands of appliances (which doesn't work anyhow, because you still need to secure the appliances, see above), that one server is also a far more interesting target for attackers than all those appliances that might not be that easy to find otherwise.

Then if the IOT box is hard to trash (a simple way would be to have no way to remotely re-flash it) you have pretty good security.

For one, no, that doesn't mean that you have pretty good security. If your heating controller can be made to burn down the house, then it is irrelevant whether it is possible to remotely re-flash it.

But also ... if an attacker can obtain remote code execution through some vulnerability, how do you prevent them from just rewriting the flash contents, assuming the MCU supports self-flashing? Unless you are talking about an MCU with MMU and unprivileged processes and stuff ... which still assumes correctness of the code that's enforcing that security boundary.

It would be highly risky to build this around some commercial service. These have a habit of ditching protocols and needing a big redesign.

Agreed.

Obviously the server will need careful firewalling.

If your server needs firewalling, you have already failed.

[AndyBeez]

With any remote device out there on the Internet, the only assumption one can make is that everything network side of the device's MAC address is insecure.
For example, the IT administrator at the ski resort might be a lazy tool and, without telling anyone else, resets every switch and access point to their default 1234 login credentials. This really happens in the real world.
How then do you keep your IoT devices safe from network attacks? Answer, you cannot. Anyone with access to the ARP can run amok. So if the IoT device can be disrupted on the network, how do you keep it from being damaged? Answer, secure design. Okay, firmwares and hardware do not need to be as tight as a crypto wallet, but it has to be impervious by design. WAN facing servers should provide minimal functionality and secure only access. There cannot be any backdoor IPs or ports either.

Do not forget how useful Bluetooth can be, where proximal access to the IoT device is required.

[zilp]

I agree with your other comments. The problem is that everybody on the internet is a security expert So, DES, PPTP, FTP, MD5, etc are all "crap". Yet practically nobody is attacking crypto itself.

But that's just a "you don't have to outrun the bear, you just have to outrun the slowest guy" kind of scenario.

As long as there is so much terrible quality stuff out there that can be compromised using trivial exploits, obviously, noone is going to bother with attacking the crypto, unless they want to get into a particular target where no easier path exists. But especially for long-lived products it's maybe not a good idea to bet on everyone else staying terrible for the lifetime of your product.

[eutectique]

- the IOT box will be detected by port sniffers within hours and attacked

How would it be attacked?

By flooding the network with bogus ACK or SYN/ACK packets. I witnessed the fleet of battery-powered devices being knocked down.

A device in question could normally live 12 hours on a battery charge, and a fleet of them should be located behind the firewall. One day the customer reported battery life of only about 5 hours for the whole fleet. The logs showed surge of power consumption in all of ~30 devices. This was traced down to continuous modem activity dealing with the flood. Which was, in turn, traced down to the absence of the firewall.

If your server needs firewalling, you have already failed.

No.

[peter-h]

zilp - you make some good points but when one gets to

If your server needs firewalling, you have already failed.

then there isn't anything one can say in response

With any remote device out there on the Internet, the only assumption one can make is that everything network side of the device's MAC address is insecure.

That isn't totally possible, however. What is supposed to happen is that an incoming packet gets to your TCP/IP stack, say LWIP, and if nothing is listening on that port then it will be discarded. If there is e.g. buffer overflow in LWIP, you can have a problem. But to get to this point, someone need to do quite a bit of work. Hacking some wifi AP to do nasty stuff takes a fair bit of expertise. You also need an extremely deep knowledge of the IOT box software. Bits of it may be open source but most won't be (hopefully).

By flooding the network with bogus ACK or SYN/ACK packets. I witnessed the fleet of battery-powered devices being knocked down.

Sure, that comes down to stuff like not allowing unrestricted RX interrupts on the ETH hardware. In my box I do it by polling the RX. One still gets 1MB/sec which is 100x fast enough for the job The industry drift is for great speeds, zero-copy drivers, etc, but that just makes it vulnerable to e.g. a misbehaving device on the LAN sending out broadcasts at 100mbps. That is a real issue with embedded and ETH. But still somebody needs real expertise and, to set up the flood, decent bandwidth.

[zilp]

How would it be attacked?

By flooding the network with bogus ACK or SYN/ACK packets. I witnessed the fleet of battery-powered devices being knocked down.

A device in question could normally live 12 hours on a battery charge, and a fleet of them should be located behind the firewall. One day the customer reported battery life of only about 5 hours for the whole fleet. The logs showed surge of power consumption in all of ~30 devices. This was traced down to continuous modem activity dealing with the flood. Which was, in turn, traced down to the absence of the firewall.

... which is relevant here how?

If your server needs firewalling, you have already failed.

No.

... because?

[zilp]

zilp - you make some good points but when one gets to

If your server needs firewalling, you have already failed.

then there isn't anything one can say in response

... because?

Like, name one thing that a firewall would be needed for that isn't the result of a failure elsewhere?!

[madires]

I would have thought it was obvious why you don't want to have an embedded system as a server on an open port, but e.g.

- the IOT box will be detected by port sniffers within hours and attacked

We can't make IoT stuff idiot-proof. There's always some professional moron connecting the control system of a water treatment plant directly to the internet and keeping the default PW. However, it's about choices. We shouldn't require the IoT device to connect to a cloud based service, it should be an option. The IoT device should also have the option to connect to a local control server or to be controlled via an on-board service. And tell the user the best current practice to protect the device.

[peter-h]

Like, name one thing that a firewall would be needed for that isn't the result of a failure elsewhere?!

One should firewall a server to accept only traffic it is supposed to be getting. So e.g. you firewall a web server to accept only 80, 443, and the other stuff like Putty (remote admin), DNS, and such like. Otherwise, in the usual virtual server scenario, absolutely everything sent to that IP will need processing.

You can also set up stuff like IP banning following a flood, etc. On one site I "run" it is behind Cloudflare and I have blocked china, russia, india which greatly reduced various nuisances. CF offers 5 such blocks for free. Obviously hackers can use VPNs but the indications are they mostly don't bother and DOS over a VPN will cost the attacker money.

Luckily, virtual servers tend to have huge incoming bw like 50Gbit/sec, so a firewall actually works.

But I am sure you knew that

This is not wholly OT because a firewall is not going to be available in an embedded box, and if it was (which would hardly be difficult) it would be of limited use because a DOS attack will just overwhelm the CPU anyway. Some CPUs with ETH do have a packet filter so LWIP will be seeing only packets destined for its IP (plus, ahem, broadcasts etc).

We can't make IoT stuff idiot-proof. There's always some professional moron connecting the control system of a water treatment plant directly to the internet and keeping the default PW.

Sure, and my original point was that in your above case the "control system" is probably a server, maybe even with a published DNS, so half of china is going to be in there, with 24/7/365 dictionary attacks at around 10Hz (I see this on some servers). But if it was a client connecting as I suggest above, it would not be attacked like that.

[madires]

I don't get the advantage of a VPN for this.

Layered security. Attacking becomes harder.

[madires]

[

If your server needs firewalling, you have already failed.

No.

... because?

... the firewall can perform rate limitting or handle other offloading tasks.

[peter-h]

I use VPNs too and have wondered about this.

A port sniffer will instantly discover VPN termination ports, whether pptp, ipsec, openvpn, whatever. Then it can send packets down there.

The assumption is that a given router (probably a consumer type, or maybe a £500 mid-level one) has fewer bugs in the VPN terminator than are elsewhere. That is probably right as in e.g. Remote Desktop is wide open and should be run only over a VPN. But is this relevant to IOT? Would you actually implement a VPN terminator in your IOT box? How much less vulnerable is that to setting up an https server on that IP?

[madires]

Sure, and my original point was that in your above case the "control system" is probably a server, maybe even with a published DNS, so half of china is going to be in there, with 24/7/365 dictionary attacks at around 10Hz (I see this on some servers). But if it was a client connecting as I suggest above, it would not be attacked like that.

Best current practice is to not connect the control system directly to the internet. You place it behind a VPN router, connect via a VPN client to the router and then access the control system (layered security).

[madires]

A port sniffer will instantly discover VPN termination ports, whether pptp, ipsec, openvpn, whatever. Then it can send packets down there.

And the VPN router will handle that. That's its job. And there are several ways to make it very hard for attackers.

The assumption is that a given router (probably a consumer type, or maybe a £500 mid-level one) has fewer bugs in the VPN terminator than are elsewhere. That is probably right as in e.g. Remote Desktop is wide open and should be run only over a VPN. But is this relevant to IOT? Would you actually implement a VPN terminator in your IOT box? How much less vulnerable is that to setting up an https server on that IP?

The IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).

[zilp]

Like, name one thing that a firewall would be needed for that isn't the result of a failure elsewhere?!

One should firewall a server to accept only traffic it is supposed to be getting. So e.g. you firewall a web server to accept only 80, 443, and the other stuff like Putty (remote admin), DNS, and such like. Otherwise, in the usual virtual server scenario, absolutely everything sent to that IP will need processing.

But a firewall doesn't change that!? Somewhere, each incoming packet needs to be matched against the set of sockets in order to figure out whether the packet should be dropped. Whether the component that does that is a "firewall" or an "IP stack" doesn't make a fundamental difference. If you are talking about a stateful firewall, you are just doubling the amount of connection state that you have to hold, as the firewall will duplicate all the socket endpoint data in its state tables rather than matching against the socket tables that are there anyway.

You can also set up stuff like IP banning following a flood, etc.

That is a DoS vector. Unless you do this only for validated paths, this allows an attacker to deny clients access by flooding your server with packets from that client's address.

On one site I "run" it is behind Cloudflare and I have blocked china, russia, india which greatly reduced various nuisances.

Yeah, you are reducing nuisances. I was talking about security. Reducing sshd logging failed passwords attempts doesn't do anything for security.

CF offers 5 such blocks for free.

You can do such blocks locally. But it's mostly pointless anyway.

Obviously hackers can use VPNs but the indications are they mostly don't bother and DOS over a VPN will cost the attacker money.

Attackers who want to exploit actual vulnerabilities in your system will bother. People trying stuff that doesn't work anyway aren't worth any mitigation effort. And also, an attacker that wants to DoS you will use booter services that use compromised machines or amplification attacks, so the traffic will not be coming from the location of the attacker.

Sure, and my original point was that in your above case the "control system" is probably a server, maybe even with a published DNS, so half of china is going to be in there, with 24/7/365 dictionary attacks at around 10Hz (I see this on some servers). But if it was a client connecting as I suggest above, it would not be attacked like that.

Which is completely irrelevant as that "attack" can not be successful anyway unless you have failed elsewhere. Relevant are attacks than can compromise your system. Dictionary "attacks" against systems that don't use terrible passwords can not compromise your system. What you have to worry about are vulnerabilities in your system, not people trying to exploit vulnerabilities that don't exist.

As far as resource consumption is concerned, you can avoid that by not making the server easily discoverable. But also, chances are that you won't see any dictionary attacks anyway against a server that doesn't use passwords. You see SSH dictionary attacks because SSH supports password authentication and there are servers out there that use weak passwords. If you use a protocol that only supports public key authentication, noone will ever try to guess a password that obviously doesn't exist.

[unseenninja]

Rate limiting is no longer a useful defence. When I exposed an ssh server through my router to test as a remote access solution to my home network, it was detected within 8 hours by several of those companies who regularly scan the Internet and publish information about all the IP addresses they find which have any open ports.

Within minutes of this, the brute force attacks started. Given that I only allowed a single key as login credentials to the server, trying to brute force your way in with a user name and password is pointless as the server immediately rejects an attempt to log in that way. Nobody will get in, but it still ends up eating my inbound bandwidth and making the ssh server a target for more advanced groups who can exploit zero day vulnerabilities in the ssh server to break in.

Initially, I thought that the good old solution of running "fail2ban" would help, but a closer examination of the incoming traffic revealed that the brute force attacks switched to a new IP address after each block of eight attempts. The first block would come (for example) from Dallas, Texas. The next block would come from Phoenix, Arizona and the next from Madrid, Spain. The crooks make good use of the vast networks of compromised home PCs that they have access to and simply jump from zombie to zombie, rendering all "fail2ban" like defences pointless. Even more simple rate limiting will only end up locking you out from your own network.

If you really want to enable access to resources inside a protected network from the Internet, the only remotely secure way to do it is to run an ssh server on a VPS. Only the VPS' IP address is allowed to connect to the ssh server at the protected network. It's not a 100% solution compared to not doing it in the first place, but it saves your local bandwidth and limits your exposure to your ability to get the ssh server on the VPS patched before those with a zero day exploit get there.

A long time ago, I had a QNAP NAS and I knew less about network security. I thought it would be a great idea to be able to access my NAS when I was at work or travelling. QNAP offered a service where a program on the NAS would connect to the QNAP company's servers and they would act as a proxy allowing me to log in to my account on their servers to access my NAS at home. No need to open any ports on my router and superficially, a secure way to give me access. It didn't take long before some hackers found a way in via the QNAP company's servers. I only found out I had been owned when I noticed that my NAS' cooling fan was running at maximum while I wasn't using it. Then I discovered that some crooks had added my NAS to a crypto mining farm. I guess I was lucky that this was in the days before ransomware.

Since then, I don't use QNAP and I have not found a use case worth the risk of allowing any way to get into my protected network from the Internet.

[madires]

fail2ban is meant to automatically block specific IP addresses (causing failed login attempts) and has a few drawbacks. Rate limitting is a different story and slows down credential stuffing or dictionary attacks. And yes, if you aren't careful you can potentially lock yourself out with a poor setup. If you're running linux you can do quite interesting things with nftables.

[peter-h]

Best current practice

I always worry about that phrase

The IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).

OK, so you are saying a VPN is used to access the internal factory/whatever LAN. That makes sense, obviously. Still would not use a £20 VPN router

But take the case of the heating controller in my original post. How would you use a VPN there? The controller would need to be a server and instead of logging directly into it, you go in via a VPN running on the controller. Back to the same problems... You are still exposing any holes in the ETH/LWIP code to the outside. Also the controller will get port sniffed very fast - same as any other device on an open port.

I suppose the controller can be on a "home LAN" and the user's router can be a VPN router. Even cheap routers, 50 quid, have VPN terminators. But this is far from universal, and requires a lot of IT expertise for the customer to set up. PPTP works out of the box, usually, but L2TP/IPSEC is a bastard to get going between kit from different companies. And everybody on the internet says PPTP is insecure, hey ho, more BS.

[SiliconWizard]

Just use LoRa instead. 

[Nominal Animal]

Whether the device is a server or client doesn't actually matter that much.  What does matter is how large the IP address range able to connect to the device is.  Limiting it to a small number of addresses means connection attempts from everywhere else can be ignored, as if a firewall dropped those packets without any further processing.  If you make the device a client, you can ignore all connection attempts, except those related to connections initiated by the device itself (assuming ftp and passive transfers, and similarly bass-ackward protocols where the server connects to the client, are not used).

Since the physical network these devices are used in are rarely secure, you still need to encrypt and secure the communications, and both ends need to authenticate each other.  I'm very interested in the Microchip ATECC608B for both the encryption and certificate-based authentication, because Mouser sells these in singles for under 1€ apiece.  Even with a dedicated crypto IC, connection attempts must be limited to a small set, as otherwise simply making lots of connection attempts works as a denial-of-service attack.

I would implement the physical user interface, the IoT interface, and the actual device as separate units, communicating using a plain serial connection.  I'd even allow disconnecting either interface while having the main device running normally.  This significantly reduces the complexity in each of these components, and thus makes bugs easier to detect and solve: fewer complex interactions with patterns and consequences that are difficult for humans to encompass, and therefore overall fewer difficult bugs.

Furthermore, instead of an IoT interface, you could actually use a small ARM/RISC-V SoC, and run a proper Linux server and firewall on that.  Many of the various FooPi Zero SBCs suffice for this, although I would avoid Raspberry Pi ones because of their fundamentally problematic hardware USB implementation (which can drop packets in a way that is supposed to not happen per the USB specs), and look for one with eMMC, M.2, or SATA storage (as opposed to microSD cards).  In most cases, a single such IoT server should suffice for a household, though, so this would need additional thinking on before deciding.

Just use LoRa instead. 

No, it definitely needs an AI that can learn your configuration preferences and heating and energy use patterns, so you don't even need to adjust the settings after a while because the AI does those aitomagically for you.  Besides, that requires we gather detailed data on each user, for quality assurance purposes; which of course means we also can "obfuscate" and then sell that data to house insurers for the Actual Big Bucks.  Might even consider negotiating package deals, insurance + heating controller, benefiting everyone. 

I'm sorry, I just threw up a bit in my mouth.

Anyway, my friends B. Urglar and B. N. Enter would be happy to pay for such data, too, for specific named clients off the books, of course.

[Siwastaja]

I'm with zilp here. People underestimate the risks of running "just a client", and then, relatively, overestimate the risks of running a server.

Client code can easily have a bug which allows the attacker to run any code, including setting up a server. You need the same scrutiny for developing client-only code. Having server code opens up more surface area for serious bugs and attacks but really it's still on the same order of magnitude.

[peter-h]

What does matter is how large the IP address range able to connect to the device is.  Limiting it to a small number of addresses means connection attempts from everywhere else can be ignored, as if a firewall dropped those packets without any further processing.

This is not possible if the property owner is mobile, which is the exact use case for heating remote control.

Otherwise yes indeed stuff like remote admin tends to be firewall-locked to accept connections only from the sysadmin's house, etc.

On a related topic, I am surprised there doesn't appear to be a firewall for LWIP. Some higher-end CPUs have a (primitive) single IP filter in the ETH hardware, but most just feed all packets to LWIP or whichever TCP/IP stack you have. I guess it has been realised that a firewall on a limited-bw interface is of little use, and nobody with any sense will be running a general purpose web server on a 32F4, H4, H7, whatever.

Client code can easily have a bug which allows the attacker to run any code, including setting up a server.

Yeah, he needs to get total access to the device first however Basically he needs to be next to it with a SWD debugger wired up I've never built an embedded system where somebody in china can remotely connect, upload some file, and it will just flash it. In that case you may as well give up. But something like a linux box might have been left in such a state, and everybody knows how to hack it.

The least anybody will have is crypto checking of firmware uploads, so it would need to be something like an ex employee of your company who stole the sources. That is probably the biggest risk, as always.

[zilp]

On a related topic, I am surprised there doesn't appear to be a firewall for LWIP. Some higher-end CPUs have a (primitive) single IP filter in the ETH hardware, but most just feed all packets to LWIP or whichever TCP/IP stack you have. I guess it has been realised that a firewall on a limited-bw interface is of little use, and nobody with any sense will be running a general purpose web server on a 32F4, H4, H7, whatever.

A "firewall" is not magic. A firewall is a piece of code that matches packets against ACLs. You might as well match packets against LWIP's socket list. You don't gain anything by duplicating the same code as a "firewall". A firewall isn't free, a firewall requires the exact same amount of processing resources to drop irrelevant packets.

If you are talking about hardware filters: Well, those potentially could reduce CPU load, but also, those are highly hardware specific, and usually extremely limited to be of any actual use, and especially so for security purposes. Mind you that an attacker is adverserial. Filtering on packet fields that the attacker can control doesn't work against adversarial attackers, because they can just change those fields to evade the filter if their goal is to overwhelm you.

Client code can easily have a bug which allows the attacker to run any code, including setting up a server.

Yeah, he needs to get total access to the device first however Basically he needs to be next to it with a SWD debugger wired up I've never built an embedded system where somebody in china can remotely connect, upload some file, and it will just flash it.

You are just demonstrating that you don't understand the problem.

If there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.

Seriously, before you build anything like this, you need to get yourself to a point where you can explain how an attacker could realistically reflash such a device from China.

Note that the point is not that it is impossible to make such a device secure. The point is that unless you understand how such an attack would work, you will have no clue whether you device is secure, and if you think you do, you stil don't. Attackers don't care about your pop-security understanding of "client behind NAT makes things secure", they will exploit vulnerabilities that can be exploited even if you mistakenly think that they can't.

The least anybody will have is crypto checking of firmware uploads,

No. Seriously, you just don't have a clue what you are talking about. I am not trying to be mean, but you really don't, and that's a problem if you need to build a secure system.

When there is a buffer overflow vulnerability in your code, and that code isn't running in an unprivileged task on an MPU with MMU, then chances are that an attacker can just use that buffer overflow to write some code into your device's RAM, with some luck overwrite the return address in the current stack frame to point to the location that they just wrote to, thus running their own code on your device, by sending a packet to that device, and then have that code, say, use the LWIP on the device to download a firmware image and write it to the flash of the device.

Cryptographic authentication of firmware images in the official firmware update mechanism of the device is completely irrelevant for that.

[peter-h]

A "firewall" is not magic. A firewall is a piece of code that matches packets against ACLs. You might as well match packets against LWIP's socket list.

Yes that is probably exactly why and what happens.

If there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.

I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever. An ex employee for sure.

This isn't a PC running windoze or linux.

As for the rest, I don't buy it as a practical scenario. You have no way of scanning the internet and discovering what is behind a NAT layer of some router you found (bugs excepted). This has just descended into a standard internet security debate... DES is insecure, etc

[madires]

The IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).

OK, so you are saying a VPN is used to access the internal factory/whatever LAN. That makes sense, obviously. Still would not use a £20 VPN router

But take the case of the heating controller in my original post. How would you use a VPN there? The controller would need to be a server and instead of logging directly into it, you go in via a VPN running on the controller. Back to the same problems... You are still exposing any holes in the ETH/LWIP code to the outside. Also the controller will get port sniffed very fast - same as any other device on an open port.

No, the user places his VPN router in front of the controller. There is no need to have VPN support on the controller itself. The typical setup for internet access is a SOHO router. Just take a model supporting VPN.

I suppose the controller can be on a "home LAN" and the user's router can be a VPN router. Even cheap routers, 50 quid, have VPN terminators. But this is far from universal, and requires a lot of IT expertise for the customer to set up.

If the user wants that setup, i.e. controller reachable from the internet without any cloud service, then it's his job to set up the VPN or he can pay someone for doing it. Or if he prefers the cloud service solution then he doesn't need a VPN. The point is that he can choose between local server, remote access via VPN or cloud service.

PPTP works out of the box, usually, but L2TP/IPSEC is a bastard to get going between kit from different companies. And everybody on the internet says PPTP is insecure, hey ho, more BS.

PPTP is insecure, it's broken for a long time. I don't know why you asscociate L2TP with IPsec. You can do L2TP over IPsec, but in our controller scenario you don't need that. Regarding IPsec, vendor interoperabilty can sometimes be a challenge, but works usually fine. And if you like something lightweight and modern use WireGuard.

[peter-h]

OK, I get this. The customer still needs to set up the VPN. My grandma had a £20 ADSL router which did PPTP VPN and it was used to "fix" her PC (over RDP) every time she broke it

It is very hard to dig out decent text on PPTP. 99% of websites just copy each other. The last one I saw which sounded credible is that PPTP is equivalent to single DES, which is hardly insecure except in the most contrived known-plaintext case.
https://www.myworkdrive.com/vpn-alternative/pptp-security-risks/

The PPTP protocol itself is no longer considered secure as cracking the initial MS-CHAPv2 authentication can be reduced to the difficulty of cracking a single DES 56-bit key, which with current computers can be brute-forced in a very short time (making a strong password largely irrelevant to the security of PPTP as the entire 56-bit keyspace can be searched within practical time constraints).

In practical terms this is BS.

More practically, PPTP uses GRE which is sometimes not supported in transit. But IME it usually works, especially when nothing else does Oh and I spent an hour yesterday on Vodafone getting them to remove an "anti p0rn" block, which they recently turned on for reasons unknown and which breaks all VPNs. Would not want a customer to have to do this; a customer support nightmare.


I would still ask: is a £20 router less likely to have a back door than "your" embedded box running a VPN terminator?

Re L2TP/IPSEC, that is the standard label in many router brands. The config is horrible. I have some device pairs which simply never work (spent all of yesterday on one case).

Probably the biggest consideration is that the "server in the middle" provides a long term revenue stream, which is key if you need to provide long term remote access support.

[zilp]

If there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.

I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever. An ex employee for sure.

Which has nothing to do with whether your stuff is "client only". That part doesn't get easier if you have a server instead.

And also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.

And also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.

And also, this sounds a lot like "I don't know how to do this, therefore it's difficult, therefore, it's secure". Which is roughly why the state of IT security is as terrible as it is.

This isn't a PC running windoze or linux.

... so?!

Are you trying to tell me that this isn't  system with privilege separation and that therefore is much easier to breach completely? Are you trying to tell me that this is using software less scrutinized than Linux, so therefore is more likely to have vulnerabilities?

Like, what is your point?

As for the rest, I don't buy it as a practical scenario.

Yeah, that's what all the people have said that built all the systems that keep getting compromised.

You have no way of scanning the internet and discovering what is behind a NAT layer of some router you found (bugs excepted).

For one: It's just wild to build the security of your product on the assumption that random equipment you encounter in the field doesn't have bugs.

But also: You still have no idea. You have no basis to make that judgement on. This is obviously just your gut feeling. And it's the exact gut feeling that got us here, with buggy and vulnerable equipment everywhere. Like, what will it take for you to realize that maybe all there vulnerabilities and hacks have a cause? And that that cause is not that everyone is building reliable, secure systems?

Even the idea that you'd have to "scan the internet" is just wild. You don't. If you know your target uses a particular device, you don't need to "scan the internet" to find it on the internet. I have explained this to you before.

And also, you are still completely ignoring that you don't even need to go through that NAT to attack your device.

Take your scenario with your heating controller being installed in an airbnb. The random clueless user will just plug that into the internet router of the place, that also provides WiFi for guests. Suddenly, you have constantly changing people connected directly to that same LAN segment that your device is connected to, with their smartphone, with tons of questionable apps installed. And all of a sudden, it's trivial for an adversarial state actor, say, to find those kinds of devices, and to establish which NAT they are behind, if any.

This has just descended into a standard internet security debate... DES is insecure, etc

That's a weird way to say that you don't care about security and refuse to learn.

[madires]

OK, I get this. The customer still needs to set up the VPN.

... if he want's remote access to his controller without a cloud service. When he's fine with a cloud based service he doesn't need a VPN. Why do you ignore the user's choice?

It is very hard to dig out decent text on PPTP. 99% of websites just copy each other. The last one I saw which sounded credible is that PPTP is equivalent to single DES, which is hardly insecure except in the most contrived known-plaintext case.
https://www.myworkdrive.com/vpn-alternative/pptp-security-risks/

https://blog.vpntracker.com/how-safe-is-pptp-vpn-and-should-i-stop-using-it/ might be a bit more informative.

More practically, PPTP uses GRE which is sometimes not supported in transit. But IME it usually works, especially when nothing else does Oh and I spent an hour yesterday on Vodafone getting them to remove an "anti p0rn" block, which they recently turned on for reasons unknown and which breaks all VPNs. Would not want a customer to have to do this; a customer support nightmare.

See above. The customer can pay someone to set up the VPN.

I would still ask: is a £20 router less likely to have a back door than "your" embedded box running a VPN terminator?

A £20 router is usually a total disaster anyway. Putting the VPN endpoint into the embedded box makes things more complex than terminating the VPN in the router (-> VPN passthrough).

Re L2TP/IPSEC, that is the standard label in many router brands. The config is horrible. I have some device pairs which simply never work (spent all of yesterday on one case).

Clueless marketing departments. The most problematic IPsec routers/clients are the ones which hide the complexity and don't document which settings they expect. Good products allow you to set everything and provide debugging info. IPsec isn't easy and you need to understand how it works. There's no magic 'click me to make it work' button.

[peter-h]

And also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.

Yet, this is exactly what one must do. If the NSA or the FSB is your enemy, you are finished. They will have penetrated your phone, your laptop, your house, your car... So the above is just a standard empty one-liner.

If you are controlling an electricity substation in Ukraine, then you need some serious kit there. But that is today; many years ago the protocol (GI74) was more or less open and the substation was on a (unpublished, hey ho) phone number

And also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.

If your source code, keys, etc, will get stolen, then you are finished too. IIRC, a Canon employee nicked their RSA key for their ink cartridges, enabling 3rd party "chipped" ones to be made.

How would you make a product secure if all your company internals get stolen periodically? Obviously, you can't.

Another "security one-liner"?

When he's fine with a cloud based service he doesn't need a VPN. Why do you ignore the user's choice?

In post 1 I discuss what you call "cloud service" - the "server in the middle" configuration, which seems to be the best way, technically and commercially. I think quite a lot of IOT products use something like this. MQTT is discussed a lot in that context but I am not sure what exactly it adds, except that there are commercial services for it (wasn't AWS getting into that game? - I would not touch that shit company with a bargepole).

[zilp]

And also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.

Yet, this is exactly what one must do. If the NSA or the FSB is your enemy, you are finished. They will have penetrated your phone, your laptop, your house, your car... So the above is just a standard empty one-liner.

If you are controlling an electricity substation in Ukraine, then you need some serious kit there. But that is today; many years ago the protocol (GI74) was more or less open and the substation was on a (unpublished, hey ho) phone number

And also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.

If your source code, keys, etc, will get stolen, then you are finished too. IIRC, a Canon employee nicked their RSA key for their ink cartridges, enabling 3rd party "chipped" ones to be made.

How would you make a product secure if all your company internals get stolen periodically? Obviously, you can't.

Another "security one-liner"?

Nice strawman you have there.

You do realize that your argument is "you probably can't achieve 100% security against all threats, therefore, it's not worth trying for anything beyond the bare minimum of cargo-cult security", and that that is obviously nonsense, right?

This is exactly the attitude why the state of IT security, and in particular of IoT security, is the mess that it is. It's just wild to see this disconnect between people being aware of it all being shit, but never realizing that that is because of the practices that they themselves promote.

Can you guarantee security if all the information of your company gets stolen periodically?

Maybe not.

Does that mean that therefore, it's useless to take steps that prevent any random disgruntled employee from compromising your installed base of thousands of appliances?

And yet, that is how you are arguing.

[peter-h]

No.

[Siwastaja]

I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever.

Don't underestimate the cleverness of attackers. They are as clever as you and me combined, PLUS they dedicated their career to come up with attacks like this. They can just buy a device, use one of the (usually nearly trivial and well documented) vulnerabilities to bypass flash protection, disassemble the code, and, because they have been doing this for years, they quickly recognize the usual patterns, and find a buffer overflow. Even if they couldn't read it out, they can run experiments on the unit.

A disgruntled ex-employee is unlikely to do any better, unless it's a true security expert (in which case they would be spending their precious time getting a new well paying job instead).

By all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.

[tellurium]

The most common pattern I've seen are:

1. Let the box connect to the MQTT server, which acts as a relay. Then, all control is built on top of MQTT. For example, request/response pattern maps to two topics per device: DEVICE_ID/rx and DEVICE_ID/tx. The "rx" topic is for requests, the "tx" topic is for responses. Requests / responses could use whatever higher level protocol - JSON-RPC, custom binary stuff, protobus, whatever.

2. Implement a proprietary, non-standard backend that relays the communication. For example, a backend could terminate all WSS (secure web sockets) connections from devices. Inside Websocket (which is a simple framing protocol with no high-level semantics) you can use whatever request-response protocol, just like above - JSON-RPC, protobufs, custom, etc. And the backend would also provide a common REST API for the UI, management, etc. So this way you can provide simple REST API for your devices, for example:
   DEVICE <-- JSON-RPC over WSS -->  BACKEND <-- REST over HTTPS --> YOU

These approaches are similar at core, in fact. The main thing it to implement a bunch of services on a device side (configuration control, firmware update, and business logic) and make it available via a backend.

Peter, to your question, what MQTT adds. Two things: first, it is a standard. You don't need to invent the wheel. Second, it is pretty lightweight, only a couple of bytes overhead per message. It also has it downsides, and perhaps that's the reason why approach #2 is taken, too.

You might take a look here for a live example, code, and tutorial, which demonstrates approach #1:
https://mongoose.ws/reference-projects/#mqtt-dashboard

[peter-h]

By all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.

I agree, and I never said I was ignoring a b or c. I just don't buy the generalising. It is specifics I am interested in, hence me posting in the first place.

A disgruntled ex-employee is unlikely to do any better,

I think it depends on what he was working on Also remember that in every case where somebody got fired, he had supporters still in the firm.

Implement a proprietary, non-standard backend that relays the communication

I wonder why people go for that? Could it be that it gives you more of a long term assurance of support?

What is "relays the communication"? That would imply the IOT box listening all the time, defeating the point of the "being a mostly hidden client" idea.

[Siwastaja]

For just communication between client and server, mqtt does not add that much. You get some tooling, like programs that print out messages of your choice, or send messages from command line, but such tooling is trivial to write anyway. Maybe the best thing you get is that broker handles access control i.e., you can configure it to use a certificate and then allow devices to only write to their own topic name. But this isn't too difficult anyway. If you want just simple point-to-point sharing of all data, then the topic name / filtration / subscription system of mqtt is zero value for you.

Then again, if you do need topic name system and subscriptions and message filtration and wildcards and all this, then mqtt is a good idea. As always, if you need a hammer, buy one. If you need something else, don't try to force it.

"Being a standard" is only of value if you are going to communicate with others who like to use this standard. For example in our case, our internal use of mqtt is of zero value, because what customers really want are HTTP restful APIs or SQL queries with the server, not access to the raw MQTT traffic.

The issue with mqtt is that if you just do something like download the PAHO EMBEDDED C, you will find out the code is completely broken and unmaintained, so you have to fix a few (simple, obvious) bugs on your own. Maybe there is a non-broken fork somewhere. Or you can try mongoose or any other stack. But how do you know what to use? Read EEVBLOG forum and trust that tellurium's solution is the right one?

A simple protocol of your own - even simpler than mqtt - bypasses all of this. Because of TCP guarantees, you can just do trivial stuff like use a header that indicates number of bytes that follow. This is what mqtt does, too. A few lines of code. For reference, PAHO is just 4000 LoC. If you only need 10% of that, then it's down to 400. Totally manageable.

Also the thing with small implementations, like PAHO, is to realize they don't correctly implement the full protocol. For example, you have to provide a code that generates a randomized client-id, a fundamental feature for mqtt to work, but no one tells you this, and failure to do so produces esoteric and weird problems. You also have to "just know" to use clean_session=1 with PAHO, which completely removes the whole point of the QoS stuff (which is, IMHO, pretty weird on top of TCP anyway; and yet you can find whole tutorials that discuss MQTT QoS system).

Then again, MQTT itself is pretty small and simple protocol, so you are not making a big mistake using it, either. It basically is "just copy this arbitrary user buffer into TCP socket with some simple header added" stuff, with some semi-fancy stuff on top.

[zilp]

By all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.

I agree, and I never said I was ignoring a b or c. I just don't buy the generalising. It is specifics I am interested in, hence me posting in the first place.

The problem is that your question is way too generic to give particularly specific advice.

Is this a one-off project that will be used by some unimportant individual running their small heat pump that's going to be in operation for three years? Then just ignore all the advice, it's pretty much impossible to fuck things up enough to cause any serious security risks.

Is this a product with millions of units expected to be deployed, many of which are expected to end up in the networks of high-value individuals, and/or controlling large gas-fired heating systems that could be made to burn the house down if the controller insists on doing stupid things, or some electric heaters with sufficient power that coordinated rapid switching of those millions of units could be used to bring dow the power grid in a country, and an expected manufacturing time of 5 years and product lifetime of 20 years? Then you really need to take things way more seriously than you currently are.

But also, you are the one who is generalizing when you insist that "client behind NAT" is a security concept. Maybe there is some security value to that in specific circumstances. Maybe that even applies for your specific requirements. But it is most certainly bad advice in general, and the security value is usually way overestimated by people who don't know much about IT security, hence my warnings. If you want to have advice for your specific requirements, you will have to flesh out your specific requirements and not complain that the advice is generic for a generic question.

[tellurium]

Then again, MQTT itself is pretty small and simple protocol, so you are not making a big mistake using it, either. It basically is "just copy this arbitrary user buffer into TCP socket with some simple header added" stuff, with some semi-fancy stuff on top.

Agreed, MQTT is quite simple.
And the publish/subscribe mechanism is very simple, easy to understand and use. It was designed for sensors dumping data, so it does not natively implements request/response, but request/response can be easily done on top of publish/subscribe.

However, if we're talking about security, the biggest issue with MQTT, IMO, is that it does not provide any mechanism to limit access to a topic space.
Simply speaking, any client that manages to authenticate with server, can subscrube to the "#" wildcard topic and get ALL the messages.

Providers of commercial MQTT services, like AWS IoT, solve this by introducing non-standard things, that restrict clients - so that they can subscribe / publish to a limited subset of topics. All those mechanisms are non-standard, thus the value of MQTT, IMO, diminishes, because the production-level system now gets vendor-locked.

IMO, it is way easier to create a custom solution based on Websocket - which I mentioned before. Why Websocket? Because:
a) it is standard, no need to invent the wheel
b) extremely lightweight - a couple of bytes header that tells the frame size, that's all, so it is just a simple "framing" wrapper on top of TCP
c) works with browser's javascript natively
d) has no semantics: frames could contain anything (unlike MQTT where packets have semantics: "subscribe", or "publish", or "connect", etc)

I am not really sure to where the discussion goes though.

[tellurium]

Or you can try mongoose or any other stack. But how do you know what to use? Read EEVBLOG forum and trust that tellurium's solution is the right one?

Yup 

[nctnico]

The context is that an IOT box should always be a client, behind NAT, and never a server, especially not on an open port.

The problem is if you want it remotely accessible. There are many such products.

This has been done for decades (I used to do one in the 1980s which was accessed over a phone line modem, so the security was a) knowing the phone # and b) knowing the password) but how do people do it nowadays?

With a central server where ET can phone home to.

You need to take a step back when designing a system that needs to be secure. Security is more than just throwing encryption onto a communication channel and call it a day. Good security starts with 3 basic pillars: Authorisation, Authentification and Accounting. The last one is the most interesting and probably the most difficult one to implement right; Accounting means monitoring how a device is used and whether the usage patterns fit with what is expected. Note that none of the pillars prescribes using encryption of some sort! That is a choice made later on when making the system's design.

Take an NFC card which has a credit balance on it as an example. Through accounting you can keep track of how much money was deposited onto the card and how much is withdrawn. If more is withdrawn than deposited, you know somebody is cheating.

When done right, one of the 3 pillars can break completely without affecting the security of the system.

[peter-h]

I agree; eavesdropping is actually a very low risk in this application where a client is "phoning home to". So encryption / authentication is not the #1 thing. Especially as anybody can buy one of these devices, and extract any keys (or certificates) from it.

But any corporate customer will demand "privacy" etc regardless of relevance. And TLS is a "standard".

[zilp]

I agree; eavesdropping is actually a very low risk in this application where a client is "phoning home to".

No, it isn't.

So encryption / authentication is not the #1 thing. Especially as anybody can buy one of these devices, and extract any keys (or certificates) from it.

You keep demonstrating that you have no idea what you are talking about.

Seriously, why did you even start this thread, asking for advice on how to implement things securely, when all your replies so far are "well, yeah, but who cares about security, and also, I know all of this better than experts in the field anyhow"?

[nctnico]

I agree; eavesdropping is actually a very low risk in this application where a client is "phoning home to".

No, it isn't.

Why not? What is secret about messages specifying the temperature setpoint and what the actual temperature is?

[zilp]

I agree; eavesdropping is actually a very low risk in this application where a client is "phoning home to".

No, it isn't.

Why not? What is secret about messages specifying the temperature setpoint and what the actual temperature is?

The addresses, ports, and sequence numbers, because those allow you to inject messages into the connection, and thus, for example, change the set point. Specific attack scenarios obviously depend on protocol details.

(If you are talking about "a protocol that protects against everything but eavesdropping", i.e., a protocol that authenticates the endpoints and provides message integrity, but doesn't provide secrecy, then you might be technically correct that that might be low risk, depending on the specific information that is being transmitted ... but then, do you think that that is anywhere close to what peter-h meant, given his obviously rather limited understanding of cryptography?)

[madires]

Maybe the OP could list the security requirements as a starting point for a more productive discussion.

[peter-h]

I mentioned a heating controller.

So you can talk about the security requirements for that.

Think about who would want to do around trashing heating controllers. Especially if they are all behind NAT so can't be casually discovered with a port sniffer. This is not like the old "AXIS IP camera on an open port, exposing an HTTP server" which could be discovered with google

[zilp]

I mentioned a heating controller.

So you can talk about the security requirements for that.

For one: That barely says anything about specific scurity requirements.

But also: I did, you dismissed it all as irrelevant because you know better anyway, so what's the point?

Think about who would want to do around trashing heating controllers.

The assumption that this is only about "trashing heating controllers" is already the first mistake.

Insecure IoT devices (depending on the specific vulnerabilities, obviously) can be used as proxies to execute attacks against other people, including for sending packets for DoS floods, they can be used to spy on the network they are connected to, they can be used to attack other devices on the network they are connected to.

Insecure IoT devices that are attached to/integrated with industrial control style systems (i.e., systems that control significant energy flows rather than just information) further are at risk for manipulation that causes malfunction in a way that causes property damage of surrounding/attached infrastructure or even injury, and for being used to generate destabilizing or even destructive load patterns on the energy supply system (think heat pumps with 9 kW resistive backup heat, if you have a million of those in a country, you can use that to generate a 9 GW load change--if needed, repeat that every 30 seconds, alternating between on and off, and I would be very surprised if the grid stayed up ... really, you could call yourself lucky if the grid just turned off without damaging large numbers of devices with overvoltage).

Which of those matter to your specific application? Well, that largely is a matter of economics. As I said, if it's a one-off project, probably none of it really matters, because noone is going to invest any effort into trying to compromise that one system for basically no gain. If you have an installed base of a few million, your device becomes way more interesting as a target. Also depends on the value of the targets themselves, of course.

Especially if they are all behind NAT so can't be casually discovered with a port sniffer.

For one, you keep repeating this term. It's not a thing.

But also ... you are just continuing to demonstrate your lack of clue when you keep going on about how NAT prevents easy dscovery of your device via port scanning, without ever even addressing any other attack vectors that can be used to discover your devices. Just because you can't think of any other way to find devices, doesn't mean that people who do such things for a living can't either. And that's completely ignoring the fact that NAT doesn't even necessarily prevent port scanning ...

Just because you haven't ever heard of ARP spoofing or DNS spoofing, doesn't mean that attackers haven't either.

This is not like the old "AXIS IP camera on an open port, exposing an HTTP server" which could be discovered with google

I mean, that is kinda trivially true. But it's also a very low bar, isn't it? I mean, that you can't find vulnerable devices with a simple google search doesn't imply that therefore, there is only negligible risk of exploitation, does it?

Obviously the server is a weak point, since it is likely to either contain a database of the units sold (for validation or firmware OTA update purposes), or if malicious code could be installed on it, such a database would eventually be compiled.

... or an attack could simply be executed via that server. Or an attacker could just rent a server with the same hoster and hope they get one on the same broadcast domain as yours and then do an ARP spoofing attack on your server to find all the devices. Or ... whatever.

The real point here is that you obviously have no idea as to how these systems can be attacked, but you still are very confident that you can dismiss all suggested security precautions as unnecessary because you can't think of a way to attack them, or you imagine that it must be extraordinarily difficult to do these things that you aren't familiar with.

[Siwastaja]

I mentioned a heating controller.

Heating controller, when widely deployed, opens up a possibility for hybrid warfare operation, e.g. by depriving people of heat. HOPEFULLY the product has a physical bypass switch, documented and obvious to end user; but even then, heating controller would be used when people are not present e.g. are on vacation, or the controller is at their second home; in such cases, lack of heating for prolonged time can cause pipes bursting, and if it's the very same product offering monitoring, then it's trivial to fake the temperature and power consumption readings so that everything appears to be right.

Another attack vector is to turn every electric heating available ON at the very same second during highest consumption moment, in an attempt to make the grid collapse. Not relevant if you have sold a thousand; very relevant after you have sold 100 000 and sold your customers as demand control to the grid operator and promised they are OFF during that very hour.

Now for something which purely monitors, the risks are smaller, but anything that controls need higher level of scrutiny.

[Nominal Animal]

Now for something which purely monitors, the risks are smaller

Actually, systematic burglaries where a team goes through a neighborhood hitting a few houses in just a few minutes each, often monitor their targets for a few days beforehand to make sure when they'll be vacant.  Having remote access to heating system statistics would make that a lot easier.
(Here in Finland, this is most typical for summer cottages raided by eastern european burglar teams just outside the main holiday seasons.  Domestic burglaries often specifically target weapons.)

I personally do consider any IoT system that can be used to easily access occupancy information a serious, serious security risk.
Vandalism, or maliciously changing the controls, may actually be a smaller practical risk than burglary profiling.

[zilp]

Now for something which purely monitors, the risks are smaller, but anything that controls need higher level of scrutiny.

I think it is important to be explicit: It's about possibilities, not about intentions. I.e., the relevant criterion here is not whether something is intended to only be used for monitoring, but whether it in fact can only be used for monitoring.

Like, if the system just hooks into the interface bus of the heating system (like, what usually the control panel is connected to) and effectively allows unrestricted access to that bus to a remote server, then it doesn't matter if this is only intended for reading sensor values, when you can in fact send messages to the bus that allow you to change any setting of the system.

Part of proper IT security is to restrict privileges at interfaces such that the user of that intrface can not do more than what is actually necessary. So, in the case of a pure monitoring system, for example, one useful approach would be to have one component that somehow extracts any relevant values from the heating system and then just continuously transmits them over a low-complexity unidirectional, ideally galvanically isolated, interface (think: tx-only UART), with that link plus some current-limited power supply being the only thing that connects the heating system itself to the circuit that is interfacing things to the internet. This way, even if the internet intrface part gets compltely compromised, the heating system still keeps doing its job unaffected.

[Nominal Animal]

Part of proper IT security is to restrict privileges at interfaces such that the user of that interface can not do more than what is actually necessary.

Yup: privilege separation, and specifically the principle of least privilege.

Modularising the system is an effective way, like I described in my suggestion earlier in this thread.  For example, you might have two buses: a control bus, and a monitoring bus, both using some plain serial protocol, perhaps I²C/SMBus or just plain UARTs.  The main device handles the heater control, and will work with the current settings even without any control or monitor modules.  Internally, the monitoring bus can be restricted to request (two-way comms) or stream (one-way comms) the statistics, whereas the control bus can change settings and request statistics (to avoid the case where a module would need to be connected to both buses).
This way, a monitoring module cannot, even if cracked/hacked, modify the settings on the heater control device.

Consider the case where you have a single embedded processor handling both heater controls and an IP/TCP stack, and how difficult it is to ensure the heater control logic is never starved of resources, even when bombarded by fake TLS connection attempts (where each handshake can take a significant fraction of a second of calculation time).  You'll basically need an RTOS or dedicated cores to get that done.  In a modular design, you only need to make sure that you handle the communication bus messages only when the heater logic allows; and since these are simple state variable requests and change requests, they can be processed in known maximum time, with pretty straightforward code.  This is why I personally prefer to split complicated devices to multiple microcontrollers, and in systems programming (mostly Linux), heavily rely on process-based privilege separation.

One of my favourite security-related rants is how both Apache and Nginx implement suEXEC fundamentally incorrectly.  It would be much better to run script interpreters that refuse to interpret scripts owned by the user they run as, because that would eliminate all script-drop attacks.  Also, because of the possible security implications in scripting in e.g. SVG files, it would be best to use a separate subdomain (virtual host) for user uploads.  But, because that means each site would need at least two DNS entries and several user accounts and groups, it is "too complicated" to the implementors of hosting control panels like Plesk who just do not understand the security scheme or privilege separation in general at all, and WordPress etc. developers believe their code should be able to rewrite ("update") itself, this is never done.  It is technically easy and simple, but the humans involved just don't want to do it.

[peter-h]

I still don't get how, if say you sold 100k of some box, which is a client which calls up some server, and they are all installed behind a standard NAT router, can somebody discover all these?

It's no good repeating that NAT is insecure blah blah. Obviously NAT won't give you protection if that server is penetrated, but that's a different discussion.

Intuitively I am sure somebody here is pulling my leg (or just doing the standard "internet security lecture" bit) because if it was possible, all PCs, laptops, even phones, would be unusable. But obviously they are not unusable, most are not on botsnets. AFAICT the normal attack is done by the owner downloading a trojan (from an infected website, or by receiving an infected email) and unfortunately those attacks have been very easy.

Oh and BTW I've just remember another way to do this kind of separation exercise: use email I have a system running (not related to anything discussed here) which polls a POP box every 5 mins for incoming emails, acts on stuff in the emails, and then emails certain data to certain recipients. Pretty bloody hard to work out how to penetrate that computer... Especially as the program polling the POP box is custom written, not MS Outlook or similar which has known exploits.

[zilp]

I still don't get how, if say you sold 100k of some box, which is a client which calls up some server, and they are all installed behind a standard NAT router, can somebody discover all these?

All of them? Maybe not. Many of them? Possibly. That depends a lot on implementation details and details of the specific NATs in place.

And I have already mentioned some possible attack vectors before, like Websites in browsers on the inside of the NAT or apps on phones on the inside of the NAT, or just guessing NAT state. Another option could be DNS poisoning.

It's no good repeating that NAT is insecure blah blah. Obviously NAT won't give you protection if that server is penetrated, but that's a different discussion.

Except it isn't really a different discussion, because good security practices help against either. If you build a monitoring-only thing the way I described above, say, then that prevents damage to the heating system regardless what way an attacker were to use to compromise the internet interface.

Intuitively I am sure somebody here is pulling my leg (or just doing the standard "internet security lecture" bit) because if it was possible, all PCs, laptops, even phones, would be unusable. But obviously they are not unusable, most are not on botsnets.

Except ... the only reason why they are not unusable is because they are nowadays largely doing exactly the things that you seem to dismiss as unnecessary!?

I mean, don't you remember the time when pretty much all Windows PCs operated by normal users had viruses on them and were participating in some botnet or another?!

That changed because vendors started caring about security at least to some degree, deployed TLS everywhere, changed code standards, did a ton of fuzzing, security audits, and fixed the vulnerabilities they found, implemented mitigations like ASLR, non-executable data pages, stack canaries, what have you ... a ton of work that makes it relatively difficult to compromise current systems.

And also, just recently, two router-botnets were taken down:

https://www.securityweek.com/us-gov-disrupts-soho-router-botnet-used-by-chinese-apt-volt-typhoon/

https://www.securityweek.com/fbi-dismantles-ubiquiti-router-botnet-controlled-by-russian-cyberspies/

Oh, and also, attackers don't have any incentive to make their victims' systems unusable. If their business is to use your router to DDos other people, being detected would hurt their business. If they are spying on you, it would undermine their goal if they made things unusable for you, when that would likely motivate you to fix the problem.

Also mind you that a lot of security depends on fixing vulnerabilities as they are being found. Requirements are very different if you expect your system to stay secure without updates for 20 years of lifetime, say. And if you want to be able to do OTA updates, then that brings its own challenges with regards to securing that update system.

AFAICT the normal attack is done by the owner downloading a trojan (from an infected website, or by receiving an infected email) and unfortunately those attacks have been very easy.

Well, that is one common attack vector, yes, but certainly not the only one. And also, that's just a matter of economics. As long as that is easier to achieve some goal, that's what attackers do. But if your product is supposed to last 20 years, say, you have to be prepared for when those easier attack vectors get shut down, so you don't end up surprised that some vulnerability in your system is now the easiest way in.

Oh and BTW I've just remember another way to do this kind of separation exercise: use email I have a system running (not related to anything discussed here) which polls a POP box every 5 mins for incoming emails, acts on stuff in the emails, and then emails certain data to certain recipients. Pretty bloody hard to work out how to penetrate that computer... Especially as the program polling the POP box is custom written, not MS Outlook or similar which has known exploits.

No, that isn't hard, if the attacker has a copy of the code, which you can assume they have if the product with the code inside is a thing you can buy.

[peter-h]

like Websites in browsers on the inside of the NAT

How would this work? The whole world is browsing websites behind NAT.

or apps on phones on the inside of the NAT

How would this work? The whole world is running apps on phones, either behind their own NAT (home wifi) or (more often) behind the telco's NAT.

or just guessing NAT state

How would this work? The NAT router opens an incoming channel only from the external IP which was accessed when the channel got opened. If this wasn't the case, with a typical NAT channel open for 180 secs from end of data, most NAT routers would be wide open, which they obviously are not. You would have to guess that the user is browsing e.g. microsoft.com and then you could send in a packet from microsoft.com's IP (actually M$ will have servers on many IPs, for load sharing reasons, so you would have to send in loads of packets all in one go) which is malformed to exploit something on the inside.

Another option could be DNS poisoning.

So now you would need to poison microsoft.com's DNS. Yeah, that must be really trivial I mean, somebody is doing this every day, which is why every time I go to microsoft.com I get a p0rn site...

OK, one could do a bit of social engineering and induce the user to go to a website whose server you control, but how would you do it with 100k users, or even 0.01% of those 100k? You won't know who they are...

because they are nowadays largely doing exactly the things that you seem to dismiss as unnecessary!?

Which is what? They are people behind NAT browsing websites and running apps on phones!!

don't you remember the time when pretty much all Windows PCs operated by normal users had viruses on them and were participating in some botnet or another?!

There was never such time. "Pretty much all"?? People caught viruses off emails / M$ Outlook, by going to infected websites, etc. Not because somebody hit them by NAT penetration. The general infection situation was greatly reduced by

- ISPs covering the domestic market filtering emails with executable attachments
- email filtering gradually getting better
- browsers having better security / fewer back doors (definitely nothing to do with https)
- email clients having fewer back doors
- email clients having a default config whereby 3rd party image links are not followed
- web servers becoming harder to compromise (various measures) so infected websites became rare

I read your Ubiquiti botnet article. Yeah, if you install some app or device which opens an incoming hole in your domestic router, then people can discover that and go in there. But that isn't "penetrating NAT". It opens a means of placing a potentially malformed packet on the user's internal LAN, so it is a similar thing to the above guessing that the user is browsing a known website. But in the vast majority of cases nothing is going to respond to that packet. So you are stacking a low probability on top of another low probability.

There are billions of devices on the internet. Sure you can hit something somewhere, but I still want to know how you are going to hit those 100k heating controllers.

You are pulling my leg, moving the goalposts, and pulling my leg again.

I still want to know how I am going to hit 100k heating controllers, when they are behind normal boring domestic-router NAT.

[zilp]

like Websites in browsers on the inside of the NAT

How would this work? The whole world is browsing websites behind NAT.

Hu? Yeah, that's why it works?!

That works by somehow making the user open a website that contains Javascript code that then can use browser APIs to access your LAN.

or apps on phones on the inside of the NAT

How would this work? The whole world is running apps on phones, either behind their own NAT (home wifi) or (more often) behind the telco's NAT.

Hu? Yeah, that's why it works?!

That works by distributing an app (say, a free game) that contains code that then can access your LAN.

or just guessing NAT state

How would this work? The NAT router opens an incoming channel only from the external IP which was accessed when the channel got opened. If this wasn't the case, with a typical NAT channel open for 180 secs from end of data, most NAT routers would be wide open, which they obviously are not. You would have to guess that the user is browsing e.g. microsoft.com and then you could send in a packet from microsoft.com's IP (actually M$ will have servers on many IPs, for load sharing reasons, so you would have to send in loads of packets all in one go) which is malformed to exploit something on the inside.

... which is all completely irrelevant for a heating controller that potentially connects to exactly one IP address, and that potentially keeps the connection open indefinitely to allow the server pushing config changes whenever the user changes a setting through their app or whatever.

The point being that you depend on a lot of details of how the specific NAT works and of how exactly your TCP stack works, and of how your specific application uses TCP, to even have that security.

Another option could be DNS poisoning.

So now you would need to poison microsoft.com's DNS. Yeah, that must be really trivial I mean, somebody is doing this every day, which is why every time I go to microsoft.com I get a p0rn site...

Yeah, I get it, you have an urge to demonstrate your cluelessness.

First of all, you obviously don't understand how a DNS poisoning attack works.

But also ... no, one reason why you don't get a porn site is because they use TLS and HSTS, with HSTS preload. And because attackers know that that is the case. So, they know that a DNS poisoning attack wouldn't gain them anything useful anyway. So they have no incentive to even try.

And also ... attackers don't present you with porn sites. How did you get this idea that the goal of attackers is to annoy you, rather than to abuse you without you noticing?

OK, one could do a bit of social engineering and induce the user to go to a website whose server you control, but how would you do it with 100k users, or even 0.01% of those 100k? You won't know who they are...

By putting your JS code on an ad network that will distribute it for you for a small fee, for example.

because they are nowadays largely doing exactly the things that you seem to dismiss as unnecessary!?

Which is what? They are people behind NAT browsing websites and running apps on phones!!

Hu? Yeah?!? Your point being?

don't you remember the time when pretty much all Windows PCs operated by normal users had viruses on them and were participating in some botnet or another?!

There was never such time. "Pretty much all"?? People caught viruses off emails / M$ Outlook, by going to infected websites, etc. Not because somebody hit them by NAT penetration.

The thing is that those aren't mutually exclusive. Yes, people caught viruses via email. But what you don't seem to realize is that that is in fact a method for penetrating NAT. Those same viruses at times then used their position on one machine behind the NAT in order to attack other machines on the same LAN. "Penetrating NAT" doesn't need to come in the form of sending packets fro the outside through the NAT, it can just as well come in the form of gaining access to the ability to send packets on the LAN via protocols/applications that can communicate freely through the NAT.

The general infection situation was greatly reduced by

- ISPs covering the domestic market filtering emails with executable attachments
- email filtering gradually getting better
- browsers having better security / fewer back doors (definitely nothing to do with https)

... except that that totally had to do with https. See prevention of successful attacks via DNS poisoning above, for example.

- email clients having fewer back doors
- email clients having a default config whereby 3rd party image links are not followed
- web servers becoming harder to compromise (various measures) so infected websites became rare

Did you notice how none of that is: Because they deemed "being a client behind NAT" good enough?

I read your Ubiquiti botnet article. Yeah, if you install some app or device which opens an incoming hole in your domestic router, then people can discover that and go in there. But that isn't "penetrating NAT". It opens a means of placing a potentially malformed packet on the user's internal LAN, so it is a similar thing to the above guessing that the user is browsing a known website. But in the vast majority of cases nothing is going to respond to that packet. So you are stacking a low probability on top of another low probability.

For one, I think you misunderstood the point that I was making, namely, that devices do get compromised in order to turn them into botnets, and that this is not just a hypothetical scenario.

But also: Yes, of course that is also a method of penetrating NAT, in that this is a way in which the security that NAT potentially provides can be undermined, and is undermined in practice, and thus, if you care about security, this is one data point that suggests that relying on NAT for security is not a good idea.

And do you maybe not realize that attacks of that sort are automated? That there isn't some hacker sitting at a computer "hacking your heating controller", but rather just some exploit payload that can be deployed to potentially hundreds of thousands of compromised devices that are part of a botnet in order to have them all look for your heating controller on their respective local network, for example?

There are billions of devices on the internet. Sure you can hit something somewhere, but I still want to know how you are going to hit those 100k heating controllers.

You are pulling my leg, moving the goalposts, and pulling my leg again.

I still want to know how I am going to hit 100k heating controllers, when they are behind normal boring domestic-router NAT.

That obviously depends on how exactly that thing is implemented?

You need to understand that security best practices are not about only doing what is absolutely necessary in order to hopefully be secure. Security best practices are about methods that reliably deliver security, without too many fragile assumptions about other systems. You seem to expect me to explain in detail how exactly I would exploit your hypothetical heating controller, which obviously is a nonsensical thing to ask, given that the thing doesn't even exist yet, in order to then go "ah, see, you haven't shown me how exactly to exploit this system, therefore, it's secure" ... and that is a nonsensical conclusion.

Security best practice is to build any system that interfaces with public networks in such a way that it does not trust the network. There are many reasons why that is best practice, some of them have been mentioned in this thread. But the general idea is to minimize fragile assumptions. Just as you always do with good engineering. You don't put fuses where you have proven that a circuit is likely to burn your house down. You put fuses everywhere. Because they are (relatively) easy, and they are very reliable. Most fuses never blow. Many fuses protect circuits that are so well designed that you could in principle prove that the fuse will never blow. You put in a fuse anyway, because it is much easier to be confident that a fuse will blow than that you didn't fail to consider some failure condition.

You are constantly asking me to prove how I would make your circuit blow the fuse, which is mostly besides the point. I have shown you various ways in which your assumptions are fragile. And if you want to do reliable security engineering, that should be the clue to you to not rely on those fragile assumptions and to put in fuses regardless.

[Siwastaja]

Day after day, I'm more and more sure peter-h must be an artistic troll account, just like treez and nctnico. Doesn't matter though, interesting stuff.

[madires]

Another option could be DNS poisoning.

So now you would need to poison microsoft.com's DNS. Yeah, that must be really trivial I mean, somebody is doing this every day, which is why every time I go to microsoft.com I get a p0rn site...

OK, one could do a bit of social engineering and induce the user to go to a website whose server you control, but how would you do it with 100k users, or even 0.01% of those 100k? You won't know who they are...

For example, there were cases of mass-compromised SOHO routers returning wrong IP addresses on DNS lookups for well known websites. The IP addresses pointed to servers which injected ads.

[nctnico]

The thing is that those aren't mutually exclusive. Yes, people caught viruses via email. But what you don't seem to realize is that that is in fact a method for penetrating NAT. Those same viruses at times then used their position on one machine behind the NAT in order to attack other machines on the same LAN. "Penetrating NAT" doesn't need to come in the form of sending packets fro the outside through the NAT, it can just as well come in the form of gaining access to the ability to send packets on the LAN via protocols/applications that can communicate freely through the NAT.

Agreed. And let's not forget hacking the Wifi router which sits on almost every local network nowadays in order to gain acces to the network.

The bottom line is that you can't trust the network your device is connected to whether it is behind a NAT or not. In some cases the threat may even come from the local network. Think about the situation where the heating controller is also used for billing OR the device is bought by somebody with the intention to find the weak spots.

[globoy]

I've just been lurking on this thread but as someone who is now often being asked to design internet connected embedded systems I have to say what zilp has been posting is both educational and bracing.  It is helpful.  Thanks zilp.

I keep having this mental image that creating these devices is like running drunk and naked into a minefield.

One problem I see are clients who don't fully appreciate this and don't want to spend the money (for example I shouldn't be the authority on security for the device - someone is an actual authority should help us).  I get it though.  This stuff is hard and hard to explain to non-technical people (hell, hard for someone like me to fully understand). 

This has all been driven home by the fact I just got hacked.  I have one of those "one-off" DIY systems monitoring a second home, and after running for six years it just got owned by someone.  I had to ask a neighbor to go over and unplug it.  Now I have to figure out an even more secure mechanism to access that house.

[peter-h]

somehow making the user open a website

Yeah, 100k users

there were cases of mass-compromised SOHO routers returning wrong IP addresses on DNS lookups for well known websites. The IP addresses pointed to servers which injected ads.

Possible, but I still want to know how you are going to hit these 100k heating controllers.

This has all been driven home by the fact I just got hacked.  I have one of those "one-off" DIY systems monitoring a second home, and after running for six years it just got owned by someone.  I had to ask a neighbor to go over and unplug it.  Now I have to figure out an even more secure mechanism to access that house.

The architecture would be interesting.

[Siwastaja]

This has all been driven home by the fact I just got hacked.  I have one of those "one-off" DIY systems monitoring a second home, and after running for six years it just got owned by someone.  I had to ask a neighbor to go over and unplug it.  Now I have to figure out an even more secure mechanism to access that house.

Do you have any idea, even guess, what happened? Would be pretty educational to hear. The reason I ask is while we can surf sites that list common vulnerabilities, we don't hear how one-offs or small systems get compromised. And one of peter-h's (false) assumptions seems something along the lines that "if you are small enough, the hackers won't invest the time in hacking you", which is, in my opinion, lacking the fact that all of this is probably automated. So how did your one-off system got A) discovered and B) hacked?

[Siwastaja]

somehow making the user open a website

That part is easy-peasy. All you have to do is to mass send email from "peter-heater support" <any_email@goes.com>, and Gmail and other bullshit email clients will "helpfully" combine your scam email in the same thread with actual emails, making users not suspect anything. The email can be something trivial like "your heat controller has been offline, please follow the link to check the current status". >80% will just click the link.

[peter-h]

So how did your one-off system got A) discovered and B) hacked?

Prob on an open port, and got sniffed.

Unless he is paying a subscription, or running a VPN, this is 100% certain.

[zilp]

somehow making the user open a website

Yeah, 100k users

Yeah, ad networks have more than 100k "users" even.

there were cases of mass-compromised SOHO routers returning wrong IP addresses on DNS lookups for well known websites. The IP addresses pointed to servers which injected ads.

Possible, but I still want to know how you are going to hit these 100k heating controllers.

By returning a manipulated address for the "cloud server" that the heating controller connects to.

[globoy]

The architecture would be interesting.

Not that interesting or perhaps helpful.  I'm sure I got bit because I am lazy and primarily counted on obscurity for protection (as was mentioned earlier in this thread).  I used a localhost forwarding service running on a Pi to make the internal webservers of some devices (cameras, temperature monitoring, etc) visible at a set of fixed domains presented by the localhost forwarding service provider to deal with changing ISP addresses.  Everything has login accounts and I hoped the fact that the domains were not registered anywhere (they were sub-domains to the service provider - one per service) would provide the obfuscation.  It worked for a while but was discovered and the Pi owned.  Everything went offline.  I had the neighbor login to the Pi locally and he found that they had somehow changed the ssh login so that it never provided a shell prompt.  Of course I kept putting off updating software on the Pi because that would be some work. So there's that too.  Fortunately nothing was control.  Only monitoring.

I'm going to figure out something new (a dedicated VPN between the houses perhaps), completely reload the Pi software from scratch and wipe and restore the router.  Also wipe and reload as much of the monitoring device's system firmware as possible (and maybe take any system I can't wipe permanently offline).

[Silenos]

I would add my story of being attacked: I was messing with pbx server on linux device, with SIP protocol (which immeasurably sucks, and it runs on plain text packets by default). Stuff was on my home network, with another pc generating test SIP traffic, all on fixed static ips.
Suddenly my logs exploded with the flood of declined INVITEs (sip packets to initiate the session). Checked the wireshark, saw the external traffic and run to kill the router, just in case.
It was a bot which likely was generating the variations of packets to either bruteforce the session or ddos or probe for no password session (I thankfully had only password ones). The bot got interested likely because it the server was correctly responding with 4xx and it never got any further in the session. And attack packets did not even bother to strip the open source pbx label used to generate the traffic or the users/uris string suggesting russian origins of this shit.
Still not sure how it got discovered, I did not have the prior traffic recorded. Though get soured once again against the internet-connected devices.

[peter-h]

SIP hacks are real and frequent. A friend lost 5 digits to this. Misconfigured (by him) Cisco gear

However one would not be running SIP on a heating controller

[globoy]

So peter-h, what are you going to do for this system?

[peter-h]

I am waiting to learn how you can port scan millions of internet IPs to locate say 100k heating controllers which are behind NAT routers

And without using obscure side attacks.

NAT has no open inbound ports, unless there is outgoing traffic. There are possible issues e.g. PnP routers having inbound ports open by some other products' installation code; I don't know how common this is. I've seen a few try it (one was a Vodafone SureSignal femtocell, and in general VOIP products might do it).

[globoy]

I'm wondering how the IOT part works.  How is it accessed/configured/updated remotely?

[peter-h]

See the earlier posts in this thread. The vendor provides a server which the IOT boxes periodically call up to collect config updates or whatever.

Providing that server also delivers a long term revenue stream to the vendor, which is important otherwise he will have no way to fund long term support of an "internet product" which will lose some big chunk of its functionality if something (beyond his control) changes (a simple example is the painful process of constantly developing an app for the constant stream of new versions of IOS or Android).

[globoy]

Thanks.  Yeah,  I saw those -  using https and talking to a web server.  I was wondering if you could share any detail about this mechanism.  How will you authenticate with the server and/or have the server authenticate with the device?  Other than encrypting updates will you do any additional authentication or validation of them?  I guess you'll have storage for at least 2 firmware images and a roll-back mechanism for failed updates.  Any other specific security related functionality?

I ask because I have a similar project for a client who a) doesn't want to spend money on security and b) has an existing web infrastructure to which he is adding the device I am working on for him so I have to follow an existing web API.  It's easy to say stuff like, "well this device will be turned off most of the time" and there's only going to be hundreds or thousands of them in the field so no-one will care so we don't have to worry about the attacks zilp has detailed but I'm still nervous, if only for my reputation, because I know how this client will react if the device is ever compromised (he'll blame everyone but himself). 

[nctnico]

@globoy: I'd walk away from that project if the customer doesn't want to be sensible. Do it right or don't do it at all.

I have a similar project on my hands (bigger amount of units though) and I took the initiative to get the security implemented properly. At some point you have to protect a customer against themselves.

[peter-h]

As I said I am not actually doing a heating controller but may one day do something which uses a similar setup for remote admin and OTA updates.

The obvious way is to use TLS (not because you have to but because it is a standard, and customers who are "internet security experts" (everybody these days) will demand it. So you would have a certificate on the server, preferably not self-signed. The client (a TLS client does not need to store any keys etc) will check this (by reference to cacert.pem or similar) and connect to the server periodically. So there is server authentication, plus session encryption. The good thing about this part is that you control both ends so no need to worry about the server suddenly not supporting say AES so all the clients need a firmware update, which could be disastrous if it goes wrong.

If you do it carefully you won't need firmware updates for many years - other than client functionality. Note that the vast majority of "retail IT" boxes are never enhanced, especially not heating controllers (I've seen remotely controlled products which are 15+ years old).

Obviously the server needs to be secure, otherwise somebody could do something malicious and trash the installed base (config damage, or firmware damage if OTA is possible). I don't see a way to 100% avoid this, other than not have the product features. There are many details you could do to tighten things up but ultimately it all hangs on things like the security of your DNS control panel. If that is compromised, you are buggered. You can't even DKIM-sign emails if your DNS is compromised. I don't know much detail on server config; I've paid others to do that.

As pointed out by others further back, the two other options which avoid this central vulnerability are

- access the IOT boxes over a VPN (needs the customer to have a VPN-capable router (too much to demand IMHO)
- have the IOT boxes as servers i.e. on an open port (and hope that none of the guaranteed flood of attacks will compromise them - way too much to hope for with an embedded system running on something like an arm32)

The 2nd option above also requires the customer to have either a PNP router (and your app installation will open a port) or if he has a half decent router he has to do this manually (and he may have an opinion on that; rightly too).

But as I've said this "central vulnerability" is also your long term revenue stream. Without this you are selling boxes, each one you sell just once, and you need to keep selling them for ever to fund long term support. It's a crap business model.

[nctnico]

Unfortunately something like TLS Interception exists:
https://english.ncsc.nl/binaries/ncsc-en/documenten/factsheets/2019/juni/01/factsheet-tls-interception/Factsheet+TLS-interceptie+1.1+EN.pdf

[zilp]

So you would have a certificate on the server, preferably not self-signed.

Can you explain what the advantage of having it not self-signed is?

[tellurium]

Thanks.  Yeah,  I saw those -  using https and talking to a web server.  I was wondering if you could share any detail about this mechanism.  How will you authenticate with the server and/or have the server authenticate with the device?  Other than encrypting updates will you do any additional authentication or validation of them?  I guess you'll have storage for at least 2 firmware images and a roll-back mechanism for failed updates.  Any other specific security related functionality?

I ask because I have a similar project for a client who a) doesn't want to spend money on security and b) has an existing web infrastructure to which he is adding the device I am working on for him so I have to follow an existing web API.  It's easy to say stuff like, "well this device will be turned off most of the time" and there's only going to be hundreds or thousands of them in the field so no-one will care so we don't have to worry about the attacks zilp has detailed but I'm still nervous, if only for my reputation, because I know how this client will react if the device is ever compromised (he'll blame everyone but himself).

One way I've described earlier:

    DEVICE  <---- Websocket over TLS --->   BACKEND <---- HTTPS ----->  OPERATOR

The backed would terminate connections from devices. Devices can be authenticated either:
  a) using TLS certificates - so-called two-way TLS. That's the way how AWS IoT authenticates devices
  b) using auth tokens, or password strings. Both device and backend would store those, and device can send either a token ot its hash in the WS handshake

For users / administrators, the backed would provide a usual HTTP REST API. Usual auth schemas are user/pass login, or Oauth.

Now, OPERATOR can authenticate with BACKEND and send REST API calls to list devices, see their status, change their config, update them, etc.

[tellurium]

I have a similar project on my hands (bigger amount of units though) and I took the initiative to get the security implemented properly. At some point you have to protect a customer against themselves.

Heh, yeah.
They pay for security either way.
One way is to pay early, and to pay less.
Another way is to pay late, and pay much more, with money and possible lost business.

[Siwastaja]

The client (a TLS client does not need to store any keys etc)

But of course the client does need to store their own private key. How otherwise would you authenticate the client?

[tru]

Unfortunately something like TLS Interception exists:
https://english.ncsc.nl/binaries/ncsc-en/documenten/factsheets/2019/juni/01/factsheet-tls-interception/Factsheet+TLS-interceptie+1.1+EN.pdf

Erm, that is so misleading.  You need client machine access to install and trust a root certificate on the client PC.  It's a known technique mainly used by web developers for security debugging purposes.  Fiddler classic is such a tool that does this.  A root certificate needs to be installed:
https://docs.telerik.com/fiddler-everywhere/installation-and-setup/trust-fiddler-ca

[peter-h]

But of course the client does need to store their own private key. How otherwise would you authenticate the client?

Obviously you know the answer

The client accesses the server by doing a DNS lookup. Now obviously the DNS could be poisoned. But the client can be disassembled anyway and any private key extracted from it. There is no perfect solution, unless you put in a secure crypto module.

However one can add additional security within the comms, and normally one would even if TLS is used. For example firmware updates would have a hash on them, although again this is not secure from client disassembly.

Ultimately any single client can be messed with, but that's OK. What you really do not want to happen is somebody trashing the installed base of say 100k of them, or any significant part of it. And that seems fairly easy to do so long as the server is not penetrated. The next thing to do is set up the server so it does not contain a database of the installed clients. I think that is easy enough because the ultimate mutual authentication could be between the client and the customer's app trying to access it. The server's purpose can be just to facilitate the clients being, ahem, clients, all behind various NAT routers so they can't all be casually discovered.

I am sure many people have been around this before (it is a very common application) but they won't be posting the details here But I also know that many don't bother; I recently stayed in accommodation where the heating was remotely controlled and the controller was a server and the router had port forwarding.

[zilp]

But of course the client does need to store their own private key. How otherwise would you authenticate the client?

Obviously you know the answer

No, obviously not, because you can't.

The client accesses the server by doing a DNS lookup. Now obviously the DNS could be poisoned. But the client can be disassembled anyway and any private key extracted from it. There is no perfect solution, unless you put in a secure crypto module.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

However one can add additional security within the comms, and normally one would even if TLS is used. For example firmware updates would have a hash on them, although again this is not secure from client disassembly.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

Ultimately any single client can be messed with, but that's OK. What you really do not want to happen is somebody trashing the installed base of say 100k of them, or any significant part of it. And that seems fairly easy to do so long as the server is not penetrated.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

The next thing to do is set up the server so it does not contain a database of the installed clients. I think that is easy enough because the ultimate mutual authentication could be between the client and the customer's app trying to access it. The server's purpose can be just to facilitate the clients being, ahem, clients, all behind various NAT routers so they can't all be casually discovered.

Yeah, we got it that you have no clue what you are talking about, why do you insist on demonstrating that over and over?

I am sure many people have been around this before (it is a very common application) but they won't be posting the details here

Obviously not for someone who dismisses all advice anyhow because they feel superior in their cluelessness and prefer insecure solutions.

[peter-h]

It is important to be aware of one's weaknesses, even if they are pointed out without offering much useful detail

[zilp]

It is important to be aware of one's weaknesses, even if they are pointed out without offering much useful detail

The useful details have already been offered. You dismissed them all.

Also, I asked you a question some posts ago that maybe could get you to reflect on where you lack understanding. Apparently, you simply ignored that question.

Also, feel free to actually spell out the reasoning behind this previous post of yours, as to why you think that all those conclusions of yours make sense. Maybe, just maybe, that will make you realize that you have no idea.

It is just blatantly obvious that you are regurgitating these ideas that you have heard somewhere without any real understanding of how they relate or what their applicability is in a given scenario. That whole post is a series of non-sequiturs that maybe is held together in your head by some vague superficial conceptual connections, but just make no sense at all if you actually had any more than surface-level understanding of the subject.

Like, there is a context in which people (experts) will recommend that you should not be using self-signed certificates, because self-signed certificates in that context have security risks. But if you had any understanding of how any of this works and why people say that in that context, you would realize that the reasons for that recommendation makes no sense in your application, and that in actual fact, a self-signed certificate might actually be the more secure choice.

If you want to design a secure system, you have to get beyond the stage of repeating rules of thumb that you've heard somewhere, and actually understand the reasoning behind those rules of thumb. You have to understand the properties of cryptographic primitives (like hashes, signatures, MACs, symmetric and asymmetric ciphers, key agreement schemes) and how they are employed in crypto protocols like TLS, and you won't get there by smugly dismissing everything people who actually understand the subject are saying.

[Siwastaja]

But the client can be disassembled anyway and any private key extracted from it.

Easily, but that is why you use unique private key for each client, so that you can't buy a product and then pretend to be another customer. But yeah, if you break in or otherwise gain physical access to the heating controller, then you can read out the private key, and pretend to be that single heating controller. This is usually the level of risk we can accept, because one could also just replace the physical heating controller with any malicious device (which could connect to another server completely).

In my opinion, authenticating the client is important so that you can't pretend being another customer, but your mileage may vary.

[peter-h]

Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?

Then you are vulnerable to staff theft.

[Siwastaja]

Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?
Then you are vulnerable to staff theft.

Well, if you offer a paid service then quite obviously you can't avoid having a list of customers, and if you have that, then it can leak. Important consideration is to avoid putting any unnecessary sensitive data into that database. For example, I think, but I'm not a security expert, that the device private key should not be stored anywhere else except the device itself.

Think about leaking list of your customers with their billing info (i.e. home addresses) combined with eavesdropping of the realtime heater controller data, from which is easy to see if they are on vacation. Burglars are happy!

[zilp]

Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?

Then you are vulnerable to staff theft.

Staff theft ... of what?

[Nominal Animal]

The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.

The only way to combat this is to assume everyone is a potential exploiter.

This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.

Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.

The customer/payment database does have the above abstract client ID, and whenever the payment state changes, only that change is pushed from the customer database to IoT server, preferably via an intermediate which separately logs such pushes, isolating the customer database from the client/IoT database in an auditable way.

[nctnico]

The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.

The only way to combat this is to assume everyone is a potential exploiter.

This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.

Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.

I don't think this will lead to a workable situation. If you have many devices in the field, you'll have a call center for customer service. People from this call center will need to be able to look at payment status and be able to change the state of the subscription quickly in order to keep customers happy. This is the area where the 'accounting' pillar of good security practises comes in. At some point the system will need to cross reference revenue in versus services rendered. When you break this down to an operator level, you can easely see if a call center employee is handing out too many freebees or not.

I'm not sure whether having seperated databases is a good idea. Lots of synchronisation issues. A long time ago I used stored procedures (which also did authentification) on a PostGresql database to keep an IoT device seperated from the actual database.

[Nominal Animal]

The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.

The only way to combat this is to assume everyone is a potential exploiter.

This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.

Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.

I don't think this will lead to a workable situation.

You mean, you think implementing sensible business practices requires actual experience and knowledge of how to do secure systems, and thus is too costly in your opinion?  That kind of thinking is exactly why so many peoples' personal details and credit card information has been "accidentally leaked".

People from this call center will need to be able to look at payment status and be able to change the state of the subscription quickly in order to keep customers happy.

Sure.  They have very limited access to the customer/payment database.  In that database, any change to the service status must trigger a related push to the IoT server within a guaranteed service interval, and here that could be seconds, say one minute maximum to see the server response change.

When you break this down to an operator level, you can easely see if a call center employee is handing out too many freebees or not.

That kind of thing is easily monitored automatically, and is not an actual problem in real life.

I'm not sure whether having seperated databases is a good idea. Lots of synchronisation issues.

There is no synchronization.  There is exactly one direction in which changes are pushed, and enforcing this is crucial for the security model.  If you have synchronization issues doing this you should not be doing anything with databases.

Even having the IoT server be technically capable of querying the customer/payment database for details instantly breaks the security model.  Separated databases are required for true privilege separation.  If the IoT server is technically able to access the customer information, then the customer information will be exfiltrated in the very first break of the IoT server.  In my opinion, any company that does that should get heavy penalties when that happens.

Security models that survive in the real world are those that do not assume every part in the model is secure.  They still provide security, albeit in degraded/reduced form, when invididual parts are compromised.  There are single point failure sources, like the customer/payment database implementation; minimizing both their number and access/exposure to them is at the core of proper security design.

[nctnico]

The point I'm making is that confining information to a few points is likely to end up with a system which isn't useable in a practical situation. You'd be unpleasantly surprised if you know how many people at a bank have access to your records. Same for the phone company. My assumption is that accesses are logged and audited though.

And there ain't no such thing as 'one way pushing' between systems. Synchronisation between distributed systems will fail at some point for some reason. So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information (whether this process is hidden from the application layer or not). And what to do when the backend database is not working? Ideally the call center operator needs to have a way to see the backend is malfunctioning so there has to be some access to the backend as well from the call center. Again, you'll need information at various places. Worse if transactions need to be corrected manually. The British Post Office has learned that the hard way (see Mr Bates vs The Post Office ).

If you look at the stacked layers of a typical system you can also draw a column spanning all layers next to it. This column is called 'logging, error and exception handling' and typically ends up asking for operator intervention. Another weak point.

[Siwastaja]

So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information

But the whole point is that the databases should have different information. If they have the same information, there is no security benefit from the separation at all and then indeed only extra complexity and syncronization issues ensue. But this wasn't at all what Nominal Animal suggested.

[nctnico]

So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information

But the whole point is that the databases should have different information. If they have the same information, there is no security benefit from the separation at all and then indeed only extra complexity and syncronization issues ensue. But this wasn't at all what Nominal Animal suggested.

Nobody is suggesting that both databases are or should be 100% identical. Only that there is shared piece of information (a subset of all information available) between the databases which needs to be kept in sync. In case of a system with IoT devices this shared information could be about the services enabled in the IoT device.

[Nominal Animal]

My phone, electricity, and insurance companies all have web portals for payments, bills, and service status.
These obviously have access to the payment database, but with very limited privileges.  They need that access to work.

However, an electricity meter (power meter) installed at a client site will not have direct access to the customer/payment database.
It will access a separate database containing only information relevant for electricity meter operation, typically with customers identified by an arbitrary ID included in the meter.  That database is synchronized periodically with the customer/payment database.

If the electricity company provides an interface for customers to monitor their power use, that will use a yet separate database, containing only electricity usage histories (typically at reduced resolution) and no payment history.

Many legal auditability requirements can only be fulfilled by separating different parts of the services into separate databases, and periodically checking for differences, and having an exact transaction record.  Many of them are used to monitor for both technical problems as well as human operator misbehaviour.

While it is technically easiest to use a single database for everything, it also means any problem anywhere in your system is potentially catastrophic.  Yes, there are companies that operate this way, but I condemn them to hell.  There is very little difference to setting up a company that really is just a front for its executives to sell the customer information on the black market.  Ineptitude should not be a legal defense at all.

And there ain't no such thing as 'one way pushing' between systems.

Bullshit.  You typically do that via an intermediary that has very limited access to the customer/payment database (typically read-only plus delete access to a single table containing pending changes), read-write access to the IoT client database, is strictly limited in its automated actions, and leaves an auditable trail of every single transaction it makes between the two databases.  Exactly how this is set up depends on the database engine used.

It is the intermediary, via its access permissions, that provides the privilege separation between the two databases.

I would expect this to include a full state update every N hours using the same mechanism but as if all customer states had changed, with the intermediary checking for differences between the IoT database and the information from the customer/payment database, as any differences would indicate some updates were lost which should never happen.

[nctnico]

And there ain't no such thing as 'one way pushing' between systems.

Bullshit.  You typically do that via an intermediary that has very limited access to the customer/payment database (typically read-only plus delete access to a single table containing pending changes), read-write access to the IoT client database, is strictly limited in its automated actions, and leaves an auditable trail of every single transaction it makes between the two databases.  Exactly how this is set up depends on the database engine used.

It is the intermediary, via its access permissions, that provides the privilege separation between the two databases.

I would expect this to include a full state update every N hours using the same mechanism but as if all customer states had changed, with the intermediary checking for differences between the IoT database and the information from the customer/payment database, as any differences would indicate some updates were lost which should never happen.

The latter is exactly what I wrote in the part you snipped.  Indeed you'll need to read the data back from the target database in order to verify it (and take action when the verification fails). That is a bit more involved than just pushing data (one way) onto the target database and call it a day. You can only say that from a very high level functional perspective.

Still, you can ask yourself the question whether you'd actually need the second database for the IoT devices. What functionality does it add to the system? Why can't they connect with the intermediary directly and fetch data from the primary database through the intermediary?

[Nominal Animal]

Still, you can ask yourself the question whether you'd actually need the second database for the IoT devices. What functionality does it add to the system? Why can't they connect with the intermediary directly and fetch data from the primary database through the intermediary?

It provides privilege separation (principle of least privilege) in the case the IoT server is penetrated.  The separate database accessible from the IoT server limits the data that is exfiltratable in the case of full penetration of the IoT server.  It also helps avoid single point of failure.

Consider the case of the intermediary requiring updates.  Let's say that takes 15 minutes.  In your suggestion, that would mean the IoT server being down for 15 minutes.  In my case, it would mean changes in the payment database take effect on the IoT server 15 minutes (plus a bit) later.

Furthermore, consider the case where the IoT servers are geographically distributed across the world (or just a continent, if you prefer).  I for sure would consider the distributed replication of the full customer/payment database a huge risk, because there is no need.  Instead, I would only distribute the IoT service databases, with the intermediaries managing the data updates.  In the case of all IoT servers being breached, customers' payment information would be absolutely safe; additional systems not accessible by the IoT servers at all would need to be compromised for that.

(To make it clear: the intermediary is an automated process, with its own limited access to each database, which also provides an auditable trail of the changes it has applied.)