How would you do a heating controller, IOT client, securely?

[madires]

A port sniffer will instantly discover VPN termination ports, whether pptp, ipsec, openvpn, whatever. Then it can send packets down there.

And the VPN router will handle that. That's its job. And there are several ways to make it very hard for attackers.

The assumption is that a given router (probably a consumer type, or maybe a £500 mid-level one) has fewer bugs in the VPN terminator than are elsewhere. That is probably right as in e.g. Remote Desktop is wide open and should be run only over a VPN. But is this relevant to IOT? Would you actually implement a VPN terminator in your IOT box? How much less vulnerable is that to setting up an https server on that IP?

The IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).

[zilp]

Like, name one thing that a firewall would be needed for that isn't the result of a failure elsewhere?!

One should firewall a server to accept only traffic it is supposed to be getting. So e.g. you firewall a web server to accept only 80, 443, and the other stuff like Putty (remote admin), DNS, and such like. Otherwise, in the usual virtual server scenario, absolutely everything sent to that IP will need processing.

But a firewall doesn't change that!? Somewhere, each incoming packet needs to be matched against the set of sockets in order to figure out whether the packet should be dropped. Whether the component that does that is a "firewall" or an "IP stack" doesn't make a fundamental difference. If you are talking about a stateful firewall, you are just doubling the amount of connection state that you have to hold, as the firewall will duplicate all the socket endpoint data in its state tables rather than matching against the socket tables that are there anyway.

You can also set up stuff like IP banning following a flood, etc.

That is a DoS vector. Unless you do this only for validated paths, this allows an attacker to deny clients access by flooding your server with packets from that client's address.

On one site I "run" it is behind Cloudflare and I have blocked china, russia, india which greatly reduced various nuisances.

Yeah, you are reducing nuisances. I was talking about security. Reducing sshd logging failed passwords attempts doesn't do anything for security.

CF offers 5 such blocks for free.

You can do such blocks locally. But it's mostly pointless anyway.

Obviously hackers can use VPNs but the indications are they mostly don't bother and DOS over a VPN will cost the attacker money.

Attackers who want to exploit actual vulnerabilities in your system will bother. People trying stuff that doesn't work anyway aren't worth any mitigation effort. And also, an attacker that wants to DoS you will use booter services that use compromised machines or amplification attacks, so the traffic will not be coming from the location of the attacker.

Sure, and my original point was that in your above case the "control system" is probably a server, maybe even with a published DNS, so half of china is going to be in there, with 24/7/365 dictionary attacks at around 10Hz (I see this on some servers). But if it was a client connecting as I suggest above, it would not be attacked like that.

Which is completely irrelevant as that "attack" can not be successful anyway unless you have failed elsewhere. Relevant are attacks than can compromise your system. Dictionary "attacks" against systems that don't use terrible passwords can not compromise your system. What you have to worry about are vulnerabilities in your system, not people trying to exploit vulnerabilities that don't exist.

As far as resource consumption is concerned, you can avoid that by not making the server easily discoverable. But also, chances are that you won't see any dictionary attacks anyway against a server that doesn't use passwords. You see SSH dictionary attacks because SSH supports password authentication and there are servers out there that use weak passwords. If you use a protocol that only supports public key authentication, noone will ever try to guess a password that obviously doesn't exist.

[unseenninja]

Rate limiting is no longer a useful defence. When I exposed an ssh server through my router to test as a remote access solution to my home network, it was detected within 8 hours by several of those companies who regularly scan the Internet and publish information about all the IP addresses they find which have any open ports.

Within minutes of this, the brute force attacks started. Given that I only allowed a single key as login credentials to the server, trying to brute force your way in with a user name and password is pointless as the server immediately rejects an attempt to log in that way. Nobody will get in, but it still ends up eating my inbound bandwidth and making the ssh server a target for more advanced groups who can exploit zero day vulnerabilities in the ssh server to break in.

Initially, I thought that the good old solution of running "fail2ban" would help, but a closer examination of the incoming traffic revealed that the brute force attacks switched to a new IP address after each block of eight attempts. The first block would come (for example) from Dallas, Texas. The next block would come from Phoenix, Arizona and the next from Madrid, Spain. The crooks make good use of the vast networks of compromised home PCs that they have access to and simply jump from zombie to zombie, rendering all "fail2ban" like defences pointless. Even more simple rate limiting will only end up locking you out from your own network.

If you really want to enable access to resources inside a protected network from the Internet, the only remotely secure way to do it is to run an ssh server on a VPS. Only the VPS' IP address is allowed to connect to the ssh server at the protected network. It's not a 100% solution compared to not doing it in the first place, but it saves your local bandwidth and limits your exposure to your ability to get the ssh server on the VPS patched before those with a zero day exploit get there.

A long time ago, I had a QNAP NAS and I knew less about network security. I thought it would be a great idea to be able to access my NAS when I was at work or travelling. QNAP offered a service where a program on the NAS would connect to the QNAP company's servers and they would act as a proxy allowing me to log in to my account on their servers to access my NAS at home. No need to open any ports on my router and superficially, a secure way to give me access. It didn't take long before some hackers found a way in via the QNAP company's servers. I only found out I had been owned when I noticed that my NAS' cooling fan was running at maximum while I wasn't using it. Then I discovered that some crooks had added my NAS to a crypto mining farm. I guess I was lucky that this was in the days before ransomware.

Since then, I don't use QNAP and I have not found a use case worth the risk of allowing any way to get into my protected network from the Internet.

[madires]

fail2ban is meant to automatically block specific IP addresses (causing failed login attempts) and has a few drawbacks. Rate limitting is a different story and slows down credential stuffing or dictionary attacks. And yes, if you aren't careful you can potentially lock yourself out with a poor setup. If you're running linux you can do quite interesting things with nftables.

[peter-h]

Best current practice

I always worry about that phrase

The IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).

OK, so you are saying a VPN is used to access the internal factory/whatever LAN. That makes sense, obviously. Still would not use a £20 VPN router

But take the case of the heating controller in my original post. How would you use a VPN there? The controller would need to be a server and instead of logging directly into it, you go in via a VPN running on the controller. Back to the same problems... You are still exposing any holes in the ETH/LWIP code to the outside. Also the controller will get port sniffed very fast - same as any other device on an open port.

I suppose the controller can be on a "home LAN" and the user's router can be a VPN router. Even cheap routers, 50 quid, have VPN terminators. But this is far from universal, and requires a lot of IT expertise for the customer to set up. PPTP works out of the box, usually, but L2TP/IPSEC is a bastard to get going between kit from different companies. And everybody on the internet says PPTP is insecure, hey ho, more BS.

[SiliconWizard]

Just use LoRa instead. 

[Nominal Animal]

Whether the device is a server or client doesn't actually matter that much.  What does matter is how large the IP address range able to connect to the device is.  Limiting it to a small number of addresses means connection attempts from everywhere else can be ignored, as if a firewall dropped those packets without any further processing.  If you make the device a client, you can ignore all connection attempts, except those related to connections initiated by the device itself (assuming ftp and passive transfers, and similarly bass-ackward protocols where the server connects to the client, are not used).

Since the physical network these devices are used in are rarely secure, you still need to encrypt and secure the communications, and both ends need to authenticate each other.  I'm very interested in the Microchip ATECC608B for both the encryption and certificate-based authentication, because Mouser sells these in singles for under 1€ apiece.  Even with a dedicated crypto IC, connection attempts must be limited to a small set, as otherwise simply making lots of connection attempts works as a denial-of-service attack.

I would implement the physical user interface, the IoT interface, and the actual device as separate units, communicating using a plain serial connection.  I'd even allow disconnecting either interface while having the main device running normally.  This significantly reduces the complexity in each of these components, and thus makes bugs easier to detect and solve: fewer complex interactions with patterns and consequences that are difficult for humans to encompass, and therefore overall fewer difficult bugs.

Furthermore, instead of an IoT interface, you could actually use a small ARM/RISC-V SoC, and run a proper Linux server and firewall on that.  Many of the various FooPi Zero SBCs suffice for this, although I would avoid Raspberry Pi ones because of their fundamentally problematic hardware USB implementation (which can drop packets in a way that is supposed to not happen per the USB specs), and look for one with eMMC, M.2, or SATA storage (as opposed to microSD cards).  In most cases, a single such IoT server should suffice for a household, though, so this would need additional thinking on before deciding.

Just use LoRa instead. 

No, it definitely needs an AI that can learn your configuration preferences and heating and energy use patterns, so you don't even need to adjust the settings after a while because the AI does those aitomagically for you.  Besides, that requires we gather detailed data on each user, for quality assurance purposes; which of course means we also can "obfuscate" and then sell that data to house insurers for the Actual Big Bucks.  Might even consider negotiating package deals, insurance + heating controller, benefiting everyone. 

I'm sorry, I just threw up a bit in my mouth.

Anyway, my friends B. Urglar and B. N. Enter would be happy to pay for such data, too, for specific named clients off the books, of course.

[Siwastaja]

I'm with zilp here. People underestimate the risks of running "just a client", and then, relatively, overestimate the risks of running a server.

Client code can easily have a bug which allows the attacker to run any code, including setting up a server. You need the same scrutiny for developing client-only code. Having server code opens up more surface area for serious bugs and attacks but really it's still on the same order of magnitude.

[peter-h]

What does matter is how large the IP address range able to connect to the device is.  Limiting it to a small number of addresses means connection attempts from everywhere else can be ignored, as if a firewall dropped those packets without any further processing.

This is not possible if the property owner is mobile, which is the exact use case for heating remote control.

Otherwise yes indeed stuff like remote admin tends to be firewall-locked to accept connections only from the sysadmin's house, etc.

On a related topic, I am surprised there doesn't appear to be a firewall for LWIP. Some higher-end CPUs have a (primitive) single IP filter in the ETH hardware, but most just feed all packets to LWIP or whichever TCP/IP stack you have. I guess it has been realised that a firewall on a limited-bw interface is of little use, and nobody with any sense will be running a general purpose web server on a 32F4, H4, H7, whatever.

Client code can easily have a bug which allows the attacker to run any code, including setting up a server.

Yeah, he needs to get total access to the device first however Basically he needs to be next to it with a SWD debugger wired up I've never built an embedded system where somebody in china can remotely connect, upload some file, and it will just flash it. In that case you may as well give up. But something like a linux box might have been left in such a state, and everybody knows how to hack it.

The least anybody will have is crypto checking of firmware uploads, so it would need to be something like an ex employee of your company who stole the sources. That is probably the biggest risk, as always.

[zilp]

On a related topic, I am surprised there doesn't appear to be a firewall for LWIP. Some higher-end CPUs have a (primitive) single IP filter in the ETH hardware, but most just feed all packets to LWIP or whichever TCP/IP stack you have. I guess it has been realised that a firewall on a limited-bw interface is of little use, and nobody with any sense will be running a general purpose web server on a 32F4, H4, H7, whatever.

A "firewall" is not magic. A firewall is a piece of code that matches packets against ACLs. You might as well match packets against LWIP's socket list. You don't gain anything by duplicating the same code as a "firewall". A firewall isn't free, a firewall requires the exact same amount of processing resources to drop irrelevant packets.

If you are talking about hardware filters: Well, those potentially could reduce CPU load, but also, those are highly hardware specific, and usually extremely limited to be of any actual use, and especially so for security purposes. Mind you that an attacker is adverserial. Filtering on packet fields that the attacker can control doesn't work against adversarial attackers, because they can just change those fields to evade the filter if their goal is to overwhelm you.

Client code can easily have a bug which allows the attacker to run any code, including setting up a server.

Yeah, he needs to get total access to the device first however Basically he needs to be next to it with a SWD debugger wired up I've never built an embedded system where somebody in china can remotely connect, upload some file, and it will just flash it.

You are just demonstrating that you don't understand the problem.

If there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.

Seriously, before you build anything like this, you need to get yourself to a point where you can explain how an attacker could realistically reflash such a device from China.

Note that the point is not that it is impossible to make such a device secure. The point is that unless you understand how such an attack would work, you will have no clue whether you device is secure, and if you think you do, you stil don't. Attackers don't care about your pop-security understanding of "client behind NAT makes things secure", they will exploit vulnerabilities that can be exploited even if you mistakenly think that they can't.

The least anybody will have is crypto checking of firmware uploads,

No. Seriously, you just don't have a clue what you are talking about. I am not trying to be mean, but you really don't, and that's a problem if you need to build a secure system.

When there is a buffer overflow vulnerability in your code, and that code isn't running in an unprivileged task on an MPU with MMU, then chances are that an attacker can just use that buffer overflow to write some code into your device's RAM, with some luck overwrite the return address in the current stack frame to point to the location that they just wrote to, thus running their own code on your device, by sending a packet to that device, and then have that code, say, use the LWIP on the device to download a firmware image and write it to the flash of the device.

Cryptographic authentication of firmware images in the official firmware update mechanism of the device is completely irrelevant for that.

[peter-h]

A "firewall" is not magic. A firewall is a piece of code that matches packets against ACLs. You might as well match packets against LWIP's socket list.

Yes that is probably exactly why and what happens.

If there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.

I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever. An ex employee for sure.

This isn't a PC running windoze or linux.

As for the rest, I don't buy it as a practical scenario. You have no way of scanning the internet and discovering what is behind a NAT layer of some router you found (bugs excepted). This has just descended into a standard internet security debate... DES is insecure, etc

[madires]

The IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).

OK, so you are saying a VPN is used to access the internal factory/whatever LAN. That makes sense, obviously. Still would not use a £20 VPN router

But take the case of the heating controller in my original post. How would you use a VPN there? The controller would need to be a server and instead of logging directly into it, you go in via a VPN running on the controller. Back to the same problems... You are still exposing any holes in the ETH/LWIP code to the outside. Also the controller will get port sniffed very fast - same as any other device on an open port.

No, the user places his VPN router in front of the controller. There is no need to have VPN support on the controller itself. The typical setup for internet access is a SOHO router. Just take a model supporting VPN.

I suppose the controller can be on a "home LAN" and the user's router can be a VPN router. Even cheap routers, 50 quid, have VPN terminators. But this is far from universal, and requires a lot of IT expertise for the customer to set up.

If the user wants that setup, i.e. controller reachable from the internet without any cloud service, then it's his job to set up the VPN or he can pay someone for doing it. Or if he prefers the cloud service solution then he doesn't need a VPN. The point is that he can choose between local server, remote access via VPN or cloud service.

PPTP works out of the box, usually, but L2TP/IPSEC is a bastard to get going between kit from different companies. And everybody on the internet says PPTP is insecure, hey ho, more BS.

PPTP is insecure, it's broken for a long time. I don't know why you asscociate L2TP with IPsec. You can do L2TP over IPsec, but in our controller scenario you don't need that. Regarding IPsec, vendor interoperabilty can sometimes be a challenge, but works usually fine. And if you like something lightweight and modern use WireGuard.

[peter-h]

OK, I get this. The customer still needs to set up the VPN. My grandma had a £20 ADSL router which did PPTP VPN and it was used to "fix" her PC (over RDP) every time she broke it

It is very hard to dig out decent text on PPTP. 99% of websites just copy each other. The last one I saw which sounded credible is that PPTP is equivalent to single DES, which is hardly insecure except in the most contrived known-plaintext case.
https://www.myworkdrive.com/vpn-alternative/pptp-security-risks/

The PPTP protocol itself is no longer considered secure as cracking the initial MS-CHAPv2 authentication can be reduced to the difficulty of cracking a single DES 56-bit key, which with current computers can be brute-forced in a very short time (making a strong password largely irrelevant to the security of PPTP as the entire 56-bit keyspace can be searched within practical time constraints).

In practical terms this is BS.

More practically, PPTP uses GRE which is sometimes not supported in transit. But IME it usually works, especially when nothing else does Oh and I spent an hour yesterday on Vodafone getting them to remove an "anti p0rn" block, which they recently turned on for reasons unknown and which breaks all VPNs. Would not want a customer to have to do this; a customer support nightmare.


I would still ask: is a £20 router less likely to have a back door than "your" embedded box running a VPN terminator?

Re L2TP/IPSEC, that is the standard label in many router brands. The config is horrible. I have some device pairs which simply never work (spent all of yesterday on one case).

Probably the biggest consideration is that the "server in the middle" provides a long term revenue stream, which is key if you need to provide long term remote access support.

[zilp]

If there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.

I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever. An ex employee for sure.

Which has nothing to do with whether your stuff is "client only". That part doesn't get easier if you have a server instead.

And also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.

And also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.

And also, this sounds a lot like "I don't know how to do this, therefore it's difficult, therefore, it's secure". Which is roughly why the state of IT security is as terrible as it is.

This isn't a PC running windoze or linux.

... so?!

Are you trying to tell me that this isn't  system with privilege separation and that therefore is much easier to breach completely? Are you trying to tell me that this is using software less scrutinized than Linux, so therefore is more likely to have vulnerabilities?

Like, what is your point?

As for the rest, I don't buy it as a practical scenario.

Yeah, that's what all the people have said that built all the systems that keep getting compromised.

You have no way of scanning the internet and discovering what is behind a NAT layer of some router you found (bugs excepted).

For one: It's just wild to build the security of your product on the assumption that random equipment you encounter in the field doesn't have bugs.

But also: You still have no idea. You have no basis to make that judgement on. This is obviously just your gut feeling. And it's the exact gut feeling that got us here, with buggy and vulnerable equipment everywhere. Like, what will it take for you to realize that maybe all there vulnerabilities and hacks have a cause? And that that cause is not that everyone is building reliable, secure systems?

Even the idea that you'd have to "scan the internet" is just wild. You don't. If you know your target uses a particular device, you don't need to "scan the internet" to find it on the internet. I have explained this to you before.

And also, you are still completely ignoring that you don't even need to go through that NAT to attack your device.

Take your scenario with your heating controller being installed in an airbnb. The random clueless user will just plug that into the internet router of the place, that also provides WiFi for guests. Suddenly, you have constantly changing people connected directly to that same LAN segment that your device is connected to, with their smartphone, with tons of questionable apps installed. And all of a sudden, it's trivial for an adversarial state actor, say, to find those kinds of devices, and to establish which NAT they are behind, if any.

This has just descended into a standard internet security debate... DES is insecure, etc

That's a weird way to say that you don't care about security and refuse to learn.

[madires]

OK, I get this. The customer still needs to set up the VPN.

... if he want's remote access to his controller without a cloud service. When he's fine with a cloud based service he doesn't need a VPN. Why do you ignore the user's choice?

It is very hard to dig out decent text on PPTP. 99% of websites just copy each other. The last one I saw which sounded credible is that PPTP is equivalent to single DES, which is hardly insecure except in the most contrived known-plaintext case.
https://www.myworkdrive.com/vpn-alternative/pptp-security-risks/

https://blog.vpntracker.com/how-safe-is-pptp-vpn-and-should-i-stop-using-it/ might be a bit more informative.

More practically, PPTP uses GRE which is sometimes not supported in transit. But IME it usually works, especially when nothing else does Oh and I spent an hour yesterday on Vodafone getting them to remove an "anti p0rn" block, which they recently turned on for reasons unknown and which breaks all VPNs. Would not want a customer to have to do this; a customer support nightmare.

See above. The customer can pay someone to set up the VPN.

I would still ask: is a £20 router less likely to have a back door than "your" embedded box running a VPN terminator?

A £20 router is usually a total disaster anyway. Putting the VPN endpoint into the embedded box makes things more complex than terminating the VPN in the router (-> VPN passthrough).

Re L2TP/IPSEC, that is the standard label in many router brands. The config is horrible. I have some device pairs which simply never work (spent all of yesterday on one case).

Clueless marketing departments. The most problematic IPsec routers/clients are the ones which hide the complexity and don't document which settings they expect. Good products allow you to set everything and provide debugging info. IPsec isn't easy and you need to understand how it works. There's no magic 'click me to make it work' button.

[peter-h]

And also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.

Yet, this is exactly what one must do. If the NSA or the FSB is your enemy, you are finished. They will have penetrated your phone, your laptop, your house, your car... So the above is just a standard empty one-liner.

If you are controlling an electricity substation in Ukraine, then you need some serious kit there. But that is today; many years ago the protocol (GI74) was more or less open and the substation was on a (unpublished, hey ho) phone number

And also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.

If your source code, keys, etc, will get stolen, then you are finished too. IIRC, a Canon employee nicked their RSA key for their ink cartridges, enabling 3rd party "chipped" ones to be made.

How would you make a product secure if all your company internals get stolen periodically? Obviously, you can't.

Another "security one-liner"?

When he's fine with a cloud based service he doesn't need a VPN. Why do you ignore the user's choice?

In post 1 I discuss what you call "cloud service" - the "server in the middle" configuration, which seems to be the best way, technically and commercially. I think quite a lot of IOT products use something like this. MQTT is discussed a lot in that context but I am not sure what exactly it adds, except that there are commercial services for it (wasn't AWS getting into that game? - I would not touch that shit company with a bargepole).

[zilp]

And also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.

Yet, this is exactly what one must do. If the NSA or the FSB is your enemy, you are finished. They will have penetrated your phone, your laptop, your house, your car... So the above is just a standard empty one-liner.

If you are controlling an electricity substation in Ukraine, then you need some serious kit there. But that is today; many years ago the protocol (GI74) was more or less open and the substation was on a (unpublished, hey ho) phone number

And also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.

If your source code, keys, etc, will get stolen, then you are finished too. IIRC, a Canon employee nicked their RSA key for their ink cartridges, enabling 3rd party "chipped" ones to be made.

How would you make a product secure if all your company internals get stolen periodically? Obviously, you can't.

Another "security one-liner"?

Nice strawman you have there.

You do realize that your argument is "you probably can't achieve 100% security against all threats, therefore, it's not worth trying for anything beyond the bare minimum of cargo-cult security", and that that is obviously nonsense, right?

This is exactly the attitude why the state of IT security, and in particular of IoT security, is the mess that it is. It's just wild to see this disconnect between people being aware of it all being shit, but never realizing that that is because of the practices that they themselves promote.

Can you guarantee security if all the information of your company gets stolen periodically?

Maybe not.

Does that mean that therefore, it's useless to take steps that prevent any random disgruntled employee from compromising your installed base of thousands of appliances?

And yet, that is how you are arguing.

[peter-h]

No.

[Siwastaja]

I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever.

Don't underestimate the cleverness of attackers. They are as clever as you and me combined, PLUS they dedicated their career to come up with attacks like this. They can just buy a device, use one of the (usually nearly trivial and well documented) vulnerabilities to bypass flash protection, disassemble the code, and, because they have been doing this for years, they quickly recognize the usual patterns, and find a buffer overflow. Even if they couldn't read it out, they can run experiments on the unit.

A disgruntled ex-employee is unlikely to do any better, unless it's a true security expert (in which case they would be spending their precious time getting a new well paying job instead).

By all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.

[tellurium]

The most common pattern I've seen are:

1. Let the box connect to the MQTT server, which acts as a relay. Then, all control is built on top of MQTT. For example, request/response pattern maps to two topics per device: DEVICE_ID/rx and DEVICE_ID/tx. The "rx" topic is for requests, the "tx" topic is for responses. Requests / responses could use whatever higher level protocol - JSON-RPC, custom binary stuff, protobus, whatever.

2. Implement a proprietary, non-standard backend that relays the communication. For example, a backend could terminate all WSS (secure web sockets) connections from devices. Inside Websocket (which is a simple framing protocol with no high-level semantics) you can use whatever request-response protocol, just like above - JSON-RPC, protobufs, custom, etc. And the backend would also provide a common REST API for the UI, management, etc. So this way you can provide simple REST API for your devices, for example:
   DEVICE <-- JSON-RPC over WSS -->  BACKEND <-- REST over HTTPS --> YOU

These approaches are similar at core, in fact. The main thing it to implement a bunch of services on a device side (configuration control, firmware update, and business logic) and make it available via a backend.

Peter, to your question, what MQTT adds. Two things: first, it is a standard. You don't need to invent the wheel. Second, it is pretty lightweight, only a couple of bytes overhead per message. It also has it downsides, and perhaps that's the reason why approach #2 is taken, too.

You might take a look here for a live example, code, and tutorial, which demonstrates approach #1:
https://mongoose.ws/reference-projects/#mqtt-dashboard

[peter-h]

By all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.

I agree, and I never said I was ignoring a b or c. I just don't buy the generalising. It is specifics I am interested in, hence me posting in the first place.

A disgruntled ex-employee is unlikely to do any better,

I think it depends on what he was working on Also remember that in every case where somebody got fired, he had supporters still in the firm.

Implement a proprietary, non-standard backend that relays the communication

I wonder why people go for that? Could it be that it gives you more of a long term assurance of support?

What is "relays the communication"? That would imply the IOT box listening all the time, defeating the point of the "being a mostly hidden client" idea.

[Siwastaja]

For just communication between client and server, mqtt does not add that much. You get some tooling, like programs that print out messages of your choice, or send messages from command line, but such tooling is trivial to write anyway. Maybe the best thing you get is that broker handles access control i.e., you can configure it to use a certificate and then allow devices to only write to their own topic name. But this isn't too difficult anyway. If you want just simple point-to-point sharing of all data, then the topic name / filtration / subscription system of mqtt is zero value for you.

Then again, if you do need topic name system and subscriptions and message filtration and wildcards and all this, then mqtt is a good idea. As always, if you need a hammer, buy one. If you need something else, don't try to force it.

"Being a standard" is only of value if you are going to communicate with others who like to use this standard. For example in our case, our internal use of mqtt is of zero value, because what customers really want are HTTP restful APIs or SQL queries with the server, not access to the raw MQTT traffic.

The issue with mqtt is that if you just do something like download the PAHO EMBEDDED C, you will find out the code is completely broken and unmaintained, so you have to fix a few (simple, obvious) bugs on your own. Maybe there is a non-broken fork somewhere. Or you can try mongoose or any other stack. But how do you know what to use? Read EEVBLOG forum and trust that tellurium's solution is the right one?

A simple protocol of your own - even simpler than mqtt - bypasses all of this. Because of TCP guarantees, you can just do trivial stuff like use a header that indicates number of bytes that follow. This is what mqtt does, too. A few lines of code. For reference, PAHO is just 4000 LoC. If you only need 10% of that, then it's down to 400. Totally manageable.

Also the thing with small implementations, like PAHO, is to realize they don't correctly implement the full protocol. For example, you have to provide a code that generates a randomized client-id, a fundamental feature for mqtt to work, but no one tells you this, and failure to do so produces esoteric and weird problems. You also have to "just know" to use clean_session=1 with PAHO, which completely removes the whole point of the QoS stuff (which is, IMHO, pretty weird on top of TCP anyway; and yet you can find whole tutorials that discuss MQTT QoS system).

Then again, MQTT itself is pretty small and simple protocol, so you are not making a big mistake using it, either. It basically is "just copy this arbitrary user buffer into TCP socket with some simple header added" stuff, with some semi-fancy stuff on top.

[zilp]

By all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.

I agree, and I never said I was ignoring a b or c. I just don't buy the generalising. It is specifics I am interested in, hence me posting in the first place.

The problem is that your question is way too generic to give particularly specific advice.

Is this a one-off project that will be used by some unimportant individual running their small heat pump that's going to be in operation for three years? Then just ignore all the advice, it's pretty much impossible to fuck things up enough to cause any serious security risks.

Is this a product with millions of units expected to be deployed, many of which are expected to end up in the networks of high-value individuals, and/or controlling large gas-fired heating systems that could be made to burn the house down if the controller insists on doing stupid things, or some electric heaters with sufficient power that coordinated rapid switching of those millions of units could be used to bring dow the power grid in a country, and an expected manufacturing time of 5 years and product lifetime of 20 years? Then you really need to take things way more seriously than you currently are.

But also, you are the one who is generalizing when you insist that "client behind NAT" is a security concept. Maybe there is some security value to that in specific circumstances. Maybe that even applies for your specific requirements. But it is most certainly bad advice in general, and the security value is usually way overestimated by people who don't know much about IT security, hence my warnings. If you want to have advice for your specific requirements, you will have to flesh out your specific requirements and not complain that the advice is generic for a generic question.

[tellurium]

Then again, MQTT itself is pretty small and simple protocol, so you are not making a big mistake using it, either. It basically is "just copy this arbitrary user buffer into TCP socket with some simple header added" stuff, with some semi-fancy stuff on top.

Agreed, MQTT is quite simple.
And the publish/subscribe mechanism is very simple, easy to understand and use. It was designed for sensors dumping data, so it does not natively implements request/response, but request/response can be easily done on top of publish/subscribe.

However, if we're talking about security, the biggest issue with MQTT, IMO, is that it does not provide any mechanism to limit access to a topic space.
Simply speaking, any client that manages to authenticate with server, can subscrube to the "#" wildcard topic and get ALL the messages.

Providers of commercial MQTT services, like AWS IoT, solve this by introducing non-standard things, that restrict clients - so that they can subscribe / publish to a limited subset of topics. All those mechanisms are non-standard, thus the value of MQTT, IMO, diminishes, because the production-level system now gets vendor-locked.

IMO, it is way easier to create a custom solution based on Websocket - which I mentioned before. Why Websocket? Because:
a) it is standard, no need to invent the wheel
b) extremely lightweight - a couple of bytes header that tells the frame size, that's all, so it is just a simple "framing" wrapper on top of TCP
c) works with browser's javascript natively
d) has no semantics: frames could contain anything (unlike MQTT where packets have semantics: "subscribe", or "publish", or "connect", etc)

I am not really sure to where the discussion goes though.

[tellurium]

Or you can try mongoose or any other stack. But how do you know what to use? Read EEVBLOG forum and trust that tellurium's solution is the right one?

Yup