The context is that an IOT box should always be a client, behind NAT, and never a server, especially not on an open port.
The context is that an IOT box should always be a client, behind NAT, and never a server, especially not on an open port.
Why?
One reason that comes into mind is that a TCP client taking a connection to a server on the Internet will always work without the user having to think about firewall/router/NAT configuration - if it didn't, pretty much nothing would work and they would notice it. It is also then irrelevant what IP address the device obtains, as long as the server address or hostname is correctly configured in the device, and this is possible to fix.
Having a local server accepting incoming TCP connections is possible, but more iffy to set up.
Both controller and IoT box are clients and connect to cloud server to relay the commands.
If so, you may use MQTT with TLS
All test access points on production units must be disabled or locked, for example by blowing on-chip fuses to disable JTAG
why does everything need a cloud connection
I would have thought it was obvious why you don't want to have an embedded system as a server on an open port, but e.g.
- the IOT box will be detected by port sniffers within hours and attacked
- its software will never be up to date security-wise (due to constant R&D cost, and commercial risk of OTA)
- auto update download (client-triggered OTA basically) is possible but risky since the mfg publishing a bad update could brick the entire installed base!
- look at how many versions of say MbedTLS come out, and the recent ones (v3+) are a big job to integrate into a product which has v2
- only a proper heavy duty server (e.g. Centos, Linux and Apache, NGINX etc) should be exposed, and the admin and security patching of such is much easier
- the server can be protected against DOS with e.g. Cloudflare
- IOT box having both client and server is a whole load of RAM; MbedTLS client alone is ~60k
- the client mode means the entire "connection" which the IOT box has to support is private and hidden, which enables operation with no software updates for years or decades
- one would use TLS for data encryption, with the server certificate having a 100 year life (one can argue that one may as well just run aes256 with a shared key; there is no difference between a shared key and a long lived certificate, is there?)
The IOT box would call up the server say every 10 mins to collect any config changes, and since it is behind NAT it would be very hard to hack it. NAT can be penetrated using a couple of methods but AFAIK both involve - at a minimum - the attacker gaining control of the server which the IOT box periodically calls up,
which is obviously possible but with good admin should be hard, otherwise the internet could never exist
Then if the IOT box is hard to trash (a simple way would be to have no way to remotely re-flash it) you have pretty good security.
It would be highly risky to build this around some commercial service. These have a habit of ditching protocols and needing a big redesign.
Obviously the server will need careful firewalling.
I agree with your other comments. The problem is that everybody on the internet is a security expert So, DES, PPTP, FTP, MD5, etc are all "crap". Yet practically nobody is attacking crypto itself.
- the IOT box will be detected by port sniffers within hours and attacked
How would it be attacked?
If your server needs firewalling, you have already failed.
If your server needs firewalling, you have already failed.
With any remote device out there on the Internet, the only assumption one can make is that everything network side of the device's MAC address is insecure.
By flooding the network with bogus ACK or SYN/ACK packets. I witnessed the fleet of battery-powered devices being knocked down.
How would it be attacked?
By flooding the network with bogus ACK or SYN/ACK packets. I witnessed the fleet of battery-powered devices being knocked down.
A device in question could normally live 12 hours on a battery charge, and a fleet of them should be located behind the firewall. One day the customer reported battery life of only about 5 hours for the whole fleet. The logs showed surge of power consumption in all of ~30 devices. This was traced down to continuous modem activity dealing with the flood. Which was, in turn, traced down to the absence of the firewall.
If your server needs firewalling, you have already failed.
No.
zilp - you make some good points but when one gets toQuoteIf your server needs firewalling, you have already failed.
then there isn't anything one can say in response
I would have thought it was obvious why you don't want to have an embedded system as a server on an open port, but e.g.
- the IOT box will be detected by port sniffers within hours and attacked
Like, name one thing that a firewall would be needed for that isn't the result of a failure elsewhere?!
We can't make IoT stuff idiot-proof. There's always some professional moron connecting the control system of a water treatment plant directly to the internet and keeping the default PW.
I don't get the advantage of a VPN for this.
[If your server needs firewalling, you have already failed.
No.
... because?
Sure, and my original point was that in your above case the "control system" is probably a server, maybe even with a published DNS, so half of china is going to be in there, with 24/7/365 dictionary attacks at around 10Hz (I see this on some servers). But if it was a client connecting as I suggest above, it would not be attacked like that.
A port sniffer will instantly discover VPN termination ports, whether pptp, ipsec, openvpn, whatever. Then it can send packets down there.
The assumption is that a given router (probably a consumer type, or maybe a £500 mid-level one) has fewer bugs in the VPN terminator than are elsewhere. That is probably right as in e.g. Remote Desktop is wide open and should be run only over a VPN. But is this relevant to IOT? Would you actually implement a VPN terminator in your IOT box? How much less vulnerable is that to setting up an https server on that IP?
QuoteLike, name one thing that a firewall would be needed for that isn't the result of a failure elsewhere?!
One should firewall a server to accept only traffic it is supposed to be getting. So e.g. you firewall a web server to accept only 80, 443, and the other stuff like Putty (remote admin), DNS, and such like. Otherwise, in the usual virtual server scenario, absolutely everything sent to that IP will need processing.
You can also set up stuff like IP banning following a flood, etc.
On one site I "run" it is behind Cloudflare and I have blocked china, russia, india which greatly reduced various nuisances.
CF offers 5 such blocks for free.
Obviously hackers can use VPNs but the indications are they mostly don't bother and DOS over a VPN will cost the attacker money.
Sure, and my original point was that in your above case the "control system" is probably a server, maybe even with a published DNS, so half of china is going to be in there, with 24/7/365 dictionary attacks at around 10Hz (I see this on some servers). But if it was a client connecting as I suggest above, it would not be attacked like that.
Best current practice
The IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).
Just use LoRa instead.
What does matter is how large the IP address range able to connect to the device is. Limiting it to a small number of addresses means connection attempts from everywhere else can be ignored, as if a firewall dropped those packets without any further processing.
Client code can easily have a bug which allows the attacker to run any code, including setting up a server.
On a related topic, I am surprised there doesn't appear to be a firewall for LWIP. Some higher-end CPUs have a (primitive) single IP filter in the ETH hardware, but most just feed all packets to LWIP or whichever TCP/IP stack you have. I guess it has been realised that a firewall on a limited-bw interface is of little use, and nobody with any sense will be running a general purpose web server on a 32F4, H4, H7, whatever.
QuoteClient code can easily have a bug which allows the attacker to run any code, including setting up a server.
Yeah, he needs to get total access to the device first however Basically he needs to be next to it with a SWD debugger wired up I've never built an embedded system where somebody in china can remotely connect, upload some file, and it will just flash it.
The least anybody will have is crypto checking of firmware uploads,
A "firewall" is not magic. A firewall is a piece of code that matches packets against ACLs. You might as well match packets against LWIP's socket list.
If there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.
QuoteThe IoT device doesn't have to support VPNs. If the user chooses to make the device accessable via the internet he has to place a VPN router in front of it. If he doesn't it's his responsibility (and problem).
OK, so you are saying a VPN is used to access the internal factory/whatever LAN. That makes sense, obviously. Still would not use a £20 VPN router
But take the case of the heating controller in my original post. How would you use a VPN there? The controller would need to be a server and instead of logging directly into it, you go in via a VPN running on the controller. Back to the same problems... You are still exposing any holes in the ETH/LWIP code to the outside. Also the controller will get port sniffed very fast - same as any other device on an open port.
I suppose the controller can be on a "home LAN" and the user's router can be a VPN router. Even cheap routers, 50 quid, have VPN terminators. But this is far from universal, and requires a lot of IT expertise for the customer to set up.
PPTP works out of the box, usually, but L2TP/IPSEC is a bastard to get going between kit from different companies. And everybody on the internet says PPTP is insecure, hey ho, more BS.
The PPTP protocol itself is no longer considered secure as cracking the initial MS-CHAPv2 authentication can be reduced to the difficulty of cracking a single DES 56-bit key, which with current computers can be brute-forced in a very short time (making a strong password largely irrelevant to the security of PPTP as the entire 56-bit keyspace can be searched within practical time constraints).
QuoteIf there is a buffer overflow in some pre-authentication part of your code, and your code is "client only" and behind NAT, that is not in any way a reliable defense against a determined attacker, including against an attacker trying to reflash your device remotely.
I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever. An ex employee for sure.
This isn't a PC running windoze or linux.
As for the rest, I don't buy it as a practical scenario.
You have no way of scanning the internet and discovering what is behind a NAT layer of some router you found (bugs excepted).
This has just descended into a standard internet security debate... DES is insecure, etc
OK, I get this. The customer still needs to set up the VPN.
It is very hard to dig out decent text on PPTP. 99% of websites just copy each other. The last one I saw which sounded credible is that PPTP is equivalent to single DES, which is hardly insecure except in the most contrived known-plaintext case.
https://www.myworkdrive.com/vpn-alternative/pptp-security-risks/
More practically, PPTP uses GRE which is sometimes not supported in transit. But IME it usually works, especially when nothing else does Oh and I spent an hour yesterday on Vodafone getting them to remove an "anti p0rn" block, which they recently turned on for reasons unknown and which breaks all VPNs. Would not want a customer to have to do this; a customer support nightmare.
I would still ask: is a £20 router less likely to have a back door than "your" embedded box running a VPN terminator?
Re L2TP/IPSEC, that is the standard label in many router brands. The config is horrible. I have some device pairs which simply never work (spent all of yesterday on one case).
And also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.
And also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.
When he's fine with a cloud based service he doesn't need a VPN. Why do you ignore the user's choice?
QuoteAnd also, it's a terrible idea to build the security of your product on the assumption that the attacker is dumb.
Yet, this is exactly what one must do. If the NSA or the FSB is your enemy, you are finished. They will have penetrated your phone, your laptop, your house, your car... So the above is just a standard empty one-liner.
If you are controlling an electricity substation in Ukraine, then you need some serious kit there. But that is today; many years ago the protocol (GI74) was more or less open and the substation was on a (unpublished, hey ho) phone numberQuoteAnd also, it's a bad idea to base your security on the assumption that (ex) insider threats don't exist.
If your source code, keys, etc, will get stolen, then you are finished too. IIRC, a Canon employee nicked their RSA key for their ink cartridges, enabling 3rd party "chipped" ones to be made.
How would you make a product secure if all your company internals get stolen periodically? Obviously, you can't.
Another "security one-liner"?
I sort of agree but there is a whole order or two of magnitude of difficulty there, unless the attacker has the source code and is pretty clever.
By all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.
A disgruntled ex-employee is unlikely to do any better,
Implement a proprietary, non-standard backend that relays the communication
QuoteBy all means, ignore CIA/NSA-tier opponents. But that doesn't mean there are no attacks beyond the dumbest script kiddie level. It's a contiuous spectrum and you can always aim a tad higher, for a higher quality, safer product. Worse, what's difficult today might be available in script kiddie toolbox of tomorrow. And you can't response quickly enough if the security is an afterthought.
I agree, and I never said I was ignoring a b or c. I just don't buy the generalising. It is specifics I am interested in, hence me posting in the first place.
Then again, MQTT itself is pretty small and simple protocol, so you are not making a big mistake using it, either. It basically is "just copy this arbitrary user buffer into TCP socket with some simple header added" stuff, with some semi-fancy stuff on top.
Or you can try mongoose or any other stack. But how do you know what to use? Read EEVBLOG forum and trust that tellurium's solution is the right one?
The context is that an IOT box should always be a client, behind NAT, and never a server, especially not on an open port.
The problem is if you want it remotely accessible. There are many such products.
This has been done for decades (I used to do one in the 1980s which was accessed over a phone line modem, so the security was a) knowing the phone # and b) knowing the password) but how do people do it nowadays?
I agree; eavesdropping is actually a very low risk in this application where a client is "phoning home to".
So encryption / authentication is not the #1 thing. Especially as anybody can buy one of these devices, and extract any keys (or certificates) from it.
I agree; eavesdropping is actually a very low risk in this application where a client is "phoning home to".
No, it isn't.
I agree; eavesdropping is actually a very low risk in this application where a client is "phoning home to".
No, it isn't.Why not? What is secret about messages specifying the temperature setpoint and what the actual temperature is?
I mentioned a heating controller.
So you can talk about the security requirements for that.
Think about who would want to do around trashing heating controllers.
Especially if they are all behind NAT so can't be casually discovered with a port sniffer.
This is not like the old "AXIS IP camera on an open port, exposing an HTTP server" which could be discovered with google
Obviously the server is a weak point, since it is likely to either contain a database of the units sold (for validation or firmware OTA update purposes), or if malicious code could be installed on it, such a database would eventually be compiled.
I mentioned a heating controller.
Now for something which purely monitors, the risks are smaller
Now for something which purely monitors, the risks are smaller, but anything that controls need higher level of scrutiny.
Part of proper IT security is to restrict privileges at interfaces such that the user of that interface can not do more than what is actually necessary.
I still don't get how, if say you sold 100k of some box, which is a client which calls up some server, and they are all installed behind a standard NAT router, can somebody discover all these?
It's no good repeating that NAT is insecure blah blah. Obviously NAT won't give you protection if that server is penetrated, but that's a different discussion.
Intuitively I am sure somebody here is pulling my leg (or just doing the standard "internet security lecture" bit) because if it was possible, all PCs, laptops, even phones, would be unusable. But obviously they are not unusable, most are not on botsnets.
AFAICT the normal attack is done by the owner downloading a trojan (from an infected website, or by receiving an infected email) and unfortunately those attacks have been very easy.
Oh and BTW I've just remember another way to do this kind of separation exercise: use email I have a system running (not related to anything discussed here) which polls a POP box every 5 mins for incoming emails, acts on stuff in the emails, and then emails certain data to certain recipients. Pretty bloody hard to work out how to penetrate that computer... Especially as the program polling the POP box is custom written, not MS Outlook or similar which has known exploits.
like Websites in browsers on the inside of the NAT
or apps on phones on the inside of the NAT
or just guessing NAT state
Another option could be DNS poisoning.
because they are nowadays largely doing exactly the things that you seem to dismiss as unnecessary!?
don't you remember the time when pretty much all Windows PCs operated by normal users had viruses on them and were participating in some botnet or another?!
Quotelike Websites in browsers on the inside of the NAT
How would this work? The whole world is browsing websites behind NAT.
Quoteor apps on phones on the inside of the NAT
How would this work? The whole world is running apps on phones, either behind their own NAT (home wifi) or (more often) behind the telco's NAT.
Quoteor just guessing NAT state
How would this work? The NAT router opens an incoming channel only from the external IP which was accessed when the channel got opened. If this wasn't the case, with a typical NAT channel open for 180 secs from end of data, most NAT routers would be wide open, which they obviously are not. You would have to guess that the user is browsing e.g. microsoft.com and then you could send in a packet from microsoft.com's IP (actually M$ will have servers on many IPs, for load sharing reasons, so you would have to send in loads of packets all in one go) which is malformed to exploit something on the inside.
QuoteAnother option could be DNS poisoning.
So now you would need to poison microsoft.com's DNS. Yeah, that must be really trivial I mean, somebody is doing this every day, which is why every time I go to microsoft.com I get a p0rn site...
OK, one could do a bit of social engineering and induce the user to go to a website whose server you control, but how would you do it with 100k users, or even 0.01% of those 100k? You won't know who they are...
Quotebecause they are nowadays largely doing exactly the things that you seem to dismiss as unnecessary!?
Which is what? They are people behind NAT browsing websites and running apps on phones!!
Quotedon't you remember the time when pretty much all Windows PCs operated by normal users had viruses on them and were participating in some botnet or another?!
There was never such time. "Pretty much all"?? People caught viruses off emails / M$ Outlook, by going to infected websites, etc. Not because somebody hit them by NAT penetration.
The general infection situation was greatly reduced by
- ISPs covering the domestic market filtering emails with executable attachments
- email filtering gradually getting better
- browsers having better security / fewer back doors (definitely nothing to do with https)
- email clients having fewer back doors
- email clients having a default config whereby 3rd party image links are not followed
- web servers becoming harder to compromise (various measures) so infected websites became rare
I read your Ubiquiti botnet article. Yeah, if you install some app or device which opens an incoming hole in your domestic router, then people can discover that and go in there. But that isn't "penetrating NAT". It opens a means of placing a potentially malformed packet on the user's internal LAN, so it is a similar thing to the above guessing that the user is browsing a known website. But in the vast majority of cases nothing is going to respond to that packet. So you are stacking a low probability on top of another low probability.
There are billions of devices on the internet. Sure you can hit something somewhere, but I still want to know how you are going to hit those 100k heating controllers.
You are pulling my leg, moving the goalposts, and pulling my leg again.
I still want to know how I am going to hit 100k heating controllers, when they are behind normal boring domestic-router NAT.
QuoteAnother option could be DNS poisoning.
So now you would need to poison microsoft.com's DNS. Yeah, that must be really trivial I mean, somebody is doing this every day, which is why every time I go to microsoft.com I get a p0rn site...
OK, one could do a bit of social engineering and induce the user to go to a website whose server you control, but how would you do it with 100k users, or even 0.01% of those 100k? You won't know who they are...
The thing is that those aren't mutually exclusive. Yes, people caught viruses via email. But what you don't seem to realize is that that is in fact a method for penetrating NAT. Those same viruses at times then used their position on one machine behind the NAT in order to attack other machines on the same LAN. "Penetrating NAT" doesn't need to come in the form of sending packets fro the outside through the NAT, it can just as well come in the form of gaining access to the ability to send packets on the LAN via protocols/applications that can communicate freely through the NAT.
somehow making the user open a website
there were cases of mass-compromised SOHO routers returning wrong IP addresses on DNS lookups for well known websites. The IP addresses pointed to servers which injected ads.
This has all been driven home by the fact I just got hacked. I have one of those "one-off" DIY systems monitoring a second home, and after running for six years it just got owned by someone. I had to ask a neighbor to go over and unplug it. Now I have to figure out an even more secure mechanism to access that house.
This has all been driven home by the fact I just got hacked. I have one of those "one-off" DIY systems monitoring a second home, and after running for six years it just got owned by someone. I had to ask a neighbor to go over and unplug it. Now I have to figure out an even more secure mechanism to access that house.
Quotesomehow making the user open a website
So how did your one-off system got A) discovered and B) hacked?
Quotesomehow making the user open a website
Yeah, 100k users
Quotethere were cases of mass-compromised SOHO routers returning wrong IP addresses on DNS lookups for well known websites. The IP addresses pointed to servers which injected ads.
Possible, but I still want to know how you are going to hit these 100k heating controllers.
The architecture would be interesting.
So you would have a certificate on the server, preferably not self-signed.
Thanks. Yeah, I saw those - using https and talking to a web server. I was wondering if you could share any detail about this mechanism. How will you authenticate with the server and/or have the server authenticate with the device? Other than encrypting updates will you do any additional authentication or validation of them? I guess you'll have storage for at least 2 firmware images and a roll-back mechanism for failed updates. Any other specific security related functionality?
I ask because I have a similar project for a client who a) doesn't want to spend money on security and b) has an existing web infrastructure to which he is adding the device I am working on for him so I have to follow an existing web API. It's easy to say stuff like, "well this device will be turned off most of the time" and there's only going to be hundreds or thousands of them in the field so no-one will care so we don't have to worry about the attacks zilp has detailed but I'm still nervous, if only for my reputation, because I know how this client will react if the device is ever compromised (he'll blame everyone but himself).
I have a similar project on my hands (bigger amount of units though) and I took the initiative to get the security implemented properly. At some point you have to protect a customer against themselves.
The client (a TLS client does not need to store any keys etc)
Unfortunately something like TLS Interception exists:
https://english.ncsc.nl/binaries/ncsc-en/documenten/factsheets/2019/juni/01/factsheet-tls-interception/Factsheet+TLS-interceptie+1.1+EN.pdf
But of course the client does need to store their own private key. How otherwise would you authenticate the client?
QuoteBut of course the client does need to store their own private key. How otherwise would you authenticate the client?
Obviously you know the answer
The client accesses the server by doing a DNS lookup. Now obviously the DNS could be poisoned. But the client can be disassembled anyway and any private key extracted from it. There is no perfect solution, unless you put in a secure crypto module.
However one can add additional security within the comms, and normally one would even if TLS is used. For example firmware updates would have a hash on them, although again this is not secure from client disassembly.
Ultimately any single client can be messed with, but that's OK. What you really do not want to happen is somebody trashing the installed base of say 100k of them, or any significant part of it. And that seems fairly easy to do so long as the server is not penetrated.
The next thing to do is set up the server so it does not contain a database of the installed clients. I think that is easy enough because the ultimate mutual authentication could be between the client and the customer's app trying to access it. The server's purpose can be just to facilitate the clients being, ahem, clients, all behind various NAT routers so they can't all be casually discovered.
I am sure many people have been around this before (it is a very common application) but they won't be posting the details here
It is important to be aware of one's weaknesses, even if they are pointed out without offering much useful detail
But the client can be disassembled anyway and any private key extracted from it.
Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?
Then you are vulnerable to staff theft.
Don't you (the mfg) need to create a database of all the devices, so you can keep track of who has a paid-up server service subscription?
Then you are vulnerable to staff theft.
The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.
The only way to combat this is to assume everyone is a potential exploiter.
This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.
Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.
The easiest way to penetrate any company currently is to bribe or extort an employee with access to provide the information.
I assume "staff theft" is used to describe this pattern.
The only way to combat this is to assume everyone is a potential exploiter.
This, in turn, leads to sensible business practices where you do not have a single customer table exposed to your IoT server at all (for client verification purposes), and instead the IoT server only knows about client IDs and what level of service that ID is entitled to.
Most employees have no access to the customer/payment database; only those involved in payments have, and they need to be monitored for financial security anyway.I don't think this will lead to a workable situation.
People from this call center will need to be able to look at payment status and be able to change the state of the subscription quickly in order to keep customers happy.
When you break this down to an operator level, you can easely see if a call center employee is handing out too many freebees or not.
I'm not sure whether having seperated databases is a good idea. Lots of synchronisation issues.
So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information
So somewhere, somehow there needs to be a verification to make sure both databases (in case of a front / backend system with separate databases) have the same information
But the whole point is that the databases should have different information. If they have the same information, there is no security benefit from the separation at all and then indeed only extra complexity and syncronization issues ensue. But this wasn't at all what Nominal Animal suggested.
And there ain't no such thing as 'one way pushing' between systems.
And there ain't no such thing as 'one way pushing' between systems.Bullshit. You typically do that via an intermediary that has very limited access to the customer/payment database (typically read-only plus delete access to a single table containing pending changes), read-write access to the IoT client database, is strictly limited in its automated actions, and leaves an auditable trail of every single transaction it makes between the two databases. Exactly how this is set up depends on the database engine used.
It is the intermediary, via its access permissions, that provides the privilege separation between the two databases.
I would expect this to include a full state update every N hours using the same mechanism but as if all customer states had changed, with the intermediary checking for differences between the IoT database and the information from the customer/payment database, as any differences would indicate some updates were lost which should never happen.
Still, you can ask yourself the question whether you'd actually need the second database for the IoT devices. What functionality does it add to the system? Why can't they connect with the intermediary directly and fetch data from the primary database through the intermediary?