The cost of "security"

[madires]

[(For instance, are their any known attacks for the WizNet embedded TCP chips?  Or the default ESP8266 AT-based TCP connections?  (DoS doesn't count.))

Some examples for known issues in TCP/IP stacks for embedded devices:
- AMNESIA:33 https://www.forescout.com/blog/amnesia33-forescout-research-labs-finds-33-new-vulnerabilities-in-open-source-tcp-ip-stacks/
- NAME:WRECK https://therecord.media/namewreck-vulnerabilities-impact-millions-of-smart-and-industrial-devices/

A dedicated TCP/IP chip can limit the impact of a vulnerability to the chip itself. In case of an MCU with a software stack the vulnerability could be fixed by an update, but for a hardware stack it can't. However, this approach could help IoT devices to be more resilient (drawback: additional costs, larger PCB).

[peter-h]

And Cloudflare had a hiccup (websites unreachable) this morning - funny coincidence.

I use CF on a number of sites and on the whole it is superb. Everybody has some downtime; it's a matter of how bad. The hosting company I use was down for ~1hr this morning.

but for a hardware stack it can't

Can work both ways; you are stuck with the firmware. And lots of p1ssed off customers who, if they are much bigger than you (which they usually are, and the big ones get pretty aggressive these days) they can apply a lot of pressure. A lot of business goes into covering one's 6 o'clock. Not long ago I was lucky to have emails going back to 1995, which saved my skin in a big way. There is no such thing as a "hardware" stack; it has to be code, which somebody wrote. AFAIK one can implement more trivial functions like IP filtering in firewalls with an FPGA but a TCP/IP stack? No way, surely.

The reality of mfg business is that one always needs to have a damage limitation strategy. If you sell 1000 boxes to 1000 separate customers, due to widely varying applications perhaps only 50 will discover an issue, and you can fix them. And a customer who had a problem and had it promptly fixed is more loyal than one who never had a problem in the first place, etc, etc. But if you sell 1000 boxes to one customer and he finds a problem, you could be forced into bankrupcy, because a "modern large company" can hit you hard; very hard. I could give actual examples but won't because they might be recognised. So a remote firmware update could be a lifesaver.

AMNESIA:33 https://www.forescout.com/blog/amnesia33-forescout-research-labs-finds-33-new-vulnerabilities-in-open-source-tcp-ip-stacks/

"or inject malicious DNS records to point a device to an attacker-controlled domain"

I wonder what is the best strategy. Going to a fixed IP is not a good idea. Probably authentication is the only way. But if somebody has enough access to do this, they could install a fake DNS server on your LAN?

[janoc]

Do companies large enough to have an IT department deploy IoT devices which are controlled remotely via the internet? And how did they manage to deploy VPNs at large scale for employees working at home (COVID 19)? BTW, I didn't say that VPNs are easy, just that WireGuard makes things easier (vs. IPsec for example).

There is a big difference between an internal project like that - e.g. having to provide VPN for employees (likely some Cisco VPN gear or Fortinet and such, forget free stuff like Wireguard, that doesn't tie in with Active Domain and other enterprise systems) and an external vendor demanding a VPN to be put in place because of some gadget that some low level manager has ordered for a project. Good luck getting something like that authorized.

IPv6 is meant to be used without NAT (It's available, and there's NAT64 as a transition method). This is what I was hinting at. I know, some telcos and companies are still living in the IP past. However, we have IPv6 for 20 years now and they all have to migrate to IPv6 sooner or later because of the lack of IPv4 address space.

Yes but sadly you have to live with what the customer has available now, not what should happen sometime in the future. Also a lot of hardware simply doesn't support IPv6 properly, can't be upgraded/replaced for various reasons, etc.

And even when IPv6 is available, NATs are commonly used as a part of firewall for security reasons, as a supplementary line of defense, with the internal networks completely isolated on non-routable addresses.  There are also compliance/legal reasons because you can't allow any traffic to go out untracked/unmonitored. Forcing everything to go out through a centralized "chokepoint" helps with that.

Dealing with corporate/enterprise IT is a very complex beast and lots of these theoretical ideas how things "are supposed to work" don't apply because there are frequently non-technical reasons (commercial, legal, regulatory, departmental politics ...) involved.

[madires]

There is no such thing as a "hardware" stack; it has to be code, which somebody wrote. AFAIK one can implement more trivial functions like IP filtering in firewalls with an FPGA but a TCP/IP stack? No way, surely.

Of course it's a combination of hardware offloading plus software. Based on the NPU the offloading can go quite far. What I've meant is that a software-only stack can be patched, something with hardware offloading not so much.

I wonder what is the best strategy. Going to a fixed IP is not a good idea. Probably authentication is the only way. But if somebody has enough access to do this, they could install a fake DNS server on your LAN?

Perhaps DNSSEC, DoH or DoT?

[madires]

And even when IPv6 is available, NATs are commonly used as a part of firewall for security reasons, as a supplementary line of defense, with the internal networks completely isolated on non-routable addresses.

RFC 1918 addresses can be routed. You would be surprised how often ISPs/telcos/carriers see those addresses from customers (usually blocked by an ingress filter, but some still haven't heard of BCPs). BTW, you can also use private IPv6 addresses (ULA).

There are also compliance/legal reasons because you can't allow any traffic to go out untracked/unmonitored. Forcing everything to go out through a centralized "chokepoint" helps with that.

It doesn't matter if IPv4, IPv6 or both, all can be filtered or monitored without any problem to comply with whatever regulation. And again, NAT isn't a security feature, but a stateful firewall is. Also, there are many flavours of NAT, some would make your security staff go berserk. Have you ever met an 'expert' telling you that his network is totally safe because of NAT and when you check the firewall settings UPnP is enabled?

[ve7xen]

IOT boxes can't be on open ports, because these are discovered by sniffers almost immediately, due to back doors and vulnerabilities which cannot be fixed ... so their "internet access" can only be as a client, calling up some server, and hidden behind a NAT router.

I am not convinced of this argument.  For an IoT box, you ought to be able to limit the "attack surface" to a manageable size, with or without "full TLS support."
To some extent, I think it's a myth spread by the "IoT Server/Service" vendors (for obvious reasons.)
And laziness by IoT developers.  "We'll just run linux!  That'll make it all easy!"  :-(

(For instance, are their any known attacks for the WizNet embedded TCP chips?  Or the default ESP8266 AT-based TCP connections?  (DoS doesn't count.))

Sure, if you are making devices with NASA or Google levels of engineering, you can be fairly sure you have built a box that isn't vulnerable, at least to a trivial exploit. It's not laziness, it's the engineering tradeoff of having to build a working product you can sell for $30 on Amazon, there just isn't time or money available to hire high quality engineers and for them to put the time into security engineering or evaluation. So that's not the situation with cheap consumer electronics that are built out of half-assed example code stuck together with scotch tape. Chances are very high that some kind of vulnerability exists, since the engineering against them hasn't been done, and then because of the nature of the product, it's often unlikely to get fixed.

In some cases the issue might not even be immediately apparent to the device owner, if it ends up getting used as an amplification vector for DDoS or something like that, which is the most likely scenario for an embedded device. The only thing saving these devices is pretty much that they are almost never exposed to the Internet directly, and that there are so many different varieties of things to exploit. If there's a product that a) sells in large enough numbers and b) is widely exposed to the Internet, I can all but guarantee it gets exploited in short order to be used as a spam cannon or whatever. And is the vendor even incentiveized to care about such issues at all? Doesn't really seem like they care once the device is sold.

But even a basic DoS is dangerous. Maybe your unethical competitor finds out about this... Pretty trivial to scan the 'net for open boxes, DoS them constantly, and cause a lot of damage to your reputation.

Add on the issues with trying to help customers set up port forwarding or whatever to access their device(s), and the path towards having them be clients for an online service is pretty cemented. It avoids most of the security concerns, issues with having multiple devices on the same network, it punches through pretty much any NAT, it's easy for the user...

There are also likely to be vulnerabilities in the application, not the network stack, so I'm not convinced that using a TCP chip helps you much here, and if there *is* a problem it's more likely to be found due to ubiquity and then you can't do anything to fix it, so I dunno, seems like a bit of a wash.

If you restrict yourself to TLS 1.3 then the range of mandatory cipher suites is much reduced, see RFC 8846 Sect 9.1.

Should be able to get away with TLS_AES_128_GCM_SHA256 and TLS_RSA_WITH_AES_128_CBC_SHA which are all that is mandatory in TLS 1.2 + 1.3, if you only care about connecting to relatively modern servers. Some old old stuff might require RC4 or MD5 or something, but meh.

[madires]

If you want to support only TLS 1.3 be aware that in some countries, ASs or corporate networks it will be blocked. Latest tests for HTTP/3 (QUIC, TLS 1.3): https://ooni.org/post/2022-quick-look-quic-censorship/.

[tellurium]

But if you sell 1000 boxes to one customer and he finds a problem, you could be forced into bankrupcy, because a "modern large company" can hit you hard; very hard. I could give actual examples but won't because they might be recognised. So a remote firmware update could be a lifesaver.

That if of course if OTA mechanism is tested, robust and won't brick devices. Experience says that sometimes, a single bricked device out of a thousand successfully updated, is enough to get in trouble. A device could be bricked because of combinations of factors, e.g. specific configuration, environment, etc. Implementing a robust OTA firmware update is a non-trivial task on its own.

[peter-h]

Indeed; OTA needs careful testing, as well as protection from malicious "updates". In practice one would deploy updates by S/N, and only to where issues have been reported. Phone makers limit the risk by deploying country by country, etc.

There are also compliance/legal reasons because you can't allow any traffic to go out untracked/unmonitored

No chance of getting the PCI/DSS remote audit if they find other services on your IP. Eventually we gave up and went to Paypal for payment processing. Another story...

NAT isn't a security feature

NAT is an excellent security feature. It works out of the box, and most people don't need to open any ports (and on VPN routers the VPN terminator bypasses NAT so that still works, for RDP or whatever) whereas firewalls need expert config which is often done wrong. Another nail in the coffin of IOT on open ports.

exploited in short order to be used as a spam cannon or whatever.

That would need the box to be running Linux or some such.

[ve7xen]

NAT is an excellent security feature. It works out of the box, and most people don't need to open any ports (and on VPN routers the VPN terminator bypasses NAT so that still works, for RDP or whatever) whereas firewalls need expert config which is often done wrong. Another nail in the coffin of IOT on open ports.

NAT is only a security feature by accident, and in some configurations (though admittedly not ones you would find in a typical home network) it offers no security at all.

It is a 'security feature' because it requires stateful monitoring of outbound sessions, and only knows what to do with traffic related to them, so unassociated packets get dropped. This is exactly what a stateful firewall gets you in the typical default 'allow all outbound, allow nothing inbound' configuration. There's no significant difference in security posture or configuration effort here.

That would need the box to be running Linux or some such.

Definitely not, all it requires is RCE.

[JohanH]

NAT is an excellent security feature. It works out of the box, and most people don't need to open any ports (and on VPN routers the VPN terminator bypasses NAT so that still works, for RDP or whatever) whereas firewalls need expert config which is often done wrong. Another nail in the coffin of IOT on open ports.

You are confused by what NAT is and isn't. It's purpose isn't security. Use a stateful firewall and it doesn't matter if there is NAT or not. You use NAT when you don't have enough IPv4 address space.

https://www.f5.com/services/resources/white-papers/the-myth-of-network-address-translation-as-security

Btw, most mobile operators have already moved to IPv6. The US has already 50% IPv6 coverage. By this I'm not saying that IPv6 will take over tomorrow, but that it doesn't matter if you use IPv4 (with or without NAT) or IPv6 from a security point of view. The network is only as secure as you have configured it to be, and that means firewalls and other layers of security.

[peter-h]

Whether NAT is a "firewall" or not, if an IOT box is behind NAT, it can't be hacked, but it can go out to access some server. Or you can access it via a VPN terminating in the NAT router.

[nctnico]

Whether NAT is a "firewall" or not, if an IOT box is behind NAT, it can't be hacked, but it can go out to access some server. Or you can access it via a VPN terminating in the NAT router.

IMHO there are actually multiple things to consider that drive the requirements:
1) Malicious acts (making the device do something it is not supposed to do -including not working-)
2) Privacy requirements (make sure third parties can't collect data)
3) Tampering with data to get some (financial) gain

[tellurium]

I'd summarise the following strategy in making a device secure, in the order of complexity

1. It must not accept incoming connections - only outgoing
2. It must use TLS for all sensitive data exchange
3. (complex) It should keep credentials on a crypto element. Or/and, use flash encryption
4. (complex) For OTA, should use checksums and secure boot

The first two generally do not require much investment.

[westfw]

1. It must not accept incoming connections - only outgoing
  :
The first two generally do not require much investment.   

To support any meaningful extraNet usage, (1) requires a whole set of cloud based infrastructure to be the servers for both the IoT devices AND the people trying to use them.   That is a very significant investment (or choice of an existing services that might go away.)  And as they say:  "Now you have TWO security problems."  :-)  (Hmm.  Three, actually, if users also need a special app to access those servers.)

[madires]

Whether NAT is a "firewall" or not, if an IOT box is behind NAT, it can't be hacked, but it can go out to access some server.

Devices behind NAT can be hacked too, it's just a bit more complex. There's no 100% security! What you can do is to define how much security your product needs and trying to reach that goal.

[peter-h]

Devices behind NAT can be hacked too, it's just a bit more complex

I've read this loads of times, and asked each time how to do it, and never got an answer So I am asking again.

[gmb42]

If you want to support only TLS 1.3 be aware that in some countries, ASs or corporate networks it will be blocked. Latest tests for HTTP/3 (QUIC, TLS 1.3): https://ooni.org/post/2022-quick-look-quic-censorship/.

TLS 1.3 != QUIC or HTTP/3

The report you quote is all about HTTP/3, that's a whole chunk of (IMHO unnecessary) extra code for an IoT device that's simply looking for secure data transfer.

[gmb42]

Devices behind NAT can be hacked too, it's just a bit more complex

I've read this loads of times, and asked each time how to do it, and never got an answer So I am asking again.

This discussion is pointless, if you have the mindset that using NAT is "secure" you'll never be convinced that actually it isn't because the proper way to do it with a statefull firewall is "too hard". 

Any time this discussion every comes up on the interwebz there's always someone claiming they have no need for anything other than NAT, they've turned off IPv6 and they've never been hacked.

[peter-h]

... and the answer to the Q is?

Any time this discussion every comes up on the interwebz, some expert appears who says he knows how to do something, but never posts the details

The first two generally do not require much investment.

I'd say TLS is a huge investment, because the code needs a lot of integration, testing and a lot of bug fixing. It also probably forces a bigger CPU (more RAM etc). On my product, TLS doubled the code space, and soaked up 90% of the then remaining RAM.

No wonder people who have done this prefer to be a client, calling up some private server, because that doesn't expose their box to the wide world, and protects it from vulnerabilities which come to light in the future.

I often say this, and MbedTLS is a large enough project for this to not be true, but open source is wonderful until the coder gets himself a girlfriend

[JohanH]

... and the answer to the Q is?

You didn't read the article I posted, did you?

For your convenience:

"While it is true that stateful ingress IPv4 NAT will reject externally initiated TCP traffic, that does not mean that an external host cannot in certain situations send traffic to internal hosts or use other methods to circumvent the NAT. In fact, most network-based attacks assume this as a requirement of the compromise.

There are several ways to accomplish this circumvention, all of which can be prevented by a firewall. First, an attacker can either use a targeted or a sweep attack to send traffic to ports that are open in the NAT device’s state table. The purpose of this attack could be to create a denial-of-service (DoS) by invalidating an existing session on the host or NAT state table, to footprint an internal network, or to inject a malware payload into a third party’s existing session in an effort to compromise the internal host. Serious implications are seen in UDP traffic that is by design stateless; however, the same could be accomplished (given host susceptibility) in TCP or other protocols. In addition, NAT may not provide protocol conformance, sequence number checking, or any other layer 2 or layer 3 DoS security measures that firewalls or advanced security devices inherently provide. NAT also provides no tools to respond should security breaches occur."

[peter-h]

Thanks. I did wonder if an attack on NAT must involve getting into it during the 180 secs of the open channel.

First, an attacker can either use a targeted or a sweep attack to send traffic to ports that are open in the NAT device’s state table. The purpose of this attack could be to create a denial-of-service (DoS) by invalidating an existing session on the host or NAT state table, to footprint an internal network, or to inject a malware payload into a third party’s existing session in an effort to compromise the internal host

That is however quite a stretch; first you get past one box and then you need to exploit back doors in another box.

all of which can be prevented by a firewall.

Assuming the firewall has no back doors... A friend used to work at Cisco and even they used to regularly patch back doors in their 5 digit priced boxes.

[JOEBOBSICLE]

I don't think there's anything wrong with SSL or TLS, I just think it shouldn't be done on a MCU.

It's a solved problem on Linux. You need to implement NTP before you even start SSL handshakes.. It's such a huge amount of work to do on a micro when on Linux it takes five minutes

[tellurium]

I don't think there's anything wrong with SSL or TLS, I just think it shouldn't be done on a MCU.

I guess then, all those vendors who ship their MCUs with network support, are just wrong. They just don't realise that simple fact that networking is for linux.

[JohanH]

A friend pointed out DTLS.

Matthew Garrett, known kernel hacker and security expert, examined an Ikea IoT device and was impressed by their effort. Usually most devices he has examined have gone down in flames.

https://mjg59.dreamwidth.org/47803.html