Where or how do you get a secure embedded TCP/IP stack

[splin]

The vulnerability of TCP/IP stacks to attacks, including buffer overflow, malformed packets and DOS attacks is a significant concern - Amazon's freeRTOS being a prime (sorry!) example. This is becoming an important issue as goverments load on the pressure onto organisations, including power and water utilities, where significant disruption to society could result from their infrastructure being hacked. So what do you do if you want a secure, and preferably cheap, TCP/IP stack for an embedded product? It seems to me you have a few choices including:

a) Use a free product such as LwIP. This should be relatively quick to implement and have a small footprint having been designed for embedded products. But are they robust and maintained and tested against new attacks?

b) Port the latest Linux or BSD stack. I expect this to be a major undertaking, especially if you don’t need to, or have the resources to, support IPv6 so extracting the IPv4 code might be a nightmare. This has the advantage that Linux and BSD TCP/IP stacks are so widely used that they get a lot of attention from both attackers and defenders and any vulnerabilities get quickly patched.

c) Buy a third party stack. But how can you be confident that they are any more secure, and remain so, than free versions? The vendors will no doubt have a long list of reasons why (only) they can be trusted because obviously they have a team of the world's foremost TCP/IP and cyber security experts who are continuously engaged in probing and testing their S/W with the most advanced techniques to reveal any possible weakness, whilst diligently monitoring CERT alerts (and extensive inside contacts in various national intelligence agencies including GCHQ, NSA etc.)

But in the real world? How do you know that the product isn’t something that was written by ‘old Fred the comms whizz’ 25 years ago and has now retired and nobody at the vendor dares to touch beyond updating the copyright dates in the header files occasionally? If they do have a large dedicated team, chances are they are engaged in porting to innumerable different MCU/platform targets with their idiosyncratic tools and peripherals with little if no time to consider security. Especially as 'security' is very nebulous and virtually impossible to prove (prior to being hacked), whereas being able to tick the box 'Paduak MCS150C as a supported target' is likely much higher up a product manager's list of priorities.

Whatever the source of the stack, there is the vexed problem of keeping the firmware in an embedded product up to date. Clearly, having a TCP/IP stack means a large part of the update problem (distribution) should be solved (or at least simplified) but the remaining issues of managing the updates remain, including keeping track of the revision state of all products in the field and interoperability with earlier versions. Using a simple stack like LwIP presumably avoids this issue because it isn’t updated (but of course your porting might have errors which need to be fixed).

If you use a widely used stack then you probably need to be able to update quickly when vulnerabilities are found, which means a lot of cost in tracking updates by the developers, and porting those into your own if necessary, and distributing and managing your new releases. On the other hand if you use a (relatively) unknown or obscure stack then it’s possible that attackers will miss your unique vulnerabilities. I expect that this ‘security by obscurity’ does protect many embedded products - from casual attacks at least. But if your customer happens to be a high value target, such as a utility company sufficiently attractive for an attacker to focus their efforts on your stack, the ‘obscurity’ protection may become minimal or non-existent.

There are many more issues of course. So what do you do? Is the security weakness of TCP/IP stacks exaggerated? Falling victim to a DOS attack is likely nowhere near as severe as a breach allowing control/access to the device. Many of the high profile vulnerabilities of many consumer devices such as routers seem to be due to backdoors for developer or maintenance access, rather than in stack code itself.

What if your customer wants evidence that your product is secure - beyond bedazzling them with your gold-plated ISOxxxx/CMM level X quality systems and processes? How much does it cost to get an embedded product tested by a third party to 'prove' that it is secure?

And how could you rely on a third party's evaluation? Given the nature of attacks, testing requires creative and innovative approaches to testing as well as rigorous and methodical testing. The latter no doubt could be certified to ISOxxxx  (whatever standards developed by whatever committees) but that may add relatively little value to the former which requires individuals motivated to understand the mindset and techniques of attackers and constantly anticipate and investigate previously unknown or unused types of attack. These skills would be almost impossible to quantify in any sort of certificated form but are likely to be much more important than any size army of people with clipboards counting bytes, measuring stack depths, complexity metrics etc. etc.

[ejeffrey]

d) don't put microcontrollers on the public internet.  Microcontrollers either use a non-ethernet inteface (USB, serial, CAN, bluetooth) or go on private/isolated subnets. Access is controlled via proxies running on an a general purpose OS with a fully featured well vetted TCP/IP stack, firewall, TLS, and authentication libraries and protocol implementations.

Because honestly, Amazon's FreeRTOS vulnerabilities aside, the reason most IoT stuff is such security garbage has nothing to do with TCP/IP stack vulnerabilities.

[Berni]

In most cases sticking to something proven in the best way to do it. I suppose baking your own TCP/IP has the advantage of being obscure so that an exploit for a known stack doesn't work on yours. But there is only so many bad things you can do to a TCP/IP stack so you are bound to repeat a security hole that some other popular stack had many years ago. Even good programmers are not perfect and will miss something somewhere.

But very rarely is the actual transport layer at fault for an exploit. Sure DOS attacks are a thing but there is nothing you can practically do when someone is opening millions of connections to you or barfing so much data at you that your Ethernet link gets clogged up. Usually the layers sitting higher are at fault, they do more complicated work and are often very configurable by the user. The best mitigation for that is to run as few services as possible and keep them simple. This is easier if your device only opens connections to other devices and never listens for connections, you can simply ignore any inbound data that you didn't ask for. But in any case whatever application is using the TCP/IP is even more at risk of having a security hole.

Not running a OS quickly makes things easier to secure since a lot of services that are common attack vectors are simply not there. And since the exploit is likely going to be inside the application code you wrote on top of the TCP/IP stack means that the attack only works on that one product family.

More of an issue with linux systems is that they never get updated. A WiFi router might come out on release day with a few year old version of linux because that's simply the latest kernel they had for that low cost SOC chip. Because its an old chip the development of the linux kernel from the manufacturer disappears and nobody wants to port it themselves so the router continue being sold for 5 more years with the same linux image. Then the router keeps working at the user for over 5 years before it gets replaced...etc So you end up with Linux builds >15 years old running on devices constantly connected to the internet in vast numbers. Even if you wanted to keep your router firmware up to date you can't, new firmware simply was never created.

But yeah part of the problem is that when management is breathing down the necks of the programing team about getting it ready on some ridiculous deadline (Because they really need to hit that marketing window and they already wound up the hype train for the product). At that point the programmers are just happy to get the thing mostly working. Lots of shortcuts are taken, random code is slapped together, nobody has time to test it properly, the codebase becomes a big tangled ball of spaghetti that nobody wants to even touch once it works. By the end of all that security is pretty much the last thing on anyones mind, besides spending 5 months on security means no shiny new features in the product and how are they going to explain that to the investors.

[richardman]

Yup, exactly the kind of things I am alluding to in the FreeRTOS thread (sorry). As a third party vendor, it is unreasonable for us to write a TCP/IP stack ourselves - it takes too much resource that we do not have. So lwIP is appealing. There is a healthy developer and user community around it so hopefully there aren't any/many exploits like the ones in FreeRTOS. However, I think that the FreeRTOS incident gives a clear directive to us commercial vendors (and in theory non-commercial suppliers) that our customers need to understand the need and allow for security updates on their firmware, and that we must provide fixes to our software in a timely manner as well.

Of course from the user side, any updates could take a long validation time and they may not like to have to update hundreds, or thousands or more of their gadgets in the field just because there are exploits that may or may not affect them.

It may be interesting to see the feasibility of using the Linux or FreeBSD etc. stack, but chances are they make too much assumption about (having virtual) memory that will make them useful for MCUs.

There are a small number of proprietary TCP/IP stacks for MCU. uC/OS had one but they are now part of SiliLabs. CMX still has one, as does Segger. OTOH, security via code opaqueness may not be any better than open source ones either. It would be interesting to find out whether the FreeRTOS exploits were discovered with or without the benefits of having the source code available.

[nctnico]

Like ejeffrey already wrote: don't connect microcontroller gadgets to internet directly.
It makes a lot of sense to offload the network stuff onto a secondary processor. In my most recent IoT-ish project I used a Wiznet chip. Even if the network chip gets hacked the device itself can still function. But ofcourse the network chip could become part of an evil botnet.

I have a lot of experience with uIP (the predecessor of lwip from the same author) and it took a lot of effort to get it in a useable state. Based on that experience I have become wary of using TCP/IP network stacks and I wouldn't want to use Lwip in particular.

[mrpackethead]

Topical thread for me at the moment.

I'm considering options for an IP stacking on Microblaze ( Xilinx ).  Theres a lot of stuff using IPWP but not a lot with FreeRTOS.. However FreeRTOS+IP looks like it will perform cosniderably better.

[Geoff_S]

Based on that experience I have become wary of using TCP/IP network stacks and I wouldn't want to use Lwip in particular.

Curious about this - anything you can share about why the reluctance to use LwIP ?  Seems to be one of the more popular stacks for microprocessors.

[nctnico]

Based on that experience I have become wary of using TCP/IP network stacks and I wouldn't want to use Lwip in particular.

Curious about this - anything you can share about why the reluctance to use LwIP ?  Seems to be one of the more popular stacks for microprocessors.

uIP has been written very poorly to allow maximum optimisation to run on a 16 microcontroller. This made it extremely hard to follow an debug. If you look at the web page of the author (of lwip and uIP) you'll see most of his software are proof-of-concepts instead of production ready software. If you need to use a TCP/IP stack in today's world you need to use one which has been designed for security from the ground up.

[westfw]

the reason most IoT stuff is such security garbage has nothing to do with TCP/IP stack vulnerabilities.

Amen to that!
The main reason that some stacks have fewer flaws is only that their flaws have already been found.  :-(
(Hey @Legacy: any well-trusted TCP/IP stacks written in Ada or other "memory-safe" language?)

[Whales]

Aside on FreeRTOS: a few years back I read some dramas about FreeRTOS' license terms prohibiting benchmarking (and some other dramas).  Now I can't find this in their published licenses.  Have things changed?

EDIT: Old reference material https://www.freertos.org/FreeRTOS_Support_Forum_Archive/January_2015/freertos_FreeRTOS_Benchmarking_f78bbf5fj.html

[splin]

d) don't put microcontrollers on the public internet.  Microcontrollers either use a non-ethernet inteface (USB, serial, CAN, bluetooth) or go on private/isolated subnets. Access is controlled via proxies running on an a general purpose OS with a fully featured well vetted TCP/IP stack, firewall, TLS, and authentication libraries and protocol implementations.

But in many cases that isn't possible. For example, you have a remote water level monitoring station; whilst some water companies do operate private radio networks for these types of application, in many cases a GSM modem is the only practical communications option. It may have to run from a solar panel and batteries limiting average power to << 1W. You could meet that power budget with a 200MHz Cortex M4 or even a 400MHz M7 design with lots of memory and AES encryption built in so plenty of resource available for implementing a robust stack with proper encrytpion.

Running a full strength OS such as Linux or BSD would require something like a Raspberry Pi or similar which would be very difficult and probably impossible with << 1W. It would also significantly increase the BOM cost and the management costs as you now have to maintain the Linux/BSD software as well as your firmware. I expect reliability would be impacted adversely too. Most importantly there's no long term security of supply for modules like the Raspberry Pi.

And security wise you may have made matters much worse. Yes the stack is well tried and tested and security fixes are generally rolled out very quickly. But you have massively increased the attack space as you have to ensure the security of all the software running on your communications gateway. Obviously you would limit running services to the absolute minimum but still you now have to follow CERT advisories/alerts for far more than just the stack. I'd argue that a firewall is also unnecessary in an embedded product as it will either not accept any incoming connections from anyone, or have a single incoming port to which only your own (hopefully) secure F/W is connected. Blacklists and whitelists likely have no place here either because of the need to manage them remotely.

And yes a remote water level monitoring system is unlikely to be much of a threat to safety or national security should hackers gain control. Until the water company requests a remotely controlled output to operate the floodgates or whatever.

Because honestly, Amazon's FreeRTOS vulnerabilities aside, the reason most IoT stuff is such security garbage has nothing to do with TCP/IP stack vulnerabilities.

Yes I'm sure this is true, but stack vulnerabilities are still a concern - even if it just the issue of persuading your customer that your stack is secure. Perhaps the solution is to test against up to date attack tools like SATAN (ok. that's probably long obsolete). Are there any such test suites and procedures that are generally accepted to demonstrate a products resistance to attacks?

[NorthGuy]

I don't think it's possible to avoid DoD attacks. Attacker can just flood you with packets. To find out if a packet is good or bad you have to receive it. And you can only receive so match. Thus, a flood will cause downtime when a legitimate user has hard time connecting. May also discharge your battery during the night.

If you have "secure" things, such as SSL, this makes DoD attacks much easier - small amount of packets causes lots of work on the CPU site. This makes much easier for the attacker to blow up your power budget, or drain batteries during the night, possibly causing permanent battery damage or even fire. Overheating and killing the CPU may also be possible.

Your own stack is likely to be more secure, because you need a limited subset of what the regular stack provides, also because your foremost goal would be security. But it'll come at much greater price - you're lucky if you create a working stack in two weeks. Even if you create some vulnerabilities, they're unlikely to be exploited - it's a lot of work for a hacker. IMHO, 99.9% of hackers are not very far from Arduino users - they take a ready-made exploit and they apply it. Such attacks are of no concern to your custom stack.

[LapTop006]

Running a full strength OS such as Linux or BSD would require something like a Raspberry Pi or similar which would be very difficult and probably impossible with << 1W. It would also significantly increase the BOM cost and the management costs as you now have to maintain the Linux/BSD software as well as your firmware. I expect reliability would be impacted adversely too. Most importantly there's no long term security of supply for modules like the Raspberry Pi.

I've been playing with the chip in the pocketbeagle, with Linux running they idle at roughly half a watt. At load, well above unsurprisingly.

[legacy]

Hey @Legacy: any well-trusted TCP/IP stacks written in Ada or other "memory-safe" language?

 

[rstofer]

But very rarely is the actual transport layer at fault for an exploit.

Exactly!  Encryption, or the lack thereof, is not the responsibility of TCP or IP.  These two protocols just deliver packets and, in the case of TCP, guarantee delivery.  But it's just packets, the contents are determined at a higher level.  Buffer overflow isn't really possible with TCP because packet size is limited by the protocol.

The maximum TCP packet is 64k but the maximum transmission unit (MTU) of Ethernet is 1500 bytes.

I wouldn't be in a hurry to blame TCP or IP for security problems.  They're just the messenger, not the message.  Time is better spent getting SSL to work.

[NorthGuy]

With C ... you have nothing, you are alone, the language doesn't help, doesn't collaborate, it's all on your shoulders ...

It's not on your way.

[nctnico]

But very rarely is the actual transport layer at fault for an exploit.

Exactly!  Encryption, or the lack thereof, is not the responsibility of TCP or IP.  These two protocols just deliver packets and, in the case of TCP, guarantee delivery.  But it's just packets, the contents are determined at a higher level.  Buffer overflow isn't really possible with TCP because packet size is limited by the protocol.

The maximum TCP packet is 64k but the maximum transmission unit (MTU) of Ethernet is 1500 bytes.

I wouldn't be in a hurry to blame TCP or IP for security problems.  They're just the messenger, not the message.  Time is better spent getting SSL to work.

Nope. SSL is not some magic sauce you can pour over a piece of code and call it secure. It doesn't work that way. At a higher level you'll need to design the system securely right from the start. As a rule of thumb a well designed system is still secure without encrypting the messages.

[legacy]

It's not on your way.

of course, and ten bottles of whiskey are on my way thanks to Ada 

[SiliconWizard]

It's not on your way.

of course, and ten bottles of whiskey are on my way thanks to Ada 

 

[ejeffrey]

d) don't put microcontrollers on the public internet.  Microcontrollers either use a non-ethernet inteface (USB, serial, CAN, bluetooth) or go on private/isolated subnets. Access is controlled via proxies running on an a general purpose OS with a fully featured well vetted TCP/IP stack, firewall, TLS, and authentication libraries and protocol implementations.

But in many cases that isn't possible. For example, you have a remote water level monitoring station; whilst some water companies do operate private radio networks for these types of application, in many cases a GSM modem is the only practical communications option. It may have to run from a solar panel and batteries limiting average power to << 1W. You could meet that power budget with a 200MHz Cortex M4 or even a 400MHz M7 design with lots of memory and AES encryption built in so plenty of resource available for implementing a robust stack with proper encrytpion.

Sure, sometimes this is just not practical.  But a lot of IoT stuff is not particularly power constrained, and people use microcontrollers out of the belief that they are "simpler".  While the microcontroller is simpler, that just means you have to do the complex stuff yourself. 

I also don't think your example is very good.  Distributed sensor networks with GSM can use carrier provided VPNs to central gateway that is the only device allowed to access them.  This is a good idea no matter what architecture you use.  Also, you can easily design such a system and keep within a low average power.  You are already going to be operating with extensive use of sleep states since GSM modems take up a fair bit of power when operating.  A totally reasonable architecture is to have a microcontroller that is monitoring water levels and can wake up the "main" processor from a deep sleep mode either at regular intervals or when there is an alert.  The main processor would then start up the modem, read a batch of data from the micro and transmit it to the central server before going to sleep.  Many smart phones do this: they have a microcontroller that operates the accelerators and other sensors, and can wake up the main processor when needed.  If for some reason you couldn't use an external VPN, but had to implement the security on the sensor, this is the architecture I would suggest.

There is no one-size fits all approach, but I strongly believe the default answer should be "if it needs to run TCP/IP, use a full size CPU and OS", and you need to carefully justify any exception to that.

[legacy]

I wouldn't be in a hurry to blame TCP or IP for security problems.  They're just the messenger, not the message.  Time is better spent getting SSL to work.

Precisely 

I remember an implementation over the transport layer where the TCP/IP was merely used to transport messages, and it was OK, but there was a bug in a third library, and under certain conditions, a too long message was able to cause leakage of precious information in the application.

A funny episode 

[rstofer]

I wouldn't be in a hurry to blame TCP or IP for security problems.  They're just the messenger, not the message.  Time is better spent getting SSL to work.

Precisely. I remember an implementation over the transport layer where the TCP/IP was merely used to transport of messages, and it was OK, but there was a bug in a third library, and under certain conditions, a too long message was able to cause leakage of precious information in the application.

These leaks have always been a problem.  Perhaps more so with PCs but it was a really big deal in the mainframe world as well.

When multitasking allowed for multiple applications to be resident at the same time, protecting the data of one application from code in another application was a really big deal.

In some ways we see in crt.S some code to clear .bss on startup.  That's pretty straightforward when there is only one app and I assume, but haven't verified, that the same thing happens when multitasking.

Base and bounds registers were used on the CDC 6400 to keep apps in their own memory space.  It was all too easy to allocate a huge array and read the contents of some other dataset so the registers put a limit on the addressable space.  In addition, the allocated memory space was always set to zero when the app was loaded. 

Classified information and all that...

You would think this stuff would be settled by now but we just keep coming up with new ways to make old mistakes.

[richardman]

First of all, do not evaluate lwIP based on what one might know from uIP. Yes, it was originally written by the same author, but lwIP was started with a much better set of requirements and specification. Furthermore, it has been enhanced by many people since then.

Second, to use lwIP effectively, and probably any TCP/IP, using an RTOS is a must. I have no personal experience of FreeRTOS+lwIP. However, we do provide an RTOS with our lwIP package. Without an RTOS, you basically have to avoid the threading issues inherent with multiple execution context that exist in a TCP/IP environment. Any workaround to use lwIP bare metal would not be worth it to avoid using an RTOS.

[rstofer]

Second, to use lwIP effectively, and probably any TCP/IP, using an RTOS is a must. I have no personal experience of FreeRTOS+lwIP. However, we do provide an RTOS with our lwIP package. Without an RTOS, you basically have to avoid the threading issues inherent with multiple execution context that exist in a TCP/IP environment. Any workaround to use lwIP bare metal would not be worth it to avoid using an RTOS.

The original mbed platform (LPC1768) had lwIP as one of the libraries.  Using an RTOS wasn't mandatory and I don't think it was available at the time (as one of the standard mbed libraries).  I had very simple needs: Open a TCP connection to a LaserJet and send packets.  At some point the print job was complete so close the connction.  Nothing exotic, no replies expected, no worry about out-of-order reception (as there was no reception) and so on.

Under at least one very limited application, lwIP can be used on bare metal without an RTOS.

I just built a simple daughter card containing a MagJack and plugged in the mbed board:

https://os.mbed.com/platforms/mbed-LPC1768/

Beyond my very simple application, I think I would use FreeRTOS and the included TCP/IP stack (based on lwIP):

https://www.freertos.org/FreeRTOS-Plus/FreeRTOS_Plus_TCP/index.html

[nctnico]

You definitely don't need an RTOS to do TCP/IP and other networking stuff. To use an RTOS or not depends entirely on the scheduling requirements of the entire application. A network stack is just another interface which inputs/outputs data.