Author Topic: Running a web server on an embedded system - risky?  (Read 8030 times)

0 Members and 1 Guest are viewing this topic.

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Running a web server on an embedded system - risky?
« on: November 12, 2021, 06:56:00 am »
It seems to me that anyone who sets up a device which has a "server" on it, and is on an open port, is going to see it trashed eventually. There are thousands of chinese and russian port sniffers running constantly, scanning IPs, so even if you don't publish the DNS it will be discovered in hours, and hit with a string of attacks.

The server code will be something which came with Cube IDE or whatever i.e. poor quality code, with whatever patches applied as they were found on stackexchange and other forums. It won't be up to date in any sense that a normal web server, or PHP or whatever language is used on it, is kept updated as new attacks come to light. Embedded systems are frozen, with no regular updates like e.g. windows gets.

If there is no hard disk or other filesystem storage, the damage probably won't be permanent, in the sense that a power cycle will get it back running again.

I know this is the standard "IOT security" debate, but most people are addressing this by requiring all sorts of authentication/encryption, so you end up running HTTPS which is a massively bloated load of code, with its own quantity of bugs, and it needs a lot of memory (two 16k buffers, for a standard server implementation) which is a struggle to find on smaller CPUs. But that isn't the point; the back doors won't be found by cracking TLS, cracking AES256, etc. They will be right there in the same places where they always were e.g. you will be able to crash the server by sending it some malformed packet which overflows the memory. Hackers get plenty of enjoyment just by crashing some device; it is not necessary to steal 1000000 credit card details :)

When one looks at how much work, over so many years, has gone into patching up the various server operating systems, the embedded scene has no chance. And once a back door is found, in most cases nobody will be deploying patches remotely. Anyone doubting this only needs to look at the world of consumer routers. They are full of bugs, mostly not fixed, and any fix, if made available, needs to be downloaded by the user, in the form of a firmware upgrade.

Most patches for embedded platforms are never published because they were developed in a company's paid time so the dev won't be uploading them anywhere.
« Last Edit: November 12, 2021, 07:32:56 am by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6172
  • Country: fi
    • My home page and email address
Re: Running a web server on an embedded system - risky?
« Reply #1 on: November 12, 2021, 10:06:52 am »
It seems to me that anyone who sets up a device which has a "server" on it, and is on an open port, is going to see it trashed eventually.
No, unless by "anyone" you mean "anyone who does not know how to do it securely".
Of course, you cannot add security on top, making an insecure thing somehow secure, so if the code you use is insecure, it is insecure.  You can cage an insecure thing, so that if compromised, only the contents of that cage get trashed, but that's it.

The first step in securing anything is to make sure it runs with minimum privileges necessary.  You don't need to run a web server as root, just because you want it to be able to bind to ports 80 (http) and 443 (https).  The (second) worst mistake, and the reason why Wordpress, phpBB etc. are inherently insecure, is giving the server write access to itself.  (The worst mistake is doing something like chmod 0777 --recursive /var/html, which is Linux/POSIX-speak for "feel free to write anything here, everybody!  I'll publish it on the net for you.")  Essentially, you can make sure that even if the server is compromised, the attacker cannot do anything, unless they already have a privilege escalation.  In Linux, this involves understanding capabilities, prctl(), and limiting resources available to a single process via setrlimit().

The second step is to use a firewall, and apply some sort of fail-to-ban.  The fail2ban is easiest to use on POSIXy machines with enough RAM, but there are other options for embedded/SBCs with limited amount of RAM.  Essentially, you use failed connection attempts from the same source to tell your firewall to completely drop any connection attempts from that address for a specified period (depending on the attempt); I use 24 hours.  I use fail2ban on all my machines, actually, triggering on at least failed SSH connection attempts.  This takes care of brute-force attacks.  Additional stuff like tripwire can be used to detect successful intrusion at the point where they manage to modify the configuration files or system binaries.
(I've been tinkering with a really lightweight daemon that listens on specific ports, and blocks (adds temporary firewall rules) for any IP addresses that attempt to connect to those ports.  It is more severe than fail2ban, but also requires very little memory and kernel resources to run; very suitable for routers and such.)

The third step, assuming Linux-based embedded system, is to enhance the server code with an unrevokable seccomp filter.  Essentially, you limit the syscalls the server process can perform, using a kernel-based filter.  If your server simply reads some files and maybe collates them somehow, and serves them to clients (a dashboard of some kind), you can limit it to an extremely small set of syscalls.  (This isn't difficult after you discover the set of syscalls you need –– and you can use strace to check at run time ––; see this example "my friend" Blabbo the Verbose has posted at SO.)  This seccomp filter minimises the attack surface in case the server itself is compromised, so that privilege escalation via a kernel bug is even less likely to happen.

The fourth step is a mix of using a separate SBC to collect the logs from the internet-facing machines, which themselves don't have any ports open to the internet, and are only accessible via your LAN.  This lets you check the status of your various machines in a centralized location.  In the case one of the server machines is compromised, it is unlikely they manage to also break into your logging machine (because you use different usernames and passwords to access the logging machine!), so they cannot clean up after themselves.

So, as you can see, although risk mitigation isn't "trivial", there are many extents to which you can go.  The very first step is to limit the attack surface in the case where an attacker manages to exploit an existing security hole.  Then you can choose additional steps to reliably detect if that happens, and to strenghten the security of existing server code.

The embedded devices I have exposed to the internet right now are all router-type devices, all of them running some variant of OpenWRT.  It is not perfect, but most of the related security advisories are to do with the management interface and not the router capabilities.  Which is why I configure mine with one dedicated physical admin port, so that the admin interfaces are not accessible at all via WLAN or from the internet port.  I prefer to have a separate outer edge firewall with an SSH service on a nonstandard port, and an instant ban on attempts to the standard SSH port (22), and my wireless routers just bridges between WLAN and (untrusted) LAN.  This means that you cannot actually access my WLAN/LAN routers from the internet (even if you get through my firewall) nor via a wireless connection, since they don't have an IP address exposed.
(There would have to be a lower, ethernet level security hole in the linux kernel, to get through.  It's such a simple and old protocol, I highly doubt such a hole exists, even to state-level actors.)

The discussion on how to write service-type code that is secure from external attacks, is somewhat related, and a topic woefully full of garbage and misinformation on the net.  DJB's qmail is a good example of how to design security in to code (although I cannot really decide whether the integer overflow bug, the only serious one ever found, ought to have earned the $500/$1000 bounty), and yet, a LOT of people really, really dislike DJB's code.
« Last Edit: November 12, 2021, 10:09:28 am by Nominal Animal »
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: Running a web server on an embedded system - risky?
« Reply #2 on: November 12, 2021, 10:39:14 am »
As far as I know due to my little personal experience with a web-server exposed to the internet 24h/24,  even if you choose an "hardened" GNU/Linux distribution, the problem is not the http server itself (1) but rather the language-support and its requirements.

For example, php suffers from some problems that can be exploited to damage a mySQL database. Python is a better option there and Ruby on Rail is the best, unfortunately many web applications are written in PHP so you face a practical choice:
  • either you "trust" a piece of PHP code that should have been "sanitized" (cleaned up)
  • or you have to study and learn Python or Ruby to rewrite the web application yourself.

(1) { Lighttpd., NGINX, apache2, ... }
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6172
  • Country: fi
    • My home page and email address
Re: Running a web server on an embedded system - risky?
« Reply #3 on: November 12, 2021, 11:48:07 am »
(I don't know everything either, obviously.  I have done web server development and maintenance and web master stuff since '97, sometimes as paid work, sometimes as a volunteer, occasionally just for fun; but always with an emphasis on the security.)

As far as I know due to my little personal experience with a web-server exposed to the internet 24h/24,  even if you choose an "hardened" GNU/Linux distribution, the problem is not the http server itself (1) but rather the language-support and its requirements.
True, but this affects all web services, not just embedded ones.

One really should run any scripts using a dedicated, less-privileged account, with the script files owned by a yet another account, so that the scripts cannot modify themselves.  With PHP and Python, this is easiest to implement using FastCGI right now.  The annoying thing is, none of the web hotels (cheap web site hosting services) support multiple OS accounts, so one needs to configure and maintain their own (virtual) server to do this right.  >:(

When one does go that route, the very first thing to do is utterly counterintuitive: you'll want to change the location of both the runtime configuration of your server (Apache, Nginx, Lighttpd, etc.), and their document roots.  The reason is that you need to be in control of the configuration, not the package manager, because only you can tell what configuration is optimal for your use case.  Keeping the original configuration directories means you can occasionally easily check if the default configuration changes anything important (thus far, basically only when switching server major versions), and "import" any useful changes to your own config.

I used to have a web page showing an example of such configuration.  I've been thinking of switching to a virtual server (I'm using a web host now), but the cost difference has thus far been bigger than how much I care, because the web is full of related garbage advice, and I don't know any way to prove (except for my own experiences in various environments, including universities) that it works better.  Just being another voice among a shouting throng isn't my scene, really.
 
The following users thanked this post: DiTBho

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #4 on: November 12, 2021, 01:24:32 pm »
The above responses - while I agree 100%, to the extent of my "proper server" involvement - are based on devices which have a "proper OS" (i.e. a target), have a hard disk (i.e. a target), have scripting languages (i.e. a nice target), run open source code (a fantastically easy target), have firewalls (often badly configured) but some embedded device won't have a fail2ban feature because that needs storage and potentially a lot of it, etc... And fail2ban works only if the first attempt was unsuccessful ;)

I guess the Q is: what could you trash on a typical embedded HTTP/HTTPS server, say a central heating controller, which has the usual bug-ridden and partly patched code from STM and bits found posted online? You could presumably crash it with malformed packets, so it needs a power cycle to run again. That is bad enough, because you can do it (the attack) remotely, and - because the code can't be patched - you can just keep doing it, every night at 3am :)

In fact the vulnerability will probably never be discovered because the customer has no access to the firmware, the vendor is not interested in supporting a 2 year old product, and in any case unless you set up a packet logger on the connection you will never find out what did it.
« Last Edit: November 12, 2021, 01:39:45 pm by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6172
  • Country: fi
    • My home page and email address
Re: Running a web server on an embedded system - risky?
« Reply #5 on: November 12, 2021, 03:46:20 pm »
devices which have a "proper OS"
Well, you can run Linux on pretty scanty resources: 4 to 8 MiB of ROM/Flash, and 32 MiB or more RAM.  You can often find Linux in devices you thought ran a simple firmware, because integrators that can cobble together something for an appliance are cheap compared to raw-metal or RTOS embedded programmers.

some embedded device won't have a fail2ban feature
True, that's why I'm fiddling with a truly minimal "connect to port 22 (SSH) / 80 (http) / 443 (https) and you'll be banned for at least a few hours" daemon;
and similar facility can be easily added to e.g. lwIP stack.  If you reserve say 2k of RAM for the IPv4 blocklist and/or 8k for IPv6 blocklist, you can block about 256 addresses (each) for a configurable duration; and if you keep these sorted, a binary search verifies if blocked in less than ten memory accesses.

(As I said, my devices currently run OpenWRT, but I do have one Teensy 4.1 with an Ethernet port, that I could play with lwIP stack.)

I guess the Q is: what could you trash on a typical embedded HTTP/HTTPS server, say a central heating controller, which has the usual bug-ridden and partly patched code from STM and bits found posted online? You could presumably crash it with malformed packets, so it needs a power cycle to run again.
That's why I use a Linux-based firewall at the edge of my home networks.

Configure the network so that the IOT things use private IP addresses (10.x.y.z, 172.16-31.y.z, 192.168.x.y), and completely block direct access to these across the firewall (note that this means not forwarding any packets TO or FROM from such addresses over the firewall, ever).  If you ever want to connect to any of the IOT things outside your network, you bounce through one of your machines inside your own network.  If that bouncer machine has fail-to-ban support, and you use non-default port for incoming SSH connections, your unreliable IOT things are pretty well shielded from attacks.

Wireless IOT things are almost impossible to secure against local wireless access, though.
 

Offline mfro

  • Regular Contributor
  • *
  • Posts: 207
  • Country: de
Re: Running a web server on an embedded system - risky?
« Reply #6 on: November 12, 2021, 04:08:26 pm »
... I guess the Q is: what could you trash on a typical embedded HTTP/HTTPS server, say a central heating controller, which has the usual bug-ridden and partly patched code from STM and bits found posted online? You could presumably crash it with malformed packets, so it needs a power cycle to run again. That is bad enough, because you can do it (the attack) remotely, and - because the code can't be patched - you can just keep doing it, every night at 3am :) ...

It appears the thing to do is to put a reasonable application-aware firewall in front of your faulty device. With a little luck, this will filter out nasty traffic and will hopefully allow you a good nap again.
Beethoven wrote his first symphony in C.
 

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #7 on: November 12, 2021, 04:33:21 pm »
Nobody will be putting firewalls in front of (today's fashionably called) IOT devices.

Also most of them can't run any form of "unix". A 32F417 has just 1MB of FLASH, and that CPU is near the upper end of consumer products.

The obvious solution is to not have the device on an open port, but then you can't really have "IOT" :)

In the domestic scenario, nearly all internet access will be via WIFI and then the house's ADSL, so it won't be on an open port, but the installation procedure will try to open ports via the P&P feature found in cheap consumer routers. This is what e.g. webcams try to do.

The funniest thing is that google eventually finds all these devices on open ports :) It's like the hobby where people look for webcams, using google search expressions, and find that a lot of them are inside peoples' houses...
« Last Edit: November 12, 2021, 04:36:29 pm by peter-h »
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14297
  • Country: fr
Re: Running a web server on an embedded system - risky?
« Reply #8 on: November 12, 2021, 04:47:47 pm »
It seems to me that anyone who sets up a device which has a "server" on it, and is on an open port, is going to see it trashed eventually. There are thousands of chinese and russian port sniffers running constantly, scanning IPs, so even if you don't publish the DNS it will be discovered in hours, and hit with a string of attacks.

Well, I would personally restrict the use of a "web server" on those embedded systems to LAN access ONLY. It's not just the security - although that in itself would be hard to get right on small systems, it's already hard to get right on full-fledged servers - but also about performance. It could potentially get the system on its knees, which could prevent it from doing anything useful at all, or crash, or whatever.

This is not what most IoT devices do anyway - they themselves push and pull data to/from outside servers, but they do not act as servers themselves.

I still find it useful for a device to act as an HTTP server, so you can access it through a web browser: its an easy way to add a fancy UI, which can be designed by people who are not software engineers! But I would add all measures so that it can't serve requests outside of a LAN.
 

Offline mfro

  • Regular Contributor
  • *
  • Posts: 207
  • Country: de
Re: Running a web server on an embedded system - risky?
« Reply #9 on: November 12, 2021, 04:59:37 pm »
Nobody will be putting firewalls in front of (today's fashionably called) IOT devices.

Didn't know my employer is a nobody - I always thought it's a large automotive supplier ;)
Beethoven wrote his first symphony in C.
 

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #10 on: November 12, 2021, 05:36:03 pm »
"restrict the use of a "web server" on those embedded systems to LAN access ONLY."

How would that be done?

I am no expert but perhaps you mean limiting responses to the subnet of the device IP received via DHCP (which will be the local router's IP e.g. 192.168.1.44).

"Didn't know my employer is a nobody - I always thought it's a large automotive supplier"

What are they using IOT devices for?
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14297
  • Country: fr
Re: Running a web server on an embedded system - risky?
« Reply #11 on: November 12, 2021, 05:55:39 pm »
"restrict the use of a "web server" on those embedded systems to LAN access ONLY."

How would that be done?

The general idea is to allow connections only from a range of IPs that can only be used for a LAN, such as 192.168.x.x

The way to implement this depends on your system of course. For Apache, you can read this: http://httpd.apache.org/docs/current/howto/access.html

But on a small embedded system, I would even filter incoming packets before handing them to the HTTP server, and would filter out anything else than 192.168.x.x, for instance.
Now of course, putting your device behind a dedicated firewall would be a better option, but the device having itself some kind of filtering, even if not as fancy, is IMO a good addition.

Now in any case, you should give all the network access part in your system a low priority compared to everything else it's doing. So that whatever the network activity it would see, all the essential tasks your system is executing are served first, and that no amount of abuse network-wise could get the system on its knees.

« Last Edit: November 12, 2021, 06:03:13 pm by SiliconWizard »
 

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #12 on: November 12, 2021, 06:48:16 pm »
A great feature idea - many thanks :)

"they themselves push and pull data to/from outside servers, but they do not act as servers themselves."

That isn't futureproof; as soon as the vendor stops running that server (which is certain to happen), you lose some part of the functionality.

"I still find it useful for a device to act as an HTTP server, so you can access it through a web browser: its an easy way to add a fancy UI, which can be designed by people who are not software engineers"

Agreed; doing this too. But there you know what the client will be doing, and you are serving a limited amount of data. The complication is that "customers want https", and the way things are going, soon there won't be any common browsers (read: chrome) which will connect to an http server at all. That, given the certificate in the product will be self-signed, causes the browser, especially chrome, emit various warnings which have to be cancelled to continue.

Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6172
  • Country: fi
    • My home page and email address
Re: Running a web server on an embedded system - risky?
« Reply #13 on: November 12, 2021, 07:47:58 pm »
The general idea is to allow connections only from a range of IPs that can only be used for a LAN, such as 192.168.x.x
With lwIP, one can use the LWIP_HOOK_TCP_INPACKET_PCB() hook to examine the address of incoming TCP packets, and drop those that are not from a private IP address range, or are currently blocked; or extend src/netif/ethernet.c:ethernet_input() to check the the Ethernet frame and the IP header if it contains an IP packet, and drop it if necessary, before it is passed on to the IPv4/IPv6 layer.
 
The following users thanked this post: SiliconWizard, DiTBho

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 21606
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Running a web server on an embedded system - risky?
« Reply #14 on: November 12, 2021, 08:20:07 pm »
What is this in regards to -- like, something you're looking at making?  Curiosity about existing solutions?  What range of things is available?  How to hack them?

Something maybe worth adding is the case of the minimum viable solution.  At its most basic, an HTTP server accepts and parses a certain amount of boilerplate text, then dumps out text in response.  It doesn't even need to respond in a dependent way; it can be essentially all client-side.

The simplest realization, then, is an MCU that, on a suitable trigger (correctly parsed request, where "parsed" may amount to checking certain fields are in order, and ignoring the rest), dumps from Flash, say, into whatever the output port is (which might be a one of those SPI-Ethernet bridges; Wiznet and Microchip have examples).  Additional bridges tie the chip into the broader networking environment (switch/wifi AP/router/etc.) and boom, you've got an "unhackable" embedded server on the internet.

Any additional functionality built upon that, also brings vulnerability.  So, do that carefully, and draw from well-known sources when needed (and possible).

So, between the extremes of this, and a full server PC, or cloud service: take your pick, right?

As for getting started fairly easily, yeah, probably something running Linux (note STM32F4s can run limited versions of Linux) and using the standard tools, probably a good idea.  It'll be subject to whatever vulnerabilities are known of the software stack (Linux, Apache or whatever, etc. are freely available and well known), as of whatever versions of them you happen to use.  Anything more stripped-down is likely harder to use; anything built from the ground up, will have who-knows how many bugs, but at least no one knows about them (security by obscurity)?

Which, if it's a one-offs kind of a thing, obscurity is perfectly fine; no one's going to develop an exploit for your one random device.  At least, if no one is motivated to do so (maybe be a bit more careful if you're a journalist or spy?).  If this is going into production to make 10^5+ of them, yeah probably a good idea to take a look at it, maybe even hire some analysis/fuzzing/etc. to be done.

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6172
  • Country: fi
    • My home page and email address
Re: Running a web server on an embedded system - risky?
« Reply #15 on: November 12, 2021, 08:28:33 pm »
The complication is that "customers want https", and the way things are going, soon there won't be any common browsers (read: chrome) which will connect to an http server at all. That, given the certificate in the product will be self-signed, causes the browser, especially chrome, emit various warnings which have to be cancelled to continue.
Have the users open/download the public certificate itself; their browser should prompt whether the user wants to trust that certificate or not.  That way the browser should not complain at all.  Of course, that needs to be done on each browser the user wants to use.

If you have multiple servers, becoming your own Certificate Authority is less work.  Basically, you create a CA certificate, and install the public CA certificate as above.  Then, when you create a certificate request, you use the private CA to sign the request, getting a server certificate signed by the CA.  As long as the CA certificate is installed on the browser, you can sign any number of server certificates with it.  This is particularly useful with private fixed IP addresses (instead of hostnames) in the server certificate, since certificate authorities won't sign certificates for private IP addresses anyway.

lwIP does have TLS support via mbed-tls when using the application layered TCP raw/callback-based API.  The actual "application" part is the same as when using plain TCP.

(In case it is unclear, lwIP is a network stack you can run on pretty darn small microcontrollers, and has nothing to do with Linux at all.)
 

Offline ralphrmartin

  • Frequent Contributor
  • **
  • Posts: 479
  • Country: gb
    • Me
Re: Running a web server on an embedded system - risky?
« Reply #16 on: November 12, 2021, 08:40:23 pm »
How big do you consider an "embedded" system to be? Anything capable of running Linux can potentially have just as much security as "serious" commercial servers which also run Linux. Certainly something like a Raspbery Pi can do that for a few £10s. In some applications, that would be far too expensive, but in others, a trivial part of the system's total cost. Trying to lump all "embedded" systems under the same umbrella is not really meaningful.

You can get free certificates for https from Let's Encrypt.
 

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #17 on: November 12, 2021, 08:59:43 pm »
From their website:

What is the lifetime for Let's Encrypt certificates? For how long are they valid? ... Our certificates are valid for 90 days.

So now you need a process running every 89 days to renew this certificate, and this is quite likely to break, sooner or later. It is the fashionable modern way to do this but it also breaks easily. Just had that at work.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline T3sl4co1l

  • Super Contributor
  • ***
  • Posts: 21606
  • Country: us
  • Expert, Analog Electronics, PCB Layout, EMC
    • Seven Transistor Labs
Re: Running a web server on an embedded system - risky?
« Reply #18 on: November 12, 2021, 10:02:00 pm »
If you're online, LE is accessible. I've never had a failure to update mine, AFAIK.  And, whatever millions of others use it?  If it goes down for more than a little while, people are going to know about it. :)

Tim
Seven Transistor Labs, LLC
Electronic design, from concept to prototype.
Bringing a project to life?  Send me a message!
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 6796
  • Country: va
Re: Running a web server on an embedded system - risky?
« Reply #19 on: November 12, 2021, 10:30:49 pm »
Quote
The server code will be something which came with Cube IDE or whatever i.e. poor quality code, with whatever patches applied as they were found on stackexchange and other forums. It won't be up to date in any sense that a normal web server, or PHP or whatever language is used on it

If this is small enough to not be running Linux and a 'proper' web server, why wouldn't it be running the server code you wrote yourself? It's not hard, and if you skip the arbitrary scripting stuff, like PHP, it's just handling inputs like any other input your code would deal with. Those toy servers may be trashed by 5K-long URLs, but your code certainly won't be.

Quote
most people are addressing this by requiring all sorts of authentication/encryption, so you end up running HTTPS which is a massively bloated load of code

ISTM there are two issues you need assistance with: authentication and, as you note, encryption. The first to ensure only the authorised user can see or, particularly, do things and the second to prevent eavesdropping. The S you probably want to outsource to make sure it's correct, but essentially it's just a shim under HTTP and between the comms media, isn't it? Doesn't mean you need a pukka 'server' to do it.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: Running a web server on an embedded system - risky?
« Reply #20 on: November 12, 2021, 11:24:27 pm »
you can run Linux on pretty scanty resources: 4 to 8 MiB of ROM/Flash, and 32 MiB or more RAM

my personal experience with PPC and MIPS32 routers

kernel+uclibc-roots in 8 MByte of ROM/Flash:
with kernel 2.4: yes
with kernel 2.6.x: { x<26 -> yes, x>25 -> maybe }
with kernel 5.x: no
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 
The following users thanked this post: Nominal Animal

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14297
  • Country: fr
Re: Running a web server on an embedded system - risky?
« Reply #21 on: November 13, 2021, 12:23:20 am »
A great feature idea - many thanks :)

"they themselves push and pull data to/from outside servers, but they do not act as servers themselves."

That isn't futureproof; as soon as the vendor stops running that server (which is certain to happen), you lose some part of the functionality.

Of course. And I don't like this IoT cloud stuff all that much anyway. But it's the only reasonably scalable solution at the moment, and it provides some safety net. A single, low power device acting as a server, yeah. Not only would it be kind of hard to secure correctly as we evoked, but it's hardly going to be scalable. Another related issue is that accessing it requires an address that may have no domain name attached and maybe even no fixed IPs. If you're targetting uses in a professional context, that's probably not a problem, but for individuals in their homes? Most private internet accesses still don't have a fixed IP. Something that can sort of be mitigated with services such as "dyndns", but that's a burden and end-users wouldn't want to mess with this.

An ad-hoc solution may not be all that much more future-proof either. Maybe at least you should consider for it to support IPv6, because IPv4 only at this point is not going to be future-proof.

First thing IMO would really be to ask yourself if this kind of remote access is a real benefit. What kind of devices are you thinking about?

 

Offline ralphrmartin

  • Frequent Contributor
  • **
  • Posts: 479
  • Country: gb
    • Me
Re: Running a web server on an embedded system - risky?
« Reply #22 on: November 13, 2021, 07:41:11 am »
What is the lifetime for Let's Encrypt certificates? For how long are they valid? ... Our certificates are valid for 90 days.

So now you need a process running every 89 days to renew this certificate, and this is quite likely to break, sooner or later. It is the fashionable modern way to do this but it also breaks easily. Just had that at work.

I use certbot for automatic certificate renewal, run in a cron job, and it has never failed for me.
 

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #23 on: November 13, 2021, 10:05:32 am »
"First thing IMO would really be to ask yourself if this kind of remote access is a real benefit. What kind of devices are you thinking about?"

I don't think the product I am working on will likely be on an open port, but it is a possibility. If it was done, the customer would need to understand firewalls etc.

" The S you probably want to outsource to make sure it's correct, but essentially it's just a shim under HTTP and between the comms media, isn't it? Doesn't mean you need a pukka 'server' to do it."

Someone else is doing that part, using MbedTLS. The STM library was as usual buggy and lots of people have struggled to fix it. It also consumes 50-60k of RAM; a bit less if you are a client not a server.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: Running a web server on an embedded system - risky?
« Reply #24 on: November 15, 2021, 07:53:10 am »
Someone or something (An alien on the roof sucking my wifi? A Loa of voodoo? A free running string of AI on the internet? who knows ... ) is knocking on port 22 (Ssh) of the automatic greenhouse for growing tomatoes in hydroponics :o

Daily quirks have been happening for a while, it seems like someone or something really wants my tomatoes.

It's an embedded Linux board based on TL-W703 exposed to the internet and when fail2ban bans its IP, somethings changes the IP, waits 661-669 seconds and tries again.

port23 ... umm, running a ssh-server on an embedded system is very risky ... it wasn't a brilliant idea, I think  :-//
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 
The following users thanked this post: Nominal Animal

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 6796
  • Country: va
Re: Running a web server on an embedded system - risky?
« Reply #25 on: November 15, 2021, 11:45:32 am »
Quote
It's an embedded Linux board based on TL-W703 exposed to the internet

Frankly, I wouldn't expose anything to the internet except a firewall at the router. One of the block lists my router subscribes to shows 78,143 connection attempts since midnight. Another, that is created from failed POP3 and SMTP logins here, shows 13,935 attempts.
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6172
  • Country: fi
    • My home page and email address
Re: Running a web server on an embedded system - risky?
« Reply #26 on: November 15, 2021, 01:28:46 pm »
DiTBho's and dunkemhigh's observations are the exact reason why I've been considering an insta-banner due to a connection attempt on specific ports on my intertubes-facing embedded devices.  It's simple, fast, extremely lightweight (runtime footprint just kilobytes!) and catches all automated attempts.  The idea is that a connection attempt to any of the specified standard ports causes the IP address to be blackholed for a specific duration, duration depending on the port.  (For example, an SSH, SMTP, POP3, or IMAP access attempt is critical and I'd ban that IP for 24 hours at least, but a HTTP access is semi-understandable, so maybe blackholing for just 15 minutes might suffice.)

The problem I'm trying to find a non-kernel-based solution is to avoid replying with a TCP ACK, and instead of treat the TCP SYN packet (to a standard port) as sufficient for the IP blackholing.  I can do that if I install a firewall rule to log such TCP SYN (and/or UDP packets), but I'd like something more robust; that if the userspace part keels over, it just stops adding new blocks and removing old ones, instead of rendering the entire system unstable/nonworking.

(Blackholing the source IP address based on a TCP SYN means the attacker gains zero information from the server: it never receives any kind of responses to its TCP connection attempts.  Blackholing or dropping UDP packets does the same for UDP connection attempts.)

In general, reverse proxying is a good option for when one can secure the physical network, and have a firewall (and reverse proxy) at the connection point to external untrusted networks.  Then, the internal traffic is unencrypted, but external access only when using an encrypted connection.  For HTTP protocol, a simple HTTPS reverse proxy works very well; for pure TCP/IP and UDP/IP based protocols, set up a VPN.
« Last Edit: November 15, 2021, 01:30:44 pm by Nominal Animal »
 
The following users thanked this post: DiTBho

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 6796
  • Country: va
Re: Running a web server on an embedded system - risky?
« Reply #27 on: November 15, 2021, 02:40:41 pm »
From my logs, what often happens is some IP attempts access and seconds later, often not quite a second, another IP from somewhere else in the world tries the same with a slightly different username. It's quite common for mail servers to blackhole addresses after n login attempts, so I presume this is a way of getting around that. It's just a distributed botnet.

There are legitimate reasons for strange systems connecting to me - I run a local mail server and DNS, for instance, so the failed logins let me block those IPs across all port on the basis that that IP is a rum one. Generally, I block the entire CIDR allocation just to be sure since some, no names mentioned OVH, seems to be legit servers at a data center rather than random end user systems. Mind, just blocking an entire country seems to do wonders sometimes :)
 

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #28 on: November 15, 2021, 03:33:12 pm »
But... how do you implement all that in an embedded system?

If you are running something big then the invisible google recaptcha is very good at blocking bots, but still they get in.

I think anybody running an embedded server (some product with a small CPU, not much RAM, etc) on an open port needs to have a firewall in front of it. Not publishing a DNS for it will help, a bit...

The earlier suggestion of limiting the permitted source IP to some IP and subnet, or an IP range, is feasible. And if the IP was obtained via DHCP then AIUI the box will also get the mask, which then gives it the entire "local LAN" IP range.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Offline PlainName

  • Super Contributor
  • ***
  • Posts: 6796
  • Country: va
Re: Running a web server on an embedded system - risky?
« Reply #29 on: November 15, 2021, 03:54:21 pm »
Quote
an embedded server (some product with a small CPU, not much RAM, etc) on an open port needs to have a firewall in front of it

Yes, which would normally be a router. Just having only a specific port contactable would do a lot, but after that you're essentially going to be reliant on user authentication - a firewall is going to let stuff through at some point which you'd've preferred it not to.

Quote
limiting the permitted source IP to some IP and subnet

That would be good if it's just local access. You don't need DHCP for that - the device will need to know the netmask however it's configured. If you need remote access, perhaps a VPN would be a better bet than letting off-LAN access direct to the device.
 

Offline DiTBho

  • Super Contributor
  • ***
  • Posts: 3796
  • Country: gb
Re: Running a web server on an embedded system - risky?
« Reply #30 on: November 16, 2021, 10:00:37 am »
the mysteries of outer space ... what rides packets on the internet these days.
The opposite of courage is not cowardice, it is conformity. Even a dead fish can go with the flow
 

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #31 on: November 16, 2021, 10:36:11 am »
What kind of "firewall" feature is common on embedded products these days?

Mine has LWIP on it. ETH uses the STM code, where the dedicated DMA controller moves data into RAM buffers. One "hopes" that the DMA length is set to not overflow the buffer ;)

The plan is to put in a feature whereby a list of IPs can be permitted. This could get quite complicated though if you also want to allow IPs with masks, etc. Maybe some code already exists but my worry is that all this stuff is so buggy. I have a part-time guy doing the ETH side and I reckon he spent a few months fixing bugs in the ST code. Then a few more months fixing MbedTLS.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 

Online Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6172
  • Country: fi
    • My home page and email address
Re: Running a web server on an embedded system - risky?
« Reply #32 on: November 16, 2021, 02:56:49 pm »
What kind of "firewall" feature is common on embedded products these days?
lwIP does not have one built-in, but as I explained above, it does have hooks for implementing one.

To filter even ARP and ICMP packets in the firewall, you do need to modify lwIP sources, though.

The lwIP stack is very, very small, and typically you would not try to implement a stateful packet filter; only a simple stateless filtering –– that is, for IP networking, based on source and destination IP addresses (and for TCP over IP, the port numbers), but without any concept of "established connection" per se, just treating each individual packet as a separate entity.

The plan is to put in a feature whereby a list of IPs can be permitted.
Implement LWIP_HOOK_IP4_INPUT() and/or LWIP_HOOK_IP6_INPUT() that return 0 if the packet has a valid/acceptable source address (including 0.0.0.0 if you use DHCP!), and nonzero otherwise.

For IPv4, for each whitelisted IP address or subnet, you can use ip4_addr_net_eq(srcaddr, addr1, mask1) to see if IPv4 address srcaddr is within the subnet defined by IPv4 address addr1 and netmask mask1.  (It expands to (srcaddr & mask1) == (addr1 & mask1).)  For specific hosts, the mask is all bits set, otherwise, the k high bits set  and 32-k low bits clear (and the compact address-netmask form is then ip.ad.dre.ss/k).  If the 32-k low bits are cleared in addr1, then test (srcaddr & mask1) == addr1 suffices.

If the addresses (and associated netmasks) are in a sorted array without overlaps, a binary search on the addresses in network byte order (most significant byte first), followed by IIRC two ip4_addr_net_eq() checks should yield the answer.

This does expose the existence of the host to all other hosts on the subnet, which should be okay.  (To avoid that for IPv4, you could modify src/core/ipv4/etharp.c:etharp_input() just after the /* send ARP response */ comment, so that *only* when if hdr->sipaddr is a valid source IP address, is the etharp_raw() call done immediately following the comment; and do a similar check in src/core/ipv4/icmp.c:icmp_input() to only respond ICMP packets from the explicitly allowed addresses.)

I'm pretty sure that if one would take the couple of hours to properly add LWIP_HOOK_IP4_FILTER(fromaddr,toaddr) and LWIP_HOOK_IP6_FILTER(fromaddr,toaddr) and LWIP_HOOK_TCP_FILTER(fromaddr, fromport, toaddr, toport) support to the proper places and the day or two it would take to test it actually works right (when different features are enabled and disabled; i.e. lots of compile tests, and basic real-life functionality test), it'd be accepted upstream.  These return 0 if the packet should be handled normally, and nonzero to blackhole (DROP) the packet.  (REJECT support would need a bit of additional code.)  This would make it easier to implement the (stateless) packet filter and even intrusion detection to otherwise unused ports (ftp/21, ssh/22, telnet/23, smtp/25, http/80, pop3/110, imap/143, ldap/389, https/443, etc).  This would seriously reduce the maintenance burden long term, because one could then use unmodified upstream LWIP stack, just implementing ones own hooks.
 
The following users thanked this post: DiTBho

Online peter-hTopic starter

  • Super Contributor
  • ***
  • Posts: 3671
  • Country: gb
  • Doing electronics since the 1960s...
Re: Running a web server on an embedded system - risky?
« Reply #33 on: November 16, 2021, 04:05:59 pm »
This is super useful info - many thanks!

Yes it needs to be just stateless. It is just to prevent crude hacking. Nobody will be attacking this product, but somebody could well try attacking the whole class of products containing a widely used code library.
Z80 Z180 Z280 Z8 S8 8031 8051 H8/300 H8/500 80x86 90S1200 32F417
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf