Running a web server on an embedded system - risky?

[PlainName]

It's an embedded Linux board based on TL-W703 exposed to the internet

Frankly, I wouldn't expose anything to the internet except a firewall at the router. One of the block lists my router subscribes to shows 78,143 connection attempts since midnight. Another, that is created from failed POP3 and SMTP logins here, shows 13,935 attempts.

[Nominal Animal]

DiTBho's and dunkemhigh's observations are the exact reason why I've been considering an insta-banner due to a connection attempt on specific ports on my intertubes-facing embedded devices.  It's simple, fast, extremely lightweight (runtime footprint just kilobytes!) and catches all automated attempts.  The idea is that a connection attempt to any of the specified standard ports causes the IP address to be blackholed for a specific duration, duration depending on the port.  (For example, an SSH, SMTP, POP3, or IMAP access attempt is critical and I'd ban that IP for 24 hours at least, but a HTTP access is semi-understandable, so maybe blackholing for just 15 minutes might suffice.)

The problem I'm trying to find a non-kernel-based solution is to avoid replying with a TCP ACK, and instead of treat the TCP SYN packet (to a standard port) as sufficient for the IP blackholing.  I can do that if I install a firewall rule to log such TCP SYN (and/or UDP packets), but I'd like something more robust; that if the userspace part keels over, it just stops adding new blocks and removing old ones, instead of rendering the entire system unstable/nonworking.

(Blackholing the source IP address based on a TCP SYN means the attacker gains zero information from the server: it never receives any kind of responses to its TCP connection attempts.  Blackholing or dropping UDP packets does the same for UDP connection attempts.)

In general, reverse proxying is a good option for when one can secure the physical network, and have a firewall (and reverse proxy) at the connection point to external untrusted networks.  Then, the internal traffic is unencrypted, but external access only when using an encrypted connection.  For HTTP protocol, a simple HTTPS reverse proxy works very well; for pure TCP/IP and UDP/IP based protocols, set up a VPN.

[PlainName]

From my logs, what often happens is some IP attempts access and seconds later, often not quite a second, another IP from somewhere else in the world tries the same with a slightly different username. It's quite common for mail servers to blackhole addresses after n login attempts, so I presume this is a way of getting around that. It's just a distributed botnet.

There are legitimate reasons for strange systems connecting to me - I run a local mail server and DNS, for instance, so the failed logins let me block those IPs across all port on the basis that that IP is a rum one. Generally, I block the entire CIDR allocation just to be sure since some, no names mentioned OVH, seems to be legit servers at a data center rather than random end user systems. Mind, just blocking an entire country seems to do wonders sometimes

[peter-h]

But... how do you implement all that in an embedded system?

If you are running something big then the invisible google recaptcha is very good at blocking bots, but still they get in.

I think anybody running an embedded server (some product with a small CPU, not much RAM, etc) on an open port needs to have a firewall in front of it. Not publishing a DNS for it will help, a bit...

The earlier suggestion of limiting the permitted source IP to some IP and subnet, or an IP range, is feasible. And if the IP was obtained via DHCP then AIUI the box will also get the mask, which then gives it the entire "local LAN" IP range.

[PlainName]

an embedded server (some product with a small CPU, not much RAM, etc) on an open port needs to have a firewall in front of it

Yes, which would normally be a router. Just having only a specific port contactable would do a lot, but after that you're essentially going to be reliant on user authentication - a firewall is going to let stuff through at some point which you'd've preferred it not to.

limiting the permitted source IP to some IP and subnet

That would be good if it's just local access. You don't need DHCP for that - the device will need to know the netmask however it's configured. If you need remote access, perhaps a VPN would be a better bet than letting off-LAN access direct to the device.

[DiTBho]

the mysteries of outer space ... what rides packets on the internet these days.

[peter-h]

What kind of "firewall" feature is common on embedded products these days?

Mine has LWIP on it. ETH uses the STM code, where the dedicated DMA controller moves data into RAM buffers. One "hopes" that the DMA length is set to not overflow the buffer

The plan is to put in a feature whereby a list of IPs can be permitted. This could get quite complicated though if you also want to allow IPs with masks, etc. Maybe some code already exists but my worry is that all this stuff is so buggy. I have a part-time guy doing the ETH side and I reckon he spent a few months fixing bugs in the ST code. Then a few more months fixing MbedTLS.

[Nominal Animal]

What kind of "firewall" feature is common on embedded products these days?

lwIP does not have one built-in, but as I explained above, it does have hooks for implementing one.

To filter even ARP and ICMP packets in the firewall, you do need to modify lwIP sources, though.

The lwIP stack is very, very small, and typically you would not try to implement a stateful packet filter; only a simple stateless filtering –– that is, for IP networking, based on source and destination IP addresses (and for TCP over IP, the port numbers), but without any concept of "established connection" per se, just treating each individual packet as a separate entity.

The plan is to put in a feature whereby a list of IPs can be permitted.

Implement LWIP_HOOK_IP4_INPUT() and/or LWIP_HOOK_IP6_INPUT() that return 0 if the packet has a valid/acceptable source address (including 0.0.0.0 if you use DHCP!), and nonzero otherwise.

For IPv4, for each whitelisted IP address or subnet, you can use ip4_addr_net_eq(srcaddr, addr1, mask1) to see if IPv4 address srcaddr is within the subnet defined by IPv4 address addr1 and netmask mask1.  (It expands to (srcaddr & mask1) == (addr1 & mask1).)  For specific hosts, the mask is all bits set, otherwise, the k high bits set  and 32-k low bits clear (and the compact address-netmask form is then ip.ad.dre.ss/k).  If the 32-k low bits are cleared in addr1, then test (srcaddr & mask1) == addr1 suffices.

If the addresses (and associated netmasks) are in a sorted array without overlaps, a binary search on the addresses in network byte order (most significant byte first), followed by IIRC two ip4_addr_net_eq() checks should yield the answer.

This does expose the existence of the host to all other hosts on the subnet, which should be okay.  (To avoid that for IPv4, you could modify src/core/ipv4/etharp.c:etharp_input() just after the /* send ARP response */ comment, so that *only* when if hdr->sipaddr is a valid source IP address, is the etharp_raw() call done immediately following the comment; and do a similar check in src/core/ipv4/icmp.c:icmp_input() to only respond ICMP packets from the explicitly allowed addresses.)

I'm pretty sure that if one would take the couple of hours to properly add LWIP_HOOK_IP4_FILTER(fromaddr,toaddr) and LWIP_HOOK_IP6_FILTER(fromaddr,toaddr) and LWIP_HOOK_TCP_FILTER(fromaddr, fromport, toaddr, toport) support to the proper places and the day or two it would take to test it actually works right (when different features are enabled and disabled; i.e. lots of compile tests, and basic real-life functionality test), it'd be accepted upstream.  These return 0 if the packet should be handled normally, and nonzero to blackhole (DROP) the packet.  (REJECT support would need a bit of additional code.)  This would make it easier to implement the (stateless) packet filter and even intrusion detection to otherwise unused ports (ftp/21, ssh/22, telnet/23, smtp/25, http/80, pop3/110, imap/143, ldap/389, https/443, etc).  This would seriously reduce the maintenance burden long term, because one could then use unmodified upstream LWIP stack, just implementing ones own hooks.

[peter-h]

This is super useful info - many thanks!

Yes it needs to be just stateless. It is just to prevent crude hacking. Nobody will be attacking this product, but somebody could well try attacking the whole class of products containing a widely used code library.