[solved] networking, a simple fake FIN package causes a lot of troubles

[legacy]

we are managing a CCC-like event, and during one of the recent preliminary meeting, we experimented an attack that shut down the whole local area network. The attacker is anonymous but we found his source code, so he operated inside our LAN. He used a long piece of code which creates fake FINs packages and open a TCP/IP ports. We are still studying the source, and it's not 100% clear.  When it tries to open a port, then it invokes these two lines

Code: [Select]

                shutdown(sock, SHUT_RDWR);
                close(sock);

Shutdown() is a socket's primitive offered by the standard UNIX networking engine, and it sends a FIN package. Combined it seems that the approach is causing problems on our local network because computers do no more responsible within 50 minutes.

edit:
I was focused on the wrong piece of code. The attack can be classified as "SYN flooding", so this topic is now cleared from useless links, and the title is edited with [solved]

edit2:
Things can also be summarized this way

TCP/IP 3-way handshake is done to establish a connection between a client and a server.

The process is :
Client --SYN Packet--> Server
Server --SYN/ACK Packet --> Client
Client --ACK Packet --> Server
The above 3 steps are followed to establish a connection between source and destination.

SYN Flood DOS attacks involve sending too many SYN packets with a bad or random source IP to the destination server. These SYN requests get queued up on the server's buffer and use up the resources and memory of the server. This can lead to a crash or hang of the server machine. After sending the SYN packet it is a half-open connection and it takes up resources on the server machine. So if an attacker sends syn packets faster than memory is being freed up on the server then it would be an overflow situation. Since the server's resources are used the response to legitimate users is slowed down resulting in Denial Of Service.

Most servers nowadays use firewalls which can handle such syn flood attacks and moreover even servers are now more immune.

For more information on TCP Syn DOS attack read up rfc-4987, titled "TCP SYN Flooding Attacks and Common Mitigations".

[madires]

as bench test, we are bombarding a subnet with a portscanner that
- tries to see if a port is open
- tries to shutdown the port by issuing a poison signal

You can't close ports remotely. It seems that you're DOSing yourself.

[xani]

It would be much easier to guess what exactly the problem is if you made a tcpdump of the traffic (wireshark is AMAZING tool to analyze dumps made by it). Trying to look at C calls and effects via port scanner is like using multimeter instead of oscilloscope. Even better if you posted the dump so we can see it (if it's nothing secret)

As for Linux side, look into:

• dmesg command or kernel log - linux kernel will generally yell there when you hit any limit
• ss --all will show all open and listening sockets in system, ss -t will do same for only TCP connections. If you see few thousand connections there you might just created a lot of unclosed connections. lsof -i if your machine doesnt have ss command
• TCP by default have LONG timeouts, and by that I mean HOURS so if daemon listening on port didn't change anything idle connection can sit unused for a long time
• normally you can't have more than 30k to ~64k connections to a given port between same pair of IPs, you just run out of source ports. A lot of connections that are not closed properly can cause that.
  

[bson]

Anyway, I still have to understand why servers stop to respond after a while they are bombarded with a FIN packet 

The way it's supposed to work is:

Client sends data to the server
Server responds with data to the client
Server, when done shuts down its write side
The client reads remaining data and when its read side is fully drained it gets an EOF (because the server sent a FIN on shutdown)
The client shuts down its write side and closes the socket
The server gets an EOF when the client has shut down its write side, and this confirms the client has seen everything sent
The server closes the socket

When a TCP connection is closed its goes into a TIME_WAIT state.  This is to prevent the same address,port pairs to be used again since they uniquely identify a connection.  If they're used too soon there is a risk of confusing TCP segments in the current connection with segments from a past connection that may have sat delayed in router queue, or got misrouted and took a while to find their way to their destination.  The TIME_WAIT state is configurable, and by default fairly long (forget off hand, several minutes).  When a connection is closed in an orderly fashion (using the shutdown semantics above), the TIME_WAIT period is dramatically shortened (to a few seconds).  If it's shutdown due to an error, the full period applies as one of the endpoints may still be trying to use it.  When you shutdown the read side, if there is any data in the socket buffer a TCP RST is issued - this signals an erroneous disconnect.  The reason is so the other part can detect that you didn't see some of the data sent.  The full TIME_WAIT applies.  When you have two hosts, or localhost, the addresses are static, plus the server port number is static, and the address, port pairs of the endpoints uniquely identify a connection, you soon have the full 65536-1024 possible anonymous ports stuck in TIME_WAIT on either the client or server or both.  No more ports are available and no more connections can be made.

The classic textbook is TCP/IP Illustrated by Richard Stevens.  Vol 1, Chapter 18.  http://www.pcvr.nl/tcpip/ (http://www.pcvr.nl/tcpip/tcp_conn.htm#18_0)
There are tons of tweaks, options and adjustments to TCP since then, but this describes very much how it still works...

[westfw]

This was my guess as well.

by default fairly long (forget off hand, several minutes).

255 (max IP TTL) seconds...

                shutdown(sock, SHUT_RDWR);
                close(sock);

So scan, open, shutdown, close?  Or just make a fake FIN?

[rstofer]

The FIN attack is not new and there is quite a bit of information about using iptables to block it.
I don't know if this video helps:



Basically, I don't see why a FIN would be performed if there wasn't an established connection.  The FIN should just be dropped in the firewall.

Google seems to have a lot of links...

Maybe some ideas here:

https://it.toolbox.com/question/how-to-block-port-scanning-tools-and-log-them-with-iptables-051115

[ejeffrey]

You aren't at all clear about what you are actually doing or what the problem is, but my guess is that you are flooding the server with ports in the CLOSE_WAIT state, and you have a defective server application.

TCP orderly shutdown requires both sides to shut down the socket.  If your client is calling shutdown(), that sends a FIN packet to the server.  The server application should be notified about the closed connection, and then call shutdown() itself.  Otherwise the socket will sit around forever.

Calling close() on the socket instead of shutdown probably sends an RST packet, which terminates the connection immediately. This can cause data loss if there is any in-flight data.

If your server application is correctly calling shutdown(), then it is possible that you are running into TIME_WAIT.  TCP forbids the reuse of the same (source IP, dest IP, source port, dest port) combination for an amount of time to make sure there are no stray packets (including retransmitted ones) still on the network.  Whichever side calls shutdown SECOND will enter the TIME_WAIT state for a couple of minutes.  It is possible to get a DoS by filling up with CLOSE_WAIT connections, but it more difficult.

[ejeffrey]

To be clear:  Calling shutdown() on a regular socket file descriptor will only send a FIN if the socket is part of an open connection.  This is not the same thing as a FIN flood where you send bare FIN packets that are not part of an established connection.  To do that you would need to use a "RAW" socket interface.  So it doesn't make sense to only look at the shutdown call, or say "shutdown creates a FIN", you have to look at the whole TCP state machine, including the connection establishment.

[ejeffrey]

Sending bare FIN packets with a raw socket (not promiscuous mode which is used for packet sniffing) is a port scanning and OS fingerprinting technique. It shouldn't really be causing the kind of slowdown you describe unless it is an absolute flood of packets.  In any case, yes it is something that firewalls should be blocking.

You should really be looking at what is actually happening on your target system to see what is actually going wrong.

[rstofer]

So, does WireShark show a flood of malformed FIN packets?
One of the big problems with network switches is that it is hard to get into the middle of a transaction.  I usually add a hub so I can see both sides of the conversation.  Alas, I don't think they make high speed hubs.  I suppose there are better ways to do this.

https://osqa-ask.wireshark.org/questions/57771/display-filtering-syn-and-fin-flags

[ejeffrey]

The modern way is to get a managed switch that can be configured with a MIRROR port.

[westfw]

You're trying to make a secure sandbox net, and you have systems running Telnet Servers?  And not even REAL Telnet Servers?
That's sort-of in the realm of "how can I make W95 secure while still allowing direct hardware access to my ISA-Bus Device Programmer.:
Sigh.  I think that most modern "corporate" routers and network switches can get pretty aggressive about filtering out "inappropriate" packets.  Even a home NAT-based router probably wouldn't allow rogue FINs on unestablished connections.

[magic]

Second hand hubs up to 100Mb/s can be found, I don't think faster ones exist.
I don't have a high-tech managed switch so when I need to intercept Ethernet traffic, I configure a Linux box with two NICs as a bridge, pass the connection through it and run my sniffer there.

As for this thread in general
 

You OP have an unhealthy tendency to read about things and let your imagination run wild about what they possibly could mean. The terminology you use is all wrong and the assumptions you make about things without really finding out what they are just aren't true. I recommend more reservation and skepticism if you don't want computers to drive you insane.

[legacy]

You OP have an unhealthy tendency to read about things and let your imagination run wild about what they possibly could mean. The terminology you use is all wrong and the assumptions you make about things without really finding out what they are just aren't true. I recommend more reservation and skepticism if you don't want computers to drive you insane.

do we need a post like this? 

[legacy]

Even a home NAT-based router probably wouldn't allow rogue FINs on unestablished connections.

Are you sure  ?

Tested two from Amazon: wrong assumption

[scatha]

Are you sure  ?

Tested two from Amazon: wrong assumption

This is surprising, what makes/models? All enterprise routers and most commodity routers will contain some form of stateful firewall behaviour which (if enabled) will absolutely block TCP traffic which is not part of an already established connection.

[Jeroen3]

You have to pass the firewall to block FIN attacks. If you stay within the switch chipset, on your LAN, your don't pass the firewall, and you can attack everyone on the subnet.

[rstofer]

You have to pass the firewall to block FIN attacks. If you stay within the switch chipset, on your LAN, your don't pass the firewall, and you can attack everyone on the subnet.

I thought client machines, like those running Linux, were normally using iptables in one guise or another.  I'm not sure how Windows Defender will respond to malformed FIN packets but if it was a large problem it would have been handled years ago.

Yes, there are  global firewalls where the LAN connects to the WAN but at this point, I would expect every client machine to have some form of filtering.  Except toys...  I suspect that packages like lwIP don't provide much in the way of defense.  But that's an entirely different deal than a mainstream OS.

It pretty hard to believe that at this point in time, malformed FIN packets wouldn't just fall in the bit bucket.

[xani]

This was my guess as well.

by default fairly long (forget off hand, several minutes).

255 (max IP TTL) seconds...

• IP does not have any timeouts
• IP TTL is number of hops. At each hop TTL is decreased by one. Yes, it is named badly
• the whole point of TCP is to handle lost IP packets in the first place so even if it was something IP related you'd just see TCP repeating dropped packet few times

[borjam]

so, we are having our local computer campus (similar to the CCC, on a scale of 1:100), Elisabeth and I have also implemented a port scanner and other security tools and ... someone has recently shown how a simple signal issued on the network can cause a lot of troubles to servers running tcp/ip

What do you mean exactly by "signal"?

specifically, the first line is really poison if overflooded on the network

Code: [Select]

                shutdown(sock, SHUT_RDWR);
                close(sock);

I still have to understand: WHY?

I don't understand the context.

as bench test, we are bombarding a subnet with a portscanner that
- tries to see if a port is open
- tries to shutdown the port by issuing a poison signal

What is that "poison signal"? Someone has pointed out that you are using non standard terms. And that's indeed true. It's impossible
to help you unless we can actually understand what you mean.

... so the whole networking of Lelly stops to work when bombarded with that signal, and within 40min the same happens to other machines, except the attacker, which continues its job of port scanning and issuing the shutdown signal, preventing other machines to resume.

You mean you are opening connections and closing them using shutdown()?

shutdown() is a system call, not a signal.

[ejeffrey]

Are you sure  ?

Tested two from Amazon: wrong assumption

This is surprising, what makes/models? All enterprise routers and most commodity routers will contain some form of stateful firewall behaviour which (if enabled) will absolutely block TCP traffic which is not part of an already established connection.

The "attack" is coming form the LAN side, not going through the firewall.  100% of  NAT routers block unassociated FIN packets simply because it is impossible to not block them -- the only work at all because of connection tracking, and without a connection it doesn't know where to send the FIN.

I still think what is happening is that the server is not calling shutdown() and leaving a lot of sockets in the half-shutdown CLOSE_WAIT state, which is then filling up the file descriptor table or some other finite resource.  But the OP has not shared any diagnostic information, reproducible example, anything other than unintelligible code fragments, or even described what specific software or hardware they are using.  It is a little surprising that busybox's telnet server would have this bug, but entirely possible.  Another possibility, since this is presumably some embedded device is that the kernel data resource limits are set very low either because of the limited system memory or misconfiguration.  Without further information this is just speculation.

[westfw]

    Even a home NAT-based router probably wouldn't allow rogue FINs on unestablished connections.

Are you sure  ?
Tested two from Amazon: wrong assumption

You would have to set them up with the "unsecured" network on the "WAN" side, and the "known ok" network on the "LAN" side.
I wouldn't expect it to do the filtering between LAN ports.

    255 (max IP TTL) seconds...

    IP does not have any timeouts
    IP TTL is number of hops. At each hop TTL is decreased by one. Yes, it is named badly

1) You're wrong.  TTL is a timeout, and IP does have timeouts (packet reassembly timeout.)  It's ALSO a hop count; each router decrements the TTL by at least one.  RFC790 is sort-of seminal.
2) TCP has a concept "Maximum Segment Lifetime" that's intimately related to the chosen IP TTL.  It's the MSL that comes into play for "zombie" connections.
3) I was wrong too.  MSL is apparently considered 2*TTL, and default TTL (in RFC793) was recommended to be one minute.  I'm not sure how that plays out in modern networks - transission times went down and hop counts went up pretty rapidly, and even back in the late 80s most hosts would use 255 as the default TTL, I think.
4) (heh.  RFC793 has some "amusing" bits in it these days...  "even if data rates escalate to 10's of megabits/sec.")


In any cases, this forum of EEs and microcontroller experts isn't the best place to discuss TCP network attacks of embedded unix.   There are a lot of well-known attacks, and most are well documented, with standard mitigation...