Linux UDP socket port reuse

[Siwastaja]

I have embedded devices which use Wiznet chips for TCP/UDP/IP stack. I use the following simple between-the-devices communication scheme, using UDP:

A leader periodically sends leader messages to broadcast address, say typically 192.168.1.255 let's say port 7777.
The follower(s) see the message, and store the leader's IP address (in this case, let's say 192.168.1.123).
After getting to know leader's IP address, followers report back whenever they see fit, by sending normal unicast UDP packets (e.g., to 192.168.1.123 port 7777 again).

Actual behavior on hardware / Wiznet is simple, understandable and works. Everybody listens to port 7777, Wiznet automagically seems to ignore "own" packets, followers get the broadcast packets, and leader gets the reply packets sent to its address.

But then we compile the firmware for simulation and testing on PC. It's a pretty low-effort thing with minimum amount of PC specific wrapper code; it's not a shipped product per se, it just needs to work. Quite obviously I don't want to use a rack of computers with actual LAN between them, but simply run multiple copies of the "pcfw", each their own OS process, and this "simplification" is where it all falls apart.

The pcfw has one notable difference, namely just use 127.0.0.1 instead of broadcasting:

leader: send UDP packets to 127.0.0.1:7777
follower: listen to UDP port 7777, store sender's (leader's) ip address (always works out as 127.0.0.1): send own stuff to 127.0.0.1:7777
leader: listen to UDP port 7777 to see follower's responses.

UDP payload itself contains unique "device" ids so that seeing one's own packets is no problem, they are trivial to ignore.

The program simply:
1) calls socket(AF_INET, SOCK_DGRAM, 0)
2) calls setsockopt setting SO_REUSEADDR
3) optionally, calls setsockopt setting SO_REUSEPORT in attempt to make this work. No difference whatsoever.
4) calls bind() with sin_port(htons(7777)) and sin_addr.s_addr = htonl(INADDR_ANY)
5) periodically calls sendto(), sending to 127.0.0.1
6) periodically calls recvfrom to see if incoming packets are in OS buffer.

Now start first copy of the program, and it will receive its own messages (what it sends). Fair enough, this is expected behavior, we are sending to localhost after all.

Then start the second copy of the program. It starts receiving the packets from the first copy, great. But at the same time, the first copy stops receiving anything, including its own messages. If I then stop and restart the first copy, it start receiving messages from the second copy (and itself), but the second copy stops receiving anything. In other words: every process can send, but only the one which started last can receive at all

So it seems SO_REUSEADDR and SO_REUSEPORT just prevent bind() from erroring out but the OS redirects the packets to only one process at the time. Internet tells me that SO_REUSEPORT in linux is by design for load balancing, i.e. dividing the messages between the processes. This is no good for this use case.

And I'm reluctant of changing the design itself (e.g. to use two different port numbers) because it is simple and Just Works on real hardware. You can of course convince me of otherwise...

I'm sure there must be a - hopefully simple! - way for multiple processes to receive all of the messages because obviously I can run:
sudo tcpdump -n udp port 7777 -i any
and receive the messages while my own software still keeps receiving them. I want similar functionality, just monitor the UDP packets without destroying them; i.e., own copy for every socket.

I usually don't like asking questions but I have been googling this for more than 10 hours total now and down quite many rabbit holes so what the heck.

[ejeffrey]

Yes generally a unicast packet goes only to one socket.  One of the reasons for this is to allow to have one socket bind to the wildcard address and another bind to an interface specific address.  Incoming packets will go to the more specific address.  Or you can have one socket "connected" to a specific remote address and it will receive packets from that peer in preference to unbound sockets.

I think this should work if you use the localhost broadcast address 127.255.255.255?  I've never tried that but this SO post suggests it works:
https://stackoverflow.com/questions/11135801/how-to-broadcast-message-using-udp-sockets-locally

And it refers to an example here: http://www.ccplusplus.com/2011/09/udp-broadcast-client-server-example.html?m=1

You might also have to have each follower process bind to a different address in the localhost network like 127.0.43.25.  I have had to do that for some situations but I don't remember exactly what the circumstances were.

If none of that works, the big hammer would  be to use docker with separate network namespaces.  It's definitely one of those "now you have two problems" solutions but it should work.  You would set up a virtual bridge device, and each container would get its own virtual interface bound to that bridge. I've definitely had to go that route when trying to simulate networks.

[Nominal Animal]

The trick is to enable SO_BROADCAST, SO_REUSEADDR, and SO_REUSEPORT socket options (all SOL_SOCKET), and then bind to the broadcast address, 127.255.255.255:7777.  I just verified on Linux Mint 20.3, kernel 5.4.0-192-generic on x86-64 that this works: each receives a copy of the broadcast UDP packet.

For sending messages, you can have each fwpc open a second socket, enable SO_BROADCAST, and use a loop to bind to the first 127.x.y.z:7777 UDP address and port that succeeds, in increasing order starting from 127.0.0.1 and trying up to 127.255.255.254.  It may sound "slow", but isn't.

The first socket is used to listen to broadcast UDP packets to port 7777.
The second socket is used to send UDP packets to a specific recipient or to broadcast address.

If you want to hide the dual descriptors, create an API or interpose your send/receive functions with ones that use fake descriptors, and poll() et al. to implement the actual functionality.


If you cannot stand the complexity of having two separate socket descriptors (and using poll() or similar to wait for incoming messages on both), you can use a separate reflector process that is the only one that listens to broadcast packets using the first socket; all your fwpc processes only use the second socket.

The reflector process simply sends a copy of each broadcast UDP packet it receives to each 127.x.y.z:7777 you might be using.  A few hundred addresses causes only an insignificant load, but for more, you'll want to monitor which addresses are actually used.

If you only need the /8 subnet, 127.0.0.x for x=1..254, then you could just use a reflector bound to 127.0.0.255.  While processes think they're sending broadcast messages, they're actually just sending the messages to the reflector, which then sends copies to each possible recipient (127.0.0.1:7777 to 127.0.0.254:7777, basically sending 254 copies of the each packet it receives).  This way you don't need to set any of the socket options, either.

[Nominal Animal]

(I personally tend to use UDP packet reflectors in my own tests: it avoids the network issues completely, and I can easily log packet reflection statistics and data on the reflector, something that is harder to do using a real Ethernet network.  I do suggest you consider the last paragraph above.)

[Siwastaja]

Thanks to both; having two linux sockets per "actual wiznet socket" is not a problem at all, there already is translation layer from Wiznet's fixed socket numbers (chosen by programmer) to dynamic socket numbers returned by socket() (and stored by programmer). Easy to add another mapping array. I'll try these suggestions.

[xvr]

I'm sure there must be a - hopefully simple! - way for multiple processes to receive all of the messages because obviously I can run:
sudo tcpdump -n udp port 7777 -i any
and receive the messages while my own software still keeps receiving them.

tcpdump (and WireShark) do not use classic sockets - they install themself as packet filters in TCP/IP stack. This interface intended to perform system wide processing of all network traffic (for example - in firewalls)

[Siwastaja]

Yeah, got it working like this:

init wrapper:
sender_sn = SO_REUSEADDR, SO_REUSEPORT, if(is_leader) then also SO_BROADCAST, bind to own_ip_addr (running from 127.0.0.1 onwards)
listener_sn = SO_REUSEADDR, SO_REUSEPORT, SO_BROADCAST, bind to 127.255.255.255

sendto wrapper:
sendto(sender_sn, ...)

recvfrom wrapper:
recvfrom(listener_sn) // gets the broadcasts
if(rc < 0 && errno == EAGAIN)
     recvfrom(sender_sn) // gets replies directed to the broadcaster's own address.


Seems to work fine, thanks all.

[Siwastaja]

... and, indeed, removing REUSEADDR and REUSEPORT from sender socket, then running loop trying sequential IP addresses until bind() succeeds, solves having to manually assign the IPs. Excellent.