Dead TCP client detection - socat on a Linux SBC - doing my head in!

[Delta]

This should be very simple, and probably is for people who know what they are doing, which I clearly don't.

I am using a Raspberry Pi 3 running Ubuntu Server 16.04 to receive an ASCII stream on a RS-232 <-> USB converter, and send it to multiple clients over a LAN using raw TCP.  (each client connects to a separate TCP port on the server)

My script uses socat to read the data from /dev/ttyUSB0 and broadcast it via UDP on the loopback device.  (This was the only way I could find that would mean that 1) the data was continually read from /dev/ttyUSB0 even if no further process was wanting it, and 2) multiple processes could read the same stream)

Code: [Select]

socat -d -U udp-sendto:127.255.255.255:29000,broadcast file:/dev/ttyUSB0,nonblock
Then multiple instances of socat listen for TCP connections, then read from the UDP broadcast and send it on

Code: [Select]

socat -u -d udp-recv:29000,reuseaddr,rcvbuf=2048 tcp-l:$port,keepalive,keepcnt=5,keepidle=10,keepintvl=2
This all works fine, when clients close their connection, my scripts sees that the socat process has stopped and starts it again ready for the next connection.  The problem is with clients that do not close their connections properly.  This is going to be used in an environment where clients WILL have their network cables unplugged or be hard-powered-down, and the server needs to make the ports available again as quick as possible.

For testing, I am using a simple

Code: [Select]

nc -vv server.ip serverport on a Ubuntu laptop.

As it stands, if a client is unplugged, the server keeps the connection open for around 1 hour. (netstat shows the connection as ESTABLISHED)
If I add the

Code: [Select]

-T 20 to the TCP socats, is will bin the dead connection after 1 minute. (The port then stays in FIN_WAIT1 for a minute, but this is bearable)
BUT, the -T option also causes the connection to be closed when the client is still connected but there is no serial data coming in!  The keepalives are being sent (when the script and thus the socats) is run as root (I have checked with Wireshark on the laptop), but I assume that as they are only on the TCP link between server and client, socat sees the link between its input (UDP) and output (TCP) as dead, and closes.

All connections are unidirectional.  Serial into server, TCP stream from server to clients.

What I want is:
Keep the TCP connection to a client open indefinitely, regardless of whether any data is being sent.
Close the connection within 30 secs if there is no ACK response from the client (either from a data packet or keepalive).
Re-open ports for future connections within 1 minute.

Sorry if this is all over the place, but I hope I have given all the required information.

[Delta]

Well I kind of have it OKish...

Code: [Select]

net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 6in /etc/sysctl.conf, and am not using the -T option on socat.

This keeps the TCP connection open constantly regardless of serial data flow, and closes the connection to unplugged clients in 30secs.

The still annoying thing is if there is no serial data flowing, the TCP connection to an unplugged client is held open until serial data is sent, whereupon it closes immediately.

I also have the feeling that it is bad practice to mess around with system settings like this...

[bson]

Then multiple instances of socat listen for TCP connections, then read from the UDP broadcast and send it on

Code: [Select]

socat -u -d udp-recv:29000,reuseaddr,rcvbuf=2048 tcp-l:$port,keepalive,keepcnt=5,keepidle=10,keepintvl=2

You should be fine with a single listen process:

socat -u -d udp-recv:29000,reuseaddr,rcvbuf=2048 tcp-l:$port,fork,keepalive,keepcnt=5,keepidle=10,keepintvl=2

It should never terminate but accept client connections indefinitely, forking a new process for each one.  So your script only needs to restart it if for some reason it terminates (like it crashed).  A better option might be to run it as a system service, using systemctl (I'm not that familiar rpi's).  This way if it dies something is logged and it's automatically restarted.  Same with the serial port broadcaster.  If you do this they will automatically run whenever the system is booted, so will always be available.

[Delta]

Negative Ghostrider! I tried the fork option, but if multiple clients then connect, the ASCII stream is split between the two clients! Each client receives approximately half of the lines! I gave up with this and just plumped for one socat process and one port number per client.

[mubes]

Delta,

If you can bear going to C then in the _client routine here https://github.com/mubes/orbuculum/blob/master/Src/orbuculum.c you will find code that does pretty much what you are looking for.

I'm afraid it's not a zero effort solution cos you will need to remove the depacketiser stuff. Fortunately it's almost only a hacking out job with virtually new code to be added to acheived the result you're looking for.

In my experience unixy (bolted together command lines) are fantastic if they do what you need, and _incredibly_ frustrating if they _nearly_ do what you need (or, like me, you just don't have the unix-fu to coax them into compliance).

Regards

Dave

[andersm]

Have you considered using an existing message broker system (eg. RabbitMQ, Mosquitto) instead of implementing your own?

[douardda]

Or the wonderful zeromq library? http://zeromq.org



Envoyé de mon GT-N7105 en utilisant Tapatalk

[bd139]

Use zeromq. We send several billion messages a day through that

Beware though, the maintainer died (RIP) so I'm not sure what the future holds for it.

[shawty]

Use zeromq. We send several billion messages a day through that

Beware though, the maintainer died (RIP) so I'm not sure what the future holds for it.

I'll back that one up.

Failing that, one other option you might want to explore is using C# under mono, which runs really well on the RPi (I use it all the time).  The TCP Client class, in the .NET base library (Which is available on the PI under mono) has everything you need.

Iv'e not got any code to hand, but what you basically need to do is use the System.Threading Library and The System.Net library.  Sit in a tight loop and listen on a single TCP Client object, then as soon as it's connect event fires, spin off a thread, create a new TCP Client, give that new TCP Client the handle from the one you where listening on, and go back into your original listening loop.

I wrote an article on this a year or two ago, on the "Dotnet Nuts and Bolts" column for devguru.com , if you go search around on devguru, you'll find it hovering around in there somewhere.

My description above doesn't do it justice though, the code was no where near as complex as I'm likely making it sound.

If you want to got he C# route, ping me I'll see if I can dig the code out.

Shawty

[bd139]

Just watch out with C#; threads take a lot of memory and kick up the context switch rate. Look at async as well.

[shawty]

Just watch out with C#; threads take a lot of memory and kick up the context switch rate. Look at async as well.

Agreed.

But you could also argue the case against the app not being real-time too.  I find it plenty enough for the daemons I use it to write, when controlling stuff.

In fact my main concern would be buffering the serial connection at a sustainable rate.

I just wrote an output routine last weekend to push data down the serial port at 115200, and sustain a 2Mhz realtime(ish) rate.  (I was trying to feed real-time chip programming instructions to a chip wired up to my arduino) [My post on Driving IC clocks in the PIC section of this forum was off of the back of it]

[grumpydoc]

Your architecture for this seems a bit screwy.

TCP servers do not normally need to have a separate port for each client, they are supposed to sit on a "well known" port listening for connections. Connections which are not fully closed do sit around for a bit in case the final packets arrive but that is only a problem if you have a high rate of incoming connections and need to free up old ones quickly - it does not sound as though this is a problem in your scenario.

The correct way would seem to be to write a small server in the language of your choice, but if you want to stick with command line tools why not use UDP to the LAN broadcast address  and pick that up at the client.

The client<->server connection is then stateless and clients dropping in and out (be it for unexpected power loss or cables being pulled out) won't affect anything but the client.

Do clients need to send anything back to the "server"?