TCP Serial servers

[coppice]

There are parts of TCP/IP stack are look unfinished, or at least never reached proper standardisation.

Support for expedited data is there, but pretty weak.

Nagles has quite a few tuning parameters at the C level and there are even swap in algorithms in the Linux Kernel if you are in that space.  Sometimes it just turns into a whole load of testing and measuring and effort to gain 0.15%.

The effect can be really big with client server type applications.

Looking at what is happening with HTTP3, I guess SCTP now has no chance to mature and become the flexible transport of choice. Things like firewalls hindered the development of streaming quite a lot, but now things like STUN have matured, and the firewalls hinder anything else that's new, it looks like NATed UDP is the way everything will go. The old firewall joke was firewalls are not a problem, do all comms on port 80. Now it seems to be firewalls are not a problem, make everything look like streaming media.

[paulca]

Its the fact that Qik w/ HTTP3 can tunnel many connections within the same stream and within the end to end encryption.  That scares firewall folks.  So much so a lot of places disable Qik and the browsers call back to HTTP2 style single connections again.

[coppice]

Its the fact that Qik w/ HTTP3 can tunnel many connections within the same stream and within the end to end encryption.  That scares firewall folks.  So much so a lot of places disable Qik and the browsers call back to HTTP2 style single connections again.

Interesting. I thought Qik was succeeding in going where no SCTP connection has managed. Maybe we will still see wide usage of SCTP.

[paulca]

[westfw]

A feature that lets you have Nagle running, but kick out the data in the buffer immediately.

Nagle does "kick out" the first packet immediately; it's subsequent packets that have a delay.
There are "interesting" interactions with "delayed ack."  :-(
Nagel and Delay Ack were both algorithms aimed at the inherent inefficiency of small-packet traffic (ie telnet), back when that was a major application on the net.  160+ bytes sent on the network for each character typed and echoed was ... undesirable.
By now, no one much cares about interactive small-packet traffic :-(

And I figure that the "slow start" and other modern "congestion-avoidance" algorithms pretty much devolve to Nagle-like behavior when you're not sending full packets, which is fine for bulk data, but not so much for the small-packet traffic.

I know of at least one TCP implementation that implemented a limit on the number of TCP packets in the retransmission queue (5 packets, IIRC.)  This had an effect similar to Nagle, only ... later.  I'd be a bit surprised if microcontroller TCP implementations don't do something similar.  (OTOH, the retransmission queue is theoretically bytes rather than packets, and maybe modern implementation are better at treating it that way.  I have not kept track, and I'm afraid to do so :-( )

Examples [of "unfinished" TCP features] are the use of "PUSH" flags on TCP segments.  Nobody agrees on what they should do and most routers remove them.

PSH and URG are pretty ambiguous (a bit of change of philosophy in how the OS communicates with the network layers, I guess), but I don't think that any routers will change that level of of a packet...
(Serial Servers, OTOH, are likely to ignore and either never set or always set PSH.)  (URG was used by the "flush output" feature of TELNET.  Usually incorrectly and/or to little effect.

[peter-h]

When I was finishing off a serial over tcp feature (code written by someone else) I found I got it to work much better if I disabled Nagle.

Nagle may be useful running stuff over mobile data (GPRS, 3G 4G) where latencies can be measured in seconds.

The old firewall joke was firewalls are not a problem, do all comms on port 80. Now it seems to be firewalls are not a problem, make everything look like streaming media.

That's funny. The more modern version would be port 443 I travel a fair amount and find that lots of wifi networks block various ports e.g. smtp, pop, etc but 443 always works. A colleague helped out and set up the Softether VPN terminating back at home.

Lots of networks block anything to do with video (a good way to trash the data allowance of an apartment you are renting is to upload a few GB to YT or Vimeo) or VOIP.

Not sure how well (back on the topic) traffic on port 10001 etc will get through, but nothing stops you using port 80 or 443 I've just tried running data over port 80, which didn't work (probably because there are so many things listening on port 80, like browsers) but 443 works fine.

[paulca]

You won't get 443 or port 80 out of my work desktop.  No sir.  "ping google.co.uk":  Host not found.  Ping 8.8.8.8  Tmeout.

No external DNS even.  The ONLY thing you have access to is the corporate proxy, which runs way, way too much high security stuff for you to sneaky a PPP over HTTPS tunnel for example.  I haven't tried as I like my job.  However, the inability to install software yourself and the total lack of any admin rights anywhere kinda make it not worth trying.

[ejeffrey]

I've always felt like they missed an element in these TCP implementations. A feature that lets you have Nagle running, but kick out the data in the buffer immediately. I've done a number of things where Nagle is useful, but there are points where I need the buffer to flush to the far end, so it can promptly construct and send its response to me. I have to turn Nagle off, put the piecemeal parts of my request together in a buffer, and send it in one go. If I'm writing all the code that's usually no big deal, but it can be clumsy when working with some libraries.

Linux at least has TCP_CORK which completely prevents partial frames from being sent, and then let's you uncork when you need to flush.  This is used in e.g. web servers that need to send some headers and then a file.

Nagle is really intended for very simple protocols, such a character and line based interactive sessions.  It was also designed for an environment when congestion was a much bigger problem than it is today like half duplex Ethernet with collisions. It's sort of expected that if your application is both complex enough and performance sensitive enough that Nagle is a problem you will take over active management of the transmitting.  Building up an entire request in an io vector and then sending it with a single system call is basically what your are supposed to do.  The main motivation for TCP_CORK was to work alongside sendfile in http servers which avoids loading the payload into user memory entirely.

[Mark0x01]

A modbus tcp gateway will do the job, so you get modbus  over tcp to modbus over serial.
No issues with packeting up the data, since it is already done.

My Homeassistand uses one with a number of devices on the RS485 cable to monitor mains power,  temperaturea and drive the irrigation system with RS485 modbus relay boards.

Mine was from Aliexpress, or you can easily find code for esp32 or arduino to do the same thing - I have an esp32 serial modbus client I build from modules.

[peter-h]

modbus tcp gateway

If these work for general data, they must be despatching the packets purely on a timeout.

The one I have is configurable, from a config file setting. I spent a lot of time playing with that... If you want a latency of a few ms or less then you can't do that though. And 3.5 characters at 9600 baud is 3.5ms.

Also you can have an instrument (a modbus slave) with a large number of registers and when the master does a block read, the "packet" can be a few k, so you need a buffer big enough for that.

I think a general converter will work for modbus in most cases, because apparently the 3.5 character packet delimiter is unimplemented.

Most modbus applications are master/slave, where the master polls and the addressed slave responds. So the master's poll will always be conveniently terminated by, ahem, no more data So if your modbus over tcp implementation just uses say a 100ms timeout to detect end of packet, it will probably "just work", and I bet this is what the chinese do.

The other way is to get involved in the modbus protocol, perhaps speculatively trying out if the last 2 bytes you have is a valid CRC... AIUI the proper modbus over tcp protocol sets up a layer underneath modbus, so whole packets are identified, packaged (potentially over multiple MTUs) and unpacked at the far end. But then such a product isn't going to work for normal data.

How else can one do it?

[paulca]

Got one today.  Hooked it up, struggled a little but got it working 90% anyway.

The VCOM driver works.  The solar controller software connects to the VCOM.  Most of the config and monitoring appears to work, only 1 or 2 less important dialogs produce "Operation failed" reading the data.

I did make a mistake though.  I read the one I bought as supporting RS485.  Oops.  No, I bought the RS485 model.  It has the RS485 to UART converter inside.

That's great, but it snubs my use as a generic network UART.  They are only £30.  I can get another later for that use case.

I figure this at least has legs to run with.  I'll check out how it behaves on a bus with other stuff as I think I'll end up with 2 or 3 RS485 devices on the bus.  I do know there is limited flow control and colisions are bound to happen, but nothing is "transactional".