Sending serial data over Ethernet - which protocols are popular?

[peter-h]

Let's say you have a box with an ETH port and some RS232/422/485 ports. You want to use TCP/IP to talk to the serial ports.

One obvious way is to make it transparent and just arrange for say port 1000 -> serial port 1, port 1001 -> serial port 2, etc. No way to remotely config the port baud rates then. But it would be routable, because it would have an IP.

Then there is RFC2217 which defines one way to package it
https://www.rfc-editor.org/rfc/rfc2217.html
There are some control packets so one can set up the baud rates etc. It doesn't appear to support RS422/485 modes (driver control).

RFC2217 dates back to about 1997. However, before that, in the 1980s, were boxes called "terminal servers", for connecting a load of VT100 etc terminals to a computer running a multiuser application e.g. accounting. What did they use? I doubt it was proprietary because terminal servers was a big business.

Everybody is making ETH-serial boxes today but nobody is saying whether they use RFC2217 or something proprietary.

A separate requirement might be to create multiple corresponding VCPs on a computer on the same LAN as the ETH port. One would be looking for open source VCP drivers but there aren't many. There is the com0com project but it has an issue with driver signing
https://gist.github.com/DraTeots/e0c669608466470baa6c
This app creates a single VCP using RFC2217
https://www.hw-group.com/software/hw-vsp3-virtual-serial-port

How does one handle driver signing? Doesn't Microsoft charge a lot of money for that?

[westfw]

  before that, in the 1980s, were boxes called "terminal servers", for connecting a load of VT100 etc terminals to a computer running a multiuser application e.g. accounting. What did they use?

“Telnet.” [size=78%]https://en.wikipedia.org/wiki/Telnet[/size]
The RFC you linked was an option added to telnet to allow remote control of port parameters, but it wasn’t terribly popular (being defined about the same time that people became more interested in slip and ppp.)


Telnet had a lot of options.  Mostly separate RFCs.  The most popular was probably “binary mode.”


Rlogin was also popular.  And a bit later, ssh (for security, you know.)
Raw TCP streams were also widely used.

[MarkF]

There are hardware solutions for serial over TCP.

Here are two examples (I'm not recommending these brands in particular. Do your own research.)
   4 port over TCP
   8 port over TCP

I used larger 16 port rack mount units at work but I can't remember the brand name any more.

[peter-h]

Yes there are many boxes out there. I am just looking for something for this board I am working on, not trying to buy somebody's box.

The 1st of those states "Can Distinguish Which Serial Port Connects To The Device Via Port Number" which is mentioned in my post #1 but I wasn't aware of anyone using it. It is probably the most obvious way, with an extra port # for port config (see below).

One could write a VCP driver for it too, once one has a source for a Windows VCP. However applications which use a VCP are not such a mystery; a VCP is just a VCP and all you are then delivering is a COMx to COMx transport solution which can be totally nonstandard in between the two serial ports.

The 2nd one doesn't say but it says you can config port parameters, which means they either use some protocol (like RFC2217) or use another port number for the config, and maybe run an HTTP server on that, which is pretty easy.

Anybody doing a Windows VCP driver must be supporting port settings, because that is a standard form for the UI on a VCP.

I actually suspect the 2nd of the above boxes is VCP-only i.e. no published ETH protocol.

If those old boxes used telnet, how was the data routed to different ports?

Encryption should be easy because you do it all internally. One end in the VCP driver and one end in the box with the serial ports. It isn't TLS, AFAICT. I am already using AES256 and have the code for it running.

[Nominal Animal]

This is not "popular", but a description of what worked well for me –– but I use Linux, not Windows.

I have a very small Odroid HC1 single-board computer, with an Samsung Exynos 5422 octa-core big.LITTLE ARM processor (four Cortex-A15 and four Cortex-A7 cores), with a built-in SATA connector for storage.  It is a variant of Odroid XU4.  Aside from USB, it has a single UART with 1.8V logic levels, and I wanted to use the UART only for my own display, voltage and current monitoring, and some UI buttons.

This meant that I needed to split the single hardware serial port into multiple virtual serial ports, sharing the bandwidth.  (I also used to do a lot of web development, so I wanted to reuse the same multiplex-demultiplex on TLS-over-TCP/IP, since it is the TLS handshake that is the costly operation, and multiplexing multiple logical streams on top of one physical stream is the common problem here.  It is slightly more complex, because it allows an arbitrary number of substreams with almost arbitrary stream identifiers, so I'll just describe the simpler one with a limited number of substreams.)

I've also been bitten by bufferbloat, so anything like netstrings or any of the existing multiplexing protocols like MIME multipart/mixed has too much latency and buffering requirements, especially since the UART is limited to 2Mbaud or so.

What I ended up with was a simple multiplexing stream protocol, and a stateless robust escaping mechanism.

Each substream has their own unique reserved start byte.  There is one additional reserved byte, that acts like an escape marker.
As an example, I'll use three substreams (thus 4 reserved bytes) below.  One of these substreams is the initial one when connection is made.

(For a number of reasons, I explicitly chose that the escape marker followed by any substream start byte is ignored, and can thus be used as padding.  If the escape marker is followed by another escape marker, the preceding one is ignored.  Thus, a sequence of escape markers followed by a substream start byte encodes nothing; and a sequence of escape markers followed by a valid escape value encodes the same thing as one escape marker and that same escape value.)

There are three cases, in increasing order of complexity, to consider in the unescaped data stream.  Let D denote any non-reserved byte value, and R one of the four reserved byte values:

• R: A single reserved byte value.  4 escape values needed.  50% overhead.
• R R: Two consecutive reserved byte values.  16 escape values needed.  0% overhead.
• R D R: Two reserved byte values, with a non-reserved byte value in between.  16 escape values needed, but the following byte is part of the escape sequence.  0% overhead.

(You can add a fourth case, R R R, to encode three reserved values in two bytes, by using 4×4×4=64 escape values, achieving compression. Because R R is encoded without overhead, D R R and R R D are encoded with that, with no overhead.)

A trivial greedy encoder/multiplexer that sees only three bytes forward, the maximum overhead is 33% for any 3-byte sequence (for D D R, D R D, and R D D, which all encode to four bytes).  For two- and one-byte sequences, the maximum overhead is 50%.  Those do not include the start byte.
Given uniform random data and the three cases above, 88% of 8-byte source sequences are encoded without overhead, with typical maximum overhead being 37.5%; and 37% of 64-byte source sequences are encoded without overhead, with typical maximum overhead being 12.5%.

As the average overhead is much lower than the maximum, I never bothered to try and reduce the absolute worst-case maximum.

The decoding and demultiplexing is very straightforward, as long as there is room for at least three output bytes, and the decoder/demultiplexer can see two future incoming bytes at the same time.  You'll want the four values to be relatively rare, but trivially detected: say, 220-223, (byte&0xFC)==0xDC, which happen to be suitably rare in both binary data and Unicode text; or for pure text, say 16-19, (byte&0xFC)==0x10.  Also note that it is then preferable that the escape values are also easily detected (contiguous in each of the three cases), but do not overlap with the four reserved values.

[Nominal Animal]

However, in this particular use case, I would consider using port pairs.

Edited: Nope.  I would use TCP urgent data for the command stream.

TCP/IP packets have an <URG> bit.  When this bit is set, there is an additional field, <Urgent pointer>, which is the offset from the sequence number to the last urgent byte in this TCP packet.  Normal, non-urgent data may follow the urgent data.

However, "urgent" is a misnomer, because it does not affect the transmission or TCP packet sequence at all.  It is just a way to mark out-of-band data.  Only the sender and recipient care about the urgent data, routers and switches completely ignore it, and treat such packets as normal TCP/IP packets.  Because they use the exact same sequence number scheme that "normal" TCP/IP packets do, it really isn't out-of-band either.

It is perfect for the command stream, because it stays in synch with the data sent: you can control exactly when the commands are executed with respect to the data sent.

The "downside" is that each command stream starts at the beginning of a new TCP/IP packet, so if they are very frequent, you'll have many packets flowing, reducing bandwidth use efficiency.  It is not a big downside, though, because even at 1Mbaud connection with a command between every byte sent, you still average only about 200,000 packets per second, even if there is long enough delay so that the command and the data bytes are sent in different packets.  And that is exceedingly rare.

This leaves the command protocol.  Responses to the command protocol would be similarly urgent TCP/IP data.

For ease of use and extensibility, I'd use a single letter for each property that can be controlled.  (For example, when setting baud rate for both rx and tx, I might use B; for input/rx only, I; and for output/tx only, O.)  The commands and queries would use

• '?' '*' – query all current settings
• '?' letter – query a property[/tt]
• '/' letter digit+ – set a property

and responses

• '/' letter digit+ – for properties set successfully
• '!' letter – when the set value was invalid or unsupported

with either whitespace (any sequence of CR (10) LF (13) HT (9) SPACE (32)) or colon (':', 58) between records.
This would allow later backwards-compatible extension by ignoring queries that start with anything other than '?' or '/' up to the first whitespace or colon, and responses that start with anything other than '/' or '!' up to the first whitespace or colon.

You'd have 27+27=54 letters for the various properties, so creating a spreadsheet or other table of all of them would be a simple way to maintain the letter-to-property mapping.

[nctnico]

I think the question of the OP is more like: how do I make my serial to ethernet product compatible with existing solutions? Or put differently: are existing products interchangeable between manufacturers and are they using an industry standard protocol to do so?

[Nominal Animal]

I think the question of the OP is more like: how do I make my serial to ethernet product compatible with existing solutions? Or put differently: are existing products interchangeable between manufacturers and are they using an industry standard protocol to do so?

Good point.

Everybody is making ETH-serial boxes today but nobody is saying whether they use RFC2217 or something proprietary.

If you have access to one, it would be rather simple to use e.g. Wireshark (free (GPL 2.0+), available for basically all OSes, and also works for USB traffic) to check.  I bet they use some kind of command protocol as TCP "urgent" data too.

[peter-h]

Exactly

Also I think I realise that if you do a VCP at the PC end, then you can cook your own protocol entirely. You could even drive the PHY directly but then you get the 1990-type of non-routable solution that people did back then (no IP, dipswitches for the MAC #) and the only "ethernet" aspect is that it uses RJ45 connectors.

AFAICT, existing products use

1 - RFC2217
2 - transparent, done as a different port number for each physical serial port, and another port number for serial port config
3 - some proprietary means
4 - telnet (1980s "terminal servers")

You can do a VCP with any of these, and it is mandatory for 3.

I have no idea of the relative prevalence of 1 and 2 and that is a good Q. One might have to implement both.

I suspect 4 is dead; its applications were all high cost and pre-dating Windows, VCPs, etc so the computer vendor was happy to write any software for driving it. There are some obscure telnet applications in server admin.

If one does VCP only then the product has poor value (zero value, really) because the chinese have already ripped off all the low hanging fruit there, at ~$30/port.

[madires]

Search for 'reverse telnet'! There are already a few open source solutions (mostly for linux).

[rstofer]

Berkeley Sockets is easy to use and the Windows WinSock library has an implementation although I haven't used it.  Linux clearly supports it as does the mbed network library.

The nice thing about the protocol is that it makes no (few?) assumptions about host or client capabilities.  I was targeting an HP LaserJet printer but I simply compiled the code with the IP of my Linux box and wrote a little program to grab the data and dump it to a terminal screen.  Debugging was a cinch.  It really was a 'write-only' application.

There would always be a 'file' full of printer commands so Character Mode wasn't necessary although one could certainly send a small packet containing just one char.  I blocked my commands to strings of 'about' 512 bytes.  'About' is an interesting concept when filling a line buffer.

I was just using my LaserJet as a plotter

There's a reason Berkeley Sockets has been around since '83.

[peter-h]

The context here is industrial instruments connected to the serial ports.

Often it might be modbus stuff, where ideally the whole modbus packet is sent in one piece (generally it should fit in one MTU anyway, but if it doesn't, the other end is supposed to wait until it has the whole packet). Modbus stipulates no breaks bigger than 3.5 character periods, although not many receivers enforce that. There are two issues with that:

- more work identifying likely packets (without knowing anything about the data)
- identifying end of packet (> 3.5 chars gap + valid CRC, or just a valid CRC in case the tx device is not doing the gaps)
- might need a large buffer (not sure of the biggest possible packet but they can be a few k)

But I think there is also a protocol for modbus over TCP, which makes the above easier.

[nctnico]

But I think there is also a protocol for modbus over TCP, which makes the above easier.

Modbus TCP/IP is quite easy to implement. An option would be to bridge Modbus but I have seen some weird behaviour from Schneider PLCs when using Modbus TCP; these tend to open & close a connection for each run cycle that queries the state of the Modbus device. Likely to avoid tcp/ip retransmit timeouts but it gets messy with creating / closing sockets all the time.

[peter-h]

In practice it probably doesn't matter because everything is so fast now. I've seen instruments, 20 years ago, which took 1 sec to respond to a Modbus query, because the code was written by 10 people in C++ and the CPU ran at 16MHz So they had to compose it all in RAM and send it back in one go.

[IanB]

However, before that, in the 1980s, were boxes called "terminal servers", for connecting a load of VT100 etc terminals to a computer running a multiuser application e.g. accounting. What did they use? I doubt it was proprietary because terminal servers was a big business.

I used terminal servers when they were a thing. The ones I used sat as a box on the network. When you plugged a terminal (or a modem) into a terminal server you could talk to it at a command prompt and execute basic commands on it, such as list the connected users, or configure the serial port parameters, or connect to a host system. When you connected to a host system your terminal appeared on the host computer as a virtual serial port (much like USB to serial adapters today create virtual serial ports). The virtual serial port had a device name and looked like any other device on the system.

It was almost standard for serial ports of the time to do automatic protocol detection. You just plugged in a terminal and hit <Enter> a few times, and it would figure out the baud rate and parity by itself.

The terminal server connected to the network over Ethernet and used what I presume to be proprietary software to communicate with the other systems on the network. The serial data from the terminals was tunnelled over the network using whatever protocol the terminal server software used, buried inside network packets.

[peter-h]

Auto baud is a can of worms. Modems always had it - it worked by every line starting with AT and the modem measured the length of the 1s which the 'A' starts with. I guess terminal servers measured the CR. If you don't do something like this, it can take a lot of data. With the right byte values, 1200 baud can look exactly like 9600 baud (or some such).

Yes I reckon the protocol was proprietary.

[westfw]

Quote

“Telnet.”

https://en.wikipedia.org/wiki/TelnetMore details:cisco Terminal Servers (which I'm intimately familiar with) in the mid-1980s supported telnet and rlogin for outgoing connections (user on a terminal connected to the terminal server would issues a command, and it would connect to the host), and an assortment of protocols for "reverse telnet" for incoming connections.  The latter were decoded by port number:
20xx - "reverse telnet" to port xx.
30xx - "reverse telnet" to "rotary" xx (where a rotary could contain any number of async ports)
40xx - "reverse TCP" raw TCP stream to port xx
50xx - "reverse TCP" raw TCP stream to rotary xx
60xx - "reverse telnet" to port xx, and negotiate binary mode
70xx - "reverse telnet" to rotary xx, and negotiate binary mode

Telnet was very much designed to support the sort of remote terminal access that was in use in the 1970s.  Half duplex, local line mode, local editing... 
It's also a pretty sucky protocol, efficiency-wise.  The CPU has to scan every byte of data checking for negotiations and special characters.   At one point I proposed the "telnet don't telnet any more" option, since usually all of the actual negotiating happened at the very start of the connection - I was only partially joking.

On a modern system, I don't really see any reason to do anything more than process raw data from packets.

Later, cisco added additional protocols and options: SLIP, PPP, LAT, SPX, tn3270, Xremote, ARAP, SSH, Async-to-UDP (uart data encapsulated in raw UDP frames.  Included IP multicast support!), enhanced line mode, the comm-port option that you referred to (although most of the time the uart connection configuration was fixed by the attached hardware, so that wasn't too-much used, except maybe for dial-out modem sharing), X.25 PAD, ...  More that I'm not remembering.

The early cisco terminal servers would "support" about 32 concurrent users running at 9600bps on an 10MHz 68000.  Although we did have our own TCP stack and an "efficient" UART board (poll 16 UART's status in one read.)  I doubt that any modern CPU needs to pay much attention to efficiency.

Documentation is still online:  https://www.cisco.com/c/en/us/support/docs/dial-access/asynchronous-connections/5466-comm-server.html
AFAIK, you can probably still buy async ports for some of the cisco routers (at what is probably an exorbitant price.)  (used boxes might be reasonably priced, though!)

[PlainName]

I'm not clear on how you expect to use this box (if you know - this would be something the end user would decide?) but the simplest solution would be telnet (or ssh for security) to connect and then transparently pass through data except for a break-in sequence - you're basically writing the front end of a modem. Make it bog-standard Hayes compatible and world+dog can connect and configure it via AT commands.

If you, or the user, prefers not to allow '+++' or whatever, just put the configuration access on a different port and still use normal AT commands.

This won't work if your user, or you, is going to write an application that expects a pukka com port, for which you would need to create a VCP driver. But if you're writing your own application then you just use telnet or SSH directly and skip the fake VCP.

I think this would be a sensible way to go (that is, connecting to telnet/ssh rather than VCP) since serial ports seem to be deprecated by the Internet nowadays. A VCP might be akin to setting up a nice CD driver now everyone is using USB sticks...

[peter-h]

I am looking to do something that would be compatible with current practice.

If one does a VCP at the internet-connected PC, then the "practice" is obvious and the entire challenge is getting a VCP driver written. That's a specialised job. Some already exist e.g. this one
https://www.hw-group.com/software/hw-vsp3-virtual-serial-port
seems to provide a VCP if you provide an RFC2217 API at your TCP/IP end.

If one does not do a VCP then one needs to implement an API which people who drive the BSD socket API directly, can use. RFC2217 is one. Mapping ports to physical COMx ports is another way, with total data transparency (and probably using another port for setting up the baud rates etc - I have not seen any examples of this).

[SiliconWizard]

You have probably already looked at what the competition was offering. For instance: https://www.nordfield.com/ethernet-serial-converter
That should give you an idea of what you'll need to offer at the very least.

[peter-h]

That one uses a VCP, so it just does the obvious thing. No support for ETH-serial.

Their text is interesting though:

The two COM ports and the entire Ethernet to serial converter system is designed around a 32-Bit ARM 7 processor, which makes this converter a very reliable and high-performance product. Another advantage of the ARM 7 processor is that we can offer the free open source virtual COM redirector software called Com0Com, which will allow you to view and even make modifications to the programming code written in C++. Alternatively a very solid virtual COM port redirector from FabulaTech is available.

This com0com thing pops up all over the place. In fact I am using it to simulate two serial ports which loop back. I guess they used the source to create their VCP.

BTW my product is not principally this. I would never want to play in a sector where the chinese have already picked off all the low hanging fruit. My box does mostly other more important stuff but I felt it would make it a bit more useful if I included this "application", since it has four serial ports, ETH, USB (and a VCP on the USB, as well as an MSC block device).

[profdc9]

ESP8266 solutions seem to be quite popular for this.

I have made a PCB of a board for a particular firmware for this:

https://github.com/profdc9/WifiModem/

esp-link is quite handy as a telnet server to connect to a remote serial device.

zimodem is useful for a device with a serial port to be able to access internet services using "AT" commands.  ESP8266 also has its own firmware for this as well.

[peter-h]

Yeah, that is for WIFI. But a lot of people wonder why they didn't use Espressif, with their wrapped-up LWIP, MbedTLS etc package, all supposedly debugged by some mysterious "forum know-everything personality". Instead of spending months debugging the crappy ST HAL libs and integration with the mostly totally unsupported open source TCP stuff.

I am happy where I am now, with a working system, but would I do this again? No way.