TCP Serial servers

[paulca]

Has anyone experience using these devices?  I have one in mind for a task.  An RS485 device on my network that's near a switch.

If am reading it write you basically pair TCP connection and series protocols.

On the other end can I assume you can add a second device to return the payload back to serial?  Do any Windows based drivers for TCP based VCOM ports exist?

Any pros/cons/gotcha's.  They don't seem very common in hobby land, most articles and videos are commercials.

[coppice]

These things are made in huge numbers. I don't know what they get used for in such numbers. They usually follow standards (e.g. RFC2217), and different models generally interwork OK.

[peter-h]

I am developing a product which has this among many other features. Mine does it transparently; it maps serial port x to tcp port # like 10000.

RFC2217 was looked at but was too much work, and very few people actually use it.

Here ya go
https://www.eevblog.com/forum/microcontrollers/sending-serial-data-over-ethernet-which-protocols-are-popular/

[cantata.tech]

Has anyone experience using these devices?  I have one in mind for a task.  An RS485 device on my network that's near a switch.

If am reading it write you basically pair TCP connection and series protocols.

On the other end can I assume you can add a second device to return the payload back to serial?  Do any Windows based drivers for TCP based VCOM ports exist?

Sure. I use them. They're usually known as socket to serial bridges.

The main thing to keep in mind is how much performance they will need to provide. How many clients etc. If you need big performance then the effort required to develop them increases appropriately with the performance expectation.


There's a few basic ones on github as gists. You can implement them in nearly any programming language that you are comfortable with.

I like python the best but Java, C# and any other language will do just fine.

[peter-h]

It's usually easy to get raw speed from serial over tcp, but latency can be an issue because most implementations will accumulate serial data before sending out a tcp packet.

[Gribo]

There are also industrial terminal server (Lantronix, Wiznet, Moxa etc), and they have a VCP driver. They are quite expensive though.

[cantata.tech]

It's usually easy to get raw speed from serial over tcp, but latency can be an issue because most implementations will accumulate serial data before sending out a tcp packet.

Knowing the protocol helps a lot.

If you do, then you can inject the appropriate terminal/tcp "breaks" that will mean that packets are sent on their original intended boundaries.

This can be done both on serial and tcp.

[westfw]

I used to be pretty intimately familiar with TCP Serial Servers.  But I've been retired for a decade, and frankly the target market is a lot different now.  The old serial servers were what would be called "very expensive" compared to modern microcontroller serial devices.

I don't know if anything beyond "remote terminal access" was ever standardized.  As someone said, there is RFC2217, but it's not widely implemented and was only vaguely aimed at "hardware serial tunneling."    And then serial connections went out of style  ("broadband internet") and all the main vendors and protocol designers lost interest.


That means if you want to tunnel some particular rs485 protocol, you'd probably be at the mercy of some proprietary implementation of ... questionable ... capabilities.  I do see a bunch of "TCP to rs485 converters" at various price points from vendors from "respectable" to "eBay", but I don't know how they work, or whether they're interoperable.  Or whether there are "host side" matching implementations.


(And "rs485" is pretty highly ambiguous.  You probably need something specifically aimed at a OSI level or two above that.)
 

[peter-h]

You generally have to buy these in pairs.

The product I have does the simplest possible: physical com port 3 = tcp port 10003 (or whatever).

The only advantage of using others is that you can get VCP emulators, commercial or free. That's easier than opening a socket and doing r/w to port 10003.

There is still a use for running serial over tcp. Most cheaper industrial hardware has RS485 because it is dirt cheap to implement, whereas ETH costs a fair bit more, and is a whole load more code.

This stuff was big in the 1980s, with boxes called "terminal servers". They supported say 8 or 16 VDUs. I think they used some sort of tcp to telnet mapping. Good money making those at 1-2k My then company was doing some stuff like that but IBM coax and twinax compatible.

RS485 is just an interface. The protocol will be whatever the product does e.g. Modbus. And there is Modbus over TCP, although if you have low enough latency (much less than the 3.5 character periods) you don't really need it.

[paulca]

The use case, other than the appealing option for an wired ethernet, switched and VLANed UART, was to gain remote access to some solar panel gear in the garage.

It all speaks RS485 ModBus and apparently at least the most common registers are standard to some degree.  Such that, and I really didn't expect this, different manufacturers displays will read others devices and vice versa.  Would be interesting to see if there is a published "standard" set of these registers somewhere.

From what I'm reading however it doesn't sound much like it will be "plug and play", connect RS485 port, connect TCP, connect app to VCP driver.

I already have an RS485 monitor installed, it's an ESP8266 originally running my code, but now running someone elses more polished version.  The downside is, it's "standard monitor" only.  You can't modify the config registers.

The only way to change a battery charge parameter in the solar charge controller is to go down and connect the laptop to the thing with a USB/RS485 dongle.  My laptop has recently expired, screens gone, not worth trying to use anymore.  Not hurrying to get a new one.  Un wiring it all, bring the controller to the PC and doing it that way is fine for a one off, not for fiddling and testing.

So one option is to write a specific Modbus TCP proxy in an ESP32 myself where I control both ends.  It means it's Wifi rather than wired but that's fine for my use.  This kind of roll-you-own can exclude 90% of the protocol overheads that I don't need and focus on making the EPEver PC app work.... or...  just use the registers myself directly with a commandline client.

[paulca]

I'd say this space still has legs.  It may no longer be used for long distance industrial and utility network comms, they have all gone GSM etc.

Where it still exists in abundance (RS485/Modbus/UART/CAN) is in solar energy tech and battery storage.  It has become the defacto interface for interconnecting these.

Many Solar PV installers will provide you a contract where by they can see your system, remote in, monitor it, fix issues etc.  While these devices may not be exactly the same as a "wired" serial server they aren't a kick in the pants off.  Any installation larger than a house might have to span dozens of meters of COM cabling and it's like to be RS485.  In an office building however with a wired backbone ethernet these devices could save a LOT on cabling.

[coppice]

I'd say this space still has legs.  It may no longer be used for long distance industrial and utility network comms, they have all gone GSM etc.

Where it still exists in abundance (RS485/Modbus/UART/CAN) is in solar energy tech and battery storage.  It has become the defacto interface for interconnecting these.

Many Solar PV installers will provide you a contract where by they can see your system, remote in, monitor it, fix issues etc.  While these devices may not be exactly the same as a "wired" serial server they aren't a kick in the pants off.  Any installation larger than a house might have to span dozens of meters of COM cabling and it's like to be RS485.  In an office building however with a wired backbone ethernet these devices could save a LOT on cabling.

About 10 years ago there were a lot of people in China looking for a cheap ethernet<->serial  solution, talking big numbers. Maybe this was specifically associated with growth in the solar industry.

[paulca]

I'd say this space still has legs.  It may no longer be used for long distance industrial and utility network comms, they have all gone GSM etc.

Where it still exists in abundance (RS485/Modbus/UART/CAN) is in solar energy tech and battery storage.  It has become the defacto interface for interconnecting these.

Many Solar PV installers will provide you a contract where by they can see your system, remote in, monitor it, fix issues etc.  While these devices may not be exactly the same as a "wired" serial server they aren't a kick in the pants off.  Any installation larger than a house might have to span dozens of meters of COM cabling and it's like to be RS485.  In an office building however with a wired backbone ethernet these devices could save a LOT on cabling.

About 10 years ago there were a lot of people in China looking for a cheap ethernet<->serial  solution, talking big numbers. Maybe this was specifically associated with growth in the solar industry.

I was watching a documentary on YT about some of the chinese factories and if you think about it's sort of expected, but rather than big white glitzy production lines making a single product in it's millions, often factories pick up whatever work they can and just retool to suit it.  That re-tooling involves a lot of room for automation.  Being as they are (stereo-typically), they made do.  So on the likes of AliExpress you will find a LOT of products which are available in strange abundance and at strange prices. 

Basically they have teams of people building micro-controller frankenstien systems to augment existing CNC, assembly belts, pick and place, etc. etc. 

The STM32H7 MCU boards I have been buying from them often come with the most bizarre set of peripherals on the board.  Like things which just don't make sense for makers on a platform like that.  One example is a load of STM32H7B0 boards which come with a camera and screen, but not a screen you would view images on a tiny oled.  I spotted a snap shot of one of these boards on a sellers website sitting in a pic and place machine, where I assume it's job was to identify and orientate a particular custom part feeder.

The RS485 boards also seem in abundance and could be related as RS485 buses are very common in factory automation links (I could be wrong).

If you want an MCU to take a photo, identify if a part if upside down on a belt and tell a flipper to flip, RS485 over a switched ethernet might be ideal.

Other bizarre peripherals include near field comms modules.  I'm still wondering what they use those for?  To simplify two or more MCUs working together?  They also tend to come with a HUGE abundance of QSPI Flash, like sometimes 8Mb or even more.  Maybe that is for storing images?  Maybe these also get used as assembly line security.

[coppice]

The STM32H7 MCU boards I have been buying from them often come with the most bizarre set of peripherals on the board.  Like things which just don't make sense for makers on a platform like that.  One example is a load of STM32H7B0 boards which come with a camera and screen, but not a screen you would view images on a tiny oled.  I spotted a snap shot of one of these boards on a sellers website sitting in a pic and place machine, where I assume it's job was to identify and orientate a particular custom part feeder.

What most MCU users don't fully realise about MCUs is 95% of them are developed for one lead application, with a mind to broadening the appeal only if it can be done at trivial cost. They are rarely general purpose in their intent. When you market an MCU and get pull from area you didn't expect that's a massive bonus, and you look like a hero. Boards are similar, even the EVM ones. Remember the MCU was intended for one market. The EVM will be similar. If you are in the application space the MCU or board was designed for you'll look and think its an entirely obvious product.

Other bizarre peripherals include near field comms modules.  I'm still wondering what they use those for?

Near field comms is huge, especially in China. The volumes on a product like transport smart card readers may be small compared to the actual cards, but its still pretty big.

[peter-h]

The chinese go for the most obvious low hanging fruit in finished products.

Especially if they can rip something off from the West.

And they implement it using the easiest way so if e.g. building a consumer router they do it with a linux box and open-wrt (or the present day variant).

Running serial over ethernet has been a totally obvious product for decades. When I started my present business in 1991 I looked at distributing a German product which ran 4 RS232 ports over ETH. Non routable; you set up a MAC # on dipswitches I didn't take it on in the end because the mfg was incredibly arrogant and unwilling to answer straight simple questions with straight answers. The TCP variants came later (the cheap ones).

[cantata.tech]

If am reading it write you basically pair TCP connection and series protocols.

Have you started writing it yet? It's a nice hobby project.

I would start with writing an emulator on the pc for the device. Are you using Linux, Windows or Mac?

Is your development environment PlatformIO, Arduino or something else?

[nfmax]

The use case, other than the appealing option for an wired ethernet, switched and VLANed UART, was to gain remote access to some solar panel gear in the garage.

It all speaks RS485 ModBus and apparently at least the most common registers are standard to some degree.  Such that, and I really didn't expect this, different manufacturers displays will read others devices and vice versa.  Would be interesting to see if there is a published "standard" set of these registers somewhere.

From what I'm reading however it doesn't sound much like it will be "plug and play", connect RS485 port, connect TCP, connect app to VCP driver.

I already have an RS485 monitor installed, it's an ESP8266 originally running my code, but now running someone elses more polished version.  The downside is, it's "standard monitor" only.  You can't modify the config registers.

The only way to change a battery charge parameter in the solar charge controller is to go down and connect the laptop to the thing with a USB/RS485 dongle.  My laptop has recently expired, screens gone, not worth trying to use anymore.  Not hurrying to get a new one.  Un wiring it all, bring the controller to the PC and doing it that way is fine for a one off, not for fiddling and testing.

So one option is to write a specific Modbus TCP proxy in an ESP32 myself where I control both ends.  It means it's Wifi rather than wired but that's fine for my use.  This kind of roll-you-own can exclude 90% of the protocol overheads that I don't need and focus on making the EPEver PC app work.... or...  just use the registers myself directly with a commandline client.

How odd - I'm currently trying to do something similar myself! In my case I have a Schüco SGI 1500 T plus 2 inverter which has a completely undocumented RS485 interface, but which I suspect is actually MODBUS. I'm using a Devantech ds3484 Ethernet relay module as a data collector: an analogue input for an outside temperature sensor, logic inputs for door closure & locked switches, logic outputs to drive status indicators, and pulse counters for the grid import and solar generation meters. It also has an RS485 port, and if it is configured to communicate using MODBUS TCP, it will forward commands to 'other' modbus addresses over this interface.

All this can be configured using the ready built application software that you can download for it, via a web interface. Or you can write your own programs in the Java-like dScript language, in which the application itself is written. All freely available.

I've had this module knocking around for years (I bought it on a whim in an end-of-line clearout sale), but it's still available: https://www.robot-electronics.co.uk/products/relay-modules/ethernet-relay/ds3484.html. It, or another model from the same supplier, may be of interest to you.

If you do get anywhere spelunking the solar inverter RS485 interface protocol, I'd be grateful if you could share what you discover.

[cantata.tech]

Here's a link to something that I used as a base and then modified:

https://github.com/GuLinux/python-socket2serial

[paulca]

Well, this might get bumped up the list of priorities.

Long story, in another thread, but my BMS-less 4S LiFePO4 pack is getting out of balance and that is being exasperated by the solar charge controller which appears to have reverted back to normal "lead acid" mode, including the 15V equalisation charge.

The problem is... I'm not 100% sure, if I unwire the SCC and take it to the lab for reprogramming over USB that it will retain those settings while powered down for the return trip.

Besides, I'd like lower level protocol access than the "monitoring" gizmo I have.

The question is... where are the pitfall is creating a bespoke solution just for the specific needs of Modbus over TCP?

I have a dev board here which includes a F411 an ESP32, a screen and a UART header.  I also might possibly have a logic level converter board and maybe even a MAX4855 converter board in a parts bin somewhere.  That should take care of the hardware, if I can find that stuff.

For software, the STM32 can handle the UART/RS485 side of things and just feed the ESP32-AT firmware buffers.  The ESP32-AT firmware supports several options on what is usually referred to as "TCP_NO_DELAY".  It's how long to wait before sending an incomplete TCP segment.  By default the TCP stack will wait for a period of time before sending the partial segment.  This is often governed by some fairly non-trivial algorythm, but for me, all I need to care about is any particular protocol timing requirements are met and the "responsiveness" is appropriate for the job.

Thus, I can tell the ESP32 TCP stack to buffer the data I send it and timeout after X miliseconds.  Thus capping the "packetisation" delay at X milliseconds.

Beyond that it should basically be a data agnostic bidirectional buffer. 

There are always going to be "But what about...."

[coppice]

For software, the STM32 can handle the UART/RS485 side of things and just feed the ESP32-AT firmware buffers.  The ESP32-AT firmware supports several options on what is usually referred to as "TCP_NO_DELAY".  It's how long to wait before sending an incomplete TCP segment.  By default the TCP stack will wait for a period of time before sending the partial segment.  This is often governed by some fairly non-trivial algorythm, but for me, all I need to care about is any particular protocol timing requirements are met and the "responsiveness" is appropriate for the job.

Thus, I can tell the ESP32 TCP stack to buffer the data I send it and timeout after X miliseconds.  Thus capping the "packetisation" delay at X milliseconds.
re always going to be "But what about...."

That is a standard part of a TCP implementation, turned on by default. Its a gotcha for many beginners who can't understand the high latency they get with their TCP communications code. It is usually referred to as the Nagle feature or algorithm, after the guy who introduced it - https://en.wikipedia.org/wiki/Nagle%27s_algorithm

[paulca]

It is usually referred to as the Nagle feature or algorithm, after the guy who introduced it - https://en.wikipedia.org/wiki/Nagle%27s_algorithm

That's the name I was looking for.  Obviously it lowers your overall through put. 

There are similar complexities around acks, selective and early versus late and contiguous and all manor in between. 

It's one of those topics I studied to the n'th degree in Uni.  I even did an exam question on the TCP sliding window algorhtm and calculating the maximum through put on a connect with a given latency and a given window size.

I do recall using this in anger in work when resolving low latency issues in the stock exchange and it was the customer who reported the Nagle aglo did not appear to be running on "some" of their sessions.

[coppice]

It is usually referred to as the Nagle feature or algorithm, after the guy who introduced it - https://en.wikipedia.org/wiki/Nagle%27s_algorithm

That's the name I was looking for.  Obviously it lowers your overall through put. 

Why would it affect your throughput? As the data flow builds up Nagle stops doing anything. The algorithm reduces the flow of inefficient small packets at low data rates, but in exchange it creates latency when you are waiting for the algorithm to kick out a packet on timeout.

[paulca]

It is usually referred to as the Nagle feature or algorithm, after the guy who introduced it - https://en.wikipedia.org/wiki/Nagle%27s_algorithm

That's the name I was looking for.  Obviously it lowers your overall through put. 

Why would it affect your throughput? As the data flow builds up Nagle stops doing anything. The algorithm reduces the flow of inefficient small packets at low data rates, but in exchange it creates latency when you are waiting for the algorithm to kick out a packet on timeout.

Sorry I was thinking about the result of shortening the time too much, rather than Nagles which tries to balance things.

[coppice]

It is usually referred to as the Nagle feature or algorithm, after the guy who introduced it - https://en.wikipedia.org/wiki/Nagle%27s_algorithm

That's the name I was looking for.  Obviously it lowers your overall through put. 

Why would it affect your throughput? As the data flow builds up Nagle stops doing anything. The algorithm reduces the flow of inefficient small packets at low data rates, but in exchange it creates latency when you are waiting for the algorithm to kick out a packet on timeout.

Sorry I was thinking about the result of shortening the time too much, rather than Nagles which tries to balance things.

I've always felt like they missed an element in these TCP implementations. A feature that lets you have Nagle running, but kick out the data in the buffer immediately. I've done a number of things where Nagle is useful, but there are points where I need the buffer to flush to the far end, so it can promptly construct and send its response to me. I have to turn Nagle off, put the piecemeal parts of my request together in a buffer, and send it in one go. If I'm writing all the code that's usually no big deal, but it can be clumsy when working with some libraries.

[paulca]

There are parts of TCP/IP stack are look unfinished, or at least never reached proper standardisation.

Examples are the use of "PUSH" flags on TCP segments.  Nobody agrees on what they should do and most routers remove them.

The idea around the PUSH flag is "Make the application aware of this segment immediately."  Rather than just carrying on filling your network stack buffer before waking your process when it feels like it, but to do it NOW.

Nagles has quite a few tuning parameters at the C level and there are even swap in algorithms in the Linux Kernel if you are in that space.  Sometimes it just turns into a whole load of testing and measuring and effort to gain 0.15%.