PSA: Wiznet chip is not a real networking stack

[voltsandjolts]

Haha, I actually meant customer (client)
But yes, you're right of course, network client logging needs tools.

[gamalot]

AI can now port and generate lwip based networking code for a target platform in minutes.

[Perkele]

If there's a way to export data from a device, and if it is allowed to export "debugging" stuff, I'm usually dumping the network traffic in the following format:
https://wiki.wireshark.org/Development/LibpcapFileFormat
It helped me a couple of times when debugging some issues with wireless stacks.

[westfw]

Claude knocked up a UDP transmit function, talking direct to the WFM200 chip, in seconds. Hardly any code, too.

Ah.  Did it include all the other bits of IP that even a UDP-only host is supposed to support (ICMP and fragmentation, for instance), or did you get exactly the sort of "non-conforming but seems to work most of the time" implementation that OP was complaining about?

I've actually got a patent claiming that a very small microcontroller could "implement tcp" by storing a sequence of packets it expects to send, and the responses it "should" get (eg in external EEPROM), essentially reducing the implementation to a pretty simple state machine.  I'm pretty happy that I never actually implemented it.  (although, it's probably just a generalization of the "header prediction" stuff that first showed up in CSLIP.)

I've also actually spent hours in a DoD contractor's office going through RFC1122/RFC1123 item by item to "verify" a product was suitable...

[peter-h]

No. This application sends defined and very small packets over wifi. Fair point though; one could not hack it like this if doing anything more sophisticated.

It is a fair point about implementing ICMP, so the presence of the UDP emitter can be checked with a ping. Thanks for that tip.

[Jeroen3]

Just a few weeks ago I was diagnosing networking issues in Windows 11.
I was not expecting this to be required. But it turns out, Windows has a very long delay before it tries to retransmit. Kind of useless if you have packet loss on modbus tcp.

So now we are using UDP and do the retransmits on modbus layer.

You can always run into issues.

[gerbay]

Those experiencing issues with TCP (with small packets) should ensure that the Nagle algorithm is disabled.

[Siwastaja]

Those experiencing issues with TCP (with small packets) should ensure that the Nagle algorithm is disabled.

Nagle exists for a reason. For generic undefined type of "issues", doing a random change is unlikely to help. Disabling Nagle makes sense when the issue, specifically, is lag in real-time application, in a network well capable of the overhead disabling it causes.

But if the network suffers from congestion, disabling the feature which reduces network use by an order of magnitude, can totally kill it.

[Jeroen3]

Networking is so difficult. How do you even know about this.

Last weekend I was also trying resolve F1 stream glitches. It was that bad that I went to look at packet captures.... 50% TCP retransmits!

Turns out, live streams are distributed in short bursts. Very high payload short burst. approx 1s each.
My 1Gbe fiber passes these fine. But then, what I didn't expect, was that this burst caused loss on the ethernet layer because there was a 2.5Gbe switch in between. Which can deliver these packet at a rate a 1 Gb switch can't pass along, and can't buffer. Dropping packets. You need to enable ethernet flow control.
Unmanaged switches won't tell you this...

This didn't fix the glitches though. TCP worked as designed, even with the loss all data was there.
---
What does wiznet run as cpu nowadays?

[nctnico]

Those experiencing issues with TCP (with small packets) should ensure that the Nagle algorithm is disabled.

That is a workaround. But the cold hard fact is that the IP stack on Windows is badly broken (and has been for decades). Some devices send TCP packets twice as a workaround so Windows responds with a next packet immediately. The IP stack on Linux doesn't have this issue; it uses a much more sophisticated algoritihm to optimise / maximise TCP packet flow rate.

[westfw]

it turns out, Windows has a very long delay before it tries to retransmit.

Modern IP/TCP stacks are engineered and optimized for networks whose major packet drop (and thus retransmit) reason is congestion at routers, rather than actual lossy link hardware, memory shortages at the final destination, or slow links.  So (re)transmission algorithms are designed/required to avoid congestion.  Which is swell for the sort of bulk data transfers that are very common in today's Internet, but not so good for any number of smaller applications.  It's something of a Pet Peeve of mine, but it's hard to get anyone to care.

Nagle is very useful for optimizing bandwidth, adn was sort-of brilliant.  So is "delayed ack."  But they assume particular data patterns, and they can interact very weirdly indeed. :-(   The required "slow start" (Jacobson) algorithms sort-of devolve into Nagle if you're sending small packets.

[Jeroen3]

Yeah, I dipped my toe in the Mikrotik queues to limit bandwidth, and there are a lot of methods.

I think in case of TCP it's good to know that it only guarantees order and correctness, not speed or latency.

[madires]

On top there are several algorithms for packet queuing, e.g. CoDel/CAKE, to 'smooth' traffic.

[nctnico]

Yeah, I dipped my toe in the Mikrotik queues to limit bandwidth, and there are a lot of methods.

I think in case of TCP it's good to know that it only guarantees order and correctness, not speed or latency.

In that aspect it amazes me that the people behind Modbus decided to use tcp/ip. It is probably the worse fit for a realtime PLC protocol. They probably wanted something similar like a serial port (a stream) and forgot that Modbus in itself is packet based which suits UDP so much better.

[peter-h]

Absolutely! There is an ack so error detection/correction is taken care of. I guess they went for TCP/IP because it is routeable, but how often does one need that?

The boxes which run modbus over tcp/ip are supposedly packing up the modbus packet (which can be huge, 10s of kbytes in the responses for example if you do a multi register read) into multiple tcp/ip packets, obviously MTU constrained as usual along the way, and reassemble them into single modbus packets at the other end. So it is a protocol specific gateway (if done properly).

[nctnico]

Absolutely! There is an ack so error detection/correction is taken care of. I guess they went for TCP/IP because it is routeable, but how often does one need that?

UDP is perfectly routable. It works with IP addresses just like TCP.

[SiliconWizard]

The main rationale was likely the benefit of guaranteed delivery order - keep in mind Modbus was originally a serial protocol and while missed packets could happen from communication errors,  out-of-order packets was just impossible. With UDP, order of delivery is not guaranted. After that, I guess the fact TCP is session-oriented and retransmission is automatic was also seen as a benefit rather than having to implement all this on a higher level.

[gspeed]

Wiznet chips do have raw mode though?

[Siwastaja]

The main rationale was likely the benefit of guaranteed delivery order - keep in mind Modbus was originally a serial protocol and while missed packets could happen from communication errors,  out-of-order packets was just impossible. With UDP, order of delivery is not guaranted. After that, I guess the fact TCP is session-oriented and retransmission is automatic was also seen as a benefit rather than having to implement all this on a higher level.

I sometimes miss a packet-oriented (like UDP) with built-in retransmission and ordering (like TCP); not on top of TCP/UDP, but at the same level, as alternative. TCP does a lot of useful things, and then forces an abstraction where the user-accessible datagram is exactly 8 bits, so if you need larger frames, as every real application in existence does, you the build a packetization system on top. Every usual layer on top of TCP, including HTTP, MQTT etc. are fundamentally packet delimiter systems (with application-specific extra features). The simplest ad-hoc packetization is a length field followed by any data; because TCP never loses anything, that suffices: you read the length field, then the correct amount of data, and are guaranteed to have the next length field. But this is still unnecessary work and it fights with layers below (Nagle, delayed ack, etc.) which try to figure out where application's data boundaries are.

Modbus TCP vendors seem to struggle somewhat, we have seen implementations which eventually crash if you keep doing polls on an open socket, forcing you to do a state machine on the application side which opens socket, communicates, checks everything went well, closes socket, and only then declares the operation finished. UDP would not be any more complicated, but simpler: still the same "retry until success" state machine, but without connection management.

[westfw]

it fights with layers below (Nagle, delayed ack, etc.) which try to figure out where application's data boundaries are.

I feel like TCP's PUSH and URGENT data were never fully explored (and are currently under-utilized.)
There's a tremendous chicken-and-egg problem.  IIRC, various reliable datagram protocols have been propsed, but they need to be implemented in the common OSes before they'd be useful on embedded systems, and there needs to be a real need on the embedded side before the OS vendors would be interested.

I suspect that the modbus folks didn't think there would be any problems at all running stuff used to 115.2kbps or less over a 100Mbps Ethernet LAN (which was probably the immediate goal.)  oops.

[nctnico]

The main rationale was likely the benefit of guaranteed delivery order - keep in mind Modbus was originally a serial protocol and while missed packets could happen from communication errors,  out-of-order packets was just impossible. With UDP, order of delivery is not guaranted. After that, I guess the fact TCP is session-oriented and retransmission is automatic was also seen as a benefit rather than having to implement all this on a higher level.

I sometimes miss a packet-oriented (like UDP) with built-in retransmission and ordering (like TCP); not on top of TCP/UDP, but at the same level, as alternative. TCP does a lot of useful things, and then forces an abstraction where the user-accessible datagram is exactly 8 bits, so if you need larger frames, as every real application in existence does, you the build a packetization system on top. Every usual layer on top of TCP, including HTTP, MQTT etc. are fundamentally packet delimiter systems (with application-specific extra features). The simplest ad-hoc packetization is a length field followed by any data; because TCP never loses anything, that suffices: you read the length field, then the correct amount of data, and are guaranteed to have the next length field. But this is still unnecessary work and it fights with layers below (Nagle, delayed ack, etc.) which try to figure out where application's data boundaries are.

Modbus TCP vendors seem to struggle somewhat, we have seen implementations which eventually crash if you keep doing polls on an open socket, forcing you to do a state machine on the application side which opens socket, communicates, checks everything went well, closes socket, and only then declares the operation finished. UDP would not be any more complicated, but simpler: still the same "retry until success" state machine, but without connection management.

Agreed. And correct me if I'm wrong, Modbus is essentially a PLC protocol where the status of inputs and outputs is updated continuously. So some packet loss is not a problem. If they added a packet sequence number to the network packets, detecting out-of-order packets is trivial.

[glenenglish]

dont use LWIP.. OMG do people stil use that ? I am amazed when dev boards come out with that as the tcp  stack.
we're not in 4 bit land anymore.

use FreeRTOS+TCP

[Siwastaja]

dont use LWIP.. OMG do people stil use that ? I am amazed when dev boards come out with that as the tcp  stack.
we're not in 4 bit land anymore.

Age and popularity itself isn't a disadvantage, rather it's the opposite: it's the strongest advantage, and exactly the reason people "still" use it. TCP itself reached kind-of maturity in 1980's, and usage patterns have remained very similar for 20 years, so the big questions are related to reliability, safety and efficiency, quality of documentation and ease of use. Directly, age isn't a factor, and indirectly, it's a positive factor. A replacement needs to fix significant issues to be better.

[nctnico]

In my experience a solution being popular says absolutely nothing about quality or suitability.

[aeg]

FreeRTOS+TCP is total amateur hour. It can't even do IP fragment reassembly! And look at ipconfigARP_STORES_REMOTE_ADDRESSES... the people behind it don't understand how ARP and routing work.

There's nothing wrong with lwIP. It's solid. The code style is idiosyncratic. It's a network stack; it requires effort to understand. There's no way around that.