PSA: Wiznet chip is not a real networking stack

[Jeroen3]

Yeah, I dipped my toe in the Mikrotik queues to limit bandwidth, and there are a lot of methods.

I think in case of TCP it's good to know that it only guarantees order and correctness, not speed or latency.

In that aspect it amazes me that the people behind Modbus decided to use tcp/ip. It is probably the worse fit for a realtime PLC protocol. They probably wanted something similar like a serial port (a stream) and forgot that Modbus in itself is packet based which suits UDP so much better.

Modbus in itself isn't intended to be realtime in the pedantic meaning. Routing it over TCP is odd because the explicitly add another seq/transaction number to Modbus TCP, even though TCP should already handle this. I assume they chose TCP as it came closest to a serial port for backwards compatibility of existing software. Which, on serial bus, packets don't get swapped.
On healthy dedicated networks, as is within the scope of modbus over serial, this shortcut would have been fine.
That we then started abusing this protocol to obtain realtime, very remote and/or high data-rate is an system engineering mistake.

Are there altnernative to LWiP yet? I remember using it and it being very easy to use incorrectly to create memory bugs. As a result, many people wrote a facade around it, but nobody wrote a new one. Wasn't amazon going to do this when they bought freertos?

[nctnico]

Are there altnernative to LWiP yet? I remember using it and it being very easy to use incorrectly to create memory bugs. As a result, many people wrote a facade around it, but nobody wrote a new one. Wasn't amazon going to do this when they bought freertos?

That would be FreeRTOS+TCP. Rumour has it Amazon is actively maintaining this tcp/ip stack.

[Siwastaja]

Modbus  ... I assume they chose TCP as it came closest to a serial port for backwards compatibility of existing software.

I believe their TCP mistake would be a misunderstanding that TCP handles resends and errors for them, so that application developers (those who program the logic) do not need to think about it. This of course turns out to be false: TCP connection can still break, and an application using TCP needs that exact logic: open connection -> send message -> poll until you get response -> detect failure codes / wait for timeout -> verify the response is related to your query, not stale data -> close connection in case of failure -> if failed, try again with same data / handle errors. They need to check that the response is related to the query anyway, verifying the function code / register address from the response, or the transaction ID. This is because a TCP stack buffers and TCP does not offer synchronization checkpoints between the TX / RX streams. It's two independent streams, buffered separately. TCP guarantees that the bytes I sent appear in order. It doesn't guarantee that the bytes I read from the OS buffer arrived after my previous write, which is exactly what modbus would need for TCP to simplify the job.

So application designer still needed to solve all the hard problems.

UDP would have given them the same guarantees, given that the maximum packet length for Modbus is ~256 bytes and hence it fits in any Ethernet MTU fine. Using it instead of TCP would remove the steps of socket creation and closure, and reduce number of different failures that can happen, and remove the extra framing that modbus TCP now needs (number-of-bytes-to-follow header), while keeping the logic which matches received data with the previous sends, that would be unchanged between TCP/UDP, but UDP would handle packet delimitation, an extra simplification.

What they really need is data integrity within the message and UDP does exactly that, with a checksum as strong as TCP. UDP has an excellent track record in exactly modbus-like fast, small poll-response protocols, DNS being the most obvious example.

[Dan N]

Wiznet chips do have raw mode though?

Yes, Mongoose uses that:

On W5500, it reads /writes raw frames, thus bypasses W5500's stack (which makes it possible to use IPv6, and implement more than 8 sockets).

It's hard to avoid these types of chips at the very low end (<$20)--it's either Wiznet or an equivalent like CH9120.  This type of bypassing takes away the locked-in risk.

[voltsandjolts]

It would be good to get to the cause of the OP's network problem(s).
Only then can we determine if it's really an unfixable wiznet stack.
Who knows, it might be as simple as increasing the retransmit back-off intervals on slow links.

[SteveThackery]

I believe their TCP mistake would be a misunderstanding that TCP handles resends and errors for them, so that application developers (those who program the logic) do not need to think about it. This of course turns out to be false: TCP connection can still break, and an application using TCP needs that exact logic: open connection -> send message -> poll until you get response -> detect failure codes / wait for timeout -> verify the response is related to your query, not stale data -> close connection in case of failure -> if failed, try again with same data / handle errors. They need to check that the response is related to the query anyway, verifying the function code / register address from the response, or the transaction ID. This is because a TCP stack buffers and TCP does not offer synchronization checkpoints between the TX / RX streams. It's two independent streams, buffered separately. TCP guarantees that the bytes I sent appear in order. It doesn't guarantee that the bytes I read from the OS buffer arrived after my previous write, which is exactly what modbus would need for TCP to simplify the job.

@Siwastaja: forgive me, for I am no expert on networking protocols, but I thought the first part of your paragraph, at least, is indeed handled by TCP. I want to quote Wikipedia here:

"TCP is a reliable byte stream delivery service that guarantees that all bytes received will be identical and in the same order as those sent. Since packet transfer by many networks is not reliable, TCP achieves this using a technique known as positive acknowledgment with re-transmission. This requires the receiver to respond with an acknowledgment message as it receives the data. The sender keeps a record of each packet it sends and maintains a timer from when the packet was sent. The sender re-transmits a packet if the timer expires before receiving the acknowledgment. The timer is needed in case a packet gets lost or corrupted."

The "poll until you get a response" and "detect failure codes/wait for timeout" you mentioned are done by TCP, not your app, as I understand it. Could you shine some light on what I've misunderstood?

[NorthGuy]

Could you shine some light on what I've misunderstood?

TCP connection may break for various reasons. For example, a NAT router in the middle may drop the connection, or a DHCP server changes the client's IP address. In such cases, TCP cannot restore the connection. The only way forward is opening a new connection.

In layman terms, if a web site does not load, it will not fix itself until you click a "reload" button.

[Siwastaja]

It would be good to get to the cause of the OP's network problem(s).
Only then can we determine if it's really an unfixable wiznet stack.
Who knows, it might be as simple as increasing the retransmit back-off intervals on slow links.

If I knew what the problems are, I would fix them. Future plan is to spend more hours on it; I have some hope left there might be a solution exactly by tuning the Wiznet registers. Retransmit back-off configurations have been adjusted a few times, no significant change. This is alone a questionable setting because the datasheet contradicts itself - it simultaneously claims it's constant time and exponential, so only way to figure out what a register setting actually does is to reproduce the issue and put some Wireshark at it.

The bottom line is: it does not work with defaults on some rare networks where Windows computers and IoT devices by any other manufacturer work. Failure rate is just 0.2-0.5% or so and I suspect it's technically fault of others - poorly designed uplink devices like some WiFi extenders etc. - but it doesn't change the outcome that others work, our doesn't.

Using a software stack would have given much more freedom on that TCP tuning business. The idea of using a single-chip solution was to avoid that work altogether, but that doesn't seem possible. I'm not sure yet how the whole thing turns out after more debugging: maybe it can be solved by changing how the Wiznet chip is configured/used, but there is no guarantee.

[Siwastaja]

forgive me, for I am no expert on networking protocols, but I thought the first part of your paragraph, at least, is indeed handled by TCP. I want to quote Wikipedia here:

You miss the fact that all abstractions leak: the real world leaks through when the abstraction, like TCP, works in conditions it can't handle: that means, TCP gives up. It's a perfectly valid condition for TCP.

Read the Wikipedia carefully. It says TCP guarantees bytes arrive in order and not changed. It does not guarantee that the bytes arrive. It also does not guarantee that all bytes get through - but it does guarantee that, if some bytes get missing, application will be notified about it - it kills the socket. Application then needs to reopen the socket, figure out in which state the systems left, and start operating again. This is the unavoidable business logic that goes with any protocol. There is no magic.

In case of Modbus over IP, UDP would be a better choice exactly because it gives those same guarantees which matter. UDP, of course, also guarantees byte ordering and integrity within the packet, modbus happens to be a packet-oriented protocol, and modbus packets happen to fit within single UDP packets, so it would be a perfect fit. TCP isn't packet-oriented (unless you think one byte, just 8 bits, a "packet") - that's the whole point why it exists, its sole purpose is to turn the underlying packetized IP layer into continuous stream of single bytes. This is exactly what Modbus does not need. Modbus traditionally works over UART / serial RS485, which also is a stream of bytes, which may have confused the designers; but here, the "stream of bytes" already is a hindrance, not a benefit; Modbus have some peculiar idle timing rules to try to enforce packet boundaries over the UART link which cannot do that inherently. That's already a hack. Modbus TCP does a similar "hack" over TCP: an extra delimiter header. UDP would have delimited the packets inherently.

Ironically enough, if you implement the modbus TCP communication state machine (write -> read -> check that it was yours and it was OK), you already implemented the ordering guarantee yourself - you had to, as part of TCP socket error handling. As such, you don't need any more special hoops against UDP message "ordering". You start a new poll only after the previous finished, and check that the response is relevant just in case anyway.

[westfw]

This is because a TCP stack buffers and TCP does not offer synchronization checkpoints between the TX / RX streams.

Surely the same problem occurs with "serial" communications, especially with modern APP/OS/USB/USBUART/UART paths inserting buffering of some kind at each stage?

I suspect that TCP was chosen because it was "close to a serial port" - they might have done better (given it's a datagram-oriented protocol), but it would have taken a lot more effort.
I'll note that there have been a bunch of attempts at defining a "reliable datagram protocol" over IP, none of which seems to have caught on.  (and few of which have addressed things like the "necessary" congestion avoidance.)

[NorthGuy]

The bottom line is: it does not work with defaults on some rare networks where Windows computers and IoT devices by any other manufacturer work. Failure rate is just 0.2-0.5% or so and I suspect it's technically fault of others - poorly designed uplink devices like some WiFi extenders etc. - but it doesn't change the outcome that others work, our doesn't.

May be we could try to troubleshoot it. Bad re-transmission timeouts are very unlikely to produce dramatic failures, more like delays or extra traffic. What exactly do you mean by not working? Can you ping your device from other devices? If you can, what is approximate packet loss rate? Also, when you ping it, what is the latency reported? Is the latency roughly the same or varies from ping to ping?

[voltsandjolts]

Just a thought: The problems appear to be occurring on small percentage of networks, which happen to be low performance networks. Not sure if you mean the local network or internet connection. Anyways, what if those 'poor' local networks are shitty enough to be using hubs rather than switches. If the wiznet is on a hub, it's already in trouble with a small rx buffer and a ton of nonsense packets to sieve through. Also, I think wiznet devices are full-duplex when on a switch but only half-duplex on a hub. So it's screwed really. Hopefully nobody is using hubs in 2026.

[Davor]

The bottom line is: it does not work with defaults on some rare networks where Windows computers and IoT devices by any other manufacturer work. Failure rate is just 0.2-0.5% or so and I suspect it's technically fault of others - poorly designed uplink devices like some WiFi extenders etc. - but it doesn't change the outcome that others work, our doesn't.

Put 0R instead of 33R and try. 33R is good for EMI and bad for connectivity.

[helius]

modbus packets happen to fit within single UDP packets

A single UDP packet is up to 65535 bytes (including the IP and UDP headers), however when carried over IPv6 it can be up to 4 gigabytes.
Fragmentation (to the MTU limit) and reassembly happen on the IP layer, not the datagram layer.

It is true that fragmentation can negatively affect UDP packet deliverability, since loss of any fragment will cause the whole packet to be lost.

[nctnico]

Common advice: "Don't reinvent TCP because TCP is very hard to do right. Use an existing stack which has been optimized during decades of experience"

So far so good, until someone advices:
"Get a Wiznet chip, it's a TCP/IP on chip solution so that you don't need to reinvent TCP because it's hard"

Fast forward to many thousands of units on the field, and you are now debugging TCP issues, the exact thing you wanted to avoid, except that you can't do anything about them. You can't fix the stack. It's locked in silicon.

It does not have exponential back-off retransmission. It doesn't have RTT measurement, Karn's algorithm, no congestion window. It's the bare minimum TCP implementation, which works on a good network. It's exactly what you would get if you implemented your own from scratch with limited time/budget. Except unfixable.

Are you 100% sure the problem is actually in the W5500 chip? Have you been able to reproduce using a traffic generator? If not, I wouldn't rule out a hardware problem like the series resistors Davor is pointing to.

[Siwastaja]

Surely the same problem occurs with "serial" communications, especially with modern APP/OS/USB/USBUART/UART paths inserting buffering of some kind at each stage?

Sure it does, and if you look at the modbus spec, it gives some peculiar timing requirements that implement ad-hoc framing over UART. This means it is often impossible to implement standard-compliant modbus through various types of buffering UARTs - it's doable only on a microcontroller, or maybe real-time kernel.

Despite that, non-standard-compliant Modbus implementations through higher level layers - say libmodbus on a linux box - are plentiful. They usually work. Just like modbus TCP usually works. What a mess.

[Siwastaja]

The bottom line is: it does not work with defaults on some rare networks where Windows computers and IoT devices by any other manufacturer work. Failure rate is just 0.2-0.5% or so and I suspect it's technically fault of others - poorly designed uplink devices like some WiFi extenders etc. - but it doesn't change the outcome that others work, our doesn't.

Put 0R instead of 33R and try. 33R is good for EMI and bad for connectivity.
(Attachment Link)

I checked, my BOM has 3.3R, not 33R here; don't remember where I got the 3.3R and what manufacturer's examples use. We use W6100L, not W5500.

But as nctnico points out, physical layer issue is a possibility I can't rule out. But the correlation now seems to be, problems are with uplinks devices that do protocol stack hacks, like extenders; simple cable-to-a-router case seems to work flawlessly. If it was a physical link issue, I would think problems would randomize over the entire fleet; the link is the same; or, concentrate on some specific router/hub. Of course it's a still possibility; maybe some LAN extender we are seeing issues with has narrower noise margin on its physical ethernet layer, and it combines badly with our board; that would mean it's just a coincidence that said extender also does the packet-level hacks I suspect to be the problem.

No, I haven't reproduced the issues in lab, that's' what I should be doing next.

[nctnico]

Problems with ethernet devices can be very specific. If you can isolate which units don't work with your device (a trick you could pull is collecting MAC addresses for the connected networks and from there determine manufacturers), then you can find out it is a particular model / brand of device causing problems or the issue is more generic. IMHO the first step is to reproduce the issue in your own lab.

[Fire Doger]

There are only a few exceptions where I would trust silicon firmware, like MP3 codec...

Old devices in my company use W5500, they also ditched internal stack, but it's huge and old IC in my opinion...
For new design we went with CH390D. For 1k pcs from Asian supplier, W5500 1.8$ LQFP, CH390D 0.7$ QFN

[Siwastaja]

The bottom line is: it does not work with defaults on some rare networks where Windows computers and IoT devices by any other manufacturer work. Failure rate is just 0.2-0.5% or so and I suspect it's technically fault of others - poorly designed uplink devices like some WiFi extenders etc. - but it doesn't change the outcome that others work, our doesn't.

May be we could try to troubleshoot it. Bad re-transmission timeouts are very unlikely to produce dramatic failures, more like delays or extra traffic. What exactly do you mean by not working? Can you ping your device from other devices? If you can, what is approximate packet loss rate? Also, when you ping it, what is the latency reported? Is the latency roughly the same or varies from ping to ping?

Two separate issues actually:

1) Total or near total silence behind cheap WiFi extenders - device tries DHCP->DNS->TCP/TLS1.3 handshake, and never succeeds, or succeeds after half an hour of trying and the connection lasts for a minute.

2) The one which motivated my post - very slow 2G uplink working for a while, then getting slower and finally dying for tens of minutes - for which I suspected retransmission / ack flooding increasing traffic when it should be decreased, leading to positive feedback loop, but that's only a working hypothesis, and figuring out the datasheet describes no exponential backoff motivated me to finally write the opening post (later realizing that the datasheet contradicts itself and there probably is exponential backoff, so the hypothesis could be wrong anyway).

1 is more serious. 2 has happened once, and on a truly slow link.

1 leaves almost no traces. When the connectivity has established, logs often show that even DNS sometimes required multiple tries. I really need to reproduce it in lab, but haven't yet been able to.

[nctnico]

It would be interesting to know if your device is subjected to lots of traffic. A Wifi extender may forward a lot of unwanted traffic which then also ends up at your port. This may be worse for streaming services relying on multicasting being active on the network. Maybe a multicast stream also ends up on your device and the stack gets swamped with packets.

[Siwastaja]

It would be interesting to know if your device is subjected to lots of traffic. A Wifi extender may forward a lot of unwanted traffic which then also ends up at your port. This may be worse for streaming services relying on multicasting being active on the network. Maybe a multicast stream also ends up on your device and the stack gets swamped with packets.

No, very little traffic; it's inherently very rate-limited, hundreds of B/second; TLS handshake being some sort of burst, but even that's limited by the poor 64MHz Cortex-M4 doing the mbedTLS encryption on software, not accelerated. In cases where "other device" has worked behind the same extender, the other device would have been light traffic as well (other IoT monitoring stuff, or a Windows laptop for a pure test, not playing any 4K video).

It's an interesting idea though that something's bombarding us with packets.

Good that I started this thread, getting motivated and will source some different extenders trying to reproduce the issue.

[NorthGuy]

Two separate issues actually:

I don't think it is easy to figure out without looking at the packets either on the wire or on the receiving server. If you compare the packets you see to what you get when Windows connects, it'll be able to see the difference and figure out what happens.

[nctnico]

Two separate issues actually:

I don't think it is easy to figure out without looking at the packets either on the wire or on the receiving server. If you compare the packets you see to what you get when Windows connects, it'll be able to see the difference and figure out what happens.

Not necessarily. Assuming these devices have some monitoring abilities, adding packet and data (byte) counters (incoming, outgoing, retries, failures, etc) can provide a lot of information on what is happening without needing to monitor the actual traffic. Typically I add these kind of counters by default in my firmware. They have helped me a lot over the years to deal with communication issues. You don't see the actual traffic but you get a good feeling on the shape.

[tellurium]

I have given the "use Wiznet chip" advice before here, so I have to apologize. It's bad advice. Avoid.

Wiznet w5500 provides "socket" interface over SPI.
Yes they have built-in TCP/IP, but you can bypass it.
They support the so-called "raw" socket, which simply sends and receives Ethernet frames.
This way, w5500 turns into a Eth transciever with SPI interface, and you can run TCP/IP stack in software.

That's exactly what we do. This example shows Mongoose running on Arduino Nano Every with 6k of RAM, using W5500 in that raw mode:

Having your stack running on host, gives you IPv6, TLS and other stuff w5500 stack does not. And of course you avoid their bugs and restrictions (like 8 sockets max). Wiznet charges premium for more modern chips that have that stuff - like TLS, etc, but as I mentioned, good old w5500 with the TCP/IP stack running on host (like Mongoose) can do the job.  So w5500 is still a neat way to add Ethernet to any micro .

So, recommending w5500 is not bad after all.