PSA: Wiznet chip is not a real networking stack

[Siwastaja]

Common advice: "Don't reinvent TCP because TCP is very hard to do right. Use an existing stack which has been optimized during decades of experience"

So far so good, until someone advices:
"Get a Wiznet chip, it's a TCP/IP on chip solution so that you don't need to reinvent TCP because it's hard"

Fast forward to many thousands of units on the field, and you are now debugging TCP issues, the exact thing you wanted to avoid, except that you can't do anything about them. You can't fix the stack. It's locked in silicon.

It does not have exponential back-off retransmission. It doesn't have RTT measurement, Karn's algorithm, no congestion window. It's the bare minimum TCP implementation, which works on a good network. It's exactly what you would get if you implemented your own from scratch with limited time/budget. Except unfixable.

Add a slow 2G link, and it fails. Add a Chinese poorly tested WiFi extender, and it fails. Only the Wiznet fails. Computers and phones on the same networks do not, because they run Windows (BSD) or linux stack; the Chinese tested with those.

I have given the "use Wiznet chip" advice before here, so I have to apologize. It's bad advice. Avoid.

Compared to Wiznet lock-in, actually even writing your own TCP stack (the thing which you should never do) would have been better - that would be fixable. Of course, the best option is to use some actually tuned and tested stack, and run that on the microcontroller, and if necessary, using a bigger controller (with MAC peripheral) to fit it.

[peter-h]

What would be the point of writing own stack when LWIP is free and with some 18 years of development behind it runs in millions of products out there?

[Siwastaja]

What would be the point of writing own stack when LWIP is free and with some 18 years of development behind it runs in millions of products out there?

I don't see much reason to use anything else, given that it's not only widely used, but also open source, which means AI will quite easily fix any bugs that arise or fine-tune it, instead of a completely from-scratch implementation.

[Karel]

What about U-blox Wi-Fi modules...

[nctnico]

What would be the point of writing own stack when LWIP is free and with some 18 years of development behind it runs in millions of products out there?

I don't see much reason to use anything else, given that it's not only widely used, but also open source, which means AI will quite easily fix any bugs that arise or fine-tune it, instead of a completely from-scratch implementation.

From my experience LWIP has it's own can of worms. I used it together with a modbus-TCP PLC but that had several issues when talking to LWIP. The core problem is that the original author of LWIP produces rather obfustigated code and tries to outsmart C compilers. His uIP is an even bigger mess. Bottom line: swapping Wiznet with Lwip is just running into different problems. If you throw Claude at LWIP, you'll probably end up with a rewrite.

[peter-h]

I did put Claude onto LWIP and it found some potential buffer overflow stuff, which could not be exploited in the 32F417 ETH subsystem context anyway where relevantly oversize packets were not possible to start with.

The code may be convoluted (I would not know - can you give examples?) but with GCC it builds and runs fine. The only issue I ever found, in years of 24/7 operation, was that if I was hammering my "homemade and using the netconn API" HTTP server, another application (GPS to NTP) which used BSD sockets was not sufficiently responsive for Windows to work its RTC sync. The workaround was trivial and probably it was just buffer sizes (that whole lwipopts.h - which is something that needs time to understand) but I was trimming the buffers aggressively to get enough RAM for the 48k MbedTLS block. Now with the extra 64k (32f437) I could revisit that but I don't have TLS in there, thank god...

[NorthGuy]

I am sure there are many private TCP/IP implementations, many of them surpass lwIP in many aspects.

I don't understand how Wiznet failures prove otherwise.

[westfw]

you are now debugging TCP issues, the exact thing you wanted to avoid, except that you can't do anything about them.

I spent a 30 year career fixing and enhancing IP/TCP (and other networking code), and have avoided having anything to do with it post-retirement for precisely this reason.  Even the open-source stacks have a huge barrier to modification and improvement; anything less than a major and obvious bug-fix is unlikely to make it past various forms of gate-keeping.  Trying to improve someone else's code, when the chances that they will listen to you are so low, is very frustrating...

[peter-h]

The problem is that, for many products,  the code size goes up 10x if you add "connectivity"

Almost nobody who has the resources to develop a Product X is going to suddenly put in 10 * x to reinvent the wheel and do it right. Networking is just incredibly complex.

[gerbay]

Has anyone tried the Mongoose network library?

[Berni]

I worked with lwIP before and it seams to work well once set up.

But yeah the source code for it is not fun to read because it pulls all sorts of tricks in an attempt to be efficient.

The unreliable part that needed fixing turned out to be the HAL driver for the STM32 Ethernet controller. It got to the point where i was inserting slight delays in a few parts and that seamed to fix it. But in order to figure that out i had to dig trough a lot of the awful lwIP code while debugging to trace down the problem to the dumb HAL driver.

[voltsandjolts]

Has anyone tried the <whatever> network library?

Need to define "tried".
Fiddle with on hobby bench?
Install on many thousands of embedded devices in diverse network environments in geographically distant locales?

As the OP points out, the nuances of non-trivial networking firmware may not become apparent until there widespread scale to show up the corner cases.

[peter-h]

But yeah the source code for it is not fun to read because it pulls all sorts of tricks in an attempt to be efficient.

I am really interested in an example.

There is a zero-copy implementation for STM32 but that's a different thing, I think, and AFAIK the only proper code was done for the 32H chips (which btw have various data caching issues too). It merges the ETH Synology subsystem buffers with LWIP's buffers. I spent a lot of time doing perf testing and the copy takes negligible time (at the 10/100 level) and anyway in most embedded usage you don't need 10 megabytes/sec...

The unreliable part that needed fixing turned out to be the HAL driver for the STM32 Ethernet controller. It got to the point where i was inserting slight delays in a few parts and that seamed to fix it. But in order to figure that out i had to dig trough a lot of the awful lwIP code while debugging to trace down the problem to the dumb HAL driver.

I would be interested too - by PM if you like. I will happily send you mine. My "low level input/output" (which I think is what you mean by HAL code) was done by another guy and I did performance stuff later. I never saw the problem you describe. Did you use FreeRTOS, or any RTOS? Non-RTOS LWIP implementations are painful.

[gerbay]

It would suffice for those who have had this experience to provide information on the subject. It would be appropriate to gather information regarding experiences of all kinds.

[madires]

The problem is that, for many products,  the code size goes up 10x if you add "connectivity"

Almost nobody who has the resources to develop a Product X is going to suddenly put in 10 * x to reinvent the wheel and do it right. Networking is just incredibly complex.

Exactly! Layers upon layers of protocols stacked. And there are always changes, like mitigations for security issues, extensions, new protocols, and new sub-protocols (e.g. TCP: CUBIC, BBR). IPv6 is the prime example of how complex and therefore slow-moving networking is.

[gerbay]

Perhaps lwIP is running inside the Wiznet chip as well.

[Berni]

I would be interested too - by PM if you like. I will happily send you mine. My "low level input/output" (which I think is what you mean by HAL code) was done by another guy and I did performance stuff later. I never saw the problem you describe. Did you use FreeRTOS, or any RTOS? Non-RTOS LWIP implementations are painful.

It is a pretty old project that i haven't worked on in years, so i forgot a lot of the details.

It was indeed a non RTOS implementation. I think the issue was that some TX packets went silently missing and and it was fixed by adding just a handful of instructions worth of delay inside the HAL driver where it works with the Ethernet controllers registers. It was on a early STM32H7 chip from what i remember.

Safe to say i spent a lot more time than i planned on getting the Ethernet working properly. But once it all worked it seamed reliable. It could also saturate the 100Mbit link no problem.

[peter-h]

Perhaps lwIP is running inside the Wiznet chip as well.

I doubt it. LWIP does implement those missing features.

It was indeed a non RTOS implementation

IMHO running a networking stack non-RTOS is Grade 1 sado masochism I can't even begin to get my head around how it could work. There are so many states, error conditions, timeouts, etc. You will need to have intimate knowledge of LWIP's internal workings, to have any hope.

The 32H7 has all kinds of data caching issues.

[Berni]

Yeah i found a lot of these the hard way.

It began with "I only need to send a bit of data to a single TCP socket, how hard can it be." and only later found out what a rabbit hole actually implementing TCP/IP is.

The H7 internal architecture similarly so. I was drawn in by the impressive specs, but only when i started programing found out what a rube Goldberg machine there is inside this MCU.

It worked in the end, but it was quite a learning experience.

[peter-h]

Tell ya a funny thing.

Recently I went to implement a UDP broadcast function (google GDL90 if you want to know).

I was going to use LWIP since it is already there. But Claude knocked up a UDP transmit function, talking direct to the WFM200 chip, in seconds. Hardly any code, too.

There is also a unicast mode, where you implement DHCP and the client (which continually broadcasts its fixed IP) tells you the IP to which to "unicast" the UDP data. The difference is, ahem, that you send UDP to that address only, not to everybody. Claude did that in a few more seconds, still direct to the WFM200.

Had I used the ESP32 route (much more popular for wifi these days) it would have been the same.

So no need for LWIP. I suppose it is obvious, for UDP. Even with ETH, you could do UDP just by dropping packets into the ETH subsystem's buffers and setting the "send this" bit.

I still have a config mode where the WFM200 connects to LWIP and then I can run my HTTP server (previously run over ETH) over wifi, and amazingly it works.

One evening's work

But if I wanted TCP/IP over WIFI I would have to use LWIP for that.

[voltsandjolts]

It does not have exponential back-off retransmission.

I'm a WizNet and IP dumb-ass, so this is a question to benefit my understanding.

IIUC for a Wiznet 5500, datasheet page 39, using default values [RTR = 2000 / 200 ms, RCR = 8], retransmit intervals are
200 400 800 1600 3200 25600ms
That's exponential back-off isn't it?

[Siwastaja]

It does not have exponential back-off retransmission.

I'm a WizNet and IP dumb-ass, so this is a question to benefit my understanding.

IIUC for a Wiznet 5500, datasheet page 39, using default values [RTR = 2000 / 200 ms, RCR = 8], retransmit intervals are
200 400 800 1600 3200 25600ms
That's exponential back-off isn't it?

You might be right, the documentation itself clearly says it's not exponential (page 38 example), but then the final timeout formula suddenly behaves as if the timeouts are exponentially increasing, so 

Maybe the failure mode over slow links is something else than excess resends blocking the channel. One needs Wireshark and patience with this chip.

[SiliconWizard]

So far so good, until someone advices:
"Get a Wiznet chip, it's a TCP/IP on chip solution so that you don't need to reinvent TCP because it's hard"

Fast forward to many thousands of units on the field, and you are now debugging TCP issues, the exact thing you wanted to avoid, except that you can't do anything about them. You can't fix the stack. It's locked in silicon.

Yes - this is a very simple, yet very limited solution. I have also suggested it before, although I would also be more measured now. It may be appropriate if the requirements for network access are extremely simple, you don't want to mess with a network stack and potential issues with network access are not critical. Maybe like a simple ethernet-enabled sensor for home use (although for that, ethernet devices are probably ultra-niche compared to WiFi ones).

The solution was neat when it got released, due to simplicity and cost, but these days, even cheapish MCUs can have an Ethernet controller and you can use a well-supported TCP/IP stack on it for possibly even cheaper (if cost is a concern) than a Wiznet chip + a MCU, and get full control.

[voltsandjolts]

One needs Wireshark and patience with this chip.

Yeh, and getting a recording from a client could be tricky.
But then you can have fun like:
tcp.analysis.retransmission && ip.src == 192.168.1.100

[nctnico]

One needs Wireshark and patience with this chip.

Yeh, and getting a recording from a client could be tricky.

For these kinds of test I have a 100Mbit/s hub. Not a switch but a good old fashioned hub where all traffic ends up on all ports. A modern day alternative would be a switch with a span (monitor) port.