Alternatives to LwIP?

[betocool]

Hi all,

this is clearly meant to be a question, and I expect some opinions (many) but hopefully also some facts. Especially, please keep it civil between comments!

The motivation for this: it seems LwIP is the de-facto TCP/IP stack used for any application where data (up to 100MBps) has to be exchanged between a PC and a µC. It makes sense in my view, Ethernet is fast and has gone a long way. Micros are capable of handling data shuffling, so in theory it shouldn't be a problem. On the other hand, in the last few weeks we've seen some threads about how to get LwIP to transfer actual streams of data, fast, between a PC and a µC and the difficulties involved, especially regarding lack of information or documentation.

Are there today any viable alternatives to LwIP that are open source, documented and do work? I've seen FreeRTOS + TCP mentioned, but I have not used it myself. Is LwIP dead or is it still being developed? Does someone know? Like someone else mentioned, it seems the latest information is always from around 2002 to 2011, and any search seems to bring up the same results.

I've been able to use LwIP with several ports, both UDP and TCP simultaneously, on F7 and H7 chips, but only to send data to a PC, which then takes care of the processing. I suppose the H7 would be able to do some pre-processing if required, but I never tested that. The fact that STM32 cube integrates it makes it very comfortable as a starting point.

Cheers,

Alberto

[betocool]

I guess I kept it vague accidentally on purpose. It's mainly about an LwIP alternative for micros, and let's just limit it to 100Mbps.

For 1Gbps stuff I'd probably go to the Linux wagon.

USB, damn, that is another can of worms I'd like to keep close at this stage, basically because the beauty of Ethernet/IP/TCP is the openness of it and no need for special drivers on all of the three most common OS's.

I'm not too worried about the complexity of the stack so much as the lack of more complex examples, say, TCP streams in either direction.

I'm curious about viable open source alternatives to LwIP as of today, that's all. Pros and cons are always welcome.

Cheers,

Alberto

[voltsandjolts]

USB, damn, that is another can of worms ..... need for special drivers on all of the three most common OS's.

Well, you can use the old hack of creating a usb hid class device which will be plug n play on all three OS's, if 64kBytes/sec full duplex is enough for you. Alternatively, on windows you can make a plug n play usb device that defaults to the winusb driver (see WCID), and same device will work on linux just fine, presumably macos too. A usb device stack is much lighter than an ethernet stack, and less hassle for non-tech users.

[nctnico]

I'd choose using one of the Wiznet chips over Lwip any day! Unfortunately chip shortage forced me to use Lwip for a project last year.

My biggest problem with Lwip is that it is written quite messy and the internals are poorly documented so it is hard to figure out how the configuration parameters affect what you want.

[tellurium]

My company is currently in the process of TCP/IP stack development, with the focus of very low RAM usage. It is not currently meant for the production use, and also it is designed to work only with our own network library, https://mongoose.ws - thus, the network stack does not provide BSD socket API to the upper layer, but Mongoose API. So it is unlikely an answer for your question, but just a FYI.

Said that, you might find some quick & dirty LWIP / Zephyr comparison benchmark interesting (see table at the bottom of the page): https://github.com/cesanta/mongoose/tree/master/examples/stm32/nucleo-f746zg-baremetal

[nctnico]

My company is currently in the process of TCP/IP stack development, with the focus of very low RAM usage. It is not currently meant for the production use, and also it is designed to work only with our own network library, https://mongoose.ws - thus, the network stack does not provide BSD socket API to the upper layer, but Mongoose API. So it is unlikely an answer for your question, but just a FYI.

Interesting but I don't like the way it seems to project structs onto buffers. That is a big no-go in my book. I've seen that going wrong (silent data corruption) or lead to the compiler generating bloated code too many times.

A BSD style socket interface isn't super complicated to add.

[tellurium]

Interesting but I don't like the way it seems to project structs onto buffers. That is a big no-go in my book. I've seen that going wrong (silent data corruption) or lead to the compiler generating bloated code too many times.

A BSD style socket interface isn't super complicated to add.

Well, lwip also uses "struct projecting onto buffers", example: https://github.com/lwip-tcpip/lwip/blob/b0e347158d8db640c6891f9f31f4e6d19dca200b/src/core/ipv4/ip4.c#L479

That is a perfectly valid technique if done right. But wasn't it you who mentioned using lwip last year? How comes, considering your big no-go?

That technique is used in very many places. For example, in CMSIS and Cube headers, peripheral structs are "projected" on memory. Correct me if I am wrong, CMSIS and Cube are also a big no-go for you? Or it is time to reconsider that piece in your book?

[nctnico]

Interesting but I don't like the way it seems to project structs onto buffers. That is a big no-go in my book. I've seen that going wrong (silent data corruption) or lead to the compiler generating bloated code too many times.

A BSD style socket interface isn't super complicated to add.

Well, lwip also uses "struct projecting onto buffers", example: https://github.com/lwip-tcpip/lwip/blob/b0e347158d8db640c6891f9f31f4e6d19dca200b/src/core/ipv4/ip4.c#L479

That is a perfectly valid technique if done right. But wasn't it you who mentioned using lwip last year? How comes, considering your big no-go?

If done right... famous last words! The kicker is that at some point you'll need to shuffle the bytes around anyway. So why not do that when the packet is being constructed / deconstructed? That way you don't have problems with alignment and network order issues. Makes everything much cleaner in the end without needing to rely on pragmas to make the code work as expected. If I move the buffer pointer in your code by 1 byte, the whole thing comes crashing down without the compiler objecting to anything. Or think about the situation where the buffer is offset by 2 bytes from a VLAN header. I think that will already break your stack at several places.

I hope you see that this is absolutely not the same as structs that are mapped specifically on a single, very specific hardware platform that is also guaranteed to be aligned in memory by design! That comparison has no merit at all.

About Lwip: Chip shortage twisted my arm... otherwise I stay clear from Lwip as far as possible. In the past I've used Uip (Lwip's smaller brother from the same creator) as well and that is a horrible piece of code that needed a lot of fixing to make it work reliable.

All in all I'd very much welcome a good tcp/ip networking stack that has been written in a clean and well structured way.

[tellurium]

@nctnico, no, I do not agree with you, and I do not see the point arguing. Your arguments are pointless.

[peter-h]

I think LWIP is OK. It just took me a while to work out some of it. It's been around for years and is pretty solid. What it needs is somebody to write up a decent readable document on how it works and how to set it up.

A fair chunk of the implementation is tying it to the low level ETH hardware, and you will have to do that for any TCP/IP stack. Most of my grief was down at that end.

Another thing is that, for typical embedded applications (like, no way your 168MHz ARM32 will be able to run ETH at even remotely near 100mbps and still actually do "something" with the data ) the buffering setup (the PBUFs etc in lwipopts.h) makes sod-all difference. STM are talking about 90mbps with their latest code but they are just shoving packets out, not doing anything, and not doing anything else.

it seems the latest information is always from around 2002 to 2011

Welcome to the world of open source It is "all enthusiasm" until the coder gets into something more interesting, like the next hot project, or a hot (or even not hot) girlfriend

But it is all we have, unless you are a big company and then you can chuck 5 digits at a commercial offering, and get a warm feeling inside that it is better. Actually it may have 100x fewer users, and that is the real test of software reliability.

[westfw]

The kicker is that at some point you'll need to shuffle the bytes around anyway.

Nah.  There are some zero-copy telnet and router implementations.  (yeah, requires big-endian CPU or compiler)

So why not do that when the packet is being constructed / deconstructed?

Speed.  Efficiency.  The number of data copies in some common and popular code is ... really depressing.  (Most recently, tinyUSB on rp2040.  But lots of networking code, too.)

[peter-h]

shuffle the bytes around

I am not actually sure that LWIP isn't zero-copy on the outbound path. Somebody claimed it does not buffer (but was never seen again) and my experiments suggest this might indeed be the case. Of course down at the bottom there has to be a data copy; you could be sending out a 20k packet which is > MTU. But, experimentally, all the lwipopts.h buffer config does practically nothing to transmit speeds.

However, in the embedded context, "zero copy" isn't usually worth much. One can use DMA and that's really fast, but even memcpy() moves 4 bytes per loop and runs very fast. I was looking at some USB code which moves 64 byte packets and thought about doing that with DMA but it is pointless, for 16 x memcpy() at ~7ns per copy.

My main issues with LWIP were to do with trying to build an application (a simple web server) which uses the netconn API, instead of the commonly used sockets API. This is finished now. Most coders follow the path of least resistance and get code off github and everything there uses sockets Whereas I started with a demo netconn application...

LWIP seems solid. I have multiple boards running 24/7, under FreeRTOS.

I have heard, however, that if you want an easy life, go to Espressif and the ESP32. Reportedly, they employed somebody from the forums who seemed to know it all (I am surprised he was employable ) and he knocked the parts together and spent time to get the usual issues out. So you get a working package out of the box, with IIRC LWIP, FreeRTOS and MbedTLS. The downside? The political and commercial risk of the ESP32 being made in China, which apart from "current troubles" is gradually disintegrating on the business landscape, with a rapidly rising gangster approach to doing business. I have spent a year extracting an injection moulding tool from there, and it turns out I am not the first one.

[betocool]

Yeah, ok, this was meant more as a water cooler question. I'm also using LwIP and tweaking it until it does what it needs to in my book. I'm familiar enough at this stage, both with the raw and the netconn API that it covers all my needs.

I will give FreeRTOS TCP a try sometime, and we'll see what new headaches I get from that.

Cheers,

Alberto

[peter-h]

We looked at FreeRTOS TCP also but decided that it was quite new and LWIP had far more users, over many years.

The code size is, generally, unimportant since micros tend to have plenty of FLASH. It is the RAM that gets you first with this stuff. We found that our 150k project became 300k (out of 1MB) and our 60k RAM became 10k, just by adding MbedTLS I have since found another 10k by working out partly how LWIP does its stuff.

Unless you are severely FLASH limited, I would not waste time on another TCP stack. You can spend half your life on this crap.

[odinthenerd]

@nctnico, no, I do not agree with you, and I do not see the point arguing. Your arguments are pointless.

I see @nctnico 's point. Just as an example the mongoose library claims to be standard conforming C++ code but the std::start_lifetime_as feature which finally makes this kind of buffer casting defined behavior was only introduced recently (C++23 I believe, C++20 introduced bit_cast which introduced a copy which is usually elided in optimized code).

If you want to read the whole backstory of how such buffer casting breaks the C++ abstract machine, and hence is undefined behavior (i.e. may work but is by no means guaranteed to) there is a good explaination in the standard paper https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p0593r6.html

[fchk]

I went for NuttX. Instead of collecting all bits and pieces together from various sources you get everything in one packet: RTOS, TCP/IP Stack, USB, ...
Works decent.

fchk

[peter-h]

Never heard of it. How many use it?

Practically everything open source is unsupported so one has to go for something which is old and has lots of users.

[Perkele]

I went for NuttX. Instead of collecting all bits and pieces together from various sources you get everything in one packet: RTOS, TCP/IP Stack, USB, ...
Works decent.

fchk

Are they now using as a default something other than uIP?

[Perkele]

Never heard of it. How many use it?

Practically everything open source is unsupported so one has to go for something which is old and has lots of users.

A good alternative to FreeRTOS, if POSIX compatibility is important.
Did some projects with it almost 10 years ago. It got improved in the meantime.

[nctnico]

I went for NuttX. Instead of collecting all bits and pieces together from various sources you get everything in one packet: RTOS, TCP/IP Stack, USB, ...
Works decent.

fchk

Are they now using as a default something other than uIP?

I just peeked at the code but it looks like the author has rewritten the uIP code to be more readable and maintainable. uIP is outright horrible (and buggy).

[peter-h]

I think there is a big difference between FreeRTOS and LWIP.

Both are open source but:

FR does everything it says it does and is pretty easy to use, so no support is needed beyond reading its excellent and clear website, and I hardly ever needed to even do that. I actually think that people who build complicated RTOS based projects which rely on a very specific priority hierarchy to work, are going to regularly get bitten in the bum. They may be mostly computer science lecturers - same people who talk for hours about different kinds of heaps   Most RTOS projects can be built with maybe 2 or 3 priority levels. Actually a lot can be done with just one (IDLE) priority; I did loads of products with an RTOS I wrote which did just that.

LWIP is damn complicated, the docs on it are basically crap (most of it is describing the API in circular language which makes sense only if you are very familiar with TCP/IP APIs, and the pile of websites which document it are just lists of API calls scraped off other websites), the mailing list is practically dead, and there is little else. Its great plus is that it is > 15 years old and any version from the last few years works just fine. Well, after you spent a lot of time trying various config options, and in the end given it plenty of RAM. It likes plenty of RAM because some weird things happen if you don't do that. The main part still works, probably with no perf change, but various odd bits do weird things.

[Jeroen3]

imho LwIP is the largest tech debt in the embedded software world. It's not friendly and often hidden in wrappers when you are using it (eg: in esp32).

Making a new one is complicated and expensive so nobody bothers really. Freertos only started with it when amazon started paying for it.
The best thing that can happen is that serious demand for ipv6 in low power devices sparks a few more big players to put effort in this and make it open source.
Maybe development around Matter will help?
Like Azure RTOS NetX Duo

In the end it's sad we're not proficient enough in software to consider standards. Looking at the mechanical trade Electronics and Software is pure chaos.
It always boils down to standards.

[peter-h]

I think the main issue is this

https://www.eevblog.com/forum/programming/a-rant-about-lack-of-support-on-open-source-software-or-anything-else-actually/msg4348588/#msg4348588

We don't need yet another open source project which people give up on long before the bugs are sorted. Especially not something from a particular manufacturer, because that will get tested only in that company's products which will be a very narrow functional profile.

For LWIP, what I would like to see is a "cookbook" with examples, and a discussion of the many config options. There is a fair bit online but it is mostly circular. A function called allocate_buffer is described with "this function allocates buffers" 

I spent a lot of time writing up such a doc for my product, but it is specific to the product and is internal. There are also many loose ends because I simply did not find the information.

[nctnico]

IMHO the structure of LWIP makes it a lost cause. You know what they say about putting lipstick on a pig...

[SiliconWizard]

Is it really this bad? Never had to use it, so, really asking.