Alternatives to LwIP?

[betocool]

Hi all,

this is clearly meant to be a question, and I expect some opinions (many) but hopefully also some facts. Especially, please keep it civil between comments!

The motivation for this: it seems LwIP is the de-facto TCP/IP stack used for any application where data (up to 100MBps) has to be exchanged between a PC and a µC. It makes sense in my view, Ethernet is fast and has gone a long way. Micros are capable of handling data shuffling, so in theory it shouldn't be a problem. On the other hand, in the last few weeks we've seen some threads about how to get LwIP to transfer actual streams of data, fast, between a PC and a µC and the difficulties involved, especially regarding lack of information or documentation.

Are there today any viable alternatives to LwIP that are open source, documented and do work? I've seen FreeRTOS + TCP mentioned, but I have not used it myself. Is LwIP dead or is it still being developed? Does someone know? Like someone else mentioned, it seems the latest information is always from around 2002 to 2011, and any search seems to bring up the same results.

I've been able to use LwIP with several ports, both UDP and TCP simultaneously, on F7 and H7 chips, but only to send data to a PC, which then takes care of the processing. I suppose the H7 would be able to do some pre-processing if required, but I never tested that. The fact that STM32 cube integrates it makes it very comfortable as a starting point.

Cheers,

Alberto

[betocool]

I guess I kept it vague accidentally on purpose. It's mainly about an LwIP alternative for micros, and let's just limit it to 100Mbps.

For 1Gbps stuff I'd probably go to the Linux wagon.

USB, damn, that is another can of worms I'd like to keep close at this stage, basically because the beauty of Ethernet/IP/TCP is the openness of it and no need for special drivers on all of the three most common OS's.

I'm not too worried about the complexity of the stack so much as the lack of more complex examples, say, TCP streams in either direction.

I'm curious about viable open source alternatives to LwIP as of today, that's all. Pros and cons are always welcome.

Cheers,

Alberto

[voltsandjolts]

USB, damn, that is another can of worms ..... need for special drivers on all of the three most common OS's.

Well, you can use the old hack of creating a usb hid class device which will be plug n play on all three OS's, if 64kBytes/sec full duplex is enough for you. Alternatively, on windows you can make a plug n play usb device that defaults to the winusb driver (see WCID), and same device will work on linux just fine, presumably macos too. A usb device stack is much lighter than an ethernet stack, and less hassle for non-tech users.

[nctnico]

I'd choose using one of the Wiznet chips over Lwip any day! Unfortunately chip shortage forced me to use Lwip for a project last year.

My biggest problem with Lwip is that it is written quite messy and the internals are poorly documented so it is hard to figure out how the configuration parameters affect what you want.

[tellurium]

My company is currently in the process of TCP/IP stack development, with the focus of very low RAM usage. It is not currently meant for the production use, and also it is designed to work only with our own network library, https://mongoose.ws - thus, the network stack does not provide BSD socket API to the upper layer, but Mongoose API. So it is unlikely an answer for your question, but just a FYI.

Said that, you might find some quick & dirty LWIP / Zephyr comparison benchmark interesting (see table at the bottom of the page): https://github.com/cesanta/mongoose/tree/master/examples/stm32/nucleo-f746zg-baremetal

[nctnico]

My company is currently in the process of TCP/IP stack development, with the focus of very low RAM usage. It is not currently meant for the production use, and also it is designed to work only with our own network library, https://mongoose.ws - thus, the network stack does not provide BSD socket API to the upper layer, but Mongoose API. So it is unlikely an answer for your question, but just a FYI.

Interesting but I don't like the way it seems to project structs onto buffers. That is a big no-go in my book. I've seen that going wrong (silent data corruption) or lead to the compiler generating bloated code too many times.

A BSD style socket interface isn't super complicated to add.

[tellurium]

Interesting but I don't like the way it seems to project structs onto buffers. That is a big no-go in my book. I've seen that going wrong (silent data corruption) or lead to the compiler generating bloated code too many times.

A BSD style socket interface isn't super complicated to add.

Well, lwip also uses "struct projecting onto buffers", example: https://github.com/lwip-tcpip/lwip/blob/b0e347158d8db640c6891f9f31f4e6d19dca200b/src/core/ipv4/ip4.c#L479

That is a perfectly valid technique if done right. But wasn't it you who mentioned using lwip last year? How comes, considering your big no-go?

That technique is used in very many places. For example, in CMSIS and Cube headers, peripheral structs are "projected" on memory. Correct me if I am wrong, CMSIS and Cube are also a big no-go for you? Or it is time to reconsider that piece in your book?

[nctnico]

Interesting but I don't like the way it seems to project structs onto buffers. That is a big no-go in my book. I've seen that going wrong (silent data corruption) or lead to the compiler generating bloated code too many times.

A BSD style socket interface isn't super complicated to add.

Well, lwip also uses "struct projecting onto buffers", example: https://github.com/lwip-tcpip/lwip/blob/b0e347158d8db640c6891f9f31f4e6d19dca200b/src/core/ipv4/ip4.c#L479

That is a perfectly valid technique if done right. But wasn't it you who mentioned using lwip last year? How comes, considering your big no-go?

If done right... famous last words! The kicker is that at some point you'll need to shuffle the bytes around anyway. So why not do that when the packet is being constructed / deconstructed? That way you don't have problems with alignment and network order issues. Makes everything much cleaner in the end without needing to rely on pragmas to make the code work as expected. If I move the buffer pointer in your code by 1 byte, the whole thing comes crashing down without the compiler objecting to anything. Or think about the situation where the buffer is offset by 2 bytes from a VLAN header. I think that will already break your stack at several places.

I hope you see that this is absolutely not the same as structs that are mapped specifically on a single, very specific hardware platform that is also guaranteed to be aligned in memory by design! That comparison has no merit at all.

About Lwip: Chip shortage twisted my arm... otherwise I stay clear from Lwip as far as possible. In the past I've used Uip (Lwip's smaller brother from the same creator) as well and that is a horrible piece of code that needed a lot of fixing to make it work reliable.

All in all I'd very much welcome a good tcp/ip networking stack that has been written in a clean and well structured way.

[tellurium]

@nctnico, no, I do not agree with you, and I do not see the point arguing. Your arguments are pointless.

[peter-h]

I think LWIP is OK. It just took me a while to work out some of it. It's been around for years and is pretty solid. What it needs is somebody to write up a decent readable document on how it works and how to set it up.

A fair chunk of the implementation is tying it to the low level ETH hardware, and you will have to do that for any TCP/IP stack. Most of my grief was down at that end.

Another thing is that, for typical embedded applications (like, no way your 168MHz ARM32 will be able to run ETH at even remotely near 100mbps and still actually do "something" with the data ) the buffering setup (the PBUFs etc in lwipopts.h) makes sod-all difference. STM are talking about 90mbps with their latest code but they are just shoving packets out, not doing anything, and not doing anything else.

it seems the latest information is always from around 2002 to 2011

Welcome to the world of open source It is "all enthusiasm" until the coder gets into something more interesting, like the next hot project, or a hot (or even not hot) girlfriend

But it is all we have, unless you are a big company and then you can chuck 5 digits at a commercial offering, and get a warm feeling inside that it is better. Actually it may have 100x fewer users, and that is the real test of software reliability.

[westfw]

The kicker is that at some point you'll need to shuffle the bytes around anyway.

Nah.  There are some zero-copy telnet and router implementations.  (yeah, requires big-endian CPU or compiler)

So why not do that when the packet is being constructed / deconstructed?

Speed.  Efficiency.  The number of data copies in some common and popular code is ... really depressing.  (Most recently, tinyUSB on rp2040.  But lots of networking code, too.)

[peter-h]

shuffle the bytes around

I am not actually sure that LWIP isn't zero-copy on the outbound path. Somebody claimed it does not buffer (but was never seen again) and my experiments suggest this might indeed be the case. Of course down at the bottom there has to be a data copy; you could be sending out a 20k packet which is > MTU. But, experimentally, all the lwipopts.h buffer config does practically nothing to transmit speeds.

However, in the embedded context, "zero copy" isn't usually worth much. One can use DMA and that's really fast, but even memcpy() moves 4 bytes per loop and runs very fast. I was looking at some USB code which moves 64 byte packets and thought about doing that with DMA but it is pointless, for 16 x memcpy() at ~7ns per copy.

My main issues with LWIP were to do with trying to build an application (a simple web server) which uses the netconn API, instead of the commonly used sockets API. This is finished now. Most coders follow the path of least resistance and get code off github and everything there uses sockets Whereas I started with a demo netconn application...

LWIP seems solid. I have multiple boards running 24/7, under FreeRTOS.

I have heard, however, that if you want an easy life, go to Espressif and the ESP32. Reportedly, they employed somebody from the forums who seemed to know it all (I am surprised he was employable ) and he knocked the parts together and spent time to get the usual issues out. So you get a working package out of the box, with IIRC LWIP, FreeRTOS and MbedTLS. The downside? The political and commercial risk of the ESP32 being made in China, which apart from "current troubles" is gradually disintegrating on the business landscape, with a rapidly rising gangster approach to doing business. I have spent a year extracting an injection moulding tool from there, and it turns out I am not the first one.

[betocool]

Yeah, ok, this was meant more as a water cooler question. I'm also using LwIP and tweaking it until it does what it needs to in my book. I'm familiar enough at this stage, both with the raw and the netconn API that it covers all my needs.

I will give FreeRTOS TCP a try sometime, and we'll see what new headaches I get from that.

Cheers,

Alberto

[peter-h]

We looked at FreeRTOS TCP also but decided that it was quite new and LWIP had far more users, over many years.

The code size is, generally, unimportant since micros tend to have plenty of FLASH. It is the RAM that gets you first with this stuff. We found that our 150k project became 300k (out of 1MB) and our 60k RAM became 10k, just by adding MbedTLS I have since found another 10k by working out partly how LWIP does its stuff.

Unless you are severely FLASH limited, I would not waste time on another TCP stack. You can spend half your life on this crap.

[odinthenerd]

@nctnico, no, I do not agree with you, and I do not see the point arguing. Your arguments are pointless.

I see @nctnico 's point. Just as an example the mongoose library claims to be standard conforming C++ code but the std::start_lifetime_as feature which finally makes this kind of buffer casting defined behavior was only introduced recently (C++23 I believe, C++20 introduced bit_cast which introduced a copy which is usually elided in optimized code).

If you want to read the whole backstory of how such buffer casting breaks the C++ abstract machine, and hence is undefined behavior (i.e. may work but is by no means guaranteed to) there is a good explaination in the standard paper https://www.open-std.org/jtc1/sc22/wg21/docs/papers/2020/p0593r6.html

[fchk]

I went for NuttX. Instead of collecting all bits and pieces together from various sources you get everything in one packet: RTOS, TCP/IP Stack, USB, ...
Works decent.

fchk

[peter-h]

Never heard of it. How many use it?

Practically everything open source is unsupported so one has to go for something which is old and has lots of users.

[Perkele]

I went for NuttX. Instead of collecting all bits and pieces together from various sources you get everything in one packet: RTOS, TCP/IP Stack, USB, ...
Works decent.

fchk

Are they now using as a default something other than uIP?

[Perkele]

Never heard of it. How many use it?

Practically everything open source is unsupported so one has to go for something which is old and has lots of users.

A good alternative to FreeRTOS, if POSIX compatibility is important.
Did some projects with it almost 10 years ago. It got improved in the meantime.

[nctnico]

I went for NuttX. Instead of collecting all bits and pieces together from various sources you get everything in one packet: RTOS, TCP/IP Stack, USB, ...
Works decent.

fchk

Are they now using as a default something other than uIP?

I just peeked at the code but it looks like the author has rewritten the uIP code to be more readable and maintainable. uIP is outright horrible (and buggy).

[peter-h]

I think there is a big difference between FreeRTOS and LWIP.

Both are open source but:

FR does everything it says it does and is pretty easy to use, so no support is needed beyond reading its excellent and clear website, and I hardly ever needed to even do that. I actually think that people who build complicated RTOS based projects which rely on a very specific priority hierarchy to work, are going to regularly get bitten in the bum. They may be mostly computer science lecturers - same people who talk for hours about different kinds of heaps   Most RTOS projects can be built with maybe 2 or 3 priority levels. Actually a lot can be done with just one (IDLE) priority; I did loads of products with an RTOS I wrote which did just that.

LWIP is damn complicated, the docs on it are basically crap (most of it is describing the API in circular language which makes sense only if you are very familiar with TCP/IP APIs, and the pile of websites which document it are just lists of API calls scraped off other websites), the mailing list is practically dead, and there is little else. Its great plus is that it is > 15 years old and any version from the last few years works just fine. Well, after you spent a lot of time trying various config options, and in the end given it plenty of RAM. It likes plenty of RAM because some weird things happen if you don't do that. The main part still works, probably with no perf change, but various odd bits do weird things.

[Jeroen3]

imho LwIP is the largest tech debt in the embedded software world. It's not friendly and often hidden in wrappers when you are using it (eg: in esp32).

Making a new one is complicated and expensive so nobody bothers really. Freertos only started with it when amazon started paying for it.
The best thing that can happen is that serious demand for ipv6 in low power devices sparks a few more big players to put effort in this and make it open source.
Maybe development around Matter will help?
Like Azure RTOS NetX Duo

In the end it's sad we're not proficient enough in software to consider standards. Looking at the mechanical trade Electronics and Software is pure chaos.
It always boils down to standards.

[peter-h]

I think the main issue is this

https://www.eevblog.com/forum/programming/a-rant-about-lack-of-support-on-open-source-software-or-anything-else-actually/msg4348588/#msg4348588

We don't need yet another open source project which people give up on long before the bugs are sorted. Especially not something from a particular manufacturer, because that will get tested only in that company's products which will be a very narrow functional profile.

For LWIP, what I would like to see is a "cookbook" with examples, and a discussion of the many config options. There is a fair bit online but it is mostly circular. A function called allocate_buffer is described with "this function allocates buffers" 

I spent a lot of time writing up such a doc for my product, but it is specific to the product and is internal. There are also many loose ends because I simply did not find the information.

[nctnico]

IMHO the structure of LWIP makes it a lost cause. You know what they say about putting lipstick on a pig...

[SiliconWizard]

Is it really this bad? Never had to use it, so, really asking.

[peter-h]

ISTM that a TCP/IP stack will never be a simple thing. An RTOS is really quite simple; what complicates it is fancy stuff like message passing which many people never actually use.

The problem with LWIP is what I've been saying: crap documentation of the structure and concepts. Some of the CPU manufacturer porting docs (e.g. ST's port of LWIP to the 32F4, and similar ones from Atmel, etc) attempt to explain how it works, but I struggled to understand it. It probably needs a video showing, in animation, the principal data flows, and then you could see how much to allocate to which buffers.

TCP/IP is complicated.

And with any complicated open source product your best strategy is to pick something that is old and has been widely used. LWIP is too complicated to just dive in and hack it around. That's what everybody needs to do with USB code, but LWIP is another level above that.

[coppice]

IMHO the structure of LWIP makes it a lost cause. You know what they say about putting lipstick on a pig...

You have to remember the LW (lightweight) in LWIP, and the u (micro) in uIP. These are highly compromised by being for very small devices. I believe both these stacks were originally produced for environments with no OS at all. They both do tricks to keep the RAM to an absolute minimum, so limited IP functionality can be provided by small MCUs using a minimal  external ethernet chip. Migrating everything on IP networks to encrypted protocols is really making this approach irrelevant. Nothing can be so small any more. A stack designed from the ground up for environments with a bit more breathing space, but not for environments with huge space (e.g. Linux or Windows type systems) would be a huge boon for small embedded machines.

[nctnico]

IMHO the structure of LWIP makes it a lost cause. You know what they say about putting lipstick on a pig...

You have to remember the LW (lightweight) in LWIP, and the u (micro) in uIP. These are highly compromised by being for very small devices. I believe both these stacks were originally produced for environments with no OS at all. They both do tricks to keep the RAM to an absolute minimum, so limited IP functionality can be provided by small MCUs using a minimal  external ethernet chip.

Yes and no. LWIP and uIP are both written in very convoluted ways. The author was thinking that typing less characters in C code and using gotos leads to less binary code being produced by the compiler. The reality is different. But the result is that nobody really understands how these TCP/IP stacks actually work. It is one big spaghetti. Somehow the author did manage to hype his code well enough to make a lot of people use it.

[coppice]

TCP/IP is complicated.

Its not that complicated. I implemented a complete TCP and UDP platform for DOS in about 1990, which interfaced with cheap network cards made for the NetWare market. I got some of the device driver code from somewhere else - probably some applications support material for the silicon - but I wrote the rest myself. It didn't take an extreme effort. It was used in some client server applications talking with Unix servers. Then windows and other platforms got their own TCP/IP support, it became irrelevant, and I don't even know if I have a copy of the code lying around any more.

I've known multiple people develop their own implementation of TCP/IP for various purposes. There was a time when commercial licences were so expensive per seat that it made sense for people to implement their own for modest volume usage.

[peter-h]

environments with no OS at all.

That is an option on LWIP, yes. Not sure how that works; AFAICS the only way to achieve the functionality would be some kind of state machine, triggered by e.g. you transmitting a packet. There was a guy here called dare who I think knows it.

Migrating everything on IP networks to encrypted protocols is really making this approach irrelevant. Nothing can be so small any more

MbedTLS needs about 50k for the worst case crypto suite requirements seen, but the buffer sizes can be to some extent controlled by whether you control one or both ends. If you control neither, you need 2 x 16k buffer. If you control one end, you can drop this to say 2k + 16k. If you control both ends then perhaps 2 x 2k would do. And if you control both ends then the crypto suite can also be shrunk.

stack designed from the ground up for environments with a bit more breathing space,

AFAICS LWIP does need of the order of 10-20k of RAM to work well. I found various cases where data was just discarded unless enough RAM was provided. IIRC, if you had say 2k for some TX buffers, and you wanted to send a 10k block, it didn't just break it up and send it in a load of pieces. This worked if you had >10k of buffers allocated. Or something like that; I don't recall the details and don't really want to now LWIP is probaby unusable with say 2k of RAM. Well, it might "work" but various peripheral stuff will be dead.

TLS is a huge problem. Another open source module and, like most, unsupported unless your question is sufficiently interesting to the devs. More basic questions are simply not replied to. And development of it is going in directions which are barely relevant to good IOT practice. And it is big - around 200k of code.

Its not that complicated. I implemented a complete TCP and UDP platform for DOS in about 1990, which interfaced with cheap network cards made for the NetWare market.

You are super clever

I've known multiple people develop their own implementation of TCP/IP for various purposes.

That is no doubt a lot easier when running in one product, using a super narrow range of the functionality. A little example of that is not bothering to do MTU negotiation; if you control both ends then you don't need it. And a lot of consumer router type boxes are broken in that department.

[nctnico]

environments with no OS at all.

That is an option on LWIP, yes. Not sure how that works; AFAICS the only way to achieve the functionality would be some kind of state machine,

Not really. Just call the maintenance function regulary. But only the low level, non-threaded API can be used. I had to resort to using LWIP in a project this way due to chip shortage. It does work though after some trial & error to figure out how the low level API is supposed to be used.

[coppice]

That is no doubt a lot easier when running in one product, using a super narrow range of the functionality. A little example of that is not bothering to do MTU negotiation; if you control both ends then you don't need it. And a lot of consumer router type boxes are broken in that department.

uIP can be extremely frugal in small MCUs. See https://www.rowley.co.uk/msp430/uip.htm  There is another very frugal TCP/IP implementation for the MSP430 in a TI app note http://www.ece.uah.edu/~jovanov/msp430/MSP430_easyWeb_slaa137a.pdf

[peter-h]

Well, I am no expert, but reading the limitations in https://www.rowley.co.uk/msp430/uip.htm it's obvious this is completely useless for anything general-purpose. It's OK for talking to the (specially tweaked) other end under your control. Precisely what an MSP430 gets used for (telemetry etc).

[nctnico]

uIP will talk to other hosts (including systems running Windows or Linux) just fine as long as they support shorter MTUs. But it needs quite a few bug fixes to work on a network. Been there, done that. I did add a BSD style socket interface onto uIP as well in order to use existing code and have a decent networking API. The guy who wrote the article on Rowley's website just played around with uIP a bit for an afternoon.

[coppice]

There was C source code floating around in the early 90s from one of the American universities for telnet, FTP and some other apps running over TCP/IP on DOS. The stack wasn't separated out as a reusable library. The stack and apps were muddled together. I spent some time looking through it, when it was too late to be useful to me. I seem to remember it being fairly well written, fairly compact and it didn't need an OS for support. If someone separated out the stack from that as a library you might have something pretty usable on larger MCUs. There were also things like the Trumpet Winsock code. That was never open sourced, but the code must be somewhere being of little value to its creator in its original form. There must surely be SOMETHING from the days when stacks had to be reasonably small that could be recycled today. Perhaps even an early BSD stack. Once upon a time that ran on some fairly small machines.

[dare]

I’ve been working closely with LwIP for over 10 years now.  I’ve shipped a number of products based on it, and become intimately familiar with its workings.  Based on my experience I can definitely say that LwIP is A MESS.  However, in the right hands, it is a fairly capable mess.

As an example, one of the products I shipped was a consumer sensor device based on a (now very old) SiLabs 802.15.4 radio MCU.  This part had 64K RAM and 512K flash.  On this we ran LwIP over a Thread network stack (IPv6 only) serving a fairly sophisticated application with time-critical sensing, status reporting and UI behaviors.  At peak activity, the system was capable of carrying on 3 simultaneous network conversations (one each with a cloud service, a local coordinator device, and the end-user’s mobile device) using 12 network buffers totaling 7.5K in size.  All interactions were encrypted and secured end-to-end with mutual certificate-based authentication, and the system ran on a modest-sized battery with a minimum 2 years lifetime.

My point is that you can use LwIP to make fairly sophisticated products that run with very modest resources.  But it isn’t easy.

As has been stated, LwIP’s biggest problem is its near complete lack of useful documentation.  Really, the only way to truly understand LwIP’s API is to know the associated networking standards really well, at which point you can look at the code and realize how the designer intended it to be used.

I’ve long considered writing some tutorial documentation for LwIP based on my experiences. Trouble is, when you go to document something as complicated as a network stack, irregularities in the design make the process super hard.

This is the other big problem with LwIP—in many areas its design is inconsistent, under-specified and needlessly opaque.  For example, trying to understand which functions can be called in what threading context, or what the consequences are of using one type of pbuf over another can be maddening. There are just too many options, and many are poorly named (so, there’s the "raw" API, and then there’s the API for sending raw packets?).

At this point, LwIP is in need of a thorough (API-breaking) clean-up.  There’s a lot of useful code there, but it’s just not setup for success.

[dare]

There must surely be SOMETHING from the days when stacks had to be reasonably small that could be recycled today. Perhaps even an early BSD stack. Once upon a time that ran on some fairly small machines.

Absolutely.  In fact, I’ve recently run 2.11BSD (c. mid-80’s) on my cobbled together PDP-11/73 and had it talking over the network to my linux PC. (Yes, I’m a retro-computing geek).

The problem with these old stacks is that networking protocols, and the devices that speak them, have moved on.  There are subtle behavior differences in modern networking systems that can stymie old code.  Even more modern code, like LwIP can have problems.  I remember an engineer on my team spending countless hours trying to understand why thousands of customer devices were struggling to download a firmware image (a harrowing situation for anyone with consumer devices in the field).  As I recall, it turned out to be Amazon AWS playing games with TCP retransmissions in a way that LwIP didn’t like.  Older stacks are likely to have these sorts of problems in spades.

[coppice]

At this point, LwIP is in need of a thorough (API-breaking) clean-up.  There’s a lot of useful code there, but it’s just not setup for success.

I think an important question is does one size fit all? I think we would all like a sockets like interface to the stack on every device. However, this becomes harder the more resource limited the platform is. An important dividing line is between machines with enough RAM to hold a full windows worth of packets, available for re-transmission, and those with less.

[coppice]

There must surely be SOMETHING from the days when stacks had to be reasonably small that could be recycled today. Perhaps even an early BSD stack. Once upon a time that ran on some fairly small machines.

Absolutely.  In fact, I’ve recently run 2.11BSD (c. mid-80’s) on my cobbled together PDP-11/73 and had it talking over the network to my linux PC. (Yes, I’m a retro-computing geek).

The problem with these old stacks is that networking protocols, and the devices that speak them, have moved on.  There are subtle behavior differences in modern networking systems that can stymie old code.  Even more modern code, like LwIP can have problems.  I remember an engineer on my team spending countless hours trying to understand why thousands of customer devices were struggling to download a firmware image (a harrowing situation for anyone with consumer devices in the field).  As I recall, it turned out to be Amazon AWS playing games with TCP retransmissions in a way that LwIP didn’t like.  Older stacks are likely to have these sorts of problems in spades.

Things really haven't changed much in these stacks. Most of the quirks have always been there, and never fixed. Sometimes they bite. Most of the time they don't. The biggest issue with an old stack would be the need to add IPv6 capabilities, which they won't have. One day IPv6 might become important on the internet. Right now its the only form of TCP/IP relevant for many wireless applications.

[westfw]

There was C source code floating around in the early 90s from one of the American universities for telnet, FTP and some other apps running over TCP/IP on DOS.

PC/IP from MIT.  I actually contributed a fast checksum algorithm for that, although I don't know if it was ever incorporated.  (PCs evolved pretty quick back then.)
Also KA9Q, especially in the Amateur Radio community.
(but, you know, putting Ethernet on a PC so that you could run TCP/IP was awfully expensive, back in the DOS days.  And SLIP was pretty painful, especially prior to IPHC.)

[coppice]

There was C source code floating around in the early 90s from one of the American universities for telnet, FTP and some other apps running over TCP/IP on DOS.

PC/IP from MIT.  I actually contributed a fast checksum algorithm for that, although I don't know if it was ever incorporated.  (PCs evolved pretty quick back then.)
Also KA9Q, especially in the Amateur Radio community.
(but, you know, putting Ethernet on a PC so that you could run TCP/IP was awfully expensive, back in the DOS days.  And SLIP was pretty painful, especially prior to IPHC.)

I had a feeling is was something Californian, but the name PC/IP sounds familiar, so that's probably the one I was thinking of. It was strange the way they mingled the apps and the stack, as it looked pretty easy to separate them, and have an really good general purpose library. If they'd got that out in the mid 80s and made it popular the mishmash of protocols people used in the late 80s would probably have shaken out really quickly.

[peter-h]

There were also things like the Trumpet Winsock code

I used that in the early days. It worked on Mondays and Wednesdays

As I recall, it turned out to be Amazon AWS playing games with TCP retransmissions in a way that LwIP didn’t like

Interesting. How was it solved?

The biggest issue with an old stack would be the need to add IPv6 capabilities, which they won't have. One day IPv6 might become important on the internet.

One view seems to be that IPV6 might become big on the internet (it is used, as I know well from running some public servers, but only at ~1% level) but far fewer people seem to think it is relevant to non public stuff like factory LANs. I've not implemented IPV6 in my current project; can do it if I get some huge customer who demands it (LWIP can be compiled with IPV6 enabled but lots of other code needs to be changed because e.g. IPV4 can be held in a uint32_t). It actually makes no sense for private LANs.

3 simultaneous network conversations (one each with a cloud service, a local coordinator device, and the end-user’s mobile device) using 12 network buffers totaling 7.5K in size

I think that's more buffering than a lot of people would have allocated.

just fine as long as they support shorter MTUs. But it needs quite a few bug fixes to work on a network. Been there, done that. I did add a BSD style socket interface onto uIP as well in order to use existing code and have a decent networking API

Clearly, you are super clever

I am happy with LWIP being "weird" because everything I am doing is working fine. But it would be worrying if sometime in the future I wanted to use e.g. existing TLS features and found they don't work with some server. This has already been found to be a risk e.g. out of the ~200 certificates in cacert.pem there are some which use a very old hash algorithm which "modern coders" will have removed support for (because they read on the internet that it is insecure, deprecated, and will let the russians and chinese blow up your product) and some of these certificates belong to very well known huge companies!

So, after all this, let's say you are Big Company X and you want something and you are willing to pay. What will you use, and how/why will it be better?

[westfw]

It was strange the way they mingled the apps and the stack, as it looked pretty easy to separate them, and have an really good general purpose library.

To do WHAT with, exactly?  A lot of other stuff really needed to be an OS service, and a DOS PC just didn't have enough memory to put the network code in RAM permanently.

If they'd got that out in the mid 80s and made it popular the mishmash of protocols people used in the late 80s would probably have shaken out really quickly.

Nah.  TCP/IP didn't have the services (like remote disk) that the mishmash provided in remarkably transparent ways.  It still doesn't, really.  Netbios and Appletalk eventually got layered on top of IP.  Netware, Vines, and etc went away when microsoft got their act more together.  But they didn't really provide the services popular in the IP world, and vis versa.
And there was little agreement in networking hardware, either.  Actual Ethernet cards were several hundred dollars.  (Jul 1988 byte: 3com Ethernet or Ethernet II card : $399 !) (but at least by then they had 10base2, so you weren't looking at another couple hundred for AUI cable and transceiver.)

[coppice]

It was strange the way they mingled the apps and the stack, as it looked pretty easy to separate them, and have an really good general purpose library.

To do WHAT with, exactly?  A lot of other stuff really needed to be an OS service, and a DOS PC just didn't have enough memory to put the network code in RAM permanently.

I developed my own TCP/IP stack as a library to put client server apps on DOS machines, talking with Unix machines. If I had seen the source code for PC/IP first I would have pulled out the actual stack from that as a library, cleaned up the API, and ended up with something very similar to my own code. I would then probably have used it with multic (a cooperative multi-tasking library for DOS), which is what I did with my own stack library. RAM was never an issue, as by the late 80s most DOS machines had at least 256k of RAM. That was more than enough to run clients, and servers with only a few client connexions.

[peter-h]

This is the basic part of my lwipopts.h file. I am allocating, roughly speaking, 6k + 9k = 15k, plus some odds little bits.

A lot of time was spent on this!

Code: [Select]

/*
* MEM_SIZE: the size of the heap memory. This is a statically allocated block. You can find it
* in the .map file as the symbol ram_heap and you can see how much of this RAM gets used.
* If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.
* If MEMP_MEM_MALLOC=1 (which is not reliable) this greatly expands and needs 16k+.
* Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.
* This value also limits the biggest block size sent out by netconn_write. With a MEM_SIZE
* of 6k, the biggest block netconn_write (and probably socket write) will send out is 4k.
* This setting is mostly related to outgoing data.
*/

#define MEM_SIZE                (6*1024)

// MEMP_ structures. Their sizes have been determined experimentally, by
// increasing them and seeing free RAM changing.

/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
   sends a lot of data out of ROM (or other static memory), this
   should be set high. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_PBUF           20 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One
   per active UDP "connection". */
#define MEMP_NUM_UDP_PCB        6 // each 1 is 32 bytes of static RAM

/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCP
   connections. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_TCP_PCB        20 // each 1 is 145 bytes of static RAM

//NC: Have more sockets available. Is set to 4 in opt.h
#define MEMP_NUM_NETCONN    10

/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP
   connections. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_TCP_PCB_LISTEN 20 // each 1 is 28 bytes of static RAM

/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP
   segments. */
// Was 8; increased to 16 as it improves ETHSER reliability when running
// HTTP server
#define MEMP_NUM_TCP_SEG        16 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active
   timeouts. */
#define MEMP_NUM_SYS_TIMEOUT    10 // each 1 is 16 bytes of static RAM


/* ---------- Pbuf dynamically allocated buffer blocks  ----------
*
* These settings make little difference to performance, which is dominated by
* the low_level_input poll period. These PBUFs relate directly to the netconn API netbufs.
*
* PBUF_POOL_SIZE: the number of buffers in the pbuf pool.
* Statically allocated, so each increment frees up 1.5k of RAM. Any value under 4 slows
* down the data rate a lot.
* 12/2/23 PH 4 -> 6 fixes TLS blocking of various RTOS tasks (5 almost does).
* The most this can go to with TLS still having enough free RAM for its 48k block is ~9.
*/

#define PBUF_POOL_SIZE           6

/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool.
** It is better to use MTU sized bufs than 8 x 512 because in say EditGetData() you are **
** much more likely to get the whole file header in the first packet. With 512          **
** byte packets the CRLFCRLF marker is only ~64 bytes before the end of the pkt         **
** Same issue in UploadGetData where we need to get the whole header in.                **
** However the above issue has been largely solved with the 200ms wait in the http svr  **
* The +2 is to get a multiple of 4 bytes so the PBUFs are 4-aligned (for memcpy_fast())
* Probably the +2 is not needed because the PBUFs are 4-aligned anyway in the LWIP code.
*/

#define PBUF_POOL_BUFSIZE  1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + 2

Some may find the TLS comments amusing...

[Siwastaja]

The legend says that TCP/IP stack is "so complex" that no one should even think about doing that from scratch. This is the advice you get everywhere, from everybody: "don't do it".

While probably somewhat true, the result of that attitude is we are stuck with a few implementations which everybody agrees suck more or less nevertheless. And despite the ratio of users : available implementations, there still is no decent documentation. Pretty weird.

I think this attitude needs to be challenged. Is the complexity of TCP/IP exaggerated after all? If it takes weeks and weeks of learning into how to use LwIP, why not try your wings with a full custom implementation? Obviously this is not for everyone. But for those smart guys; according to peter-h many smart people exist.

Code reuse is more cult than science. Best thing with code is, code is easy to discard and new code can be written. Working around poor and old designs is a lot of work.

[peter-h]

And despite the ratio of users : available implementations, there still is no decent documentation. Pretty weird.

Actually, to me, this is completely obvious: most bugs are found and fixed in corporate scenarios and the people doing it are just leechers. They google all over the net to find free code, they get it working, the boss is super impressed, but 99% of them never post their fixes to help others. Some are prohibited per company policy from posting, too...

Practically all fixes are posted by hobbyists, which is why almost everything on github is (putting it politely) "work in progress"

But for those smart guys; according to peter-h many smart people exist.

Yes, and you are extremely clever So, go ahead and write one!

In reality, any non-hobbyist has a very limited time to get something up and running. They don't actually want to understand it.

On the project I am working on (nearly finished now) the part-timer who set it up (Cube IDE + ST ports of LWIP, FreeRTOS, FatFS, USB CDC+MSC, etc) had a web server running very soon. It looked awfully impressive. LOOK we have tcp/ip and a web server showing a WEBSITE!!! Well, he was way more clever than me... Of course I was super impressed. Little did I know that this demo was basically meaningless and much more time, god knows how many man-months had to be sunk into it to have something useful.

And clearly a lot of people have been up this road. They buy some ST DISCO-XX demo board, project 1 flashes an LED, tick that, project 2 writes text on the big LCD, tick that, project 3 is the HTTP server demo, which works but is near-impossible to mod to do something useful so they post questions on the ST forum which nobody replies to because a) there is almost nobody there and b) there is no way to help these people.

In a corporate environment you get paid all the way down the line. It can be a good job. One guy, yesterday, was saying what a great job software is. He's produced almost nothing for 10 years.

[Siwastaja]

I loathe that kind of tech demos because they are completely unsustainable. The result is almost always a complete crash of the business. 99.9% of crowd funded startups are like that (I think the worst Kickstarter times are already behind): build a demo using Arduino/Rapsberry Pi/existing code, without any knowledge into either hardware or software design, or even idea how long it takes. Promise a quick delivery of a product based on the demo only to find out the demo does not scale and can't be manufactured. The company goes bust and no one truly understands what happened.

This is why I work bottom-up by demonstrating something by doing it from scratch, it's the polar opposite. Now this have had downsides, too, because adding functionality and features then might take longer than expected as you either need to keep doing all by yourself, or introduce reuse of existing work at later stage (when it's harder to integrate), but it is not an order-of-magnitude mismanagement thing, and I (usually) survive.

Best results are achieved when there is a team who can both build from-scratch solutions efficiently and reuse existing parts without breaking sweat. But the viewpoint of reuse is highly overvalued (in CVs, for example) whereas being able to solve problems from simple fundamentals is undervalued. Former is important, but latter is absolutely essential.

Regarding skills and "cleverness", being able to write things from scratch or reuse things made by others are IMHO equally challenging, neither is easy. Again, ease of reuse is a typical management misassumption while management fears novel solutions would be too difficult.

[coppice]

The legend says that TCP/IP stack is "so complex" that no one should even think about doing that from scratch. This is the advice you get everywhere, from everybody: "don't do it".

While probably somewhat true, the result of that attitude is we are stuck with a few implementations which everybody agrees suck more or less nevertheless. And despite the ratio of users : available implementations, there still is no decent documentation. Pretty weird.

I think this attitude needs to be challenged. Is the complexity of TCP/IP exaggerated after all? If it takes weeks and weeks of learning into how to use LwIP, why not try your wings with a full custom implementation? Obviously this is not for everyone. But for those smart guys; according to peter-h many smart people exist.

Code reuse is more cult than science. Best thing with code is, code is easy to discard and new code can be written. Working around poor and old designs is a lot of work.

In the 80s TCP/IP was hard. People didn't have enough background to make sense of what is needed, so the learning time could be huge. These days, if you've been working with networks in detail for some time, it isn't that hard. It not a job for the less able people out there, but for a capable person its an approachable job. It can get harder if you really want to crank the performance, but we are only talking about small scale systems exchanging modest amounts of data. Skills in compact design might be more interesting that skills in networking.

[coppice]

Does anyone know where I can find a copy of the source code for PC/IP? I wanted to take a fresh look at it, but I can't find it by Googling. I can only find a copy of the manual from 1986.

[nctnico]

The legend says that TCP/IP stack is "so complex" that no one should even think about doing that from scratch. This is the advice you get everywhere, from everybody: "don't do it".

While probably somewhat true, the result of that attitude is we are stuck with a few implementations which everybody agrees suck more or less nevertheless. And despite the ratio of users : available implementations, there still is no decent documentation. Pretty weird.

I think this attitude needs to be challenged. Is the complexity of TCP/IP exaggerated after all? If it takes weeks and weeks of learning into how to use LwIP, why not try your wings with a full custom implementation? Obviously this is not for everyone. But for those smart guys; according to peter-h many smart people exist.

Code reuse is more cult than science. Best thing with code is, code is easy to discard and new code can be written. Working around poor and old designs is a lot of work.

In the 80s TCP/IP was hard. People didn't have enough background to make sense of what is needed, so the learning time could be huge. These days, if you've been working with networks in detail for some time, it isn't that hard. It not a job for the less able people out there, but for a capable person its an approachable job. It can get harder if you really want to crank the performance, but we are only talking about small scale systems exchanging modest amounts of data. Skills in compact design might be more interesting that skills in networking.

Compact design leads to hard to maintain & understand code which is exactly why the current 'popular' stacks are so hard to use. Where it comes to writing your own stack: it is not just tcp/ip but you'd need to support ARP & ping and UDP ethernet protocols. On top of that there has to be support for basic things like DHCP and DNS. But the most difficult part is to make the stack interoperable with all kinds of network devices. LWIP has been exposed to a broad range of devices and fine tuned over the many years it has been out there. So the likelyhood it doesn't play nice with a device is small. From my own experience with implementing protocols, the most time consuming part is dealing with devices that respond differently compared to what you expend.

[dare]

As I recall, it turned out to be Amazon AWS playing games with TCP retransmissions in a way that LwIP didn’t like

Interesting. How was it solved?

IIRC we made a change to the server code to avoid the behavior.  Since devices were already in the field, this was the best solution.

3 simultaneous network conversations (one each with a cloud service, a local coordinator device, and the end-user’s mobile device) using 12 network buffers totaling 7.5K in size

I think that's more buffering than a lot of people would have allocated.

Well, at least in this situation, their devices would have failed or performed poorly.  As I'm sure you know, running with very small amounts of network buffer space requires careful analysis of application behavior, and lots of testing.  We spent a lot of time optimizing this.

This is where a deep understanding of network protocols is essential.  And unfortunately this is not something that a better network stack can make up for (although a poorly implemented stack can obviously make this harder).

[SiliconWizard]

Regarding skills and "cleverness", being able to write things from scratch or reuse things made by others are IMHO equally challenging, neither is easy. Again, ease of reuse is a typical management misassumption while management fears novel solutions would be too difficult.

I couldn't agree more. Both are equally challenging - at least if you do things half "right". You'll face different issues.

Engineering management is tough. Managers tend to fall into the silver bullet trap, because it looks easier... on them.
As a manager, if you pick one well-known, industry-standard solution, even if it's suboptimal or even plain sucks, then you'll be safe personally if something goes wrong. No one in upper management will be able to blame your choices. Now if you decide to let the team develop their own solution and things go sour - then it'll be 100% on your shoulders, and both your team and you upper management will point fingers at you. Which option do you think managers tend to pick unless they are ready to find a new job?

Other than that, sometimes reusing some third-party code is not an option depending on context. If you're working in an environment in which you need to follow strict processes, a given code style/code guidelines and have to *validate* your software, then you often are not too keen on just reusing third-party stuff - it would make your life miserable - unless it comes with full guarantees and a lot of paperwork, which pretty much excludes anything but expensive "certified" commercial solutions.

[coppice]

Regarding skills and "cleverness", being able to write things from scratch or reuse things made by others are IMHO equally challenging, neither is easy. Again, ease of reuse is a typical management misassumption while management fears novel solutions would be too difficult.

Doing anything really well is not easy. Even if you are doing something that's not very challenging there will be other smart people doing a similar not very challenging thing, and you need to outperform them to win. Reuse should speed things up, but doesn't necessarily make high skill levels less of an advantage in producing a competitive result.

[dare]

I think we would all like a sockets like interface to the stack on every device.

Well, no so much for me.   After many decades of using the sockets API I’ve come to consider it a miserable and error-prone interface for anything but the most simplistic of applications.   As soon as you want to do something even slightly sophisticated—say, like making a UDP server that will respond to any client, including those using link-local addresses—then the arcana has to come out.  Even things that should be simple, like: Which sock_addr structure should I use? What should I do with the result from write()? or When should I call close() vs shutdown()? are subtle and confusing to newcomers.  Sockets perpetuates the illusion that networking is easy while quickly leading devs into territory that can get them in trouble.  I can’t count the number of times I’ve heard a dev say "I copied this example sockets code [because understanding how to implement it from scratch is just too hard] and now it isn’t working [because I don’t understand that my situation isn’t as simplistic as the sunny-day sockets example would have me believe]"

On top of this, there is no one definition of the sockets interface.  Linux, Android, MacOS, iOS—all have subtle differences.  And as far as I can see none of the embedded stacks have anywhere near full support for interface (e.g. the IP_PKTINFO / IPV6_RECVPKTINFO socket options).  LwIP’s socket interface sure doesn’t.

Circling back to the topic at hand, neither of the alternate APIs in LwIP are particularly good (netconn or "raw").  We choose to create a portable veneer over the raw API because it was the most efficient of the three and gave us greater control over how asynchronous I/O was handled which was important for our power-constrained devices.

[westfw]

we are stuck with a few implementations which everybody agrees suck more or less

Well, you should note that most of the complaints are of the form "The documentation is poor and it's hard to use", rather than "When I plugged in my device my whole LAN melted down, and then I got a call from the people who run the server at the other side of the country that I had crashed their system."  That sort of stuff used to actually happen.

And then there's 100+ pages of "requirements for Internet Hosts, Communications layers" that a vendor MIGHT be required to certify that their implementation meets...

[coppice]

I think we would all like a sockets like interface to the stack on every device.

Well, no so much for me.   After many decades of using the sockets API I’ve come to consider it a miserable and error-prone interface for anything but the most simplistic of applications.   As soon as you want to do something even slightly sophisticated—say, like making a UDP server that will respond to any client, including those using link-local addresses—then the arcana has to come out.  Even things that should be simple, like: Which sock_addr structure should I use? What should I do with the result from write()? or When should I call close() vs shutdown()? are subtle and confusing to newcomers.  Sockets perpetuates the illusion that networking is easy while quickly leading devs into territory that can get them in trouble.  I can’t count the number of times I’ve heard a dev say "I copied this example sockets code [because understanding how to implement it from scratch is just too hard] and now it isn’t working [because I don’t understand that my situation isn’t as simplistic as the sunny-day sockets example would have me believe]"

On top of this, there is no one definition of the sockets interface.  Linux, Android, MacOS, iOS—all have subtle differences.  And as far as I can see none of the embedded stacks have anywhere near full support for interface (e.g. the IP_PKTINFO / IPV6_RECVPKTINFO socket options).  LwIP’s socket interface sure doesn’t.

Circling back to the topic at hand, neither of the alternate APIs in LwIP are particularly good (netconn or "raw").  We choose to create a portable veneer over the raw API because it was the most efficient of the three and gave us greater control over how asynchronous I/O was handled which was important for our power-constrained devices.

I agree. The basics of sockets are simple, but the details are extremely confusing when you first encounter them. Error handling can be downright funky. The big benefits are a lot of people are familiar with sockets programming, and most code you might want to reuse will have been written for sockets.

[nctnico]

we are stuck with a few implementations which everybody agrees suck more or less

Well, you should note that most of the complaints are of the form "The documentation is poor and it's hard to use", rather than "When I plugged in my device my whole LAN melted down, and then I got a call from the people who run the server at the other side of the country that I had crashed their system."  That sort of stuff used to actually happen.

Yeah. I recall crashing a high-reliability, core infrastructure grade telephone switch by sending it a malformed packet over an ISDN link  Only a power cycle of the interface would make it recover.

[Jeroen3]

My recommendation after attempting a few projects LwIP was that if your project requires more then only a few UDP packets you should use a linux based single board computer or soc (eg: rpi cm4 or wiznet).
The current trend with encryption everywhere only makes this more relevant.

[peter-h]

That is true for "knocking up working hardware" quickly, and the chinese do that a lot, but it is most suitable for short-life products, after which you get constant production hassles. And unless the CPU is fast, with lots of RAM, you end up with a painfully slow box.

Whereas if you build an ARM32 box with LWIP, you should get 10 (and possibly 20, with some "last time buy" stock grabbing) years out of the design. I've got 25 years out of a H8/323 box (with considerable LTB grabbing).

TLS is something else, with crypto suite fashions changing faster than a *****'s knickers But if you make a suitable field upgrade provision, then you will have control for

- as many years as the crypto can be run in software (already a 32F417, not that old, has no "useful" hardware crypto apart from AES)
- as many years as somebody in your company is able to maintain a machine for rebuilding the firmware (that is what kills many products in reality, as people leave, etc)

I am no expert but it's already clear to me that you have to pick your market.

- retail IOT, talking to a 3rd party server -> you will get shafted unless you build a "PC"
- retail IOT, talking to your private server-> you can do what you like (even IPV4 will be around for ever)
- industrial IOT -> talking to your private server -> got to do crypto to impress the customer but basically whatever you want in terms of details
- industrial IOT -> talking to anything else -> got to build a "PC"

My box is #3, with dire warnings to not use it for the others