Alternatives to LwIP?

[peter-h]

ISTM that a TCP/IP stack will never be a simple thing. An RTOS is really quite simple; what complicates it is fancy stuff like message passing which many people never actually use.

The problem with LWIP is what I've been saying: crap documentation of the structure and concepts. Some of the CPU manufacturer porting docs (e.g. ST's port of LWIP to the 32F4, and similar ones from Atmel, etc) attempt to explain how it works, but I struggled to understand it. It probably needs a video showing, in animation, the principal data flows, and then you could see how much to allocate to which buffers.

TCP/IP is complicated.

And with any complicated open source product your best strategy is to pick something that is old and has been widely used. LWIP is too complicated to just dive in and hack it around. That's what everybody needs to do with USB code, but LWIP is another level above that.

[coppice]

IMHO the structure of LWIP makes it a lost cause. You know what they say about putting lipstick on a pig...

You have to remember the LW (lightweight) in LWIP, and the u (micro) in uIP. These are highly compromised by being for very small devices. I believe both these stacks were originally produced for environments with no OS at all. They both do tricks to keep the RAM to an absolute minimum, so limited IP functionality can be provided by small MCUs using a minimal  external ethernet chip. Migrating everything on IP networks to encrypted protocols is really making this approach irrelevant. Nothing can be so small any more. A stack designed from the ground up for environments with a bit more breathing space, but not for environments with huge space (e.g. Linux or Windows type systems) would be a huge boon for small embedded machines.

[nctnico]

IMHO the structure of LWIP makes it a lost cause. You know what they say about putting lipstick on a pig...

You have to remember the LW (lightweight) in LWIP, and the u (micro) in uIP. These are highly compromised by being for very small devices. I believe both these stacks were originally produced for environments with no OS at all. They both do tricks to keep the RAM to an absolute minimum, so limited IP functionality can be provided by small MCUs using a minimal  external ethernet chip.

Yes and no. LWIP and uIP are both written in very convoluted ways. The author was thinking that typing less characters in C code and using gotos leads to less binary code being produced by the compiler. The reality is different. But the result is that nobody really understands how these TCP/IP stacks actually work. It is one big spaghetti. Somehow the author did manage to hype his code well enough to make a lot of people use it.

[coppice]

TCP/IP is complicated.

Its not that complicated. I implemented a complete TCP and UDP platform for DOS in about 1990, which interfaced with cheap network cards made for the NetWare market. I got some of the device driver code from somewhere else - probably some applications support material for the silicon - but I wrote the rest myself. It didn't take an extreme effort. It was used in some client server applications talking with Unix servers. Then windows and other platforms got their own TCP/IP support, it became irrelevant, and I don't even know if I have a copy of the code lying around any more.

I've known multiple people develop their own implementation of TCP/IP for various purposes. There was a time when commercial licences were so expensive per seat that it made sense for people to implement their own for modest volume usage.

[peter-h]

environments with no OS at all.

That is an option on LWIP, yes. Not sure how that works; AFAICS the only way to achieve the functionality would be some kind of state machine, triggered by e.g. you transmitting a packet. There was a guy here called dare who I think knows it.

Migrating everything on IP networks to encrypted protocols is really making this approach irrelevant. Nothing can be so small any more

MbedTLS needs about 50k for the worst case crypto suite requirements seen, but the buffer sizes can be to some extent controlled by whether you control one or both ends. If you control neither, you need 2 x 16k buffer. If you control one end, you can drop this to say 2k + 16k. If you control both ends then perhaps 2 x 2k would do. And if you control both ends then the crypto suite can also be shrunk.

stack designed from the ground up for environments with a bit more breathing space,

AFAICS LWIP does need of the order of 10-20k of RAM to work well. I found various cases where data was just discarded unless enough RAM was provided. IIRC, if you had say 2k for some TX buffers, and you wanted to send a 10k block, it didn't just break it up and send it in a load of pieces. This worked if you had >10k of buffers allocated. Or something like that; I don't recall the details and don't really want to now LWIP is probaby unusable with say 2k of RAM. Well, it might "work" but various peripheral stuff will be dead.

TLS is a huge problem. Another open source module and, like most, unsupported unless your question is sufficiently interesting to the devs. More basic questions are simply not replied to. And development of it is going in directions which are barely relevant to good IOT practice. And it is big - around 200k of code.

Its not that complicated. I implemented a complete TCP and UDP platform for DOS in about 1990, which interfaced with cheap network cards made for the NetWare market.

You are super clever

I've known multiple people develop their own implementation of TCP/IP for various purposes.

That is no doubt a lot easier when running in one product, using a super narrow range of the functionality. A little example of that is not bothering to do MTU negotiation; if you control both ends then you don't need it. And a lot of consumer router type boxes are broken in that department.

[nctnico]

environments with no OS at all.

That is an option on LWIP, yes. Not sure how that works; AFAICS the only way to achieve the functionality would be some kind of state machine,

Not really. Just call the maintenance function regulary. But only the low level, non-threaded API can be used. I had to resort to using LWIP in a project this way due to chip shortage. It does work though after some trial & error to figure out how the low level API is supposed to be used.

[coppice]

That is no doubt a lot easier when running in one product, using a super narrow range of the functionality. A little example of that is not bothering to do MTU negotiation; if you control both ends then you don't need it. And a lot of consumer router type boxes are broken in that department.

uIP can be extremely frugal in small MCUs. See https://www.rowley.co.uk/msp430/uip.htm  There is another very frugal TCP/IP implementation for the MSP430 in a TI app note http://www.ece.uah.edu/~jovanov/msp430/MSP430_easyWeb_slaa137a.pdf

[peter-h]

Well, I am no expert, but reading the limitations in https://www.rowley.co.uk/msp430/uip.htm it's obvious this is completely useless for anything general-purpose. It's OK for talking to the (specially tweaked) other end under your control. Precisely what an MSP430 gets used for (telemetry etc).

[nctnico]

uIP will talk to other hosts (including systems running Windows or Linux) just fine as long as they support shorter MTUs. But it needs quite a few bug fixes to work on a network. Been there, done that. I did add a BSD style socket interface onto uIP as well in order to use existing code and have a decent networking API. The guy who wrote the article on Rowley's website just played around with uIP a bit for an afternoon.

[coppice]

There was C source code floating around in the early 90s from one of the American universities for telnet, FTP and some other apps running over TCP/IP on DOS. The stack wasn't separated out as a reusable library. The stack and apps were muddled together. I spent some time looking through it, when it was too late to be useful to me. I seem to remember it being fairly well written, fairly compact and it didn't need an OS for support. If someone separated out the stack from that as a library you might have something pretty usable on larger MCUs. There were also things like the Trumpet Winsock code. That was never open sourced, but the code must be somewhere being of little value to its creator in its original form. There must surely be SOMETHING from the days when stacks had to be reasonably small that could be recycled today. Perhaps even an early BSD stack. Once upon a time that ran on some fairly small machines.

[dare]

I’ve been working closely with LwIP for over 10 years now.  I’ve shipped a number of products based on it, and become intimately familiar with its workings.  Based on my experience I can definitely say that LwIP is A MESS.  However, in the right hands, it is a fairly capable mess.

As an example, one of the products I shipped was a consumer sensor device based on a (now very old) SiLabs 802.15.4 radio MCU.  This part had 64K RAM and 512K flash.  On this we ran LwIP over a Thread network stack (IPv6 only) serving a fairly sophisticated application with time-critical sensing, status reporting and UI behaviors.  At peak activity, the system was capable of carrying on 3 simultaneous network conversations (one each with a cloud service, a local coordinator device, and the end-user’s mobile device) using 12 network buffers totaling 7.5K in size.  All interactions were encrypted and secured end-to-end with mutual certificate-based authentication, and the system ran on a modest-sized battery with a minimum 2 years lifetime.

My point is that you can use LwIP to make fairly sophisticated products that run with very modest resources.  But it isn’t easy.

As has been stated, LwIP’s biggest problem is its near complete lack of useful documentation.  Really, the only way to truly understand LwIP’s API is to know the associated networking standards really well, at which point you can look at the code and realize how the designer intended it to be used.

I’ve long considered writing some tutorial documentation for LwIP based on my experiences. Trouble is, when you go to document something as complicated as a network stack, irregularities in the design make the process super hard.

This is the other big problem with LwIP—in many areas its design is inconsistent, under-specified and needlessly opaque.  For example, trying to understand which functions can be called in what threading context, or what the consequences are of using one type of pbuf over another can be maddening. There are just too many options, and many are poorly named (so, there’s the "raw" API, and then there’s the API for sending raw packets?).

At this point, LwIP is in need of a thorough (API-breaking) clean-up.  There’s a lot of useful code there, but it’s just not setup for success.

[dare]

There must surely be SOMETHING from the days when stacks had to be reasonably small that could be recycled today. Perhaps even an early BSD stack. Once upon a time that ran on some fairly small machines.

Absolutely.  In fact, I’ve recently run 2.11BSD (c. mid-80’s) on my cobbled together PDP-11/73 and had it talking over the network to my linux PC. (Yes, I’m a retro-computing geek).

The problem with these old stacks is that networking protocols, and the devices that speak them, have moved on.  There are subtle behavior differences in modern networking systems that can stymie old code.  Even more modern code, like LwIP can have problems.  I remember an engineer on my team spending countless hours trying to understand why thousands of customer devices were struggling to download a firmware image (a harrowing situation for anyone with consumer devices in the field).  As I recall, it turned out to be Amazon AWS playing games with TCP retransmissions in a way that LwIP didn’t like.  Older stacks are likely to have these sorts of problems in spades.

[coppice]

At this point, LwIP is in need of a thorough (API-breaking) clean-up.  There’s a lot of useful code there, but it’s just not setup for success.

I think an important question is does one size fit all? I think we would all like a sockets like interface to the stack on every device. However, this becomes harder the more resource limited the platform is. An important dividing line is between machines with enough RAM to hold a full windows worth of packets, available for re-transmission, and those with less.

[coppice]

There must surely be SOMETHING from the days when stacks had to be reasonably small that could be recycled today. Perhaps even an early BSD stack. Once upon a time that ran on some fairly small machines.

Absolutely.  In fact, I’ve recently run 2.11BSD (c. mid-80’s) on my cobbled together PDP-11/73 and had it talking over the network to my linux PC. (Yes, I’m a retro-computing geek).

The problem with these old stacks is that networking protocols, and the devices that speak them, have moved on.  There are subtle behavior differences in modern networking systems that can stymie old code.  Even more modern code, like LwIP can have problems.  I remember an engineer on my team spending countless hours trying to understand why thousands of customer devices were struggling to download a firmware image (a harrowing situation for anyone with consumer devices in the field).  As I recall, it turned out to be Amazon AWS playing games with TCP retransmissions in a way that LwIP didn’t like.  Older stacks are likely to have these sorts of problems in spades.

Things really haven't changed much in these stacks. Most of the quirks have always been there, and never fixed. Sometimes they bite. Most of the time they don't. The biggest issue with an old stack would be the need to add IPv6 capabilities, which they won't have. One day IPv6 might become important on the internet. Right now its the only form of TCP/IP relevant for many wireless applications.

[westfw]

There was C source code floating around in the early 90s from one of the American universities for telnet, FTP and some other apps running over TCP/IP on DOS.

PC/IP from MIT.  I actually contributed a fast checksum algorithm for that, although I don't know if it was ever incorporated.  (PCs evolved pretty quick back then.)
Also KA9Q, especially in the Amateur Radio community.
(but, you know, putting Ethernet on a PC so that you could run TCP/IP was awfully expensive, back in the DOS days.  And SLIP was pretty painful, especially prior to IPHC.)

[coppice]

There was C source code floating around in the early 90s from one of the American universities for telnet, FTP and some other apps running over TCP/IP on DOS.

PC/IP from MIT.  I actually contributed a fast checksum algorithm for that, although I don't know if it was ever incorporated.  (PCs evolved pretty quick back then.)
Also KA9Q, especially in the Amateur Radio community.
(but, you know, putting Ethernet on a PC so that you could run TCP/IP was awfully expensive, back in the DOS days.  And SLIP was pretty painful, especially prior to IPHC.)

I had a feeling is was something Californian, but the name PC/IP sounds familiar, so that's probably the one I was thinking of. It was strange the way they mingled the apps and the stack, as it looked pretty easy to separate them, and have an really good general purpose library. If they'd got that out in the mid 80s and made it popular the mishmash of protocols people used in the late 80s would probably have shaken out really quickly.

[peter-h]

There were also things like the Trumpet Winsock code

I used that in the early days. It worked on Mondays and Wednesdays

As I recall, it turned out to be Amazon AWS playing games with TCP retransmissions in a way that LwIP didn’t like

Interesting. How was it solved?

The biggest issue with an old stack would be the need to add IPv6 capabilities, which they won't have. One day IPv6 might become important on the internet.

One view seems to be that IPV6 might become big on the internet (it is used, as I know well from running some public servers, but only at ~1% level) but far fewer people seem to think it is relevant to non public stuff like factory LANs. I've not implemented IPV6 in my current project; can do it if I get some huge customer who demands it (LWIP can be compiled with IPV6 enabled but lots of other code needs to be changed because e.g. IPV4 can be held in a uint32_t). It actually makes no sense for private LANs.

3 simultaneous network conversations (one each with a cloud service, a local coordinator device, and the end-user’s mobile device) using 12 network buffers totaling 7.5K in size

I think that's more buffering than a lot of people would have allocated.

just fine as long as they support shorter MTUs. But it needs quite a few bug fixes to work on a network. Been there, done that. I did add a BSD style socket interface onto uIP as well in order to use existing code and have a decent networking API

Clearly, you are super clever

I am happy with LWIP being "weird" because everything I am doing is working fine. But it would be worrying if sometime in the future I wanted to use e.g. existing TLS features and found they don't work with some server. This has already been found to be a risk e.g. out of the ~200 certificates in cacert.pem there are some which use a very old hash algorithm which "modern coders" will have removed support for (because they read on the internet that it is insecure, deprecated, and will let the russians and chinese blow up your product) and some of these certificates belong to very well known huge companies!

So, after all this, let's say you are Big Company X and you want something and you are willing to pay. What will you use, and how/why will it be better?

[westfw]

It was strange the way they mingled the apps and the stack, as it looked pretty easy to separate them, and have an really good general purpose library.

To do WHAT with, exactly?  A lot of other stuff really needed to be an OS service, and a DOS PC just didn't have enough memory to put the network code in RAM permanently.

If they'd got that out in the mid 80s and made it popular the mishmash of protocols people used in the late 80s would probably have shaken out really quickly.

Nah.  TCP/IP didn't have the services (like remote disk) that the mishmash provided in remarkably transparent ways.  It still doesn't, really.  Netbios and Appletalk eventually got layered on top of IP.  Netware, Vines, and etc went away when microsoft got their act more together.  But they didn't really provide the services popular in the IP world, and vis versa.
And there was little agreement in networking hardware, either.  Actual Ethernet cards were several hundred dollars.  (Jul 1988 byte: 3com Ethernet or Ethernet II card : $399 !) (but at least by then they had 10base2, so you weren't looking at another couple hundred for AUI cable and transceiver.)

[coppice]

It was strange the way they mingled the apps and the stack, as it looked pretty easy to separate them, and have an really good general purpose library.

To do WHAT with, exactly?  A lot of other stuff really needed to be an OS service, and a DOS PC just didn't have enough memory to put the network code in RAM permanently.

I developed my own TCP/IP stack as a library to put client server apps on DOS machines, talking with Unix machines. If I had seen the source code for PC/IP first I would have pulled out the actual stack from that as a library, cleaned up the API, and ended up with something very similar to my own code. I would then probably have used it with multic (a cooperative multi-tasking library for DOS), which is what I did with my own stack library. RAM was never an issue, as by the late 80s most DOS machines had at least 256k of RAM. That was more than enough to run clients, and servers with only a few client connexions.

[peter-h]

This is the basic part of my lwipopts.h file. I am allocating, roughly speaking, 6k + 9k = 15k, plus some odds little bits.

A lot of time was spent on this!

Code: [Select]

/*
* MEM_SIZE: the size of the heap memory. This is a statically allocated block. You can find it
* in the .map file as the symbol ram_heap and you can see how much of this RAM gets used.
* If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.
* If MEMP_MEM_MALLOC=1 (which is not reliable) this greatly expands and needs 16k+.
* Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.
* This value also limits the biggest block size sent out by netconn_write. With a MEM_SIZE
* of 6k, the biggest block netconn_write (and probably socket write) will send out is 4k.
* This setting is mostly related to outgoing data.
*/

#define MEM_SIZE                (6*1024)

// MEMP_ structures. Their sizes have been determined experimentally, by
// increasing them and seeing free RAM changing.

/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
   sends a lot of data out of ROM (or other static memory), this
   should be set high. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_PBUF           20 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One
   per active UDP "connection". */
#define MEMP_NUM_UDP_PCB        6 // each 1 is 32 bytes of static RAM

/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCP
   connections. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_TCP_PCB        20 // each 1 is 145 bytes of static RAM

//NC: Have more sockets available. Is set to 4 in opt.h
#define MEMP_NUM_NETCONN    10

/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP
   connections. */
//NC: Increased to 20 for ethser
#define MEMP_NUM_TCP_PCB_LISTEN 20 // each 1 is 28 bytes of static RAM

/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP
   segments. */
// Was 8; increased to 16 as it improves ETHSER reliability when running
// HTTP server
#define MEMP_NUM_TCP_SEG        16 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active
   timeouts. */
#define MEMP_NUM_SYS_TIMEOUT    10 // each 1 is 16 bytes of static RAM


/* ---------- Pbuf dynamically allocated buffer blocks  ----------
*
* These settings make little difference to performance, which is dominated by
* the low_level_input poll period. These PBUFs relate directly to the netconn API netbufs.
*
* PBUF_POOL_SIZE: the number of buffers in the pbuf pool.
* Statically allocated, so each increment frees up 1.5k of RAM. Any value under 4 slows
* down the data rate a lot.
* 12/2/23 PH 4 -> 6 fixes TLS blocking of various RTOS tasks (5 almost does).
* The most this can go to with TLS still having enough free RAM for its 48k block is ~9.
*/

#define PBUF_POOL_SIZE           6

/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool.
** It is better to use MTU sized bufs than 8 x 512 because in say EditGetData() you are **
** much more likely to get the whole file header in the first packet. With 512          **
** byte packets the CRLFCRLF marker is only ~64 bytes before the end of the pkt         **
** Same issue in UploadGetData where we need to get the whole header in.                **
** However the above issue has been largely solved with the 200ms wait in the http svr  **
* The +2 is to get a multiple of 4 bytes so the PBUFs are 4-aligned (for memcpy_fast())
* Probably the +2 is not needed because the PBUFs are 4-aligned anyway in the LWIP code.
*/

#define PBUF_POOL_BUFSIZE  1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + 2

Some may find the TLS comments amusing...

[Siwastaja]

The legend says that TCP/IP stack is "so complex" that no one should even think about doing that from scratch. This is the advice you get everywhere, from everybody: "don't do it".

While probably somewhat true, the result of that attitude is we are stuck with a few implementations which everybody agrees suck more or less nevertheless. And despite the ratio of users : available implementations, there still is no decent documentation. Pretty weird.

I think this attitude needs to be challenged. Is the complexity of TCP/IP exaggerated after all? If it takes weeks and weeks of learning into how to use LwIP, why not try your wings with a full custom implementation? Obviously this is not for everyone. But for those smart guys; according to peter-h many smart people exist.

Code reuse is more cult than science. Best thing with code is, code is easy to discard and new code can be written. Working around poor and old designs is a lot of work.

[peter-h]

And despite the ratio of users : available implementations, there still is no decent documentation. Pretty weird.

Actually, to me, this is completely obvious: most bugs are found and fixed in corporate scenarios and the people doing it are just leechers. They google all over the net to find free code, they get it working, the boss is super impressed, but 99% of them never post their fixes to help others. Some are prohibited per company policy from posting, too...

Practically all fixes are posted by hobbyists, which is why almost everything on github is (putting it politely) "work in progress"

But for those smart guys; according to peter-h many smart people exist.

Yes, and you are extremely clever So, go ahead and write one!

In reality, any non-hobbyist has a very limited time to get something up and running. They don't actually want to understand it.

On the project I am working on (nearly finished now) the part-timer who set it up (Cube IDE + ST ports of LWIP, FreeRTOS, FatFS, USB CDC+MSC, etc) had a web server running very soon. It looked awfully impressive. LOOK we have tcp/ip and a web server showing a WEBSITE!!! Well, he was way more clever than me... Of course I was super impressed. Little did I know that this demo was basically meaningless and much more time, god knows how many man-months had to be sunk into it to have something useful.

And clearly a lot of people have been up this road. They buy some ST DISCO-XX demo board, project 1 flashes an LED, tick that, project 2 writes text on the big LCD, tick that, project 3 is the HTTP server demo, which works but is near-impossible to mod to do something useful so they post questions on the ST forum which nobody replies to because a) there is almost nobody there and b) there is no way to help these people.

In a corporate environment you get paid all the way down the line. It can be a good job. One guy, yesterday, was saying what a great job software is. He's produced almost nothing for 10 years.

[Siwastaja]

I loathe that kind of tech demos because they are completely unsustainable. The result is almost always a complete crash of the business. 99.9% of crowd funded startups are like that (I think the worst Kickstarter times are already behind): build a demo using Arduino/Rapsberry Pi/existing code, without any knowledge into either hardware or software design, or even idea how long it takes. Promise a quick delivery of a product based on the demo only to find out the demo does not scale and can't be manufactured. The company goes bust and no one truly understands what happened.

This is why I work bottom-up by demonstrating something by doing it from scratch, it's the polar opposite. Now this have had downsides, too, because adding functionality and features then might take longer than expected as you either need to keep doing all by yourself, or introduce reuse of existing work at later stage (when it's harder to integrate), but it is not an order-of-magnitude mismanagement thing, and I (usually) survive.

Best results are achieved when there is a team who can both build from-scratch solutions efficiently and reuse existing parts without breaking sweat. But the viewpoint of reuse is highly overvalued (in CVs, for example) whereas being able to solve problems from simple fundamentals is undervalued. Former is important, but latter is absolutely essential.

Regarding skills and "cleverness", being able to write things from scratch or reuse things made by others are IMHO equally challenging, neither is easy. Again, ease of reuse is a typical management misassumption while management fears novel solutions would be too difficult.

[coppice]

The legend says that TCP/IP stack is "so complex" that no one should even think about doing that from scratch. This is the advice you get everywhere, from everybody: "don't do it".

While probably somewhat true, the result of that attitude is we are stuck with a few implementations which everybody agrees suck more or less nevertheless. And despite the ratio of users : available implementations, there still is no decent documentation. Pretty weird.

I think this attitude needs to be challenged. Is the complexity of TCP/IP exaggerated after all? If it takes weeks and weeks of learning into how to use LwIP, why not try your wings with a full custom implementation? Obviously this is not for everyone. But for those smart guys; according to peter-h many smart people exist.

Code reuse is more cult than science. Best thing with code is, code is easy to discard and new code can be written. Working around poor and old designs is a lot of work.

In the 80s TCP/IP was hard. People didn't have enough background to make sense of what is needed, so the learning time could be huge. These days, if you've been working with networks in detail for some time, it isn't that hard. It not a job for the less able people out there, but for a capable person its an approachable job. It can get harder if you really want to crank the performance, but we are only talking about small scale systems exchanging modest amounts of data. Skills in compact design might be more interesting that skills in networking.

[coppice]

Does anyone know where I can find a copy of the source code for PC/IP? I wanted to take a fresh look at it, but I can't find it by Googling. I can only find a copy of the manual from 1986.