difference between LwIP's Raw API and Socket API?

[tilblackout]

Hello, I am accessing the internet using a 4G module in RNDIS mode (https://winprotocoldoc.blob.core.windows.net/productionwindowsarchives/WinArchive/%5BMS-RNDIS%5D.pdf).

In LwIP's contrib/apps/ping/ping.c, you can choose between RAW API and SOCKET API by defining the PING_USE_SOCKETS macro.

One strange thing is that when I use the SOCKET API, only the first ping is successful, and the following ones fail. However, when I use the RAW API, every ping works just fine.

Since I enabled LWIP_TCPIP_CORE_LOCKING and FreeRTOS, I'm not sure how to adapt the RAW API. I would like to understand the difference between these two APIs.

Additionally, I tried accessing the internet using an Ethernet chip, and both the RAW API and SOCKET API always worked for pinging, which made me suspect that the RNDIS protocol might only support TCP. But, given that the RAW API works fine, it makes me think the issue might be with LwIP.

I'm a bit puzzled. Does anyone have any suggestions? Thanks.

[tellurium]

I explain how things work in my https://github.com/cpq/embedded-network-programming-guide - go read it.

Long story short:
1. LWIP has 3 APIs: raw, netconn, and socket
2. raw API is a callback based API: TCP/IP stack receives data, and calls your function. Works in RTOS and non-RTOS environment, cause LWIP just calls your function, so it is simple as a pancake.
3. netconn API is just a small layer on top of raw API that implements connection data buffering for a given connection, that's all, so you can consider it as a raw API, too
4. socket API implements BSD interface, designed like files - cause that's what those UNIX people used to use. They thought, hey, we represent things in UNIX as files - kernel objects with numeric IDs in userland (file descriptors), and API like read/write/open/close. So why don't we do the same for network connections? And they did.
5. network connections they called sockets, and assigned a similar numeric ID (socket number), and did the same open/close/read/write API as for files.
6. And here goes a problem - whilst you can just read from a file and expect data synchronously, you can't from a socket. you can wait for data forever. So they thought - oh shit, we can block forever, so let's introduce O_NONBLOCK for sockets. And since we can have multiple sockets, we can read them all in a loop using O_NONBLOCK, fine! But wait, that's going to be a busy loop.... What we gonna do? Ah, let's introduce a multiplexing syscall, select/poll, which will just block until any of the sockets is ready
7. That's convoluted API, but that's how it was designed. And it is a standard now. All because read/write/poll/select can block.
8. A proper API for network connections of course is event based.
9. But everyone uses BSD API - just because, and so to do so, they MUST use RTOS, because read/write/poll/select can block, so they run a TCP/IP task in a separate task, and user network tasks in a separate task, and use RTOS queues for passing data between them - like, recv() from a socket uses RTOS queue to send data from a socket buffer in LWIP task, to your user task. And then yeah, your task is likely going to buffer the data again, just like the socket buffer did one layer below.
10. But that's exactly how socket API works.

So I think that gives you a clue about the API difference.

[peter-h]

Do a search here on LWIP. Some great posts in the past, especially from a guy called dare. I also asked a load of questions

LWIP is poorly documented and the few people who really understand it rarely post in forums.

[PinkPearl]

That is a good explanation, thank you.
Now, are you saying one MUST use an RTOS to do sockets, or that is just what people got used to?
I am running a simple TCP server on an 32bit microcontroller, no RTOS, and need to support a single socket connection and I am not sure where to start. I would imagine this is doable with timers and such without the use of any OS?

[betocool]

You can do TCP and UDP using bare metal and LWIP. UDP is easier in terms of fire-and-forget mechanisms, but you don't know if your packet actually arrived. TCP is more complicated.

But, in a nutshell, and apologies I don't have the source code with me here, for UDP you have your bare metal "send" functions, and you must define a udp_receive callback function when the micro receives data. You'll have to implement your own strategy if you need to receive large amounts, but that's unusual for a micro (unless you have some sort of bootloader function I guess). TCP is more complicated in the sense that you have to define callbacks for every state TCP transitions into, say, listening, accepting, binding, etc (point of view of a TCP server). You need to keep track of the state you're in in TCP mode (I'm guessing LWIP does it to, but I never read those states when I did that project). Also, you need to again define a TCP receive callback.

There are a few examples on the 'net to get you started, from dodgy websites and old as hell as peter-h says, but it's possible. You just need to make sure that you poll or have an interrupt when the TCP packet arrives.

TCP and UDP over an RTOS are a bit easier to implement as they use -nix like function calls. I've used it a lot with FreeRTOS in the past.

Hope it helps.

Cheers,

Alberto

[westfw]

we can block forever, so let's introduce O_NONBLOCK for sockets. And since we can have multiple sockets, we can read them all in a loop using O_NONBLOCK, fine! But wait, that's going to be a busy loop.... What we gonna do? Ah, let's introduce a multiplexing syscall, select/poll, which will just block until any of the sockets is ready

It sounds like you're blaming sockets for O_NONBLOCK, but there were many things that you could do non-blocking IO on long before TCP/IP.  (notably TTYs.)

A proper API for network connections of course is event based.

Note that the TCP philosophy is that the applications SHOULD NOT be aware of packet boundaries.  "TCP is a byte stream protocol."  (Not that everyone paid attention.  Especially Unix...)

[nctnico]

That is a good explanation, thank you.
Now, are you saying one MUST use an RTOS to do sockets, or that is just what people got used to?
I am running a simple TCP server on an 32bit microcontroller, no RTOS, and need to support a single socket connection and I am not sure where to start. I would imagine this is doable with timers and such without the use of any OS?

I vaguely recall that using sockets with LWIP requires more resources and you might need an RTOS to do pre-emptive multi-tasking. The latter could be solved by calling the LWIP maintenance function from a timer interrupt and implement mutexes by disabling interrupts momentarily. But the raw (low level) API will work but be prepared having to figure out how to configure LWIP. The documentation is very poor. Therefore I very highly recommend to use the TCP-stack-in-a-chip devices from Wiznet (W5500 or W6500 IIRC). These are much easier to work with.