Any experts on LWIP and the netconn API? (not looking for free; can pay)

• Any experts on LWIP and the netconn API? (not looking for free; can pay)
  Posted by peter-h on 02 Aug, 2022 10:13
• 
  Hi All,
  
  I am implementing a simple web server. The code is all over the place. It came with ST Cube IDE, and an identical version with same comments is here
  https://github.com/particle-iot/lwip/blob/master/contrib/apps/httpserver/httpserver-netconn.c
  
  However, they assume incoming data (or the part of it of interest) is contained in just the one buffer. The code is like this
  
  >  err_t res = netconn_recv(conn, &inbuf);
  >  if ((res == ERR_OK) && (inbuf != NULL))
  >  {
  >    if (netconn_err(conn) == ERR_OK)
  >    {
  >      netbuf_data(inbuf, (void**)&buf, &buflen);
  >      /* Is this an HTTP GET command? (only check the first 5 chars, since
  >      there are other formats for GET, and we're keeping it very simple )*/
  >      if ((buflen >=5) && (strncmp(buf, "GET /", 5) == 0))
  >      {
  
  netconn_recv appears to be a blocking function which tells you some data has arrived.
  
  netconn_err is a macro.
  
  netbuf_data reads in a buffer. There is no way to know how big this buffer is, but you are told how much data is in there.
  
  The problem I have is that I am extending this "server" (on which I have spent at least a week full-time and which works solidly) with a file upload feature, and that obviously involves getting a bit more incoming data than just a few strings. There are almost no examples on the web of using netconn for receiving a data stream - in this case up to 2MB.
  
  I am wondering if the buffer pointed to by netbuf_data is actually one of these, defined in lwipopts.h
  
  >/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
  >#define PBUF_POOL_SIZE           8
  >
  >/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
  >// If small, set it so that say 3 of them hold a complete largest packet (1500+stuff)
  >#define PBUF_POOL_BUFSIZE        512
  
  which would mean no more than 512 bytes. It kind of makes sense since all the netconn code refers to pbufs.
  
  There is also a function netbuf_next and I don't know how it fits into this scheme.
  
  I have a suspicion this scheme involves a linked list of buffers (up to 8 if my suspicion above is right; also why the double indirection in the buffer address??) and you somehow have to step through them if you want to receive a lot of data.
  
  Basically I need some code which implements data receive, with a timeout. The timeout I can do in various ways, with a FreeRTOS timer being the most "modern" way. The buffer from which I will be writing the eventual file is 512 bytes long. I can write less but can't write more.
  
  On a related topic, am I right in saying that if
  
  LWIP_TCPIP_CORE_LOCKING=1
  then the LWIP API is thread safe for raw, netconn and sockets, whereas if
  LWIP_TCPIP_CORE_LOCKING=0
  then the LWIP API is thread safe for netconn and sockets only? There is a vast range of disagreement online about this topic.
  
  Thank you in advance for any comments.
  
  Peter
  
• #1 Reply
  Posted by betocool on 03 Aug, 2022 00:14
• I don't know if this is what you're looking for, but I've been able to successfully stream audio data over Ethernet using an STM32H7, LWIP and FreeRTOS. The architecture and speed may differ, but I believe the API is mostly the same.
  
  The STM32 acts as a server, when it gets a connection request, it streams data over TCP. It's using a StreamBuffer to store the incoming data, and reading from it as soon as there are so-many bytes available.
  https://github.com/betocool-prog/Audio_Analyser_FW_OS/blob/master/tasks/Src/controller.c
  The server starts at line 188.
  
  To get the throughput I needed, just under 10Mbits, I modified the buffer sizes on lwipopts.h as well, maybe that also helps you. The H7 has tons of available memory, so it's a no brainer to increase the LWIP buffer sizes and have larger packets. That's always worked well for me in different projects.
  
  If you're stuck somewhere let me know, happy to help you if I can.
  
  Cheers,
  
  Alberto
• #2 Reply
  Posted by peter-h on 03 Aug, 2022 08:42
• Thank you Alberto!
  
  Your code is great but it is transmitting data. That is no problem. I am happily transmitting data e.g. this is a file download function on my server:
  
  

  Code: [Select]

  // Download a file that was clicked on.

static void Download(struct netconn *conn, char *name)
{

#define DBUF 512

uint8_t pagebuf[DBUF];
uint32_t offset=0;
UINT numread=0;
FIL fp;
FILINFO fno;
FRESULT fr;
//err_t netconn_err;
static const char montab[36+1]="JanFebMarAprMayJunJulAugSepOctNovDec";
char datebuf[40];

// Send standard file download header
netconn_write(conn, DOWNLOAD_HEADER, strlen((char*)DOWNLOAD_HEADER), NETCONN_COPY);

// We have the filename, now get its parameters (if it still exists, which is virtually certain)

fr = f_stat( name, &fno );

    if ( fr == FR_OK )
    {

// send filesize, as "Content-Length: 1910916\r\n"
strcpy((char*)pagebuf, "Content-Length: ");
itoa(fno.fsize,(char*)&pagebuf[16],10); // place size after "Content-Length: "
strcat((char*)pagebuf,"\r\n");
netconn_write(conn, pagebuf, strlen((char*)pagebuf), NETCONN_COPY);

// Send date/time, as "Date: Mon, 21 Oct 2015 07:28:00 UTC\r\n"
// To save the hassle of calculating the day of week (which isn't stored in the directory anyway)
// we use Mon in case the client is validating the presence of the day string ;)
// For jpegs, FF and Edge extract this from exif, interestingly, if header is missing.
// Otherwise browsers appear to ignore this, and same with Last-Modified.

strcpy((char*)pagebuf, "Date: Mon, ");
int monidx = 3*(((fno.fdate >> 5) & 15)-1);
snprintf(datebuf,sizeof(datebuf),"%2u %c%c%c %4u %02u:%02u:%02u%c",
fno.fdate & 31,
montab[0+monidx],
montab[1+monidx],
montab[2+monidx],
(fno.fdate >> 9) + 1980,
fno.ftime >> 11,
(fno.ftime >> 5) & 63,
2*(fno.ftime & 0x1f),
0);
strcat((char *)pagebuf, datebuf);
strcat((char *)pagebuf, "\r\n\r\n");   // 2xCRLF is the last thing before the binary file data
netconn_write(conn, pagebuf, strlen((char*)pagebuf), NETCONN_COPY);

// send file

if (f_open(&fp, name, FA_READ | FA_OPEN_EXISTING) == FR_OK)
{
do
{
if ( f_read(&fp, pagebuf, DBUF, &numread) != FR_OK )
{
numread=0;
break;
}
netconn_write(conn, pagebuf, numread, NETCONN_COPY);
offset+=DBUF;
}
while (numread==DBUF);

f_close(&fp);
}
}

        // The browser has now disconnected after the file transfer, anyway...

}


  Transmit is easy. You just repeatedly call netconn_write().
  
  However I need to read (from the client browser).
  
  I found some examples online e.g. this lot
  https://cpp.hotexamples.com/examples/-/-/netbuf_next/cpp-netbuf_next-function-examples.html
  but if it works it probably does only by accident because the API doc for e.g. netbuf_data shows this
  https://lwip.fandom.com/wiki/Netbuf_data
  and says it can return discontinuous data
  
  For example, if you get a netbuf object as a result to a call to "netconn_recv()", you could use this function to access the data you received. Note that the memory may not be contiguous. You might have to use "netbuf_first()" and "netbuf_next()" to scan through all the memory.
  Alternatively you could use "netbuf_copy" to get all of the data at once.
  
  I reckon this code is working because it is trivial to receive just the first packet! And a lot of people's code does just that. They are sending one packet. For example DHCP, DNS, NTP are done with just one packet. Hence the hacked server I posted above i.e.
  https://github.com/particle-iot/lwip/blob/master/contrib/apps/httpserver/httpserver-netconn.c
  uses that trick, to get the start of the client message ("GET /filename" etc). That code is exactly identical to what is supplied in ST's LWIP port, and that is what I started with. It works fine until one tries to receive lots of data...
  
  In practice, with ETH being "pretty fast", that first packet will almost certainly be a whole MTU by the time you discover you got it. Or perhaps one whole LWIP PBUF, say 512 bytes depending on your lwipopts.h - me reading between the lines.
  
  The trick is how to stream reception. There are examples using the socket API but for historical reasons (this project is nearly finished) I am trying to get it done using the netconn API.
  
  I am reading data into a 512 byte buffer and I suspect netbuf_copy() will be the key. And indeed it does work - for the first packet again This is my code for getting the first packet, extracting the filename and filesize from it (again the big assumption is that the first ~300 bytes of the client header is in that packet, but doing it properly is quite a bit more work)
  
  

  Code: [Select]

  static void EditGetData(struct netconn *conn, struct netbuf *inbuf)
{

struct netconn *newconn;
u16_t buflen;
uint16_t bytesread=0;
char* buf;
FILINFO fno;
err_t err = ERR_RST; // some initial value ("connection closed")
char filebuf[512];
char filename[20];
char filesizebuf[20];
uint32_t filesize=0;

// Fill up filebuf with the first load of data. This assumes
// - the relevant part of the client's header fits into filebuf
// - enough data has actually arrived (might need a short delay)
bytesread=netbuf_copy( inbuf, filebuf, sizeof(filebuf) );

// Extract filename
strncpy(filename,1+memchr(filebuf,'=',sizeof(filebuf)),sizeof(filename));

// Extract data size
strncpy(filesizebuf, 15+strnstr(filebuf,"Content-Length:",sizeof(filebuf)),sizeof(filesizebuf));
filesize=atoi(filesizebuf);

// If we got a duff size then filename is probably duff too, so do nothing
if ( (filesize!=0) && (filesize<(2048*1024)) && (bytesread!=0) )
{


  It is after the above that I need to set up a loop which gets the next packet, and repeats until done.
  
  I found this
  https://doc.ecoscentric.com/ref/lwip-api-sequential-netbuf-next.html
  but that shows netbuf_data being used with what looks like the wrong 2nd parameter (it is **data i.e. a pointer to a pointer, allowing some sort of multi pointer structure pointing to a sequence of packets which may not be contiguous) and together with netbuf_next which again I don't get, reading the API.
  
  Working at LAN speeds it is easy for this stuff to work by luck.
  
  Experimentation suggests that interleaving netconn_recv() and netconn_copy() may work but netconn_recv() is blocking, which makes timeouts difficult. One can still get a timer callback from FreeRTOS but there is no way to do an orderly return from that function, deallocating its stack as normal.
  
  Seemingly, with the netconn API, the normal sequence is a loop with
  
  netconn_recv - blocks until >=1 byte arrives
  netbuf_copy - copy up to x bytes into your own buffer (avoids complexity of discontinuous buffers)
  
  and repeat. But netconn_recv is blocking, so it will hang on the last packet. And I have found the above doesn't work. You get the same packet every time.
• #3 Reply
  Posted by betocool on 03 Aug, 2022 12:11
• Ahhh... I see... apologies for that! I must've skimmed over the part where the streaming in was the problem... Unfortunately I haven't done any data streaming into a micro! It's usually the micro that gathers stuff and sends it upstream.
  
  I just had a quick look at netconn_recv()... a couple of ideas here that I would try out.
  
  Increase the receive buffer size to fit a full (or so) Ethernet frame, 14xx bytes or so, that should decrease the frequency at which you receive data. Also, have two netbuf ptrs at the ready to swap them out after each one arrives.
  Can you put the netconn_recv function in a while(ERR_OK == err) loop, within the task that receives the data? I suspect the task processing and storing the data should have high priority.
  
  The thing about TCP, it should be able to auto-throttle speed, that is supposed to be the selling point of TCP over UDP, no (little) loss of data. So even if you're too slow to read it, you wouldn't receive the next set of packets until you ACK the first set.
  
  Do you connect/establish a new TCP connection each time you want to write to the micro? I've seen cases where establishing a TCP connection and leaving it idle turns into a mess because the PC usually tends to time out and the micro is still holding the connection for ransom. I would connect -> send -> close the connection and leave it open only while streaming.
  
  Another thing, the PC should send small packets. I usually use Python, my go-to way would be to grab 1.2 kbyte chunks or so of data to send with each tcp_write(data) instead of trying to do that with the complete buffer. I'm sure between python apps that'd work ok, but dividing it in pieces also gives you better control of how and when to send the new chunk.
  
  This is an anecdotal observation with wireshark. I noticed that when using TCP to a micro and you're streaming into the PC the micro ALWAYS sends two packets before waiting for the ACK from the PC.
  
  Oh, yes, and Wireshark is your friend, once you filter out all the cr@p on the network you can get a pretty good insight on what's happening on the TCP comms.
  
  If all of the above sounds like I'm repeating stuff someone else already said, it's not my intention... I have not read the complete threads... it's stuff that has worked for me in general, after a lot of tweaking.
  
  I'd be interested to know how this plays out for you, good luck!
  
  Cheers,
  
  Alberto
• #4 Reply
  Posted by peter-h on 03 Aug, 2022 13:55
• I could post the whole http server source (it isn't big ) but basically what happens is this:
  
  

  Code: [Select]

  /* Create a new TCP connection handle */
conn = netconn_new(NETCONN_TCP);
 
if (conn!= NULL)
{
/* Bind to port 80 (unless modified in config.ini) with default IP address */
err = netconn_bind(conn, NULL, http_server_port);
   
if (err == ERR_OK)
{
// Put the connection into LISTEN state
netconn_listen(conn);
 
while(1)
{
// Accept the connection after netconn_listen
// This function reports if there is some rx data (non blocking)
err_t res = netconn_accept(conn, &newconn);

if (res == ERR_OK)
{
// Something arrived from the client
debug_thread("http incoming connection");

// Respond to the client request
http_server_serve(newconn);

// Delete the connection
netconn_delete(newconn);
}
}
}
else
{
debug_thread("cannot bind netconn");
}
}
else
{
debug_thread("cannot create netconn");
}
}

  and there you can see how the connection is set up. Then, if something arrives, you go to http_server_serve() and in there, in brief, is
  
  

  Code: [Select]

  void http_server_serve(struct netconn *conn)
{
  struct netbuf *inbuf = NULL;
  char* buf;
  u16_t buflen;
  FILINFO fno;
 
  // Read the data from the port. BLOCKING if nothing there (should never happen
  // because we get here only if netconn_accept() reported there is something).
  // We assume the request (the part we care about) is in one netbuf.

  err_t res = netconn_recv(conn, &inbuf);
 
  if ((res == ERR_OK) && (inbuf != NULL))
  {
    if (netconn_err(conn) == ERR_OK)
    {
      // this function can return a list of linked buffers, but we ignore that
      netbuf_data(inbuf, (void**)&buf, &buflen);

      /* Is this an HTTP GET command? (only check the first 5 chars, since
      there are other formats for GET, and we're keeping it very simple )*/

      if ((buflen >=5) && (strncmp(buf, "GET /", 5) == 0))
      {

      // Check if this is a return string from the login page
      // in the form of GET /kde485.html?un=username&pw=password some-junk.....
      if( (strncmp(buf, "GET /kde485.html?un=", 20) == 0) && (!logged_in) )
      {
      LoginCheck(conn, buf, conn->pcb.ip->remote_ip.addr);
      }

  so you can see that once we got some data, we assume it is a big enough chunk to go looking for the "GET /..." stuff (requests from the browser).
  
  Whether that is always true, I would question (been doing comms for a few decades). I think the way TCP/IP works is that the source sends a packet whole (obviously - that is the API) but along the way it can get split up. Obviously if you send 10k that will definitely get split up into at least 6 or so packets, due to the 1500 max MTU. But even if you sent one 1k packet, that might also get split up, I think, if you send data slowly enough. I might be wrong on that. Does anyone know? How would Modbus over TCP/IP work then?? But it is a fact that the web nowadays simply does not work at all on GPRS (~2kbytes/sec) and only just barely works on 3G (say 10-100k bytes/sec), presumably because various timeouts chuck the line away. Although a lot of that will be due to massive amounts of data exchanged under HTTPS.
  
  Anyway, this works. I need to get ~300 bytes of this first packet for it to work. If this was a problem, say accessing this http server over GPRS, or where the link was delivering 30 byte packets, it is fairly simple to change, to get enough data to get the filename and the filesize, and dump data until the CRLFCRLF has been received (the PUT header). The real data starts after that.
  
  So the function which actually does the stuff, in the above case LoginCheck(), is entered with the connection already set up, and with some data having definitely arrived (we found the GET... in there). So it calls netconn_recv() to get this data into inbuf. Note that inbuf is not a buffer you own; it is inside LWIP, and this may be handy if you don't have any RAM to play with. But I have, 512 bytes at least.
  
  Then it calls netbuf_data() and the API description for that is, to me, quite unclear. Lots of google as usual but it could return a list of non contiguous linked buffers. Yet I have not found any examples online where somebody is picking their way through these multiple buffers. I would speculate that generally the data you want will be in the first of these, which is how all the example code online works (by luck). OTOH, the API for netbuf_copy() clearly states that that will return contiguous data, so clearly that is the better function to use. And it does work.
  
  Here is where I get stuck. This is one of the functions like LoginCheck
  
  

  Code: [Select]

  static void EditGetData(struct netconn *conn, struct netbuf *inbuf)
{

struct netconn *newconn;
u16_t buflen;
uint16_t bytes_read=0;
char* buf;
FILINFO fno;
err_t err = ERR_RST; // some initial value ("connection closed")
char filebuf[512];
char filename[20];
char filesizebuf[20];
uint32_t filesize=0;

// Fill up filebuf with the first load of data. This assumes
// - the relevant part of the client's header fits into filebuf
// - enough data has actually arrived (might need a short delay)
bytes_read=netbuf_copy( inbuf, filebuf, sizeof(filebuf) );

// Extract filename
strncpy(filename,1+memchr(filebuf,'=',sizeof(filebuf)),sizeof(filename));
filename[strcspn(filename," ")]=0; // null-terminate filename

// Extract data size
strncpy(filesizebuf, 15+strnstr(filebuf,"Content-Length:",sizeof(filebuf)),sizeof(filesizebuf));
filesize=atoi(filesizebuf);

// If we got a duff size then filename is probably duff too, so do nothing
if ( (filesize!=0) && (filesize<(2048*1024)) && (bytes_read!=0) )
{

/* receive data until the other host closes the connection */
do
{

netbuf_data(inbuf, (void**)&buf, &buflen);
if (buflen!=0)
{
netconn_recv(conn, &inbuf);
    bytes_read=netbuf_copy( inbuf, filebuf, sizeof(filebuf) );
        debug_thread_printf("size=%3d %3d",buflen, bytes_read);
        osDelay(100);
    }

}
while (bytes_read==buflen);

    /* the connection has now been closed by the other end, so we close our end */
    netconn_close(conn);

    // deallocate the buffer
    netbuf_delete(inbuf);

  and it works, up to the do-while loop, and that is junk and gets stuck. In that loop I need to read data from the client browser, until the byte count has been exhausted, and (this bit is easy) write it to a file in the filesystem.
  
  It uses netbuf_copy() which, per API, delivers a complete contiguous buffer, and you specify the max #bytes the retrieve (your buffer size, obviously).
  
  And that works, for this first buffer, but I just can't find any examples of working code after that.
  
  Normally this sort of thing is done by a loop, and testing for incoming data (non blocking), if there is some you get it (can be blocking because it will never block), process it by writing it to a file, reload a timeout timer, and go back to check if more arrived, and if the timer expired before the byte count has been reached you delete the temp file and return some error, otherwise you return 200 OK. Handshaking is taken care of by TCP/IP. The error conditions will be a) the timeout (say 5 secs), or b) an error condition from any of the netconn* functions.
  
  This might work
  https://doc.ecoscentric.com/ref/lwip-api-sequential-netconn-recv.html
  but netconn_recv() is blocking so this is completely useless.
  
  I have other code in the project but it all uses the socket API (btw a google shows it was lifted from the internet like most things today ) so it isn't useful to me.
  
  I've put this up on freelancer.com
  
• #5 Reply
  Posted by peter-h on 05 Aug, 2022 19:22
• Amazingly, nobody came up with anything here, on the ST forum (which is mostly useless anyway due to vast traffic and zero mfg participation, but has great SEO which reliably yields hundreds of hits from desperate people getting no help), on the LWIP mailing list (from which I was privately told to not bother), and on freelancer.com where a few claimed to know LWIP and netconn and even Cube but (as is common there) no meaningful comms could be established.
  
  In the end, a friend with a lot of unix expertise, and a good analytical brain, worked it out, and it works. I will post the details here in case somebody is googling and looking for the solution.
  
  You start with something like
  
  

  Code: [Select]

  void initialise(void)
{

        struct netconn *conn, *newconn;
err_t err;

debug_thread_printf("Starting HTTP server");

/* Create a new TCP connection handle */
conn = netconn_new(NETCONN_TCP);
 
if (conn!= NULL)
{
/* Bind to port 80 with default IP address */
err = netconn_bind(conn, NULL, 80);
   
if (err == ERR_OK)
{
// Put the connection into LISTEN state
netconn_listen(conn);
 
while(1)
{
// Accept the connection after netconn_listen
// This function reports if there is some rx data (non blocking)
err_t res = netconn_accept(conn, &newconn);

if (res == ERR_OK)
{
// Something arrived from the client
debug_thread("http incoming connection");

// Respond to the client request
http_server_serve(newconn);

// Delete the connection
netconn_delete(newconn);
}
}
}
else
{
debug_thread_printf("cannot bind netconn");
}
}
else
{
debug_thread_printf("cannot create netconn");
}

  
  then you have
  
  

  Code: [Select]

  // Some data has arrived.

void http_server_serve(struct netconn *conn)
{
  struct netbuf *inbuf = NULL;
  char* buf;
  u16_t buflen;
 
  // Read the data from the port. BLOCKING if nothing there (should never happen
  // because we get here only if netconn_accept() reported there is something).
  // We assume the request (the part we care about) is in one netbuf.

  err_t res = netconn_recv(conn, &inbuf);
 
  if ((res == ERR_OK) && (inbuf != NULL))
  {
    if (netconn_err(conn) == ERR_OK)
    {
      // this function returns one buffer, out of a possible number of them, according to lwipopts.h settings
      // #define PBUF_POOL_SIZE           4
      // #define PBUF_POOL_BUFSIZE        1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN
 

      netbuf_data(inbuf, (void**)&buf, &buflen);

      /* Is this an HTTP GET command? (only check the first 5 chars, since
      there are other formats for GET, and we're keeping it very simple )*/

      if ((buflen >=5) && (strncmp(buf, "GET /", 5) == 0))
      {

      // Check if this is a return string from the login page
      // in the form of GET /xxxxxx.html?un=username&pw=password some-junk.....
      if( (strncmp(buf, "GET /xxxxxx.html?un=", 20) == 0) && (!logged_in) )
      {
                  // this function does more digging around buf, and in this case picks up the client IP too for checking
      LoginCheck(conn, buf, conn->pcb.ip->remote_ip.addr);
      }

   // and so on for other client requests

      }
    }
  }

  /* Close the connection (server closes in HTTP) */
  netconn_close(conn);
 
  /* Delete the buffer (netconn_recv gives us ownership,
   so we have to make sure to deallocate the buffer) */
  netbuf_delete(inbuf);
}


  
  The above has picked up ONE buffer only and we have been parsing it for some client requests. This is ok for one-packet usage, and most examples online cover only that, but is no good if you want to receive loads of data. For that, you do the following, which shows reception of data and writing it to a file. This function contains code irrelevant to netconn; it deals with a PUT... request from the client (which has to be generated by a JS script) and with the fact that the first packet we got contains not only the PUT... header (with filename and filesize) which is CRLFCRLF terminated as standard, but also with some file data.
  
  

  Code: [Select]

  /*
 *
 * This function receives the PUT data from a JS-submitted textarea box.
 * The PUT etc is generated by a JS script EDIT_HEADER_JS.
 * We get here only if data has arrived from client browser i.e. from netconn_recv
 * Buffer buf already contains first packet and starts with:
 *
 * PUT /ufile=BOOT.TXT HTTP/1.1..Host: 192.168.3.73:81..User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64;
 * x64; rv:103.0) Gecko/20100101 Firefox/103.0..Accept: .....Accept-Language: en-US,en;q=0.5..Accept-En
 * coding: gzip, deflate..Content-Type: text/plain;charset=UTF-8..Content-Length: 198..Origin: http * p
 * ://192.168.3.73:81..Connection: keep-alive..Referer: [url]http://192.168.3.73:81/efile?BOOT.TXT....boot[/url]
 * time: 2022-08-02 15:39:28.app name: appname_1.1
 *
 * The likelihood of the 1st packet containing the entire header including the CRLFCRLF
 * is dependent on the value PBUF_POOL_BUFSIZE on lwipopts.h, and on the program sending
 * the data (the browser).
 *
 * We get here with the filename at buf[11] (buf = "PUT /ufile=BOOT.TXT ...)
 * To get data size, search for Content-Length:
 * To find the start of the data, search for CRLFCRLF, and extract Content-Length of it.
 *
 * The actual file data is after a CRLFCRLF and is quite likely in this buffer!
 *
 * This function writes data directly from LWIP's buffers to the filesystem, so there
 * is no 512 byte etc limitation there.
 *
 */

static void EditGetData(struct netconn *conn, char *buf, uint16_t buflen)
{

struct netbuf *nbuf = NULL; // address of a netbuf
err_t err = ERR_RST; // some initial value ("connection closed")
char filename[20];
char filesizebuf[20];
uint32_t filesize=0; // size extracted from client header
bool found=true; // false if any of various items not found in header
char * ptr = NULL;
FIL fp;
UINT actual_length=0; // #bytes actually written to file
uint32_t total_written=0; // accumulation of above
bool wr_fail = false;
char TEMPFILE[] = "$$$$TEMP.TMP";

// Buf already holds the first load of data. Length is buflen. Parsing this assumes that
// - the relevant part of the client's header is in buf
// - enough data has actually arrived for the above to be true (might need a short delay)
// The lwipopts.h PBUF_POOL_BUFSIZE parameter has a direct bearing on this and needs to be >500

// Extract filename
strncpy(filename,&buf[11],sizeof(filename)); // copy over filename, ' ' terminated
ptr = strnstr(filename," ",sizeof(filename));
if (ptr != NULL)
{
*(uint32_t*) ptr = 0; // null-terminate filename (replace '=' with 0)
}
else
{
found=false;
}

// Extract file size; limit search for "Content-Length:" to some plausible value (MTU)
ptr = strnstr(buf,"Content-Length:",1500);
if ( (ptr != NULL) && found )
{
strncpy(filesizebuf, 15+ptr, sizeof(filesizebuf));
filesize=atoi(filesizebuf);
}
else
{
found=false;
}

// Extract the portion of the file in buf. Typically this is at buf+390 or so
uint32_t foffset=0; // offset of where file data starts in buf
uint32_t flen=0; // size of file data
ptr = strnstr(buf,CRLFCRLF,1500);
if ( (ptr != NULL) && found )
{
foffset = 4+(uint32_t)(ptr-buf); // 4 to skip CRLFCRLF
flen = buflen-foffset;
}
else
{
found=false;
}

//debug_thread_printf("flen=%d offset=%d, found=%d", flen, foffset, found);

// If we got a duff data size then filename is probably duff too, so do nothing
if ( found && (filesize!=0) && (filesize<(2048*1024)) )
{
// First we need to write to the file the data in the above buffer, from
// buf[offset], len=flen
// Write to a temp file first

// Delete any file of same name
f_chmod(TEMPFILE, 0, AM_RDO|AM_SYS|AM_HID); // This is in case file exists and is R/O etc
  f_unlink(TEMPFILE); // Erase it

// Open the file
if ( f_open(&fp, TEMPFILE, FA_CREATE_ALWAYS | FA_WRITE) != FR_OK )
{
wr_fail=true;
}
else
{
//  Write this initial piece of the data
if (f_write(&fp, &buf[foffset], flen, &actual_length) != FR_OK )
{
wr_fail=true;
}
else
{
total_written+=flen;

// Now write the rest of the data to the file
while((err = netconn_recv(conn, &nbuf)) == ERR_OK)
{
char *data = NULL;
uint16_t len = 0;

netbuf_first(nbuf);
do
{
netbuf_data(nbuf, (void**)&data, &len);
if ( f_write(&fp, data, len, &actual_length) != FR_OK )
{
wr_fail=true;
break;
}
total_written+=len;
}
while ( (netbuf_next(nbuf) >= 0) && (len==actual_length) );
netbuf_delete(nbuf);
}
}
f_close(&fp);
}

// Return status to client

if ( !found || wr_fail || (total_written!=filesize) )
{
netconn_write(conn, EDIT_WRITE_BAD, strlen((char*)EDIT_WRITE_BAD), NETCONN_COPY);
}
else
{
netconn_write(conn, EDIT_WRITE_GOOD, strlen((char*)EDIT_WRITE_GOOD), NETCONN_COPY);
}

// Close connection
    netconn_close(conn);
}

}

  
  The last code above does not show the renaming of the temp file to the real file, etc. It also does not show checking if the temp file is the same size as the filesize extracted from the header, but these are irrelevant to the topic.
  
  So actual netconn code is simple. What it is not showing is that the netconn API is clearly (though I have not seen this spelt out anywhere) tightly connected to the above mention two values in lwipopts.h. This loop
  
  

  Code: [Select]

  // Now write the rest of the data to the file
while((err = netconn_recv(conn, &nbuf)) == ERR_OK)
{
char *data = NULL;
uint16_t len = 0;

netbuf_first(nbuf);
do
{
netbuf_data(nbuf, (void**)&data, &len);
if ( f_write(&fp, data, len, &actual_length) != FR_OK )
{
wr_fail=true;
break;
}
total_written+=len;
}
while ( (netbuf_next(nbuf) >= 0) && (len==actual_length) );
netbuf_delete(nbuf);
}
  
  picks the next buffer out of the possibly many rx buffers specified in lwipopts, and then netbuf_next advances to the next buffer, until you reach the last one. Then netbuf_delete tells lwip that you have extracted data out of that buffer and it can be refilled.
  
  I have posted a lot more code above than was needed but I wanted to show the whole sequence of netconn calls and don't have the time to cut out the other stuff.
  
  One interesting aspect in the above code is that all the rx data gets copied straight out of lwip buffers (which in this case are 1 MTU in size) and you don't need another buffer of your own, for which memory would need to be allocated.
• #6 Reply
  Posted by tellurium on 05 Aug, 2022 19:52
• Thanks Peter!
  
  A tiny nitpick:
  
  

  Code: [Select]

  netbuf_data(nbuf, (void**)&data, &len);
if ( f_write(&fp, data, len, &actual_length) != FR_OK )
{
wr_fail=true;
break;
}
total_written+=len;

  That piece can write incomplete data. The "f_write()" call returns the number of written bytes in "actual_length", which can be smaller than requested "len", whilst the return code is success (FR_OK). The code assumes that actual_length is always equal to len, which may or may not be true. If not true, the file will have missing pieces, and will be smaller than the uploaded original.
• #7 Reply
  Posted by peter-h on 05 Aug, 2022 20:08
• Are you saying that in
  
    while ( (netbuf_next(nbuf) >= 0) && (len==actual_length) );
  
  I should have only
  
   while ( netbuf_next(nbuf) >= 0)
  
  ?
  
  The only scenario where inequality would occur is if disk is full. Then my next test (not shown) will fail; the filesize in the header will not match the filesize in the FAT12 directory. And no recovery is possible anyway. Maybe I misunderstand?
  
  On some other topics, I have been trying to work out how the lwipopts.h settings affect things. I have concluded that the two values mentioned above are for receive only. And that when transmitting, lwip does no buffering internally and simply passes every buffer you send it to low_level_output (where the ETH copying code splits it up into MTU sized packets as necessary). I have not seen this aspect documented anywhere (well, one poster said LWIP has no TX buffers but would not reply to my followup questions) but it makes sense.
  
  And I still have not worked out what MEM_SIZE is for exactly. All the stuff online just copies each other like clickbait. I think MEM_SIZE is the size of the private heap which LWIP sets up for the (mainly) receive structures, notably for the two packet values above. But not entirely just for receive.
  
• #8 Reply
  Posted by nctnico on 05 Aug, 2022 21:49
• MEM_SIZE is the minimum length of a block of memory that Lwip will allocate using it's own memory allocator. Typically you'd need to set this to a value larger than the maximum packet size you expect to deal with. The default value is 1600.
• #9 Reply
  Posted by peter-h on 05 Aug, 2022 21:56
• Experimentally, I found this
  
  

  Code: [Select]

  // MEM_SIZE: the size of the heap memory. This is a statically allocated block.
// If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.
// If MEMP_MEM_MALLOC=1 (which is not reliable) this greatly expands and needs 16k+.
// Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.
// 6k yields a good speed and going to 8k+ makes a minimal improvement.
#define MEM_SIZE                (6*1024)

  IOW, it needs to be a lot more than the max packet size. I found that at 3k the system stops running. That was with 8 x 512 byte buffers; now I am using 3 x MTU size buffers.
  
  Reading between the lines, my view is that while LWIP is old now and thus reliable (15 years?) it also means that there is little discussion online of how it actually works, and those who know are out of the game and have retired, or moved on to new stuff. So e.g. posting on the LWIP mailing list yields zero most of the time.
  
• #10 Reply
  Posted by peter-h on 06 Aug, 2022 06:09
• Done re-testing of the above particular point (the MEM_SIZE private heap) and get the same result as before:
  
  With MEMP_MEM_MALLOC=0
  
  #define PBUF_POOL_SIZE           4
  #define PBUF_POOL_BUFSIZE     1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN
  
  are obviously statically allocated.
  
  The following values of MEM_SIZE give me the following transfer speeds
  
  16k 285 kbytes/sec
  6k 250
  3k 149
  2k 2.3
  
  The above speeds are actually capped by something else (polling packet rate in low_level_input, which with TCP equally throttles the output speed, but this is accepted on this project for other reasons)
  
  So below MEM_SIZE=3k the system just collapses. My attempts to find out how this is made up have not been successful, but probably it contains these
  
  

  Code: [Select]

  // MEMP_ structures.
/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
   sends a lot of data out of ROM (or other static memory), this
   should be set high. */
#define MEMP_NUM_PBUF           5
/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One
   per active UDP "connection". */
#define MEMP_NUM_UDP_PCB        6
/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCP
   connections. */
#define MEMP_NUM_TCP_PCB        5
/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP
   connections. */
#define MEMP_NUM_TCP_PCB_LISTEN 5
/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP
   segments. */
#define MEMP_NUM_TCP_SEG        8
/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active
   timeouts. */
#define MEMP_NUM_SYS_TIMEOUT    10
  but despite extensive posts and searches it has been impossible to find out how big these are individually. Some are probably small structs only.
  
  With MEMP_MEM_MALLOC=0, MEM_SIZE is statically allocated and this is obvious in the Build Analyser free RAM display.
  
  I spent a lot of time playing with MEMP_MEM_MALLOC=1 and that prevents the static allocation of the above buffers (each of which is worth about 1.54k) but then MEM_SIZE needs to be greatly increased (6k to say 16k) because these buffers come out of the LWIP private heap. There is various stuff online about this, with warnings that it slows the system down because each packet involves basically a malloc and a free. So this seems pointless since there is no overall gain, and a reported speed loss. Static allocation is always safer.
  
  On a fast LAN, PBUF_POOL_SIZE can be anything, down to 2, with no difference to performance. In fact the same is true for the low level ETH DMA buffers (more than 2 is pointless) but that's another story.
  
• #11 Reply
  Posted by nctnico on 06 Aug, 2022 09:48
• Interesting. In a project that uses Lwip on a very minimal system, I have set MEM_SIZE to 768 but also greatly limited the number of PBUFs. I do see a large effect on the amount of memory that Lwip takes when changing packet sizes and number of PBUFs so there is more going on besides MEM_SIZE. That project has MEM_USE_POOLS set to 0 though.
  
• #12 Reply
  Posted by peter-h on 06 Aug, 2022 10:35
• I have #define MEM_USE_POOLS 0 so same as you.
  
  I don't think MEM_SIZE needs to include all the PUBFs i.e.  PBUF_POOL_SIZE * PBUF_POOL_BUFSIZE because those are clearly statically allocated. But nobody has been able to shine any light on what exactly MEM_SIZE does. For example I have MEMP_NUM_TCP_PCB=5 which is almost certainly not needed, but I don't know how big these are and neither does anyone else. I went looking and ended up with
  
  

  Code: [Select]

  /** the TCP protocol control block */
struct tcp_pcb {
/** common PCB members */
  IP_PCB;
/** protocol specific PCB members */
  TCP_PCB_COMMON(struct tcp_pcb);

  /* ports are in host byte order */
  u16_t remote_port;

  tcpflags_t flags;
#define TF_ACK_DELAY   0x01U   /* Delayed ACK. */
#define TF_ACK_NOW     0x02U   /* Immediate ACK. */
#define TF_INFR        0x04U   /* In fast recovery. */
#define TF_CLOSEPEND   0x08U   /* If this is set, tcp_close failed to enqueue the FIN (retried in tcp_tmr) */
#define TF_RXCLOSED    0x10U   /* rx closed by tcp_shutdown */
#define TF_FIN         0x20U   /* Connection was closed locally (FIN segment enqueued). */
#define TF_NODELAY     0x40U   /* Disable Nagle algorithm */
#define TF_NAGLEMEMERR 0x80U   /* nagle enabled, memerr, try to output to prevent delayed ACK to happen */
#if LWIP_WND_SCALE
#define TF_WND_SCALE   0x0100U /* Window Scale option enabled */
#endif
#if TCP_LISTEN_BACKLOG
#define TF_BACKLOGPEND 0x0200U /* If this is set, a connection pcb has increased the backlog on its listener */
#endif
#if LWIP_TCP_TIMESTAMPS
#define TF_TIMESTAMP   0x0400U   /* Timestamp option enabled */
#endif

  /* the rest of the fields are in host byte order
     as we have to do some math with them */

  /* Timers */
  u8_t polltmr, pollinterval;
  u8_t last_timer;
  u32_t tmr;

  /* receiver variables */
  u32_t rcv_nxt;   /* next seqno expected */
  tcpwnd_size_t rcv_wnd;   /* receiver window available */
  tcpwnd_size_t rcv_ann_wnd; /* receiver window to announce */
  u32_t rcv_ann_right_edge; /* announced right edge of window */

  /* Retransmission timer. */
  s16_t rtime;

  u16_t mss;   /* maximum segment size */

  /* RTT (round trip time) estimation variables */
  u32_t rttest; /* RTT estimate in 500ms ticks */
  u32_t rtseq;  /* sequence number being timed */
  s16_t sa, sv; /* @todo document this */

  s16_t rto;    /* retransmission time-out */
  u8_t nrtx;    /* number of retransmissions */

  /* fast retransmit/recovery */
  u8_t dupacks;
  u32_t lastack; /* Highest acknowledged seqno. */

  /* congestion avoidance/control variables */
  tcpwnd_size_t cwnd;
  tcpwnd_size_t ssthresh;

  /* sender variables */
  u32_t snd_nxt;   /* next new seqno to be sent */
  u32_t snd_wl1, snd_wl2; /* Sequence and acknowledgement numbers of last
                             window update. */
  u32_t snd_lbb;       /* Sequence number of next byte to be buffered. */
  tcpwnd_size_t snd_wnd;   /* sender window */
  tcpwnd_size_t snd_wnd_max; /* the maximum sender window announced by the remote host */

  tcpwnd_size_t snd_buf;   /* Available buffer space for sending (in bytes). */
#define TCP_SNDQUEUELEN_OVERFLOW (0xffffU-3)
  u16_t snd_queuelen; /* Number of pbufs currently in the send buffer. */

#if TCP_OVERSIZE
  /* Extra bytes available at the end of the last pbuf in unsent. */
  u16_t unsent_oversize;
#endif /* TCP_OVERSIZE */

  /* These are ordered by sequence number: */
  struct tcp_seg *unsent;   /* Unsent (queued) segments. */
  struct tcp_seg *unacked;  /* Sent but unacknowledged segments. */
#if TCP_QUEUE_OOSEQ
  struct tcp_seg *ooseq;    /* Received out of sequence segments. */
#endif /* TCP_QUEUE_OOSEQ */

  struct pbuf *refused_data; /* Data previously received but not yet taken by upper layer */

#if LWIP_CALLBACK_API || TCP_LISTEN_BACKLOG
  struct tcp_pcb_listen* listener;
#endif /* LWIP_CALLBACK_API || TCP_LISTEN_BACKLOG */

#if LWIP_CALLBACK_API
  /* Function to be called when more send buffer space is available. */
  tcp_sent_fn sent;
  /* Function to be called when (in-sequence) data has arrived. */
  tcp_recv_fn recv;
  /* Function to be called when a connection has been set up. */
  tcp_connected_fn connected;
  /* Function which is called periodically. */
  tcp_poll_fn poll;
  /* Function to be called whenever a fatal error occurs. */
  tcp_err_fn errf;
#endif /* LWIP_CALLBACK_API */

#if LWIP_TCP_TIMESTAMPS
  u32_t ts_lastacksent;
  u32_t ts_recent;
#endif /* LWIP_TCP_TIMESTAMPS */

  /* idle time before KEEPALIVE is sent */
  u32_t keep_idle;
#if LWIP_TCP_KEEPALIVE
  u32_t keep_intvl;
  u32_t keep_cnt;
#endif /* LWIP_TCP_KEEPALIVE */

  /* Persist timer counter */
  u8_t persist_cnt;
  /* Persist timer back-off */
  u8_t persist_backoff;

  /* KEEPALIVE counter */
  u8_t keep_cnt_sent;

#if LWIP_WND_SCALE
  u8_t snd_scale;
  u8_t rcv_scale;
#endif
};

  
  but that contains more structs... I will try to add them up later.
  
  OTOH it is clear LWIP "works" with practically no buffering available, and if sending just small stuff (DHCP etc) this may work plenty fast enough. A speed of 2.5kbytes/sec may be totally un-noticed in most embedded applications.
  
  EDIT: no time to do this fully but each MEMP_NUM_TCP_PCB is probably 200 bytes, so reducing this will make a difference to RAM usage. By examing that LWIP private heap (it is prefilled with 0) I find that 5k out of the declared 6k is getting used. Changing some of the above values e.g. MEMP_NUM_TCP_PCB does not change this.
  
  EDIT2: I did many recompilations with greatly increased values for the stuff in lwipopts.ini, to see the effect on static RAM usage. This file has the effects in comments:
  
  

  Code: [Select]

  /**
  ******************************************************************************
  * @file    LwIP/LwIP_HTTP_Server_Netconn_RTOS/Inc/lwipopts.h
  * @author  MCD Application Team
  * @brief   lwIP Options Configuration.
  ******************************************************************************
*
* This sort of explains the memory usage
* [url]https://lwip-users.nongnu.narkive.com/dkzkPa8l/lwip-memory-settings[/url]
* [url]https://www.cnblogs.com/shangdawei/p/3494148.html[/url]
* [url]https://lwip.fandom.com/wiki/Tuning_TCP[/url]
* [url]https://groups.google.com/g/osdeve_mirror_tcpip_lwip/c/lFYJ7Fw0Cxg[/url]
* ST UM1713 document gives an overview of integrating all this.
*
*
*
*
  */
#ifndef __LWIPOPTS_H__
#define __LWIPOPTS_H__

/**
 * NO_SYS==1: Provides VERY minimal functionality. Otherwise,
 * use lwIP facilities.
 */
#define NO_SYS                  0

// Flag to make LWIP API thread-safe. The netconn and socket APIs are claimed
// to be thread-safe anyway. This *may* make the raw API thread-safe too.
#define LWIP_TCPIP_CORE_LOCKING    1

// This places more objects into the static block defined by MEM_SIZE.
// Uses mem_malloc/mem_free instead of the lwip pool allocator.
// MEM_SIZE now needs to be increased by about 10k.
// It doesn't magically produce extra memory, and causes crashes.
// There is also a performance loss, apparently. AVOID.
#define MEMP_MEM_MALLOC 0


//NC: Need for sending PING messages by keepalive
#define LWIP_RAW 1
#define DEFAULT_RAW_RECVMBOX_SIZE 4

/*-----------------------------------------------------------------------------*/

/* LwIP Stack Parameters (modified compared to initialization value in opt.h) -*/
/* Parameters set in STM32CubeMX LwIP Configuration GUI -*/

/*----- Value in opt.h for LWIP_DNS: 0 -----*/
#define LWIP_DNS 1

/* ---------- Memory options ---------- */
/* MEM_ALIGNMENT: should be set to the alignment of the CPU for which
   lwIP is compiled. 4 byte alignment -> define MEM_ALIGNMENT to 4, 2
   byte alignment -> define MEM_ALIGNMENT to 2. */
#define MEM_ALIGNMENT           4

// MEM_SIZE: the size of the heap memory. This is a statically allocated block.
// If MEMP_MEM_MALLOC=0, this holds just the PBUF_ stuff.
// If MEMP_MEM_MALLOC=1 (which is not reliable) this greatly expands and needs 16k+.
// Empirically this needs to be big enough for at least 4 x PBUF_POOL_BUFSIZE.
// 6k yields a good speed and going to 8k+ makes a minimal improvement. The main
// factor affecting speed is the poll period in ethernetif_input().
#define MEM_SIZE                (6*1024)

// MEMP_ structures. Their sizes have been determined experimentally, by
// increasing them and seeing free RAM changing.

/* MEMP_NUM_PBUF: the number of memp struct pbufs. If the application
   sends a lot of data out of ROM (or other static memory), this
   should be set high. */
#define MEMP_NUM_PBUF           5 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_UDP_PCB: the number of UDP protocol control blocks. One
   per active UDP "connection". */
#define MEMP_NUM_UDP_PCB        6 // each 1 is 32 bytes of static RAM

/* MEMP_NUM_TCP_PCB: the number of simultaneously active TCP
   connections. */
#define MEMP_NUM_TCP_PCB        5 // each 1 is 145 bytes of static RAM

/* MEMP_NUM_TCP_PCB_LISTEN: the number of listening TCP
   connections. */
#define MEMP_NUM_TCP_PCB_LISTEN 5 // each 1 is 28 bytes of static RAM

/* MEMP_NUM_TCP_SEG: the number of simultaneously queued TCP
   segments. */
#define MEMP_NUM_TCP_SEG        8 // each 1 is 20 bytes of static RAM

/* MEMP_NUM_SYS_TIMEOUT: the number of simulateously active
   timeouts. */
#define MEMP_NUM_SYS_TIMEOUT    10 // each 1 is 16 bytes of static RAM


/* ---------- Pbuf dynamically allocated buffer blocks  ---------- */

// These settings are probably for RECEIVE only and make negligible difference
// to performance, which is dominated by the low_level_input poll period. These
// PBUFs relate directly to the netconn API netbufs etc.

/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
// Statically allocated, so setting this to 2 frees up another 3k of RAM
#define PBUF_POOL_SIZE           4

/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
// ** It is better to use 4 x MTU than 8 x 512 because in say EditGetData() you are **
// ** much more likely to get the whole file header in the first packet. With 512   **
// ** byte packets the CRLFCRLF marker is only ~64 bytes before the end of the pkt  **
#define PBUF_POOL_BUFSIZE        1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN

/* --------------------------------------------------------------- */


/* ---------- TCP options ---------- */
#define LWIP_TCP                1
#define TCP_TTL                 255

/* Controls if TCP should queue segments that arrive out of
   order. Define to 0 if your device is low on memory. */
#define TCP_QUEUE_OOSEQ         0

/* TCP Maximum segment size. */
#define TCP_MSS                 (1500 - 40)   /* TCP_MSS = (Ethernet MTU - IP header size - TCP header size) */

/* TCP sender buffer space (bytes). */
#define TCP_SND_BUF             (4*TCP_MSS) // no effect on static RAM

/*  TCP_SND_QUEUELEN: TCP sender buffer space (pbufs). This must be at least
  as much as (2 * TCP_SND_BUF/TCP_MSS) for things to work. */

#define TCP_SND_QUEUELEN        (2* TCP_SND_BUF/TCP_MSS)

/* TCP advertised receive window. */
// Should be less than PBUF_POOL_SIZE * (PBUF_POOL_BUFSIZE - protocol headers)
#define TCP_WND                 (2*TCP_MSS) // no effect on static RAM


/* ---------- ICMP options ---------- */
#define LWIP_ICMP            1


/* ---------- DHCP options ---------- */
#define LWIP_DHCP               1


/* ---------- UDP options ---------- */
#define LWIP_UDP                1
#define UDP_TTL                 255


/* ---------- Statistics options ---------- */
#define LWIP_STATS 0

/* ---------- link callback options ---------- */
/* LWIP_NETIF_LINK_CALLBACK==1: Support a callback function from an interface
 * whenever the link changes (i.e., link down)
 * 8/2022 this is done from the low_level_input RTOS task.
 */
#define LWIP_NETIF_LINK_CALLBACK        0


• #13 Reply
  Posted by peter-h on 15 Aug, 2022 20:28
• I had a bit of fun today with netconn_write() - a simple function which doesn't give any trouble. It just sends out the stuff...
  
  And various people have said that the outgoing stream in LWIP is zero-copy.
  
  I found the function hangs if I try to send out a block bigger than about 4k.
  
  It should have been obvious that LWIP cannot be zero copy on the way out because it has to re-pack packets like that into MTU units, and short of a very clever DMA controller one can't do that on the fly. One could do it in software (assemble the header, send that to low_level_output, then copy the MTU bytes to LLO, then assemble the back end and send that to LLO). But it's obvious that this doesn't happen, because LLO gets only "finished data", which gets copied (with memcpy) to the ETH buffers which the ETH DMA picks up and sends out (a list of buffers).
  
  I checked the RTOS task stacks and they aren't getting this, so it must be LWIP's own internal storage.
  
  Further investigation revealed that increasing the LWIP private heap (MEM_SIZE) lifts this max data size figure. However increasing PBUF_POOL_SIZE etc makes no difference. My PBUF_POOL_BUFSIZE is already 1500+ (full MTU).
  
  I am only sending big chunks of JS to a browser so the simple way was to send the stuff out 1 byte at a time Still takes a tiny fraction of a second.
  
  Still have not found out whether the RAW API is thread-safe. It definitely isn't if LWIP_TCPIP_CORE_LOCKING=0 but nobody knows if LWIP_TCPIP_CORE_LOCKING=1. The LWIP mailing list is dead.
• #14 Reply
  Posted by betocool on 17 Aug, 2022 11:29
• Don't quote me on this, but I remember reading somewhere, a few times, that the raw API is not thread safe.
  
  I don't know why. But TCP on the raw API is a pain if you've never used it before. It's based on callbacks for different things, and examples are few and far between.
  
  Cheers,
  
  Alberto
• #15 Reply
  Posted by peter-h on 17 Aug, 2022 13:44
• That was my conclusion too.
  
  Makes one wonder what exactly LWIP_TCPIP_CORE_LOCKING=1 does, then, given that the netconn and the socket APIs are supposed to be thread-safe without that:
  
  https://www.nongnu.org/lwip/2_1_x/multithreading.html
  
  I do have one example of RAW, in some code produced by someone else
  
  

  Code: [Select]

    fd = socket(AF_INET, SOCK_RAW, IP_PROTO_ICMP);
  to transmit ICMP packets, for 4G etc modem keep-alive.
  
  Somebody posted on the ST forum that using LWIP_TCPIP_CORE_LOCKING=1 is "more efficient" but as usual was never seen again when I asked for detail. My guess would be that it enables mutexing at a higher level, which could be regarded as "more efficient" according to where you are coming from...
  
• #16 Reply
  Posted by nctnico on 17 Aug, 2022 20:29
• raw Lwip API is not a raw socket!
• #17 Reply
  Posted by ttt on 18 Aug, 2022 03:42
• 
  Maybe completely off topic when looking through this thread, I use slightly different approach using HTTP POST for (larger) file uploads. This relies on the LWIP_HTTPD_SUPPORT_POST LwIP feature. Though it seems you are avoiding using the httpd_xxx stuff.
  
  https://github.com/tinic/lightkraken/blob/master/bootloader.cpp
  
  I use this to directly update the firmware from my bootloader through a standard web upload, i.e. an HTTP POST to <yourip>://upload/ The multipart parser is pretty much key here to make this work nicely.
  
  Don't mind the .cpp extension, this is mostly C. This code is not thread safe at all in its current form, but it gets me by.
  
• #18 Reply
  Posted by peter-h on 24 Aug, 2022 09:24
• I avoided the LWIP HTTPD code because I could not work it out, and since this is my own little business and I have to do all support myself, and currently I am writing nearly all the code myself, I can't sensibly go down that road. A subcontractor started on doing an HTTP server but the project was terminated after it turned out that it didn't work properly and was costing too much. He also found the multipart code didn't work so he had to incorporate some multipart code from Github and in the end it was a bit of a mess.
  
  The issue with HTTP POST is that the server has to parse long delimiter strings.
  
  My simple HTTP server, using the netconn API, is now working great. It does exactly the job and I understand almost every bit of it I got a lot of help from others, some free and some on freelancer.com for about $100 (JS scripts for uploads, etc).
  
  A thread safe HTTP server is going to need a lot more RAM. But the other approach is to make it stateless and fulfil each client request immediately. Mine does actually do that; you can have 5 browser tabs all displaying a 1Hz-auto-refresh page.
  
• #19 Reply
  Posted by peter-h on 28 Sep, 2022 20:29
• Does anyone know about the outgoing data flow when transmitting a packet via LWIP?
  
  I think it is relatively simple (compared to receiving). Let's say you want to send out a 10k block. That's obviously bigger than the MTU (typ. 1500) so LWIP must break it up into 1500 byte packets, stick the preamble on each one, and pass them down to the low level output code which passes them to the 32F4's ETH controller which is constantly scanning a list of packet pointers.
  
  What I don't know is whether LWIP sends out 1500 byte packets via ETH, or whether it sends out smaller (PBUF sized) packets as defined here
  
  

  Code: [Select]

  /* ---------- Pbuf dynamically allocated buffer blocks  ---------- */

// These settings are probably for RECEIVE only and make negligible difference
// to performance, which is dominated by the low_level_input poll period. These
// PBUFs relate directly to the netconn API netbufs etc.

/* PBUF_POOL_SIZE: the number of buffers in the pbuf pool. */
// Statically allocated, so setting this to 2 frees up another 3k of RAM
#define PBUF_POOL_SIZE           4

/* PBUF_POOL_BUFSIZE: the size of each pbuf in the pbuf pool. */
#define PBUF_POOL_BUFSIZE        1500 + PBUF_LINK_ENCAPSULATION_HLEN + PBUF_LINK_HLEN + PBUF_IP_HLEN + PBUF_TRANSPORT_HLEN


  I have 4xMTU there but often I see code posted where they have 8 x 512 byte buffers. I am not 100% sure if the above is really for RX only.
  
  If these PBUFs are used for TX too, then where does the MTU value come in?
  
  This config seems to make no difference to performance, until you have just 1 buffer, which is to be expected.
• #20 Reply
  Posted by betocool on 29 Sep, 2022 10:49
• Hey, I've looked at TCP data using Wireshark for a long time a few years ago. Still do it when I want to confirm what happens.
  
  I suggest you give it a try, you can filter out IP addresses you're not interested in. Run it as admin in Linux or extend your user permissions.
  
  From what I remember (pinch of salt here please) LWIP improves when you have two large buffers at least. I tend to change buffer size to 1420 bytes (I think that is Ethernet frame size but I could be wrong on the details) and have two, and have the firmware send data in chunks of that, or less.
  
  When I look at Wireshark, I can see that two packets are always sent one directly behind the other, and acknowledged if all went well. Then LWIP proceeds to send another set of two of whatever is in your queue, until it ends transmission.
  
  Details may be wrong, but that's the general idea.
  
  Cheers,
  
  Alberto
• #21 Reply
  Posted by peter-h on 29 Sep, 2022 11:11
• My target talks directly to a switch so an ETH debugger running on my PC would not see the packets. I would need to get a hardware debugger and put it in the ETH lead from the board.
  
  Or put in some debugs in the low level ETH code. Maybe that is a better way.
  
  I too found that double buffering speeds things up a lot but additional buffering beyond that does almost nothing. That is what one would expect. So one can save a lot of RAM, relative to "typical" configs one finds on the internet.
  
  The actual transmission onto a LAN with the usual gigabit switches, is very fast. Probably approaching the 100mbps physical data rate. And this means that spending a load of time on lwipopts.h is basically pointless - for a LAN connected product.
• #22 Reply
  Posted by nctnico on 29 Sep, 2022 11:21
• Quote from: peter-h on 29 Sep, 2022 11:11

  My target talks directly to a switch so an ETH debugger running on my PC would not see the packets. I would need to get a hardware debugger and put it in the ETH lead from the board.
  

  Buy an old ethernet 100Mbit hub or a managed switch with a span port. That way you can monitor all traffic.
• #23 Reply
  Posted by peter-h on 29 Sep, 2022 12:52
• That's very smart and I do have an old hub
  
  However, after some digging around in low_level_output I found a spot where the eventual packets appear in their final length, and put a debug there. And... it is as I suspected. Outgoing data is not packaged into those PBUFs. It gets sent straight out, in len=MTU or actual data, whichever is smaller. The PBUF config is purely for incoming data.
  
  Can save another few k RAM there.
  
  However, reducing the number of PBUFs below 4 dramatically slows down receive data.
• #24 Reply
  Posted by paf on 29 Sep, 2022 13:14
• A small switch with port mirroring (like the 5 port TP-LINK TL-SG105E) has a cost between 15 and 20 UKP, and is a very useful tool if you are "messing around" with network packets...