STM32F7 H7 LWIP webserver problem

[cal]

Hi,
Can anyone point me in the right direction ?
I designed a complicated project for my company consisting of an STM32H743 webserver using C, FreeRTOS,SSI,CGI and LWIP based on STM32cubeide compiler. I need technical help on the upload from browser straight to  serial port on STM32 via the STM32. It is partially working, but there are problems with intermittent corruption of the binary file during upload.
Basically I have SSI CGI etc working OK but the binary file upload for large files sent from the browser has never worked consistently.

In desperation, I used an STM32F746 Nucleo board without RTOS/SSi/CGI etc. - just a simple cubeide project using LWIP configured only for HTTPD.
Exactly the same problem.
I realise that I could strip out the content-disposition /type fields etc., but why are there no examples around that give a fault free performance without corruption, especially during large binary or text file transfers?
Any ideas? Anyone?
Thanks
Chris

I use this index.shtml:

  <form action ="/index.s html" enctype="multipart/form-data" method="post">
  <p>Simple upload. Please specify a  file:
   

<input type="file" name="datafilemnu" size="40">
   <input type="submit" value="Upload ppf">
 </br>

The reception of file data is as follows: (probably hacked about a bit now)
#define ser  // for serial port on usart3

err_t httpd_post_begin(void *connection, const char *uri, const char *http_request,
                       uint16_t http_request_len, int content_len, char *response_uri,
                       uint16_t response_uri_len, uint8_t *post_auto_wnd)
{

   return ERR_OK;
}


err_t httpd_post_receive_data(void *connection, struct pbuf *p)
{ //multipart/form-data is used
   int32_t i,len=0;
   uint32_t DataOffset, FilenameOffset;
   char *data, *ptr, filename[20];
   char readval[4800];
    DataOffset =0;

    data = p->payload;

    len = p->tot_len;

   // HAL_UART_Transmit(&huart3,data,len , 0xFFFFFF);

    if (DataFlag ==0) //has not entered before - must be first pbuf
    {  m1++;
        /* parse packet for Content-length field */
              //   size = Parse_Content_Length(data, (uint32_t)(p->tot_len));
      // parse packet for the octet-stream field
      for (i=0;i<len;i++)
      {
         if (strncmp ((char*)(data+i), octet_stream, 13)==0)
         {
           DataOffset = i+16;
           DataFlag++;
           break;
         }
      }

      // case of MSIE8 : we do not receive data in the POST packet
      if (DataOffset==0)
      {
         DataFlag++;
         BrowserFlag = 1;
         pbuf_free(p);
         return ERR_OK; // returns here but this is firefox

      }
      // case of Mozilla Firefox v3.6 : we receive data in the POST packet

       for (i=0;i<len;i++)
       {
         if (strncmp ((char*)(data+i), "filename=", 9)==0)
         {
            FilenameOffset = i+10;
            break;
         }
       }
       i =0;
       if (FilenameOffset)
       {
         while((*(data+FilenameOffset + i)!=0x22 )&&(i<20))
         {
           filename = *(data+FilenameOffset + i);
           i++;
         }
         filename = 0x0;

       }
       if (i==0)
       {

         pbuf_free(p);

         DataFlag=0; // RESET HERE
         return ERR_OK;

       }


     ptr = (char*)(data + DataOffset);
     len-= DataOffset;

  //   memcpy(str, p -> payload, p -> len);//added

m2++;
#ifdef ser
HAL_UART_Transmit(&huart3, ptr,len , 0xFFFFFF);
#endif

     pbuf_free(p);
     return ERR_OK;
    }//if dataflag==0


    else
    {

       // if last packet need to remove the http boundary tag
       // parse packet for "\r\n--" starting from end of data

          i=0; //??? 4?
       while ((strncmp ((char*)(data+ i),http_crnl_2 , 4))&&
             (strncmp ((char*)(data+ i),http_crnl_3 , 4))  && (i<len))
       {
         i++;
       }
    //   char *ip;
     //  ip=p->payload;
     //  removeSubstr(ip, "dndata=");
    //   memcpy(str, p -> payload, p -> len);//added
       m3++;
#ifdef ser
       HAL_UART_Transmit(&huart3, p->payload,i , 0xFFFFFF);
       #endif
           if(i!=len)
              DataFlag =0;
    }
    // not last data packet

    pbuf_free(p);

    return ERR_OK;
}

void httpd_post_finished(void *connection, char *response_uri, u16_t response_uri_len)
{
     connection = NULL;
}

   

[tellurium]

If commercial solution is OK, you can take a look at https://mongoose.ws/tutorials/stm32/nucleo-f746zg-cube-freertos-lwip/
File uploads are covered in https://mongoose.ws/tutorials/file-uploads/

[cal]

Thanks but tutorials are 'blocked due to Trojan' according to Malwarebytes - so I'm a bit apprehensive!
Chris

[peter-h]

I wrote an HTTP server for a 32F417 based product. Took a while but now is rock solid, file up and downloads via HTTP. Had to pay someone on freelancer.com to produce javascript for the file transfers - can't be done without that if you want a progress count of some sort. Also much easier to use PUT and you need JS for that.

A contractor started on that project using the ST supplied HTTPD source and after a few k GBP I killed that project (and paid him for time spent, obviously). That source is crap.

Most people develop fixes in company time so they don't publish them. They just leach from the web, github, everywhere

Basically none of that ST-supplied code actually works for serious use. LWIP itself is OK but you spend a lot of time doing the ETH hardware interface, and setting up the opts files with sensible values. FreeRTOS is good.

My HTTP server is a bare cut down job, using the netconn API instead of the socket API (that wasn't such a great idea, in case I want to use TLS one day) and I avoided the multipart stuff because it is messy to parse on a small micro.

Check out
https://www.eevblog.com/forum/microcontrollers/anyone-here-familiar-with-lwip/msg4306798/#msg4306798
https://www.eevblog.com/forum/microcontrollers/any-experts-on-lwip-and-the-netconn-api-(not-looking-for-free-can-pay)/msg4336729/#msg4336729
https://www.eevblog.com/forum/programming/what-is-the-http-server-client-interaction-to-do-file-transfers/msg4303780/#msg4303780

[betocool]

I wrote and audio analyzer based on the STM32H7 and use it from time to time. Lately I've used it to record music directly into a PC. Works a treat.

https://github.com/betocool-prog/Audio_Analyser_FW_OS

I have not updated the documentation, and it requires a daughter card to work. However, if you look at "tasks/Src/controller.c" you'll find a TCP Server on line 188 "controller TCP task". What it does is reading data from a streambuffer (the audio from the ADC) and streaming it to a TCP client on the PC. You could use netcat to connect to the TCP server and it would send everything without a hitch.

Of course you'll need to modify the code so that your send buffer is not ADC data but whatever it is you're sending. But the controller task itself should work well I think.

Hope it helps.

Cheers,

Alberto

[ttt]

      for (i=0;i<len;i++)
      {
         if (strncmp ((char*)(data+i), octet_stream, 13)==0)
         {
           DataOffset = i+16;
           DataFlag++;
           break;
         }
      }

Please don't take this the wrong way but have an experienced C developer review your code. The above code will read past allocated buffers resulting in undefined behavior.

At the very least do something like this:

Code: [Select]

      const size_t octet_stream_len = strlen(octet_stream); // Assuming octet_stream is a const char * string somewhere.
      if (len >= octet_stream_len) { // ensure we don't read past the length of the buffer
        for (i=0;i<(len - octet_stream_len);i++) // '- octet_stream_len' to prevent comparing past the buffer
        {
           if (strncmp ((char*)(data+i), octet_stream, octet_stream_len)==0)
           {
             DataOffset = i+16; // Where does the 16 come from?!
             DataFlag++;
             break;
           }
        }
      }

There are issues like this all over in your code. I highly recommend to at least run static analyzers on your code to catch these kind of problems.

C is a footgun.

[zilp]

If you want people to actually look at your code, please put in some minimal effort to make it readable. Like, format it as a code block, remove unused code, make sure the declarations for all variables are included, ...

ttt has already pointed out one problem - but also, more fundamentally, you seem to assume there is such a thing as a "POST packet", which there simply isn't, so you probably need to restructure your code. TCP doesn't transport records, it's just a stream of bytes, which is chopped up into packets for transmission over IP, and that you see those packet boundaries at all is simply an artefact of how low-level TCP implementations tend to be constructed, but how the stream is chopped up is completely abitrary, so your parsing state machine must be built such that it can handle packet boundaries between any two bytes in the stream.

Also, you really should read the HTTP RFC. You can't build interoperable software by looking at examples and guessing, only by reading, understanding, and following the relevant standards - and internet standards luckily are freely available, so no reason to not read them.

[tellurium]

The web server / uploads / multipart encoding is something that is not trivial.
A robust hand-crafted ad-hoc solution can be made only by an experienced engineer who understands the area well, and have done it before.
LWIP examples are not something that are easy to follow and modify.

OP, if you're not experienced in this area, then the following choices are reasonable:

a) educate yourself, which will take quite a bit of effort and time that far exceed asking a question in this forum (pointers are already given), or
b) seek for help from someone more experienced, or
c) use a ready-made code written by those experienced in the area (pointers are already given)

[peter-h]

There is no need for multipart for file upload and download via HTTP.

[tellurium]

There is no need for multipart for file upload and download via HTTP.

That's right!

Also, a client code that uploads a file (which is often a piece of JS code) can split the file into small-ish chunks, and send them sequentially - next chunk when a response from the previous one arrived. That makes the whole thing under control in terms of RAM usage, and also quite simple on the server side.

The same tactics can be done over Websocket, which can speed up uploads significantly if e.g. TLS is used - because it avoids multiple expensive TLS handshakes and does everything inside a single WS connection.

[peter-h]

a client code that uploads a file (which is often a piece of JS code) can split the file into small-ish chunks, and send them sequentially - next chunk when a response from the previous one arrived. That makes the whole thing under control in terms of RAM usage, and also quite simple on the server side.

No need for that even because tcp/ip does end to end flow control. So no large buffers are needed on the server side either. There is a thread here on how to do that. It was much simpler than I expected.

Lots of people tell you that you have to send in chunks and ack each chunk. Even some experienced web coders have told me that. You absolutely do not need to do that. You can just send the data - in either direction - and it "just works". What is not possible without JS is file transfer client -> server with an indication of progress (a bar graph, a % reading, whatever). Going server -> client, most browsers provide some sort of progress indication, even if primitive, and no JS is needed. I am doing file transfers with 512 byte blocks and I am getting about a megabyte per second down (limited by lwip buffer config, ETH low level compromises, and the filesystem, FatFS, being in SPI flash with a 21MHz clock), and the up speed is limited by the flash write speed to about 30kbytes/sec (actually I do a pre-read and if the 512 byte block is same, then I skip the write, which speeds up file uploads about 50x).

The thing I can't do much about is browser caching, which does weird stuff in some browsers, so a 512k file may appear to upload in 1 second, but you have to wait until the transfer is completed, which is not exactly dumb-user-friendly. If you want to break browser caching, in a browser-independent way, then you do need to handshake each block. I suspect image upload libs like Dropzone do this, but good luck getting that to run on an embedded target.

However I am not doing it with TLS. No idea how I would do that. I would need to change from netconn to bsd sockets for a start, I think. But a server open to the outside is a very bad idea for IOT, and if you aren't opening it, then HTTPS is pointless. TLS uses a huge amount of code - roughly 200k of my 450k total codespace - and a huge amount of RAM - I am allocating a 48k block for MbedTLS's private heap, and this seems not too generous - plus the setup crypto takes ~3 secs on a 168MHz 32F417. So maybe 1 second at 480MHz.

[tellurium]

I am pretty aware of the TCP flow control mechanism. As server slows down reading from the socket/connection, then the TCP stack shrinks the TCP windows size, signalling the client to slow down too. That's a common knowledge.

Yet even taking that into consideration, there are cases when chunking is the way to go.
Why? Because when one uses TCP flow control, the buffering / TCP window is determined by the network stack config. To put it simply, the socket/connection buffer size drives it. With chunking, the control is on the application layer.

I don't see any point arguing further. My remark was that the most controllable way is chunking, and I stand on that point of view.

[peter-h]

That's a common knowledge.

Obviously, I knew that you know all this, but not everybody does

Because when one uses TCP flow control, the buffering / TCP window is determined by the network stack config. To put it simply, the socket/connection buffer size drives it. With chunking, the control is on the application layer.

What is the benefit? It's a lot more work all around, including setting up buffers in client JS and implementing the handshaking there.

I don't see any point arguing further. My remark was that the most controllable way is chunking, and I stand on that point of view.

Nobody is arguing with you.

I was trying to help out the other guy, who is trying to do this with that ST-ported junk code.

I've had a huge amount of help on eevblog, working alone on pretty complicated stuff, so I try to put some back by helping out when I can (which is not often).

[tellurium]

The client side is not complex. Here is the example, under 50 lines of code:
https://github.com/cesanta/mongoose/blob/master/examples/file-upload-multiple-posts/web_root/app.js

Since the client code knows the total length and already sent length, it knows the progress %, too.