What is the HTTP server-client interaction to do file transfers?

[Nominal Animal]

I have concluded that Chrome and FF extract the EXIF data from the file (a jpeg), if there is no date in the header. I will test this later. Edge doesn't do it and saves the file under current date.

Interesting; I definitely did not know that.  For ordinary files, the Date header should suffice, but not all browsers will honor it.

Regarding a login, one interesting Q is whether it can support multiple concurrent clients. Most of it is stateless (client strings are immediately processed) so it can, provided they don't do stuff at literally the same time. There is only one RTOS thread. So obviously this is a hack, but I am ok with that.

I would use a separate file with for example

Code: [Select]

struct user {
    unsigned char  user[12];  /* Length dictated by making this structure 64 bytes long */
    uint32_t  access;  /* Bit mask, each bit set grants access to something */
    uint32_t  salt[4];  /* Random number */
    uint32_t  hash[8];  /* SHA256((access) (user) \0 (password without padding) \0 (salt)) */
};
At login time, you read the file until you find the first matching user.  Then, you calculate the SHA256 hash of the access word, username with a single terminating nul byte, password (obtained via url-encoded POST) with a single terminating nul byte, and the salt stored in the file.  If the hash matches the stored hash, the user is authorized.  You generate a cryptographically secure pseudorandom number or string (i.e., unguessable), and add an entry in the logged in users table in RAM for this time, this user, with the pseudorandom number.  In the login response, you include a HTTP header,
    Set-Cookie: peterslogin=pseudorandomnumber; Max-Age=seconds; SameSite=Strict; Secure

In future requests the client makes, it will include a HTTP header
    Cookie: peterslogin=pseudorandomnumber
although there could be additional values (separated by a semicolon and a space) on the same line too.  The name of the cookie must be preceded by a space, and the value must be succeeded either by a semicolon or a CR (of a CR LF pair), or be enclosed in doublequotes.

If the cookie has been in use for sufficiently long time (typically, a fraction of the lifetime; this is also the precision at which these cookies expire), you add a new one by adding in the response headers another
    Set-Cookie: peterslogin=pseudorandomnumber; Max-Age=seconds; SameSite=Strict; Secure
with hopefully a new pseudorandom number; remembering both old one and new one for at least a while, because the requests are not strictly ordered, and especially when you serve just one request at a time, they can be delayed so that a later request can easily still have the "wrong", old, cookie value.

Whenever you get a request, you first decide whether the access needs authorization or not.  If it does, you consult the Cookie header(s) in the request, and verify that it has a valid peterslogin=pseudorandomnumber pair. The value may or may not be double-quoted; that is, peterslogin="pseudorandomnumber" would be equally valid.

The cookie name should only contain letters A-Z and a-z, digits 0-9, and any of ! # $ & ' * - . / ? ^ _  ` | ~ . (This means there are 26+26+10+15 = 77 allowed ASCII characters.)
The value can contain ASCII codes 33, 35-43, 45-58, 60-91, 93-126; or all non-whitespace non-control-characters except doublequote (34), comma (44), semicolon (59), or backslash (92).  (This means there are 90 allowed ASCII characters, so one can use Base85 (or own variant) to encode each 32-bit part in five characters, making the 256-bit pseudorandom cookie value 40 characters long. I'd keep the encoded value in RAM, so that one can do a string comparison instead of decoding/encoding it each time.)

I've spent a lot of time running file downloads and trying to see which packet / buffer size etc options made any difference. I am getting interesting results: 120kbytes/sec and even ridiculous amounts of buffering improve this only slightly (to 140). Reducing the buffers makes little difference until you get down to ridiculous levels e.g. 1 buffer at low level ETH which halves the speed.

What kind of bandwidth is that MCU supposed to achieve, then?

[peter-h]

As a quick reply:

Interesting; I definitely did not know that.  For ordinary files, the Date header should suffice, but not all browsers will honor it.

This turns out to be very complicated. If no Date or Last-Modified header gets sent, the EXIF data seems to be getting used to save the file. But if either of these two headers is included, this feature disappears and the browsers save the file under the current date, disregarding the date in the header. Well, that's unless I am doing something else wrong, but I have tested it very carefully, stepping through the code and checking all the data being sent, and doing an Inspect in both Chrome and FF shows the correct dates which the browsers then ignore. IOW, neither of the two headers actually works, with any browser. And I get the same result with most files I download from elsewhere. This turns out to have been another rabbit hole. This link shows it to be a deliberate policy.
https://bugs.chromium.org/p/chromium/issues/detail?id=4574
The EXIF behaviour varies, too. FF and Edge use it to save the file if it is a jpeg with EXIF. Chrome does not. All 3 browsers disregard the Date: header.

For ordinary files, the Date header should suffice, but not all browsers will honor it.

I think they all ignore it. They see it (use Inspect on the file and you see it) and I didn't test obscure stuff like Opera. I spent way too many hours learning that this doesn't actually work, by design.

What kind of bandwidth is that MCU supposed to achieve, then?

No idea but I would expect a lot faster. The flash read for a 2MB file is ~2 secs, so the diff between that and about 15 secs actual time looks like LWIP+ETH low level stuff. The problem is that for all the GB posted on the net about this, there are few clues, and the standard buffer config makes almost no diff.

EDIT: One curious issue is that after a file transfer the page could do with a refresh. The reason for this is the structure of the code (not multi threaded etc). I have done a ton of googling on how to do a "server-side F5" but without javascript. Some hits suggested a header and I tried this

Code: [Select]

// Header for refreshing the browser
static const uint8_t REFRESH_HEADER[] =
"<!DOCTYPE html>\r\n"
"<html>\r\n"
"<head>\r\n"
"<title>xxxxxx</title>\r\n"
"<meta http-equiv=refresh content=2>\r\n"
"</head>\r\n"
"<body>\r\n"
"<META HTTP-EQUIV=\"refresh\" CONTENT=\"1\">\r\n"
"</body>\r\n"
"</html>\r\n"
;

where the META tag is the relevant one, but it doesn't work. I guess one can't just send data to a browser... Some posts above suggest that one has to send a complete HTML page for anything to work, but I thought the above does that.

[peter-h]

I managed to double the file download speed (to 250kbytes/sec) by polling for receive data 2x more often. This is the wrong data path so it should not make any difference, but evidently during a target -> PC file transfer there is a lot of data going in the other direction

Code: [Select]

/**
  * This function is the ethernetif_input task. It uses the function low_level_input()
  * that should handle the actual reception of bytes from the network
  * interface. Then the type of the received packet is determined and
  * the appropriate input function is called.
  *
  * This is a standalone RTOS task so is a forever loop.
  *
  */
void ethernetif_input( void * argument )
{
struct pbuf *p;
struct netif *netif = (struct netif *) argument;

// This mutex position is based on an idea from Piranha here, with the delay added
// https://community.st.com/s/question/0D50X0000BOtUflSQF/bug-stm32-lwip-ethernet-driver-rx-deadlock

do
    {
sys_mutex_lock(&lock_eth_if_in);
p = low_level_input( netif );
sys_mutex_unlock(&lock_eth_if_in);

if (p!=NULL)
{
if (netif->input( p, netif) != ERR_OK )
{
pbuf_free(p);
}
}

osDelay(10);   // this polling period has a corresponding effect on data flow speed in *both* directions

    } while(true);

}

Somebody should find this very funny. I should have realised that TCP/IP is a constant packet flow in both directions

[Nominal Animal]

I managed to double the file download speed (to 250kbytes/sec) by polling for receive data 2x more often. This is the wrong data path so it should not make any difference, but evidently during a target -> PC file transfer there is a lot of data going in the other direction

Somebody should find this very funny. I should have realised that TCP/IP is a constant packet flow in both directions

Well, we did already discuss the fact that each TCP data packet received needs to be acknowledged as received..

If the files are 2 MiB or less in size, that's around eight seconds, so I guess it'll do.. but still, less than a megabyte per second does seem low, considering how even an 8-bit AVR ATmega32u4 running at 16 MHz can easily do 1 MiB/s over USB Serial.

I just haven't played enough with the innards of IP stacks to know how to squeeze everything out of lwIP, for example.

[peter-h]

Yes, for this application, 250kbytes/sec is plenty. I am now getting that with a 10ms poll period. And still playing with the buffer settings makes sod-all difference. With a 5ms poll period and tweaking buffers way up (to the point of wasting a lot of RAM) I can get nearly 1mbyte/sec.

Having the rx interrupt driven would obviously be smarter but there is no free lunch, an it exposes you to hanging the product with fast input, unless you do one of the hacks of throttling the packet rate with ISR+timers etc. In this case it isn't needed and the osDelay(10) serves the important purpose of allowing other tasks to run. I tried taskYIELD() which should be ideal but immediately the whole thing more or less hangs.

With a poll period of 1 (which with osDelay actually means anything from 0 to 1, AIUI) I get 1.2mbyte/sec. There you go... with an ISR that would be even better.

Anyway, enough time spent on the download header date/time rabbit-hole, and enough time spent on the ETH speed rabbit hole. File UPload next

[mariush]

EDIT: One curious issue is that after a file transfer the page could do with a refresh. The reason for this is the structure of the code (not multi threaded etc). I have done a ton of googling on how to do a "server-side F5" but without javascript. Some hits suggested a header and I tried this

Code: [Select]

// Header for refreshing the browser
static const uint8_t REFRESH_HEADER[] =
"<!DOCTYPE html>\r\n"
"<html>\r\n"
"<head>\r\n"
"<title>xxxxxx</title>\r\n"
"<meta http-equiv=refresh content=2>\r\n"
"</head>\r\n"
"<body>\r\n"
"<META HTTP-EQUIV=\"refresh\" CONTENT=\"1\">\r\n"
"</body>\r\n"
"</html>\r\n"
;

where the META tag is the relevant one, but it doesn't work. I guess one can't just send data to a browser... Some posts above suggest that one has to send a complete HTML page for anything to work, but I thought the above does that.

See https://en.wikipedia.org/wiki/Meta_refresh

1. It must be between  <head> </head> tags
2. It should be written correctly, with quotes where they have to be  :  <meta http-equiv="refresh" content="5">

Some tips if you want to save some bytes ... 

<!DOCTYPE declaration is not required if you write valid HTML 5 content. So that whole <!DOCTYPE html> is redundant, majority of browsers will treat your pages as html 5 content by default.

The browsers don't care about those new lines between tags. Could use just \n or you could use nothing ... this is just as valid:
<html><head><title>abc</title><meta http-equiv="refresh" content="5"></head><body><p>Text</p></body></html>

[peter-h]

None of this seems to work, for the post file download page refresh.

Code: [Select]

// Header for refreshing the browser
static const uint8_t REFRESH_HEADER[] =
"<!DOCTYPE html>"
"<html>"
"<head>"
//"<meta http-equiv=\"refresh\" content=\"5\">"
"<meta http-equiv=\"refresh\" content=\"0; url=/&files.html\">"
"</head>"
"</html>"
;

Even just a means of loading the /&files.html URL would do the job.

I have been trying delays before/after sending that stuff.

Note that the file data is going to the client, which saves it off to a file, and then this refresh header gets sent. Is the client browser really receptive to HTML data after it has had the file data?

[Nominal Animal]

One possibility is for the upload to respond with a Status: 303 See Other with a Location: header specifying the URI.

For example, the response from a successful upload could be

Code: [Select]

HTTP/1.1 303 See Other
Date: Thu, 21 Jul 2022 13:52:07 GMT
Location: /files.html
Connection: Close


However, usually the JavaScript approach is taken, because the notification of an upload success is really useful for users:

Code: [Select]

HTTP/1.1 200 Ok
Date: Thu, 21 Jul 2022 13:52:07 GMT
Content-Type: text/html; charset=utf-8
Connection: Close

<!DOCTYPE html><html><head><title>Upload successful</title><meta http-equiv="Content-Type: text/html; charset=utf-8"><script type="text/javascript">
window.addEventListener('load', function (){ setTimeout(function (){ window.location = "/files.html"; }, 3000); });
</script><style type="text/css">
p { text-align: center; }
</style></head><body><p>Upload successful!</p><p>If you are not redirected to the file list in three seconds, please <a href="/files.html">click here</a>.</p></body></html>
Again, I'd just use a "hidden" file and emit it as if requested via a normal GET request.

[peter-h]

This is after a download. I haven't got to the upload yet.

I just want the browser to either "do an F5" on the server, or request a particular URL.

I am fairly sure the packet gets sent to the browser, because there is no error return from netcomm. But it looks like the browser is not seeing it.

[mariush]

Try with javascript ...add this between the body tags

<script type="text/javascript">
  setTimeout(function () { location.reload(true); }, 5000);
</script>


5000 is the number of milliseconds to wait from the moment the javascript code is parsed by the browser

after 5000 ms, the function sent as a parameter will run which does location.reload()  , reloading the page.

In theory reload() has no parameters, so the true shouldn't do anything.  But in Firefox they added a forceGet parameter which if set to true, forces the browser to ignore cached content and force reload the page
See https://developer.mozilla.org/en-US/docs/Web/API/Location/reload

[peter-h]

No luck; tried with and without the CRLF, and same for the <head> tags.

Code: [Select]

// Header for refreshing the browser
static const uint8_t REFRESH_HEADER[] =
"<!DOCTYPE html>"
"<html>"
"<head>"
"<body>"
//"<meta http-equiv="refresh" content="5">"
//"<meta http-equiv="refresh" content="0; url=/&files.html">"
"<script type="text/javascript">"
"setTimeout(function () { location.reload(true); }, 5000);\r\n"
"</script>"
"</body>"
"</head>"
"</html>"
;

Googling around, I also tried window.location.reload(); unsuccessfully.

But perhaps I am supposed to send this to the browser before the file data is sent to it. The problem is that the transfer could take a long time.

Using the various Inspect tools (I am no expert there) I see no evidence that this piece of data is reaching the browser. I suspect this is the same thing as the comments earlier on a progress bar with a file upload.

This actually raises another point: I have a Reboot link on this server, and post-reboot I would like the browser to periodically refresh that URL so it finds the server again.

[mariush]

But perhaps I am supposed to send this to the browser before the file data is sent to it. The problem is that the transfer could take a long time.

wait, what?

No, once you output </html> you're done, you close the connection and the page transfer is complete.
If you want to put something on the page, you put it within the <body> </body> tags.

the javascript bit (what's between <script> </script> tags) ideally should be at the bottom of the page, because the timer starts as soon as the </script> tag is parsed.  You can change the javascript code to be triggered only on page load complete event or some other events, but in this version, it starts as soon as it's  parsed by browser.

Any html is made out of these bits

<html>  <- tells browser it's html content

<head >   
 ^- in this block there's properties of the html page that are not visible on the actual "page" , like the title that will show up on title bar , favicon ,
  here you also define stylesheets (css files) , third party fonts,  description and keywords for the page if desired and so on
 here you can also link to javascript files  (*.js) that a browser could load and cache
</head >

<body >

between these tags you output anything that you want to be visible on the page to the user
text, tables, forms, pictuers
you can also put invisible stuff like inline javascript , code between <script> </script> tags   

</body>

</html> - this tells the browser the page is done.

You don't output all that as a header, and then continue to print stuff or whatever. Once </html> is printed, you close connection. Maybe the page doesn't refresh because the browser is still waiting for you to close the connection or because the html page is invalid as you write stuff after  the </html> is sent?

Last but not least,  that meta refresh should be considered at best unreliable .. some browsers may even ignore it as they could treat it as annoying (like those joke pages or scripts that show up a popup after popup to annoy user or prevent user from closing page  (they added a checkbox to disable showing more popup messages on a page for this reason,  or one could hold shift down to abort popup messages)

Another annoyance with the fact you have little memory is that it's hard for you to buffer the whole page in ram in order to determine its actual length (number of bytes) and send the proper header Content-Length before sending the html page.


As for the reboot and show page to refresh once reboot is done I don't see that without javascript.
You can set up a timer, a function that launches another javascript function on an interval, let's say every 5 seconds.
That function could attempt to retrieve a file that's guaranteed to be there and timeout in a few seconds if the connection fails. If the connection is successful, your javascript function tells the browser to go to default index / login page.
 

[peter-h]

I must be getting my tags mixed up, but actually everything is running ok. It is merely that after the file download (to the browser) the browser has not refreshed the page. I notice this because the 2 sec auto refresh has disappeared. If I press F5, it returns.

The first header I send to the browser (when it goes to the base URL) is

Code: [Select]

// This one is at the top of every page. This header is not terminated and must be
// followed by e.g. PAGE_HEADER_MAIN or PAGE_HEADER_STATUS.
static const unsigned char PAGE_HEADER_ALL[] =
"HTTP/1.1 200 OKrn"
"Server: XXXXXX rn"
"Content-Type: text/htmlrnrn"
"<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">"
"<html>"
"<head>"
"<title>XXXXXX </title>"
"<meta http-equiv=Content-Type content="text/html; charset=utf-8">"
#ifdef BLOCK_FAVICON
"<link rel="icon" href="data:,">"
#endif
;
Next I send this one, which terminates the page with </html>

Code: [Select]

// This one is the main page
// (eventually this will appear only after a login)
static const unsigned char PAGE_HEADER_MAIN[] =
"<p><font size= 2><b>XXXXXX HTTP Server</b></font></p><p>"
"<a href="&files.html">Files</a><br>"
"<a href="&status.html">Status</a><br>"
"<a href="&reboot.html">Reboot</a><br>"
"<a href="&test.html">Test</a><br>"
"</body></html>nr";

Then if somebody clicks on the &files.html link, they get a file listing, which sends the first one above and then this one

Code: [Select]

// This one is the file listing
static const unsigned char PAGE_HEADER_FILES[] =
"</head>"
"<body>"
"<p><font size= 2><b>XXXXXX Files</b></font></p><p>"
"<meta http-equiv=refresh content=2>" // 0.5Hz refresh
"<pre>";
Then if somebody clicks on one of the files, they get this one which starts a new page

Code: [Select]

// Header for file download
static const uint8_t DOWNLOAD_HEADER[] =
"HTTP/1.1 200 OKrn"
"Content-Type: application/octet-streamrn"
"Content-Disposition: attachment>rn"
"<meta http-equiv=refresh content=1000>rn" // cancels out the file listing refresh
"Pragma-directive: no-cachern"
"Cache-directive: no-cachern"
"Cache-control: no-cachern"
"Pragma: no-cachern"
"Expires: 0rn"
;
and that page is not closed with </html>. Then this is the download code which should be fairly obvious

Code: [Select]

static void Download(struct netconn *conn, char *fname)
{

#define DBUF 512

uint8_t pagebuf[DBUF];
uint32_t offset=0;
UINT numread=0;
FIL fp;
FILINFO fno;
FRESULT fr;
//err_t netconn_err;
static const char montab[36 1]="JanFebMarAprMayJunJulAugSepOctNovDec";
char datebuf[40];

// We have the filename, now get its parameters (if it still exists, which is virtually certain)

fr = f_stat( fname, &fno );

    if ( fr == FR_OK )
    {

// send filesize, as "Content-Length: 1910916rnrn"
strcpy((char*)pagebuf, "Content-Length: ");
itoa(fno.fsize,(char*)&pagebuf[16],10); // place size after "Content-Length: "
strcat((char*)pagebuf,"rn");
netconn_write(conn, pagebuf, strlen((char*)pagebuf), NETCONN_COPY);

// Send date/time, as "Date: Mon, 21 Oct 2015 07:28:00 UTCrn"
// To save the hassle of calculating the day of week (which isn't stored in the FAT12 directory anyway)
// we use Mon in case the client is validating the presence of the day string ;)
// For jpegs, FF and Edge extract this from exif, interestingly, if header is missing
                // Otherwise browsers appear to ignore this, and same with Last-Modified.

strcpy((char*)pagebuf, "Date: Mon, ");
int monidx = 3*(((fno.fdate >> 5) & 15)-1);
snprintf(datebuf,sizeof(datebuf),"%2u %c%c%c %4u u:u:u%c",
fno.fdate & 31,
montab[0 monidx],
montab[1 monidx],
montab[2 monidx],
(fno.fdate >> 9)   1980,
fno.ftime >> 11,
(fno.ftime >> 5) & 63,
2*(fno.ftime & 0x1f),
0);
strcat((char *)pagebuf, datebuf);
strcat((char *)pagebuf, "rnrn");  // 2xCRLF is the last thing before the binary file data
netconn_write(conn, pagebuf, strlen((char*)pagebuf), NETCONN_COPY);

// send file; this is faster than KDE_file_read() which gradually slows down for big files

if (f_open(&fp, fname, FA_READ | FA_OPEN_EXISTING) == FR_OK)
{
do
{
if ( f_read(&fp, pagebuf, DBUF, &numread) != FR_OK )
{
numread=0;
break;
}
netconn_write(conn, pagebuf, numread, NETCONN_COPY);
offset =DBUF;
}
while (numread==DBUF);

f_close(&fp);
}
}

        // An attempt to reload the page after the transfer
netconn_write(conn, REFRESH_HEADER, strlen((char*)REFRESH_HEADER), NETCONN_COPY);

}

What I am not doing is closing the page (with </body> and/or with </html>) after the transfer, but is that necessary? IOW, is the file transfer a part of the body? I tried the foregoing but it makes no difference.

The Q seems to be: what state is the browser in after the file download? I don't think you can just send it data.

Maybe the page doesn't refresh because the browser is still waiting for you to close the connection or because the html page is invalid as you write stuff after  the </html> is sent?

Indeed.

I think the file data must be within the body, so I sent the JS stuff

Code: [Select]

"<script type="text/javascript">"
"setTimeout(function () { window.location.reload(true); }, 5000);rn"
"</script>"
and then closed the page with </body></html> but that doesn't work either.

At the end of the file download, the browser still has the download form on the screen. If one presses F5, that form disappears and the page gets refreshed.

Note that this website is dropping some chars above, like the backslash in \r.

[mariush]

Oh... some issues there.

First of all, drop the DOCTYPE, stop using that. It's only needed for compatibility purposes, if you have some html content that actually uses deprecated html tags and other bad things from the past.
By default, browsers will assume you use HTML 5, which doesn't require you to specify the DOCTYPE, or in other words <!DOCTYPE html> is implicit. It's not needed.
Then, you don't need to say meta http-equiv = "content-type" because you already tell the browser through the HTML Response Header

( --- assume [ENTER] or [enter] means "\r\n" everywhere I say it, I type [enter] faster than \r\n  --- )

You should have a function that does the HTTP Response, and a separate function that does the actual html headers:

Code: [Select]

function generate_headers( contentType="text/html", filename="", filesize=-1, extraHeaders = "" )
Your HTTP Response would be 

Code: [Select]

HTTP/1.1 200 OK [enter]
Content-Type: text/html [enter]  <= this "text/html" could be passed as parameter to the function that builds the string so you could reuse this function with "octet-stream" as well
[enter]
This is your basic header.
If you want to serve a file as a download, then you can add some lines after changing the content-type to octet-stream or something else:

Code: [Select]

Content-Disposition: attachment; filename="file.xyz" [ENTER] 
Content-Length: <filesize> [ENTER]

 <---  you can add the file name, file size as parameters to your generate header function, see above

In your generate_headers function you could do something as basic as this

if filesize != -1  then you know you have a download, so add those two lines.  Optionally, only add "; filename="###" segment IF filename is not an empty string (filenane != ""). If it is, leave just "attachment"

You can add those no-cache tags as well, if you want.

You can also have an extra parameter to your generate_header function called "extraHeaders" which could be useful for example when you want to add an extra response line in your header, like Set-Cookie for example, to set the session info , token to determine if user is logged in or not.

You have to be careful how you glue these bits together, so that you'll only have an empty line when you're done with the Response Header lines.

ex generate_header("text/plain", "", -1, "Set-Cookie: token=value; Secure; Max-Age=3600 [ENTER]")

Once you're done sending the Response Header,  you send the content ... which can be a html page, or it can be a download.  As soon as you're done, you close the connection and everything ends here.
Your webserver will receive another request.

For your html responses, AFTER the Response Header lines, you will have the basic HTML header, which would show up in all html files :

Code: [Select]

<html>
  <head>
    <title> this title is optional but shows up in the title bar </title>
    <link rel="shortcut icon" href="/favicon.ico" or href="data; ... " - see https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs -  />
    <link rel="stylesheet" href="filename.css" > <-- if you choose to serve a stylesheet as a separate CSS file (will be cached)
    <script src="file.js"></script">  <--  if you choose to serve a javascript each time a html page is loaded (will be cached)
 </head>
 <body>
After body, you can put actual content that's visible on a page.
You could have everything up to <body> including it, in another function, let's call it generate_pageHeader(title="Main Page") 

Then you print the stuff you want to be seen on the page, and last you close the body tag and the html tag.

The body tag, the html tag,  some optional inline javascript if you wish ... you could have a function called generate_pageFooter() that outputs

Code: [Select]

</body>
</html>
After you output this, you close the connection, the server's job is done.


NOW ... for your main page with the menu ... & is a bad character to use, and you shouldn't rely on  it to make distinction between menu options and actual downloads.
I would basically set a rule, for example :

anything that's in the root of the web server is a command or something that returns a html / picture response
* index.html = show main page
* login.html =  login user
* logout.html = log out
*upload.html = upload file
* edit.html  = edit a file
* delete.html = delete a file
* script.css = serve a css file by setting the proper mime type in the header function and then sending the css text after the response
* script.js = serve a javascript file, like with css file
* favicon.ico = serve favicon file

If the request has a folder in the url, then you know it's a download  ex   

* /FILES/CONFIG.INI  -  you receive the request, you see there's FILES/ there, so you know a download was requested, send proper response header then send the file, close the connection

If you do this, then in your html header, you would need to set any address as fixed to the root

anyway, we're going off topic... back to your default  index page

B tags and <font> tags are kind of deprecated, it's best to use style= " " , or move what you put in the style, into separate CSS or  inline CSS in the head

So your page after body could be this :

Code: [Select]

<h1>Menu</h1>
<a href="files.html">Files</a><br/>
<a href="status.html">Status</a><br/>
<a href="reboot.html">Reboot</a><br/>
<a href="test.html">Test</a><br/>
then you send the footer  aka </body></html>  and connection is closed.

Code: [Select]

<table>
<tr><td>Filename></td><td>Size</td><td>Attr.</td><td></td></tr>

<tr>
 <td><a href="/FILES/CONFIG.INI">CONFIG.INI</a></td>
 <td>954B</td>
 <td>---a-</td>

 <td> <a href="edit.html?filename=CONFIG.INI">edit</a>  <a href="delete.html?filename="CONFIG.INI">delete</a></td>
</tr>

</table>
In the files list, there's no reason why you would refresh the file list often ... are files gonna be created all the time? No. Once you dump the list of files, connection is closed, job done. User can hit reload to refresh the page if needed.

When user clicks on the file name, the browser creates a new connection to your server and requests the file by asking for /FILES/CONFIG.INI  or /FILES/WHATEVER ... your code sees that "FILES" and it can extract the file name desired to download from the string.
Or, you could have  download.html?filename=CONFIG.INI I guess. But you still have to parse the URL in the request headers.

There's no reason to add that meta refresh to disable the refresh because it's not possible, it's like this request for a file is loaded in a different browser, or another computer, it has no connection to the previous page.
You can't mix that meta tag with the HTML Response Header lines, it doesn't work like that.

For downloads, you call your generate response header function but you DO NOT call the generate html header function and your generate html footer function, you simply outpout the proper response headers, then dump the file and close the connection.

[peter-h]

In the files list, there's no reason why you would refresh the file list often ... are files gonna be created all the time?

As a bit of explanation: I have a 2MB FAT12 filesystem in a piece of SPI FLASH. This is accessible from embedded code (for which FatFS implements a 2MB drive) and also from Windows over USB (which just sees a 2MB removable drive with 512 byte sectors).

So files can just pop up from nowhere for this HTTP server file listing They can be created by one or more RTOS tasks (this http server is just an RTOS task) or by Windows over USB.

FatFS has been configured to support 8.3 names only (no LFN) and a decision was made to support only root file ops in the FatFS API. No directories will be supported, although ones already created in the root are visible:



You will point out that Windows can do what the hell it likes in that 2MB drive (it doesn't know there are other players) and that does work for the windoze->embedded direction; FatFS presents windoze-created LFNs as 8.3 using the standard alternate names and embedded code (including this server, of course) will see all windoze creations immediately, as soon as windoze has written the FAT entry. The bit which has no good solution is that embedded->windoze; windoze won't see filesystem changes due to its internal caching (it assumes it totally owns any removable device); there are crude workarounds like a periodic dismount of the drive, and there is an embedded function for that...

Hence the auto refresh on the file listing is desirable. I will document that it is lost after a file transfer.

& is a bad character to use

I have removed that; the URLs like files.html cannot appear accidentally in the 8.3-only filesystem

Favicon.ico could but I check for that before checking for potential filenames, so a file called that will not be picked up.

you send the content ... which can be a html page, or it can be a download.  As soon as you're done, you close the connection and everything ends here.

I realised the download is a part of the body so I am doing a </body></html> after the download is done. Not that it does anything. I think the browser has disconnected anyway after it got the specified # of bytes in the file.

B tags and <font> tags are kind of deprecated, it's best to use style= " " , or move what you put in the style, into separate CSS or  inline CSS in the head

There is also <strong> but I am not using any style sheets. I pay others for CSS work

then you send the footer  aka </body></html>  and connection is closed.

This is curious. I am sure LWIP (the TCP/IP embedded stack) does not implement any of that so it must be the browser which detects the </head> and close the connection at the windoze socket API.

Or, you could have  download.html?filename=CONFIG.INI

I had that originally but you can see that whole string when you hover over the filename. Still, my Edit and Delete links do show that stuff...

There's no reason to add that meta refresh to disable the refresh because it's not possible, it's like this request for a file is loaded in a different browser, or another computer, it has no connection to the previous page.
You can't mix that meta tag with the HTML Response Header lines, it doesn't work like that.

OK, good to have this confirmed. Still, why is that? The browser must be closing the connection after it has got the specified file size. I guess it does not do that if the content is embedded in the page e.g. an inline image. But for application/octet-stream (which is what triggers the browser to open the Save dialogue) it is evidently closing the connection right afterwards.

With an all-JS approach one could do anything but I need to do this within limits of my knowledge, more or less. I spent all day today playing with re-establishing the refresh after a file download and none of the zillion suggestions online work, HTML or JS.

Moving to uploads, presumably the server doesn't know what is sending the data, so one use some JS in the browser to implement the sending, and have a progress bar. I can see stuff like this HTML5 solution https://codepen.io/PerfectIsShit/pen/zogMXP but as usual there are key parts missing, because all these people are running with proper servers.

[mariush]

I realised the download is a part of the body so I am doing a </body></html> after the download is done. Not that it does anything. I think the browser has disconnected anyway after it got the specified # of bytes in the file.

No. When you do a download you do not transmit any html , no <html>....<body>, no </html>, you send the response header, you send data, you close connection or browser closes it because you don't send the Connection: Keep-Alive in response header (which is no longer allowed in http/2)
You're messing around with the dates and extracting from files... no need... send as binary data.

that meta refresh is not your solution... go with javascript, you can have something basic as last file system update or something like that, your javascript requests that value and if different than previous value it has, something has changed and force the page reload using javascript.
Parsing the fat table to get newest creation date shouldn't be hard.

I'd suggest leaving this refresh for last.

you have a whole http server with multiple connections and lots of features you don't need in lwip/src/apps/http/httpd.c  , check functions there.

[peter-h]

You're messing around with the dates and extracting from files... no need... send as binary data.

The file being downloaded is sent as binary. That works. Interesting to learn that that closes the connection afterwards.

you have a whole http server with multiple connections and lots of features you don't need in lwip/src/apps/http/httpd.c  , check functions there.

The cancelled server project used that. I cancelled it after it ran to a huge number of hours; could not afford to pay so much. That ST stuff is mostly junk which doesn't work properly and needs a lot of time spent fixing.

Agreed; JS must be the way to do refresh and I will dig around again at the end of this.

Re uploads, I am going to get someone on freelancer.com to write me some JS. It won't be completely trivial.

I have that "Upload file" link (see image above) and when somebody clicks on that, I can emit whatever data I want and send that to the browser. The browser JS can then return the file (with a byte count header at the start) and I can save the data to the filesystem as it arrives.

There are no doubt more standard ways to do this but I am having difficulty working out which side sends what, and then I would like the progress bar which does need JS. And with JS I have control of both ends of the link. Presumably JS can also pop up the standard client file picker.

[mariush]

The browser prefers to keep the connection alive, because that initial handshake between client and server takes time.  If you don't specify that in your response header, it will default to closing the connection.
At the same time, browsers are optimized and assume the average website will have extra resources to download (css, js files, images on the page etc) so some will be ready to open 2-3 or even more connections in parallel as soon as the html page tells it there's some extra file to be retrieved (ex a img tag, a favicon address) even if the server says it supports recycling the connection to serve more content. It won't wait until the whole html page is downloaded to reuse the connection to get those extra resources.

Your server listens on an IP and on port 80 (default for unencrypted http).
Client's browser picks a random unused port (let's say 50120) and a connection is created between  server ip : 80  <-> client ip : 50120
if there's more resources on the page being requested, the browser may decide to open a second connection by picking another free port , let's say 50300 .. and now there's connection serverip:80 <-> clientip:50300
It's your server's job to keep track of these connections and manage their state, and close the connection when you're done serving something, so that entry in the pool can be reused for another connection.
A simple way would be to have a pool of 2-8 connections and allow a maximum of two or three connections per IP (this would allow one upload or download, while browsing page, refreshing page using the other connection)
If your server doesn't distinguish between connections, it could happen that while your server pushes a download to a client, you get another request and you serve the reply to that in your download corrupting the download.
My guess is that right now, if you send something you're blocking anything else or ignoring any incoming data which is not good. 

I'd suggest looking some more into that httpd.c file, strip whatever you don't need from it , ex ipv6 support, https support, but try to figure out how it keeps the state of connections and how it parses the request header and all that.

[peter-h]

I have this thing nearly done but I am getting stuck on simple things like file deletion. Displaying the updated Files page after a deletion turned out to be really hard. It was solved with these two

static const unsigned char PAGE_HEADER_ALL[] =
   "HTTP/1.1 200 OK\r\n"
   "Server: XXXXXX\r\n"
   "Content-Type: text/html\r\n\r\n"
   "<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01//EN\" \"http://www.w3.org/TR/html4/strict.dtd\">"
   "<html>"
   "<head>"
   "<title>XXXXXX</title>"
   "<meta http-equiv=Content-Type content=\"text/html; charset=utf-8\">"
#ifdef BLOCK_FAVICON
   "<link rel=\"icon\" href=\"data:,\">"
#endif
   ;

static const uint8_t REFRESH_HEADER[] =
      "<meta http-equiv=\"refresh\" content=\"0; url=files.html\" />"
      "</body></html>\r\n";
      ;

However I have been doing some version of this for ages. Some part of that first header is key to make the refresh/redirect line work, and it isn't the obvious bits like <head>. I reckon it is some deprecation.

I asked someone about JS for uploading (it seems to me that once you are running JS in the client, you can run whatever private protocol you like for the data transfer) and I will now have a go at editing:

Editing of a document can be done just the same with a POST form, only you can use a TEXTAREA - https://developer.mozilla.org/en-US/docs/Web/HTML/Element/textarea -  around the contents of the document and the user will see the text in a editable text box on the screen
When you hit the save button, the browser will send you the whole contents of that textarea and whatever other parameters are in the form (hidden on purpose by you or not)

Note that the contents of the document would have to be escaped ex < > and & at the very least :  < would be &lt; , > would be &gt; and & would be &amp; - the browser will parse the html and show the characters properly on the screen and send you the actual characters in the POST form data, not the escaped ones.

There's some other gotchas, like if you don't specify a text encoding for your html files, it's assumed your html file and whatever is in it is UTF-8, so if you dump a .txt file that was written with character encoding ISO-8859-1 then some valid characters in ISO-8859-1 would be invalid in UTF-8 (invalid code points, those specific characters get two byte combinations in utf-8) and not rendered properly on screen.  See for example https://mincong.io/2019/04/07/understanding-iso-8859-1-and-utf-8/ for explanation

I have the text box opening ok, 80x25



Now I have to work out how to send it back. Will have to check there is enough room for the temp file and flag an error or something... can this be done with just HTML? It needs a bidirectional protocol of some sort. I need to send back a message (which again must be a complete HTML page) advising whether the write passed or failed.

This
http://marc.merlins.org/htmlearn/cgitutor/textareas.html
seems to have examples of POST to return the textarea to the server. And I guess that is it - you POST the textarea back to the server, and it sends back a page with some message, or nothing.

Am I right that a server can accept a POST at any time? If not, how is the session kept alive? Is there some underlying keep-alive process running when a textarea is opened?

[peter-h]

A colleague reckons I should use JS for both returning the textarea and for the file upload.

JS would enable a progress bar in both cases; relevant due to the slow 30kbyte/sec flash filesystem writing speed.

He's having a go at it.

[peter-h]

I've been getting on well with this http server. Quite a learning exercise, doing this with the basic TCP/IP API in LWIP (I am using netconn), and byte-banging everything. 99% of stuff on google is of little use because it assumes a proper web server...

From the POV of "embedded", two things which are quite messy are returning a TEXTAREA edit box, and file upload. These both use the POST method which involves searching incoming data for huge delimiting strings, which is tricky if your RX buffer is say 512 bytes. The string itself should not be > 512 in length but they can be > 512 bytes from the start of the incoming data. The result is quite tacky code, if you are going to correctly handle partial matches.

The alternative is to use PUT which is much simpler (you just get a filename and byte count and search for 2xCRLF) but a browser can't transmit a PUT. You have to use JS.

[tellurium]

The alternative is to use PUT which is much simpler (you just get a filename and byte count and search for 2xCRLF) but a browser can't transmit a PUT. You have to use JS.

The safest strategy is to load the whole file on the client side (browser) using JS, split it by small pieces, and POST/PUT piece by piece (again, using JS xhr or fetch), using binary encoding. This way, the next chunk gets sent only when the previous is acknowledged, which guarantees a limit on RAM usage.

A standard HTML form upload is quite difficult to implement on a RAM-limited device.

[peter-h]

The safest strategy is to load the whole file on the client side (browser) using JS, split it by small pieces, and POST/PUT piece by piece (again, using JS xhr or fetch), using binary encoding. This way, the next chunk gets sent only when the previous is acknowledged, which guarantees a limit on RAM usage.

You know far mor about this stuff than I ever will, but AIUI you can upload to a server in one go. TCP/IP does the flow control automatically so if a browser sends stuff at 10mbytes/sec, but the server is taking data out of the netconn or winsock API at 30kbytes/sec (flash write speed) that will "just work".

I am actually reading a file in 512 byte blocks and sending each one via the netconn lwip api. Optimising some things to the limit I can get 1.2mbytes/sec out of my board, to the browser, which is entirely respectable, and this is genuine application to application, going through god knows how many protocol layers.

There is bidirectional traffic anyway, as part of TCP/IP - as I soon discovered when I found that the RX ETH data rate (which on my system is limited to 100 low level packets/sec, for various reasons) determines the TX data rate quite precisely, to 250kbytes sec

A standard HTML form upload is quite difficult to implement on a RAM-limited device.

AFAICT the messy bit about POST is looking for the long delimiter strings, with limited RAM, and doing it correctly in case they appear in the data by accident. Whereas with a PUT you get a simple header with a byte count (but need to use JS to send the stuff).

But one needs JS to do a progress bar or any kind of progress report, on any upload to the server, anyway.

[ejeffrey]

The delimiter should never appear in the body.  The client is supposed to ensure that, although AFAIK on file uploads it will only do so statistically (it doesn't read the file in advance to see if the boundary is there).  The boundary is also required to be no longer than 70 characters and on a line by itself (plus the two implicit leading hyphens).  I don't remember if the 70 character limit includes the hyphens, double check that.

This should be enough to parse it even on a low memory machine but you need to do so carefully, especially if you are also trying to have good performance.  One way is to read in chunks that always are either max size or end in a newline.  That way you will always start each line at the beginning of a buffer and the delimiter can never be split across blocks.

One important thing to consider for small memory devices is how to handle a failed POST.  If the client drops the connection you should make sure that the entire POST is reverted.

[tellurium]

Do whatever you wish

Just to note - as far as I know, the HTTP method (POST or PUT), is irrelevant to the encoding.
The multipart encoding (which browser does on "standard" form upload) can be done with either PUT or POST.
Likewise, a binary encoding, without multipart chunks/boundaries, can be done with either PUT or POST.

What matters is the body format, which is specified in the Content-Type together with Transfer-Encoding headers. Thus, there is nothing messy about POST and good about PUT.