Anyone here familiar with MbedTLS?

[peter-h]

With a colleague, I am working on a product which previously used PolarSSL but was later changed to MbedTLS which, in the ST ARM 32F417 implementation, is believed to be less buggy.

It is working ok as far as we have got, but we are finding that as well as needing some 100k more FLASH, it needs nearly 50k of RAM, which on the 32F417 (128k RAM) would normally be OK but due to other required functionality it leaves us just 10k, which is sure to bite us in the bum one day.

The architecture we are using for MbedTLS is a 48k (48k was found to be the smallest that works) static buffer within which TLS runs its own heap. I am aware that in the most generic case one has to have
enough for one 16k buffer (plus a bit) for HTTPS, but what concerns me is that there appears to be no way to determine the worst case memory usage, across various usage scenarios.

It would be great to reduce this 48k to say 30k.

Some of the protocols which come from ST we know we don't need (e.g. PPP) but within any of them there is the question of which cipher suite needs supporting.

What is the minimum cipher suite required to be able to use MbedTLS as an HTTPS client for use with typical current cloud-based file storage or data logging APIs such as Dropbox, Google Drive, Loggly etc? 

And what is the recommended minimum cipher suite required to implement an HTTPS server that would be able to negotiate a session with most current web-browsers?

The goal is to ideally reduce both RAM and code size requirements for use in an embedded device that needs to work both as a general purpose server and client, without requiring support for multiple
 concurrent sessions.

The other thing is that even the ST port of MbedTLS doesn't appear to make use of ST CPU hardware features such as AES256, DES, etc, which the 32F417 has in hardware. This speeds things up hugely but much more importantly in our case, saves a lot of RAM. For example AES256 can use about 10k if done in software.

I think the main questions are

1) Which cipher suites are worth supporting? For example DES arguably isn't, but you never know what some customer might want, and if everybody else thinks DES is dead, there will be money in doing it

2) What can be done to reduce the amount of RAM? Digging around the internet, lots of people have been up this path. It does seem strange that this library takes up so much space, given its name...

3) What is the best way to determine the RAM requirement? For example one can't test an HTTPS server with every possible client (although AIUI a 16k RX buffer should always do).

Thank you very much in advance for any input.

[soFPG]

I've used MbedTLS during work and we ended up using TLS v1.2 because, afaik, TLS v1.3 is not supported (at least not in the version which came with Mbed OS).

I had to use a lot of debug functions to track down why MbedTLS would crash after a request.

However, we had an external SDRAM attached to the STM32F4 so enough memory was available.

Unfortunately, we couldn't get it to work 100% reliably because sometimes either the HTTPS JSON-response was faulty (invalid character somewhere in the string) or the board crashed with a Hardfault upon parsing the response.

Up to this day I believe that either there is a bug in the HTTPS code which handles the response or the STM32F4 memory protection unit can't handle unaligned access even if it should be enabled.

In the end, the experience wasn't too great and we ended up with a Qt-App on Raspberry Pi which worked flawlessly after 2 days of work.


We probably did something wrong but I think as soon as you go from HTTP to HTTPs you should think about switching to an embedded linux device which has a complete network stack which is tested for decades.

I remember how many google searches it took to figure out that you have to add a parameter string "st0" to get the DNS working otherwise Mbed wasn't able to resolve the URL 

[peter-h]

I guess most people implementing this stuff are doing it in a company, in a paid job, so can't talk about it in detail. The usual problem... anyone working in this field ends up googling for many hours, finding forum after forum full of posts with zero replies, occassionally coming across a post which points you to a bug fix.

I am not working on this part myself but as the owner of the business I am sort of the "project manager" so need to know roughly what is going on, and I am writing the documentation so I need to understand it. Various things amaze me, especially how non-viable this HTTPS stuff is on typical embedded micros such as a 32F4, whose 168MHz speed, 1MB FLASH and 128k/196k of RAM are otherwise extremely powerful for embedded work. Just the collection of root certificates comes to 200k, and MbedTLS is supposed to be able to traverse through this lot when establishing the identity of a server it is connecting to. There is not enough RAM for a start. It's not fast, either, taking around 5 seconds for the simplest session setup (a "I am alive" ping to a server). It can just about be made to work if you are running a single session, or if you just run TLS to establish a secure session but without server certificate verification (unless the server is a private one in which case you need just 1 certificate stored - probably most "IOT" products are doing it this way).

So no wonder so many people ditch this stuff and stick a board in there running Linux. Then you have proven code, GB of RAM, and can do "everything". An embedded product trying to do this will have a very limited functionality, carefully chosen to do a specific job.

[soFPG]

Yes, 100% agreement.

I think about it like this: At least there is an option to run HTTPS if you are willing to connect an external SDRAM.

But why did you chose the STM32F4 if you are aware that this would probably be solved much easier on an embedded linux machine?

[peter-h]

The problem is that if you are going to build a box which can run linux, and run it at a performance level sufficient to even produce config screens which take less than a few seconds to load (ever played with low-end linux boxes like Linksys?), then you are looking at hardware which will cost quite a bit more, and draw quite a bit more power.

The product I am working on has way more than enough performance for its intended application, which includes various "web" (or "IOT", in modern fashionable language) functions. It is just TLS which is proving to be a problem, but we don't need any real performance on TLS; it just "needs to work".

[Foxxz]

One thing I would recommend and have seen others do to lighten the load on the CA cert size is not to install any and provide a method for installing just the one specific for the server you will connect to. Or if you ultimately control the server it will use install your own CA with a 50 year expiration.

I've seen IOT devices just accept any certificate valid or not. But I'm assuming if you are running TLS you want actual security.

If you don't need to encrypt the information but still verify it's authenticity you might go for a more lightweight approach such as a MAC/HMAC/signature based authentication of messages.

[peter-h]

" is not to install any and provide a method for installing just the one specific for the server you will connect to. Or if you ultimately control the server it will use install your own CA with a 50 year expiration."

Yes indeed; this is the conclusion we came to as well.

If talking to a private server under your control (probably the most common solution for "IOT" devices) then you load just that one certificate, and you put the max lifetime on it. Then users can have both encryption and host authentication.

If talking to some "public" site (e.g. microsoft.com although obviously one would not actually be doing that one ) then encryption is easy enough because the site will give you that certificate allright, but if you want to be sure you are talking to "microsoft.com" and not a spoofed version then you have much more work. But more importantly you need a secure means of periodically updating the root certificates (around 130 of them, AIUI) because if anybody can get in there to update them, you can be spoofed. Browsers deal with this by grabbing new certificates during a browser update (or maybe the OS itself does it? - if that was actually the case then no browser would now work on winXP or even win7, but FF still runs ok on XP, and win7 is fully functional with all including Chrome). Anyway, this aspect is going to get dropped because we don't have enough RAM for the 200k block. And the current certificate for microsoft.com will change all the time so you can't just preload it and in effect treat microsoft.com as a private server. We do have a serial FLASH file system (FAT-FS) into which we can load a 200k file (so it isn't taking up 200k in the CPU FLASH) and I don't know why MbedTLS can't just read one certificate at a time into RAM - 2k would be enough - as it is stepping through them. I am not the person working on this bit and I only just understand how it is supposed to work. But if MbedTLS (which AFAICT is the typical totally unsupported open source product, with user forum(s) which almost nobody participates on) really needs 200k of RAM for this, it is pretty useless for "embedded" work.

There is no obvious way to securely update the root certificate block in an embedded system which is an industrial product, with no keypad/LCD, and has to work for many years. The most obvious option (holding a button at power-up, or similar, or doing it via USB) needs physical access. And needs this to be documented, but any docs will be lost after a few years, if not immediately. Same problems with a remote update; the product will either have to be on an open port (which raises all sorts of issues) or has to fetch a certificate bundle from a private server (which needs to be maintained, "for ever"). Then the customer ends up with a time bomb. In the early 1980s I designed a product which had code to generate day of week from the date (an industrial multizone heating controller, which had different programs for different days, so needed to know the day of week) and the way I did it would break after 29 years. So they would all stop working in 2012 The company selling it went bust in 1993 so nobody faced the music for it but I am sure a fair number (1000s were sold) were still installed and running in 2012. But that product had a user interface and the fix was to set the date 29 years back

BTW we don't have enough pins left on the 32F417 (QFP100) to connect an external RAM chip.

[peter-h]

Back to this old one, and looking for opinions on how long before TLS 1.3 is likely to be mandatory for an embedded system.

MbedTLS supports 1.3 but doesn't yet offer it as a solid branch of the code. So it seems to early to consider it anyway.

Obviously if one is (as a client) connecting to a server under one's control then this is moot because you can use whatever crypto you like (unless the customer makes some demands).

Also MbedTLS has the issue that it can't process a chain of certificates without getting enough RAM to load the whole lot - potentially about 200k. There is a patch here
https://github.com/mongoose-os/mbedtls
which enables processing of a chain of certificates but it was fairly complicated to make it work. If one does not do this patch, one is limited to either 1) not authenticating the server or 2) connecting only to a server under one's control (and then you have just one cert to deal with). Or find a CPU with another 256k RAM (which is a whole new game)... Updating MbedTLS means doing this again.

Stuff like TLS is always going to be a problem with embedded development. It isn't a browser which gets updated every x weeks or months.

How many years do people here think before TLS 1.3 will become de facto mandatory?

[Siwastaja]

Back to this old one, and looking for opinions on how long before TLS 1.3 is likely to be mandatory for an embedded system.

I have now been going on this MbedTLS... trip since three days ago, what an experience! Just the number of Google searches needed is absolutely baffling, as one needs to understand every freaking TMLA (Three or More Letter Acronym) to understand what it does, and whether it is needed or not. My constraint was 150k of flash and 40-50k of RAM, for the TLS client part that is. Payload is short, infrequent mqtt messages. One server with self assigned "CA" "certificate".

Of course after just three days I have nothing running, but I have the library compiling (and if it compiles, it is verified, right!?), and afforded even a massive 24KB buffer for the mbedtls_memory_buffer thing (which replaces calloc/free) after all the statically allocated stuff. I have absolutely no idea if that will be enough or not before I try it out. Had to do slight modifications to the sources to remove completely unnecessary calls to stuff like snprintf which is not used and would be optimized out during link anyway, but prevents compilation. Just extend the already existing #ifdef #endif stuff to a few leaf functions only used from functions already getting removed, but not marked static.

Anyways, I decided to go this hey, $current_year is 2011+11+1, so only support TLS1.3 way because admittedly, TLS1.3 drops a whole lot of legacy and dangerous bullshit. OTOH, it did not make the size optimization that much easier, because you still have to go through every mbedtls feature out separately and disable everything not supported by TLS1.3; they did not spent the effort of adding another #if !defined(MBEDTLS_SSL_PROTO_TLS1_2) to turn all the crap off automagically.

But at least when you encounter QWERTYXYZ, you can Google "QWERTYXYZ TLS1.3" and find someone on the internetz saying that "oh, QWERTYXYZ was removed from TLS1.3".

Oh I have so much wanted to open some peter-h style threads on this journey so far but have restrained myself. And from this day, I will sell myself as a world-class TLS expert consultant who knows every freaking acronym.

I still don't know if the key size with TLS_AES_128_GCM_SHA256 is 128 bits or 256 or something else, since the Internetz has contradictory information about such primitively simple questions, but I guess I either find out the correct answer, or the fact that the question itself is stupid, within a few days of more googling.

So maybe time is now ripe for TLS1.3-only solutions; at least when you manage the devices and the server. So far it looks the support in mbedTLS is... ok? Without testing, of course. But at least it compiles.

Of course, downloaded from the development branch. I have learned never to trust so called "stable" branches, but always go for the latest. Sometimes stable branches are good, of course, but often they are just a nice pipe dream, and in reality are just so freaking obsolete, possibly not even without critical bugfixes, that any experimental new branch will be better.

[peter-h]

Welcome to the club

With TLS you need much more RAM than with HTTP unless you control the machines at both ends. This is because

- TLS needs RAM for whatever combination of crypto algorithms gets negotiated between the two ends. This is not deterministic and you don't find two servers doing the same thing, some use RSA, some use EC, many versions of these, some use AES256, some chachapoly, curiously Cisco use AES128, etc, etc.

- TLS (or HTTPS?) needs two 16k buffers as standard. You can reduce one of these if you only transmit small bits, and you can reduce the other if you know the other end will only transmit small bits. You can see this is why "IOT" tends to talk to private servers only

- the overall operation is a mystery to almost everybody, and while there is a mailing list on which some people do respond (unlike LWIP which died ~10 years ago as far as any community support goes) you get a certain amount of response to narrow questions but no more.

In my project MBEDTLS takes up maybe 200k. It went up quite a bit after a recent update to V2.16.2. But we aren't doing TLS 1.3 which seems completely pointless, mainly because almost nobody supports only 1.3 so you can get old suites requested. We found that when trying to ping some ransom sites. Some use very old (supposedly deprecated, according to 97.5% of all the amateur brain surgeons / crypto experts) hashes, so you have to support all the old stuff too. Maybe not DES/3DES (but I would not bet on it) but everything else. We did finally move AES from software (which was quite fast at almost 1MB/sec) to hardware on the 32F417 purely to save about 12k. Moving to hardware DES saves 7k.

Also MBEDTLS can handle only one certificate at a time i.e. you can (as a client) authenticate just one private server (which has a self signed cert which is unacceptable for "public" use). If you want to support the more usual cert chain method you need to store ~200k of root certs- the whole cacert.pem file. There is a patch for this on github which took quite a while to integrate, but it works.

I will send you some notes via PM.

BTW TLS 1.3, and limiting to 1.3, does not enable you to avoid the older crypto suites because you can still be presented with a server certificate which specifies a hash algorithm which is found in the pre-1.3 TLS We got caught with this. I don't know any more details.

[Siwastaja]

Well, indeed, if you are just a client, and if you only connect to servers managed by yourself*, you can actually limit to one (1) crypto suite if you so wish. Anyway, TLS1.3 goes down from the nearly hundred(!) or so to just five, which, according to the internets, are all usable and safe, go figure. From this set of five, I dropped out two more, and also dropped some of the elliptical curves.

*) even that does not matter - servers on the public Internet must be kept up to date. If some server does not yet support TLS1.3 in 2023, it's a good sign you should not connect to it.

If you go down to one crypto suite, then the negotiation will be determisistic, too. Nothing wrong with that.

But life is made more difficult because same acronyms can mean kind of different things in different contexts. For example, RSA is deprecated from TLS1.3, but you can still use "RSA certificates". So do you need RSA enabled or not? I don't know. I disabled it, of course. I can generate all certificates with ecparams. Would it work if someone else created an RSA certificate for me? I don't know. Is "RSA certificate" even a meaningful phrase? I don't know. I loathe people who talk about shit they don't understand properly, working with cargo cult like expressions. Yet I now find myself being in that position.

[peter-h]

I am no expert on this, but...

Yes if the server is wholly under your control, then you can do what crypto you like. You could just forget TLS, use a shared key, and save yourself a massive amount of hassle. Lots of people do that...

I appreciate you won't be connecting to cisco.com one day and microsoft.com another day, but there are loads of servers which are for industrial data acquisition (MQTT "brokers"?) and you don't control their config in the long term, unless again you totally own it.

If talking to own server that does not mean that server must be public-facing. You could firewall it to accept only certain kinds of traffic. You don't have to use port 443 (although chinese sniffers will find whatever port you do use). You can do port knocking from your embedded box. Probably the other side of that server will be public-facing (for data viewing or whatever) but your box won't be connecting to that.

Nearly everything written online about security is BS, and if you need real security (against say, Russia hacking an electricity substation) then you won't achieve it with a little box. The crypto is not the weak link, and probably never was.

If some server does not yet support TLS1.3 in 2023, it's a good sign you should not connect to it.

A view expressed by the chief dev of MBEDTLS was that the embedded world is going to be on 1.2 for many more years.

[Siwastaja]

I appreciate you won't be connecting to cisco.com one day and microsoft.com another day, but there are loads of servers which are for industrial data acquisition (MQTT "brokers"?) and you don't control their config in the long term, unless again you totally own it.

You can require the servers to use up-to-date TLS implementations, that's completely sane requirement that can be fully rationalized with security. If security is unimportant, then why to encrypt at all? Either use plain unecrypted TCP, or use - and require others to use - up-to-date cipher suites. The whole idea of TLS1.3 was to remove all the deprecated, dangerous ciphers.

If you support just two ciphersuites, say TLS_AES_256_GCM_SHA384 and TLS_AES_128_GCM_SHA256 on the client, a server which does not have these would be either too old to be considered safe, or configured very weirdly.

The "muddiness" whether to use TLS1.3 or not is going away pretty quickly as we speak. But even if you decide to support TLS1.2, you can still choose just a few known-good cipher suites.

[Siwastaja]

If some server does not yet support TLS1.3 in 2023, it's a good sign you should not connect to it.

A view expressed by the chief dev of MBEDTLS was that the embedded world is going to be on 1.2 for many more years.

There is no contradiction.

Servers must support TLS1.3; but they also need to support TLS1.2 because many old clients that are harder to update. But servers must be kept up to date.

Servers (usually) have to support both. Clients are fine supporting just a limited set. As the embedded client designer, you have the luxury of choice. As the embedded client user, you don't have that.

[peter-h]

Good luck

[Siwastaja]

Maybe I'll just update here if someone's interested.

Got MbedTLS running and working, obviously it was non-trivial, but no more than 2-3 days of extra work. This is partial list of issues I faced:
* As usual, the few important interface functions user needs to supply are completely or partially undocumented. Would have been 10 minutes job: for example: mbedtls_hardware_poll (RNG callback): function prototype is non-trivial and the four arguments are completely undocumented, you need to guess
* Examples are either too simple or too complex. You have to shuffle between ssl_client1 and ssl_client2 examples.
* PSA is mandatory in TLS1.3, which means both enabling it and remembering psa_crypto_init()
(I still have no freaking idea what PSA is. I Google every stupid acronym for 2-3 minutes after which I timeout myself. For PSA, the best explanation was that it exists for "demystification".)
* With read callback functions, timeout of 0 does not mean instant return, but infinite timeout (blocking call) instead.
* Many different errors return the same error code, so you need to either write a logging system suitable for your use case, or modify the code to spread the error codes out
* Documentation makes it appear like TLS1_3_COMPATIBILITY MODE is for some weird middlebox thing. In reality, OpenSSL (so basically all servers) seem to require compatibility mode enabled. Enable it.
* SESSION TICKETS must be enabled; documentation makes it appear like some sort of optional feature. Doesn't work without. No need to write any callbacks.
* MbedTLS returns various error codes from mbedtls_ssl_read even when there is no error. You need to treat them as valid zero count reads. There is no maintained list of all possible "FYI" "errors". MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET is one of the fun ones; called experimental error code which might get modified or removed any time. MBEDTLS_ERR_SSL_WANT_READ is at least understandable and somewhat analogous to WOULDBLOCK from the POSIX world.
* With the limited RAM in embedded, it's a good idea to enable SSL_MAX_FRAGMENT_LENGTH extension, and configure MBEDTLS_SSL_IN_CONTENT_LEN and _OUT_CONTENT_LEN to something less than the default 16384.

But hey, the thing seems to work. And I can now confirm that with all EC keys, none of the RSA stuff is required.

[Siwastaja]

Another random comment.

For ****s sake, I have been banging my head on the wall about the incompatibility of EC curve support, as mbedTLS and openssl seemingly do not agree at all what is available. This was supposed to be standardized.

Turns out, people have just invented different nicknames for the exact same curves, and those who know are either lazy or sadistic not to ever mention it anywhere.

Now I truly hope Google will find this ****ing post when the next victim googles this:
Keywords: TLS, EC curves, ECDHE, ECDSA, secp256r1, prime256v1
Fact: secp256r1 IS THE SAME AS prime256v1

No wonder how all the key generation tutorials used this mysterious "nonstandard" prime256v1 curve. It was standard after all, and definitely supported by mbedtls.

[Nominal Animal]

For ****s sake, I have been banging my head on the wall about the incompatibility of EC curve support, as mbedTLS and openssl seemingly do not agree at all what is available. This was supposed to be standardized.

I feel your pain.  I have felt this exact pain too.

Pretty exactly a decade ago I had issues with OpenSSL and GnuTLS cipher suite compatibility selection (on the server software side, software supporting both libraries).  I even went as far as look into rewriting GnuTLS's cipher suite listing spec and selection algorithm, but as the proper cipher selection was more important to me than my users, and there was no way in hell the OpenSSL folks would have accepted any suggestions from me no matter how small or backwards-compatible, I decided I had to let it go.

(The GnuTLS devs were very nice and friendly, though.)