I want to design my own STM32-based device, but I'm running into lots of issues

[SparkyTD]

What are you doing with all the flash?

Storing some logging data in the internal flash? Don't you have the SD card for that?

Nice and large graphics for UI? Consider compression (or accept slightly lower image quality).

Complete large adventure games with quite nice-looking graphics, sounds and music fit in 2MB just fine in early 1990's.

Are you really sure this is a project that should run on a microcontroller? Maybe an application processor would be a better fit? Or an existing SBC?

In other words, what are you doing? How much can you tell about your project?

The project is a smart electricity meter that should log consumption data from 7 channels with 1Hz frequency on an SD card, but one of the big challenges is the ability to also host a website that can serve that data quickly and reliably. For this I would need to use an sqlite3 database on the SD so I can request partial data, or to write more advanced queries. The webserver part was causing me a lot of problems (due to large libraries like jQuery and bootstrap) on the esp32 prototype, so that's why I would like to redesign the whole thing on a more powerful chip.

I've thought of building it around a Raspberry Pi Zero, but the chip shortage hit those pretty hard too, so they're even more difficult to find than STM chips.

As for the Flash requirements, my current ESP32 codebase that implements 80% of the features I want, compiles to a 1.1MB binary, so I need at least 2MB for extra headroom.

My current prototype is using a small 1.8" TFT display with no fancy graphics, and that would be enough for me, but the display still needs an SPI interface to be able to work in parallel with all the other devices.

[Siwastaja]

If you want to run existing networking, web serving and database stack, I'd do it (and have done similar projects) as a combination of any cheapest MCU doing the HW measurement things and communicating through UART with any SBC which runs standard linux for non-timing-critical web serving, logging, etc. Being standard linux computer with standard interfaces, chip shortage isn't a big problem, pick any SBC and write standard POSIX compliant code for it so that little or no porting is required (just to make UART work, for example).

Raspi isn't the only one. There are more robust and more industrial computers available, always been. Obviously, they cost more, but for one-off (or making a few dozen), does it matter?

For low cost, you would be aiming to produce hundreds of thousands of devices and should be able to afford a professional to make it all happen on the single cheap microcontroller. But then you really need to know what you are doing to be able to make the databases and web servers happen within, say, half a meg of flash. And do that reliably.

[SiliconWizard]

The project is a smart electricity meter that should log consumption data from 7 channels with 1Hz frequency on an SD card, but one of the big challenges is the ability to also host a website that can serve that data quickly and reliably.

Can you detail this a bit?

You mean that you'll want to access the meter via a web page when connecting to it on the network with a web browser, so that the device must act as a basic http server?
If so, this is not uncommon at all and you don't need all that much resources either. Unless you want it to be able to serve many client connections concurrently...
There are small http servers that are a good fit for that. Of course you should then probably avoid using things like PHP. Which, if the device builds its own web pages, wouldn't make sense whatsoever anyway IMHO. So maybe a little more details?

[Siwastaja]

Generally, you can simplify the HW requirements of small IoT gadgets by not making them full-blown web servers (that's problematic anyway - you'd need to install them in networks without firewalls to allow access to the web server from public Internet), but instead push the data using some simple interface into "cloud" i.e., the server somewhere else. That server elsewhere can then run whatever fancy database system to enable queries; and serve web pages for the whole planet.

[poorchava]

The first issue you have to solve is: where will you buy an actual STM32 in question?

[SparkyTD]

The project is a smart electricity meter that should log consumption data from 7 channels with 1Hz frequency on an SD card, but one of the big challenges is the ability to also host a website that can serve that data quickly and reliably.

Can you detail this a bit?

You mean that you'll want to access the meter via a web page when connecting to it on the network with a web browser, so that the device must act as a basic http server?
If so, this is not uncommon at all and you don't need all that much resources either. Unless you want it to be able to serve many client connections concurrently...
There are small http servers that are a good fit for that. Of course you should then probably avoid using things like PHP. Which, if the device builds its own web pages, wouldn't make sense whatsoever anyway IMHO. So maybe a little more details?

Yes, I want to implement a very basic http server that serves static files (i.e. no server-side processing like PHP), and the rest would be done in the client's web browser with javascript and a WebSocket connection to the MCU. I've tried doing this on the esp32 prototype, and it did sort of work with a very basic html file, but as soon as I tried adding large libraries like jQuery, amCharts and bootstrap, the esp32 started to struggle serving the files due to memory issues, and probably poor file handling (e.g. the minified jquery.js file is ~90kB).

It's okay if the web interface takes a second to load, it doesn't have to be instant, and also I'm not expecting more than 2 clients accessing it simultaneously. I think a more powerful STM32, let's say an H7 series should have no problem serving that web interface, given that I can find a nice and fast wifi chip.

[SparkyTD]

Generally, you can simplify the HW requirements of small IoT gadgets by not making them full-blown web servers (that's problematic anyway - you'd need to install them in networks without firewalls to allow access to the web server from public Internet), but instead push the data using some simple interface into "cloud" i.e., the server somewhere else. That server elsewhere can then run whatever fancy database system to enable queries; and serve web pages for the whole planet.

This is what I'm doing in my current working esp32 prototype. The website and the database are hosted on my AWS instance. The problem is that I really don't like the idea of internet-dependent gadgets, even if I'm in total control of the cloud service. Also, if I wanted to "debug" my home's excessive power usage by unplugging things and seeing how the consumption graph changes in real time (which was the original goal of this project), it would be very difficult because there are two different routers separating the meter from the internet, and if any of them was to go down, the whole device would become useless.

It would be much nicer if all of it was self contained withing the MCU, including the web server and the database, and it could act as a WiFi hotspot so I can walk around in the house with a tablet in my hand, testing the consumption of each individual wall plug and extension cord, while still being able to see the live data.

[SparkyTD]

The first issue you have to solve is: where will you buy an actual STM32 in question?

There is currently a H7 series chip with 5 units available on stock on one of the online electronics stores, with 220 more arriving in a few days. I am thinking about picking up a couple before it's gone for years, but at the same time, I'm also trying to figure out if it could even solve my problem, hence this forum post.

[apurvdate]

You can offload the display interface to a smaller AVR controller. Communicate AVR with your host controller via UART & update display via AVR SPI. Any arduino compatible AVR will ease your coding part. Additional IOs / peripherals can also support your host controller.

[Siwastaja]

You can offload the display interface to a smaller AVR controller. Communicate AVR with your host controller via UART & update display via AVR SPI. Any arduino compatible AVR will ease your coding part. Additional IOs / peripherals can also support your host controller.

No, don't do that. Using an undersized 8-bit microcontroller not really suitable for graphics, to generate graphics, then force yourself into the realm of multi-mcu design (keeping two firmwares in sync) makes absolutely zero sense when you can just pick any modern ARM microcontroller which comes with proper display abilities (including DMA) out of box and can do the graphics fine, and everything else at the same time.

[SparkyTD]

You can offload the display interface to a smaller AVR controller. Communicate AVR with your host controller via UART & update display via AVR SPI. Any arduino compatible AVR will ease your coding part. Additional IOs / peripherals can also support your host controller.

I don't think I have to do that. I plan on using a display with an SPI interface, so as long as the STM32 has a free SPI port, I see no need for offloading it to a second MCU.

[SiliconWizard]

Seems that the only real performance/memory factor that causes you questions here is this web server thing? Am I right?
As I suggested, while I think it's a good idea, you may want to work on that part a little bit to maybe make it less resource-hungry, so you have more options for an MCU.

As to the display, I don't see a single reason to implement it on another MCU either, since it's probably going to be relatively modest in terms of CPU use, at least I suppose so (and then certainly any very low-processing power one would not cut it anyway!), there shouldn't be any concern. As I see it, the web access will allow to give a pretty UI interface, while the display will be to just display the immediately useful information without needing to look particularly fancy.

[thm_w]

Yes, I want to implement a very basic http server that serves static files (i.e. no server-side processing like PHP), and the rest would be done in the client's web browser with javascript and a WebSocket connection to the MCU. I've tried doing this on the esp32 prototype, and it did sort of work with a very basic html file, but as soon as I tried adding large libraries like jQuery, amCharts and bootstrap, the esp32 started to struggle serving the files due to memory issues, and probably poor file handling (e.g. the minified jquery.js file is ~90kB).

It's okay if the web interface takes a second to load, it doesn't have to be instant, and also I'm not expecting more than 2 clients accessing it simultaneously. I think a more powerful STM32, let's say an H7 series should have no problem serving that web interface, given that I can find a nice and fast wifi chip.

A lot of sites just hotlink jquery.js, thats what I did for ESP8266.
Are you saying: you don't want to be dependent on an internet service, or you don't want to be dependent on internet being available at all?

[SparkyTD]

A lot of sites just hotlink jquery.js, thats what I did for ESP8266.
Are you saying: you don't want to be dependent on an internet service, or you don't want to be dependent on internet being available at all?

I don't want to be dependent on an internet connection at all. If I could find a memory-efficient way to serve ~1.6MB of javascript and some html from the SD card with reasonable speed, that would be perfect. Currently the biggest library would be echarts with a 1MB minified js file. Sure, I could probably find a smaller chart library, but I want my web app to look nice and modern. And the current cloud-hosted web app is already using this library.

[SparkyTD]

Seems that the only real performance/memory factor that causes you questions here is this web server thing? Am I right?
As I suggested, while I think it's a good idea, you may want to work on that part a little bit to maybe make it less resource-hungry, so you have more options for an MCU.

Yes, that is currently my biggest concern. The database solution is also a bit concerning, because in the current ESP32-based prototype the user can select all sorts of different time ranges on the consumption chart (hours, days, weeks, even an entire month) which means that in order to calculate the accurate kWh, it has to be able to quickly calculate the sum of up to millions of DB records. Even my PC with a relatively decent i7 CPU takes about ~300 milliseconds to query all the data necessary from an SQLite database file, and I'm worried about how a 480MHz ARM core would perform. I might have to look into implementing a custom, more efficient storage format.

[SiliconWizard]

I don't mean to judge and I don't really know how your system works at this point, but I'm just wondering if this amount of data (millions of records) is really needed for a smart meter.
You can probably get as useful and as accurate results while lowering your sampling rate a lot. Not sure you really need to *store* all this data.

[fchk]

I would split the whole system in two parts: a sensor part that grabs measurements and sends them periodically, and a processing unit that stores, processes and displays the measurements from one or more sensors.

The processing unit may be a PI or a small PC (NUC etc). For the sensor platform you may keep your ESP but keep the software there as simple as possible and just do measurements.

You also could use these sensors

https://www.eastroneurope.com/products/view/sdm120modbus (ok, these are 230V parts, but things like these should also exist for 120V systems)

or others, and you could later on process also other sensors like temperatures or water consumption etc etc.

fchk

[peter-h]

It is perfectly possible to implement a web server on a 32F4 device. I have one running here, and even TLS/HTTPS is almost working... The libraries out there for this stuff are full of bugs, and while lots of devs have independently fixed the bugs, they aren't publishing the fixes because they were paid by their employer to do this They probably spent a man-year just fixing ST's ethernet libs, polar SSL or EmbedTLS.

However, nearly all applications where data is being collected for retrieval elsewhere, upload data to a central server, from where it can be retrieved. This has various advantages e.g.

- simpler to run a web client than a web server
- can use a 3G/4G (or even GPRS) connection to the web
- IP doesn't have to be fixed
- no public visibility of the server port (80/443 etc) avoids constant hacking attempts by chinese/russian hackers
- no storage limit at the server
- no need to always be powered
- much more secure, due to no fixed IP, and any public network effectively placing it behind NAT
- can use nonstandard protocol to push the data to the server e.g. just bare UDP, avoiding all the TCP/IP complexity (no DHCP, etc)
- etc

You can set up your own server, which is very cheap these days, although you do need to have a backup strategy, and somebody needs to be in charge of that. But this is a well worn route - check out pricing at say Linode, for the virtual server options which most people use these days unless they need massive storage (terabytes). Or you can use e.g. Loggly https://www.loggly.com/ for a ready made server.

The hacking issue is very real. The IP will be discovered within hours, more quickly if you publish the DNS, and the server will be hammered with thousands of attacks per day. Eventually they will find a back door in your server code, which after all was just some old code with patches applied as they were found on forums

A smart meter may need to take fast measurements because it is calculating real time power (watts, and reactive VA) by V x I multiplication and that needs a lot of samples per 50Hz cycle. But you don't need to upload the data. Only aggregated samples.

[asmi]

SD card and display often get their own SPI buses for the obvious bandwidth reasons. Could be QSPI for the SD.

SD card does not support QSPI, only a single-bit SPI. SDIO != QSPI, it's an entirely different protocol.