Quick ESP32 project - Arduino, ESP IDF, Micropython, Tasmota?

[Warhawk]

Hello everyone,
I've been around embedded programming professionally as well as for hobby. My hobby projects use typically 8-bit microcontrollers (MSP430, ATmega, PIC..). I prefer vanilla C, registers access, and simplicity.

However, I would like to do a little project for home automation. The device shall decode a serial bus from Buderus heating system and report parameters to my home assistant server that uses MQTT. For the transport layer, I could use ZigBee but WiFi seem to be easier. I like the ESP32 device.

I came to the conclusion that writing vanilla C code for the said application would be enormous effort. I would never finish this project. For this reason, I am looking at other options.

• Tasmota in ESP32 + small MCU (that I know well) for decoding the bus. I would then just stream JSONs over UART to ESP32 and let it do its job.
• Micropython in ESP32 - there are some examples online that seem to be easy
• Arduino - I've been always hesitant to use arduino. It is something that professionals avoid. However, I must admit that I blinked an LED with ESP32 in 5 minutes. Still, it feels like a toy.

Here are my prirorities:
1. Frustration free
2. Stable end-application (Tasmota?)

This device should run 24/7. When it fails it is not the end of the world. It is not controlling anything. However, I hate when I need to babysit electronics.

Thank you for any tips and other suggestions.

Best regards, Jiri

[globoy]

I can't speak to Tasmota but either Arduino or Micropython running on the ESP32 would be fine for this task.

If you know python then Micropython would probably be a quick way to get this done and you wouldn't have to code for another micro-controller.

Arduino would probably be more code but for the exact reason people sometimes discount it would be less code than a straight C Espressif IDF project.  Aside from the intentionally simplistic IDE (since it's aimed at a very broad audience) it has two characteristics that help project like this.  There are libraries with simple API's that wrap fairly complex systems such as WiFi and mqtt.  You can get onto wifi and communicate with literally only a few lines of code.  And if you want the entire Espressif IDF API is available as well (under the covers, so-to-speak).  I've written multi-CPU programs for ESP32 using Arduino with all sorts of features including semaphores and access to IDF libraries.

There's nothing inherently un-stable about either environment, given your code is reliable.  I've never shipped a product that's built using the Arduino IDE but I have had systems I've built for other purposes that run 24/7 for years at a time.  One thing to contemplate no matter what your solution are WiFi disconnections and having your code know to re-connect since you aren't in control of the other end of any WiFi connection or the servers you might be talking with.

Seems to me that all computer work is frustrating at times :-)  I find biting off a little at a time and slowing building an app like this works reasonably well.  Good luck with your project!

[Warhawk]

I can't speak to Tasmota but either Arduino or Micropython running on the ESP32 would be fine for this task.

If you know python then Micropython would probably be a quick way to get this done and you wouldn't have to code for another micro-controller.

Arduino would probably be more code but for the exact reason people sometimes discount it would be less code than a straight C Espressif IDF project.  Aside from the intentionally simplistic IDE (since it's aimed at a very broad audience) it has two characteristics that help project like this.  There are libraries with simple API's that wrap fairly complex systems such as WiFi and mqtt.  You can get onto wifi and communicate with literally only a few lines of code.  And if you want the entire Espressif IDF API is available as well (under the covers, so-to-speak).  I've written multi-CPU programs for ESP32 using Arduino with all sorts of features including semaphores and access to IDF libraries.

There's nothing inherently un-stable about either environment, given your code is reliable.  I've never shipped a product that's built using the Arduino IDE but I have had systems I've built for other purposes that run 24/7 for years at a time.  One thing to contemplate no matter what your solution are WiFi disconnections and having your code know to re-connect since you aren't in control of the other end of any WiFi connection or the servers you might be talking with.

Seems to me that all computer work is frustrating at times :-)  I find biting off a little at a time and slowing building an app like this works reasonably well.  Good luck with your project!

Thank you, this is extremely valuable feedback. A friend of mine also mentioned that MicroPython could be the right thing to start with. I saw the C Espressif IDE but honestly, I am not willing to bother with it. I admit that I have limited Python knowledge but I am willing to learn something new. 

[Lindley]

As you want something quick and ready to run, have a look at this site which covers mainly Arduino, but also some MicroPython MQTT.
https://randomnerdtutorials.com/

[paulca]

Not sure about micropython, but I did make a serial RS485 device monitor.

https://www.eevblog.com/forum/projects/nodemcu-esp8266-rs485-epever-solar-monitor-diy/

Bizarrely, this is still sitting in the garage and I don't think it has ever, ever been switched off, it is connected to solar power of course and it's never missed an update, never gave a rubbish value either. The only minor issue I had was an ESP8266 out in the garage in a tiny plastic box does not do well with Wifi reception, buts that's not its fault.

Annoyingly, it has yet to under go the MQTT upgrade the rest of my stuff got.  So it still uses a bridge script which republishes it as individual MQTT topics.

[paulca]

As another option I have the code base to link the ESP32 to the STM32 as a pure Wifi+TCP+MQTT module, including the UART command and data channels.  It's not on public github, but if you are interested, the STM32s will make a much better option to the arduino

..although I keep forgetting the MicroPython requirement.  Sounds like tractor or snail racing to me.  I don't quite get why you would take one end of the spectrum in terms of power and resources and run the other extreme language on it.  To say that micro-phython is SLOW is such an understatement it's a joke.  It takes it half a second to count to 100,000!

I'm trying to think of the right analogy.  It's like buying a moped to tow a caravan or something.

[bidrohini]

I think Tasmota will be just fine.

[tooki]

Hello everyone,
I've been around embedded programming professionally as well as for hobby. My hobby projects use typically 8-bit microcontrollers (MSP430, ATmega, PIC..). I prefer vanilla C, registers access, and simplicity.

However, I would like to do a little project for home automation. The device shall decode a serial bus from Buderus heating system and report parameters to my home assistant server that uses MQTT. For the transport layer, I could use ZigBee but WiFi seem to be easier. I like the ESP32 device.

I came to the conclusion that writing vanilla C code for the said application would be enormous effort. I would never finish this project. For this reason, I am looking at other options.

• Tasmota in ESP32 + small MCU (that I know well) for decoding the bus. I would then just stream JSONs over UART to ESP32 and let it do its job.
• Micropython in ESP32 - there are some examples online that seem to be easy
• Arduino - I've been always hesitant to use arduino. It is something that professionals avoid. However, I must admit that I blinked an LED with ESP32 in 5 minutes. Still, it feels like a toy.

FYI, “Arduino” in reality refers to three distinct things:
1. Arduino-branded dev boards.
2. The Arduino frameworks (a bunch of C++ libraries, basically).
3. The Arduino IDE.

What’s not necessarily obvious to people is that these things are not indivisible. In fact, you can cherry-pick which you want to use. (Other than trying to use a different software framework from within the Arduino IDE.)

Wanna program an ESP32 using the Arduino framework in the Arduino IDE? Done. ESP32 using Arduino framework in PlatformIO, or any of the numerous other IDEs that support Arduino? Sure! Arduino Uno board with Microchip Studio? Why not! Wanna use the Arduino framework, but write directly to registers? Go for it!

ESP32 is an interesting case, in that its Arduino framework support runs atop its native ESP-IDF framework, so you can use both at once just by including the right headers.

Since you’re an experienced embedded programmer, I’d suggest using PlatformIO. It has wide support for tons of MCU and board types (the only popular ones that are completely unrepresented are the PICs), tons of different software frameworks, and lives within a nice IDE (Visual Studio Code). Plus it has a robust, grown-up library manager. ESP32 in PlatformIO is an extremely popular combination, whether you use it with or without Arduino. $10 gets you the official hardware debugger, the ESP-PROG board. (Cheap, but a smidgen fiddly to set up.) I’ve built all my recent personal projects with this combination.

I have been meaning to order some STM32 Nucleo boards, since many of them have really nice software support (lots of frameworks) and really good built-in debuggers. But for WiFi, ESP32 is a no-brainer.

[Davor]

ESPhome maybe?
https://community.home-assistant.io/t/buderus-lagamatic-2107-km271-replacement-with-wifi-using-esp32-and-esphome/497543

[paulca]

I have been meaning to order some STM32 Nucleo boards, since many of them have really nice software support (lots of frameworks) and really good built-in debuggers. But for WiFi, ESP32 is a no-brainer.

The Blue and Black Pill are probably an easier to work with options, unless you really need 64 or 144 pin devices.  Although there are some of the L4 series in Nucleo 32 form (think Nano).

The best "general" board, IMHO is the F411CE BlackPill.  ... or if you need the extra 16 pins, the Nucleo 64 version with the F411RE.

[rstofer]

Would a Raspberry Pi Pico W (wifi) play into this?  Networking is trivial via wifi and the device can be coded in MicroPython.  Thonny is the IDE and it works well.

Here's a little sample of making a wifi connection:

Code: [Select]

import network
import time
from secrets import secrets

wlan = network.WLAN(network.STA_IF)
wlan.active(True)

ssid = secrets['ssid']
pw = secrets['pw']
# print (ssid, pw)

wlan.connect(ssid, pw)

Pretty easy!

[Warhawk]

Would a Raspberry Pi Pico W (wifi) play into this?  Networking is trivial via wifi and the device can be coded in MicroPython.  Thonny is the IDE and it works well.

Here's a little sample of making a wifi connection:

Code: [Select]

import network
import time
from secrets import secrets

wlan = network.WLAN(network.STA_IF)
wlan.active(True)

ssid = secrets['ssid']
pw = secrets['pw']
# print (ssid, pw)

wlan.connect(ssid, pw)

Pretty easy!

Thank's for the tip! It seems very similar to ESP32. I have normal Pi Zero there now. And .. well... sdcards suck

[ralphrmartin]

I use Tasmota for simple things like timed switches, coloured lighting control, etc.

I've also built a simple system where someone enters information on a webpage, which is sent to an MQTT server, and a remote thermal printer connected by wifi using an ESP32 prints out the information. I used the Arduino IDE, as
- there are libraries for MQTT and the thermal printer, but tasmota lacks the latter,
- there is less to get familiar with than if using esp-idf directly, or using the Berry language of tasmota
- maybe micropython would be fine too, but I am not familiar with it, and in general terms, there seem to be more libraries and support for Arduino

As for stability - I did have a problem with the wifi connection dropping, and not coming back up, despite trying to programmatically reconnect in what looked like the right manner. I worked around it in the end by using the following strategy in the program:
if the wifi connection is down, then turn the wifi off completely, reboot
For good measure, the device also uses a timer to reboot itself every 12 hours, in case there are memory leaks.

[Geoff-AU]

Arduino is fantastic for doing a simple job very easily.

Arduino (usually) sucks majorly trying to do 3 different jobs on the same microcontroller.

I love fiddling registers sometimes too but I also like not having to fiddle registers every single time I want to make something.  I also like having a C compiler rather than hand-holding the CPU's every instruction.

The device shall decode a serial bus from Buderus heating system and report parameters to my home assistant server that uses MQTT. For the transport layer

I'd use Arduino all the way for that.  Particularly since you admit if you need to reinvent the wheel you'll never get around to it.

Hard work pays off sometime but laziness pays off right now

[paulca]

If you want IoT NoT type stuff done.  All you need to do is buy any device that contains an ESP8266 / ESP8285 (ideally with flash+).

That opens you up to pretty much any of the pre-canned firmware options, write your own or mix and match.

When I say "any device", almost all of them will be flashable.  Things like smart plugs, LED controllers, power monitors etc. etc.   Browse for Tasmota hacking products.

You don't even need to write any code or build any hardware!  Though you can add custom code of course.  Just download Tasmota into VSCode and add what you want as a plugin.

I even used a 12V smart switch which contained an ESP8266 and had a spare GPIO to not only switch the RGB LEDs on and off, but run the whole of WLED too to configure and program them.  Was a bit of a bear squeazing everything into the flash, but I got it working.    (Unfortunately I cooked the board with a miss configured open drain GPIO which was meant to be an input pin... oops.  Bought a new one and as dumb as I am reflashed the same firmware and killed hte second one instantly).  "What, why is it pulling half an amp!?",  "Oh, it's dead.  Ohhh ouch!  It's hot too!")

The lesson there is...  when you are hacking in your own firmware on an already made device, look up the schematic and check what it expects the GPIOs to be at boot... or pay the price.

[tooki]

Arduino is fantastic for doing a simple job very easily.

Arduino (usually) sucks majorly trying to do 3 different jobs on the same microcontroller.

So… like basically any MCU?

Anyhow, are you familiar with TaskManagerIO? It’s a nifty little scheduling/event handling library for Arduino that runs on a bunch of board types. On the boards that “natively” run the Arduino libraries atop a RTOS (like ESP32 or RPi Pico), it leverages the underlying RTOS, otherwise it handles it on its own. 

[james_s]

I've been using Tasmota for a few years now, it's been great. Most of my applications are pretty vanilla but it's written using the Arduino framework and it's modular so it should be fairly easy to modify to suit your needs. 

[Geoff-AU]

So… like basically any MCU?

My point is, the Arduino ecosystem is great right up until you spend a bunch of time fighting with libraries because you've blown your timing budget.  That's usually a very complex sketch though, and now that the SAMD21 and ESP32 processors are cheap and fast you can waste a lot of cycles on sinful programming shortcuts and still have enough headroom to be OK.  Arduino lets you be lazy, and fast MCUs let you be even lazier.

Haven't seen TaskManagerIO specifically but there are a ton of schedulers out there that do similar jobs.

[james_s]

My point is, the Arduino ecosystem is great right up until you spend a bunch of time fighting with libraries because you've blown your timing budget.  That's usually a very complex sketch though, and now that the SAMD21 and ESP32 processors are cheap and fast you can waste a lot of cycles on sinful programming shortcuts and still have enough headroom to be OK.  Arduino lets you be lazy, and fast MCUs let you be even lazier.

Haven't seen TaskManagerIO specifically but there are a ton of schedulers out there that do similar jobs.

Arduino has plenty of warts for sure, but "x is great right up until you spend a bunch of time fighting with y because you want to do z" can apply to virtually every embedded platform.

[Warhawk]

Thank you everyone for follow up discussions. I read it and learn new things. 

[ralphrmartin]

Arduino (usually) sucks majorly trying to do 3 different jobs on the same microcontroller.

I have had no problems using  FreeRTOS with Arduino, with 18 tasks running updating info for an LCD.
This book provides a good way in:
https://www.elektor.com/freertos-for-esp32-arduino

[paulca]

I'm still enjoying my love/hate relationship with manually coding my buffer and task synchronisation.  Well, caveat, I am on single core MCUs.  I expect I will get bored of that and it will become a chore.  So RTOS is on the list to get round to exploring.  Buffer/task synchronisation between cores really requires careful ASM coding or... using some form of OS based IPC support... which is ESP32 IDF/Arduino's approach.  Semaphores, queues, mutexes etc.

It's been a long while since I studied low level distributed IPC, but there has to be atomic test+set instructions, cache invalidation and such to ensure correct synchronisation.   Best left to something like a microkernel.

EDIT:  I wonder if ARM architecture's "conditional statement variants" could be atomic enough to provide test+set operations.  My worry would be they would only be register level operations and not memory based, making the synchronisation a lot more difiicult as cores don't share registers.

The C code such as:

_lock = get_mutex(thing);
if ( _lock != NULL ) {}

Or aggressively
while( NULL == get_mutex(thing) ) {}

can be implemented in get_mutex as a test+set operation.  In one atomic instruction the memory address will be tested (is it already locked) and immediately set if it's found in a certain state (not already locked).  Such that the return in C land is a pointer to your hold on the mutex or a NULL if it was found to be already locked.  It does not matter how many cores or CPUs or PCs in a cluster attempt to do the same thing at the same time, the test+set atomic instruction will mean only one of them gets the lock.

[tooki]

My point is, the Arduino ecosystem is great right up until you spend a bunch of time fighting with libraries because you've blown your timing budget.  That's usually a very complex sketch though, and now that the SAMD21 and ESP32 processors are cheap and fast you can waste a lot of cycles on sinful programming shortcuts and still have enough headroom to be OK.  Arduino lets you be lazy, and fast MCUs let you be even lazier.

Haven't seen TaskManagerIO specifically but there are a ton of schedulers out there that do similar jobs.

Arduino has plenty of warts for sure, but "x is great right up until you spend a bunch of time fighting with y because you want to do z" can apply to virtually every embedded platform.

Exactly! That’s the point I was trying to make.