Worst "in production" microcontrollers

[hans]

Yes I fully agree. A firmware developer with a computer science background may get overwhelmed with this VHDL/Verilog rubbish. It's more of a hardware design wet dream.

And the FPGAs with onboard CPUs are much more capable of running Linux, are also a different order of magnitude complexity for the kind of systems they're involved in.. so the imaginary fence to throw things across is much higher..

Now I do see some very talented people in OSS community doing RISC-V+FPGA fabric experiments on ice40/ECP5/etc. , and that are able to make both domains work (like porting Doom to a RISC-V softcore CPU and making memory/graphics/peripherals etc. all work out), but I think you need a solid computer engineering background plus good software skills to pull it off.

[Berni]

Yeah in my experience the software people really don't like HDL languages. Even just looking at what the language looks like they go "eww". Then the next trap is that once you figure out the odd syntax it starts to look a lot like a weird multitasking C, however trying to use it as such does not end up going well. People who know programing apply regular programing concepts on it that don't work here, or result in HDL code that runs slowly and eats lots of resources. Coupled on top that you can easily just clock your HDL too fast and have it stop working in the weirdest and glicthyest ways (Yes i know timing analysis exists, the new guy wont tho). The more effective way is to write HDL is to imagine what kind of logic functions need to be wired together to do the job, then write that down as HDL. A software developer never worked with 74xx logic, all they know is what a AND,OR,XOR look like because they can do that in software as bitwise operations.

Heck even when it comes to drivers you can loose some of the software developers that are used to thinking in more high level terms. For example way back we had a case where the hardware engineer connected a MPU interface LCD to the external memory controller on a ARM MCU, some time later a software developer comes back in furious as to why the data bus was jumbled up across 3 different GPIO banks, he would have to do so much bit shuffling to actually get data on them. Only to be informed that the LCD data bus was wired up like that so that it would appear in the memory map and let him DMA straight into it.

[mikeselectricstuff]

Yeah in my experience the software people really don't like HDL languages.

The problem is HDL looks on the surface like software, but is a completely different concept

[Sal Ammoniac]

For example way back we had a case where the hardware engineer connected a MPU interface LCD to the external memory controller on a ARM MCU, some time later a software developer comes back in furious as to why the data bus was jumbled up across 3 different GPIO banks, he would have to do so much bit shuffling to actually get data on them. Only to be informed that the LCD data bus was wired up like that so that it would appear in the memory map and let him DMA straight into it.

The best way to avoid issues like that is for the software team to work with the hardware team during board definition. Sadly, too many teams fail to do that and it results in designs that the software team has to jump through hoops to get around, or doesn’t understand why something was implemented in the manner it was (as in the case described above).

[Bassman59]

For example way back we had a case where the hardware engineer connected a MPU interface LCD to the external memory controller on a ARM MCU, some time later a software developer comes back in furious as to why the data bus was jumbled up across 3 different GPIO banks, he would have to do so much bit shuffling to actually get data on them. Only to be informed that the LCD data bus was wired up like that so that it would appear in the memory map and let him DMA straight into it.

The best way to avoid issues like that is for the software team to work with the hardware team during board definition. Sadly, too many teams fail to do that and it results in designs that the software team has to jump through hoops to get around, or doesn’t understand why something was implemented in the manner it was (as in the case described above).

The converse of that is when the hardware engineers have to jump through hoops to implement something in a weird way because the software team defined something and can't/won't change it.

[SiliconWizard]

Embedded software developers who don't understand hardware should not be trusted IMHO anyway.
But hardware engineers who don't understand software shouldn't either for embedded development.
Tough shit.

Then there are the (very) few who are equally not too bad at both.

[Berni]

Yeah not always are the right people in on making decisions. Sometimes there is just a brain flash of a product idea of something to be made, there is no software guy assigned to it yet, but the brilliant idea gets dropped off in the hardware department so that work on it can begin ASAP, sometimes with too much hurry to even set down some solid performance targets. Then perhaps a rough prototype is build and handed over, but then a lot of code ends up developed for the prototype and then the final product has to be very close to what the prototype was since otherwise a lot of code has to be rewritten, so a quick and dirty solution to something might now be a mandatory feature in it...etc Most likely the same thing will also have to be in another similar product because they already have code to do it.

But it is clear that whenever dealing with embedded systems you need a bit of cross discipline knowledge on both sides. The programmer better have an idea of how the hardware works and at least what a pullup resistor looks like and how to read schematics. On the other hand the hardware engineer should have some insight into how the software plans to run the product so that the hardware can be designed in a way that makes doing those tasks easy.

But the one thing i have learned is that software developers will always want the most powerful over the top hardware possible. They appear to need 200MHz and 64KB of RAM to do the most basic of things these days.

[nctnico]

Yeah in my experience the software people really don't like HDL languages. Even just looking at what the language looks like they go "eww". Then the next trap is that once you figure out the odd syntax it starts to look a lot like a weird multitasking C, however trying to use it as such does not end up going well. People who know programing apply regular programing concepts on it that don't work here, or result in HDL code that runs slowly and eats lots of resources. Coupled on top that you can easily just clock your HDL too fast and have it stop working in the weirdest and glicthyest ways (Yes i know timing analysis exists, the new guy wont tho). The more effective way is to write HDL is to imagine what kind of logic functions need to be wired together to do the job, then write that down as HDL. A software developer never worked with 74xx logic, all they know is what a AND,OR,XOR look like because they can do that in software as bitwise operations.

No. For HDL development you need an HDL engineer. That is somebody who knows how to apply a software development oriented workflow to programmable logic / ASIC development. If you let a software engineer or hardware engineer write HDL, you'll get unmaintainable rubbish with very long development times.

To make a comparison: The software engineer will try to use Java and the hardware engineer will use assembly while the right tool for the job is C.

[Kjelt]

But the one thing i have learned is that software developers will always want the most powerful over the top hardware possible. They appear to need 200MHz and 64KB of RAM to do the most basic of things these days.

That is just as BS as saying that HW engineers want 12 layer boards so they don't have to sweat when laying out the traces.

The unfortunate thing seperating HW from SW is requirements and product features that say are frozen for hardware at start time 0+1 year.
Then the board is finished tested and done.
With SW the initial requirements should also be frozen at that time.
Then you can get an optimized design.

But for SW nothing is frozen^*.
After testing or releasing the product, customers or company engineers find new features thus changing requirements and having witnessed many products myself from birth to maturity this could have a huge impact.

Product A from 2011: initial product hardware M3 128kB ROM 64kB RAM.
Used at release v1.0:  98kB ROM 42kB RAM.
2016: v3.24  124kB ROM 64kB RAM and having some new features rejected due to HW limitations. Plus aprox. 14 manmonths "waisted" on optimizing code, rewriting some parts to be able to release v3.x because it did not fit.

We even had an issue that for debugging we already ran out of space in 2012 so we put a bigger chip with unfortunately a different package to a conversion board for debugging.

The reason we were not allowed to use a bigger chip at release was because it was not needed and it was $0.90 more expensive. Sales predicted 6M pcs in 8 years so that would be a loss of $5.4M.

Now I totally agree that if there is no need to use an very large chip you shouldn't do that but make sure you got enough space to expand when the feature list expands over time or convince mgt to release a new product, actually then putting up a middlefinger to the first group of customers.

^* in case of firmware update possibility that is.

[IDEngineer]

Embedded software developers who don't understand hardware should not be trusted IMHO anyway. But hardware engineers who don't understand software shouldn't either for embedded development.

I've long told my son, who is working on his EE at CalPoly, that the best Engineers are those who understand BOTH hardware and software. Such people understand the tradeoffs of both and can most accurately choose where the dividing line should be between hardware and software. The folks who only know one discipline can create some laughably stupid results.

Like the pure software guy who once insisted a design needed to add an A/D for input, and a D/A for output, so they could apply fixed gain to an analog signal in firmware. Honestly he didn't really understand converters, he just thought of them as "black boxes" that gave his code access to the real world. In my best stage whisper I said "You can do that for twenty five cents with an opamp and a few discretes". He gave me this look of pity, like I didn't really understand the problem. In the end the analog hardware solution was almost free because there was a spare opamp available in a dual or quad package that was already on the board.

My story is I started in analog hardware, then digital hardware, then microprocessors, which led to Assembly language, which led to systems architecture, which led to C/C++ and a smattering of other languages. I'm very comfortable with everything and constantly think about the interdisciplinary tradeoffs. And I still automatically worry about clock cycle count, having come from the days when single digit MHz clocks were considered "fast" and every cycle mattered.

[brucehoult]

Yeah in my experience the software people really don't like HDL languages.

The problem is HDL looks on the surface like software, but is a completely different concept

I disagree with that. There are many different kinds of "software".

HDL is a programming language that builds a flow graph.

There is plenty of other software that does the same thing. Compiler optimisers and back-ends, for example.

Modern compilers are manipulating the flow graph to a shape that is wide (has as many live values) as possible -- at least if possible up to the decode/execute width of the CPU back end, and at most up to the number of registers in the CPU.

This is the same basic task of organising a hardware flow graph into stages with each stage taking values from physical registers (flip-flops), doing combinatorial logic that is as wide as possible but not deep, and going back into registers for the next clock cycle to start from. Or maybe sometimes you want to minimise the number of live values (register bits). In any case you're manipulating the shape of a flow graph.

These things are SO SIMILAR that there is a project called CIRCT that uses the LLVM compiler infrastructure to optimise hardware. The Chisel -> FIRRTL -> Verilog workflow does the same kind of thing, with FIRRTL -> FIRRTL transformations to optimise the flow diagram in various ways.

This is very different to business software programming, or web site development using Javascript, certainly.

[SiliconWizard]

Agree with Bruce there.

What most people mean when saying that HDLs are not "software" is that they are not imperative programming languages, which is what most people are still familiar with.

Yes, HDLs describe "structures" and not a direct sequence of operations, but so do many programming languages with paradigms different from imperative. A pure functional language for instance. So yes, a series of statements in HDLs is *not* a sequence, but we can say the same for many other programming paradigms, with which I'm sure many people would struggle just as much.

[Siwastaja]

Agree with Bruce there.

What most people mean when saying that HDLs are not "software" is that they are not imperative programming languages, which is what most people are still familiar with.

Yes, HDLs describe "structures" and not a direct sequence of operations, but so do many programming languages with paradigms different from imperative. A pure functional language for instance. So yes, a series of statements in HDLs is *not* a sequence, but we can say the same for many other programming paradigms, with which I'm sure many people would struggle just as much.

But, finally, even imperative language like C, and HDL like VHDL share a lot. In C abstract machine, order of instructions is something comparable to just arithmetic operation precedence. Compiler does not replicate the imperative sequence 1:1, but reads the code and implements something which produces the correct results.

Compare this to something under the process keyword in VHDL: exactly the same thing, the language is imperative, compiler (that of synthesizer or simulator) follows the order of "instructions" to figure out which the correct result must be, and then implements the logic in whatever way is most efficient.

While it is possible to explicitly write logic gates in VHDL, no one does that, and just instantiating a bare flipflop is so much boilerplate it makes no sense. So VHDL is pretty much written "like software", and even more, pretty much like an imperative language such as C (especially if VHDL variables are used).

[hans]

Logic fabrics map very well to functional programming languages like Haskell. If you come from a C or other imperative language background, then Haskell is a huge brainf*k to start out with, but so can HDLs be..

I think the difference is that HDLs try to do both. You have combinational logic and process, while IMO a process is such a weird beast. It's one of the easiest way to have logic area explode on you because logic functions may all be naively implemented in parallel, perhaps with duplicates.. You really need to think graphically and get a feeling of where area/time/power bottlenecks may come from.

Say you have this piece of code:

Code: [Select]

difference_calculation: process(clk, rst) begin
if rst='1' and clk'event then
    if calculation_selector = '1' then
        output_data <= input_b - input_a;
    else
        output_data <= input_a - input_b;
    end if;
end process;
How is the synthesizer going to implement this? Is it going to have 2 subtractors with muxed output data? (most likely) Is it going to make an area trade-off in which it could apply 'retiming' to shove the mux through the inputs and synthesize only 1 subtractor? (unlikely)

If you want 1 subtractor, you will most likely to explicitly implement that:

Code: [Select]

difference_calculation: process(clk, rst)
variable sel_a : std_logic_vector(7 downto 0);
variable sel_b : std_logic_vector(7 downto 0);
begin
if rst='1' and clk'event then
    if calculation_selector = '1' then
        sel_a := input_b;
        sel_b := input_a;
    else
        sel_a := input_a;
        sel_b := input_b;
    end if;
    output_data <= sel_a - sel_b;
end process;
Or use combinational logic outside the process if you dislike the variable.

There is a good reason why you must be so explicit in HDL, because both examples will not behave the same at all. Not just from an area aspect, but also power due to switching activity from propagation delays and combinational logic. In order to optimize properly, you would need to know how often input_a, input_b or calculation_selector changes. If calculation_selector changes (a lot) more often than input_a/b, and power is important, then I would probably choose the first implementation because it has less spurious switching activity.

[Berni]

Yes HDL languages are still programing languages but the big difference is that the end result of compiled code is a logic circuit, not a sequence of instructions.

Main point of a HDL language is to save you from having to do everything with logic gates. You can describe your intent using better readable structured code, then the language figures out a way to actually implement it. But in the end it all becomes logic again. Some seemingly simple things in code are actually very tough to implement in logic, so a single line of HDL code could actually cause a giant pile of logic to be placed into the final result that takes up a lot of valuable FPGA fabric space and might run really slow too.

This is especially important because in FPGAs the tradeoff between size and speed can be enormous. In a CPU you have a certain number of registers available and you want to use most of them, then they get reused for other tasks anyway. But in a FPGA most the circuitry you put down is dedicated to doing only a single task and there is no limit to how much resources you can dedicate to a single task. So if you need to add up an array of 512 numbers in 32bit you could run them 1 by 1 trough a 32bit adder and take 512 clock cycles to do it, or alternatively make 512 adders and do the calculation in 1 clock cycle. Similar decisions have to be made inside HDL compilers, yet not all of your logic should be optimized for purely speed or for space. So as a result you have to be a lot more aware of what you want the final output to look like. This is why i often tend to look at the synthesized output of my HDL code to see if i agree with what the compiler decided to do. If it got the wrong idea i might change the code around a bit to be more specific about the way i want it done.

When it comes to complex HDL designs like processors you definitely want professional specialized HDL designers. But when it comes to simple things like making some glue logic or a simple video format converter or something, you can in most cases teach a digital engineer to be good enough in HDL in a few days.

But the one thing i have learned is that software developers will always want the most powerful over the top hardware possible. They appear to need 200MHz and 64KB of RAM to do the most basic of things these days.

That is just as BS as saying that HW engineers want 12 layer boards so they don't have to sweat when laying out the traces.

The unfortunate thing seperating HW from SW is requirements and product features that say are frozen for hardware at start time 0+1 year.
Then the board is finished tested and done.
With SW the initial requirements should also be frozen at that time.
Then you can get an optimized design.

Well yes some modern products are complex to a point where they need some some large complex code to do what they do, or perhaps they are doing a job where there is no way around lots of horsepower such as driving a high resolution display.

I am talking more about simple applications where it doesn't actually have to do much. Things like reading 4 analog voltages and 2 digital inputs and sending it over CAN. Or read some MEMS IMU sensor values and send them over RS485. Or a battery monitor that calculates the SOC from current and voltage and makes that available on CAN etc... All of this are examples for some reason required a pricey high performance ARM MCU with a a FPU and memory in the triple digit KB. When they ware told to stay in sleep mode as much as possible to not drain the battery, the result was the device sleeping for 1 second followed by running at max CPU clock for 3 seconds.

On the other hand in one of my projects i end up using a PIC16F with 256 bytes of RAM and about 1.5KB of ROM to implement a motor controller with PID regulation, UI, battery management etc. The software side of things did need a fair bit of math trickery to run fast (since this MCU doesn't even have a 8bit fixed point multiply instruction) but it worked and it was dirt cheep and very low power.

But not that i would be the kind of person that ONLY writes in assembly for ancient obsolete chips. I do also make use of high performance ARM MCUs or even Linux SoCs when the application needs more oomph. I just use a crappy little MCU when there is no need to use something bigger.

[Kjelt]

I am talking more about simple applications where it doesn't actually have to do much. Things like reading 4 analog voltages and 2 digital inputs and sending it over CAN. Or read some MEMS IMU sensor values and send them over RS485. Or a battery monitor that calculates the SOC from current and voltage and makes that available on CAN etc... All of this are examples for some reason required a pricey high performance ARM MCU with a a FPU and memory in the triple digit KB. When they ware told to stay in sleep mode as much as possible to not drain the battery, the result was the device sleeping for 1 second followed by running at max CPU clock for 3 seconds.

Then you clearly have a mismatch for the chosen engineering team. Were theses guys embedded SW engineers or conventional SW engineers that sometimes need an embedded Linux stack before they can do anything? 

Any and I mean any embedded SW engineer I have encountered the last decades knows from the above two words : battery & sleepmode that there is a need for a low power uC. This does not always have to mean low performance though, nowadays there are Arm 32 bits cortex uCs that have way lower standby and sleep power usage than conventional 8 bit uCs.

So yeah I see your point in this example.

[emece67]

.

[Siwastaja]

I think that was the case years ago but, the same as GCC/clang generate code that may be hard to correlate with the original sources, nowadays HDL synthesizers do generate circuits that may be hard to correlate with the original code.

Exactly, and this is pretty obvious, and one must understand VHDL/Verilog is higher level language, not FPGA configuration language.

If you target FPGA's and you think you are outsmarting the synthesizer by writing explicit muxes and and/or/not gates etc., you already are mentally lost because FPGAs do not come with muxes and gates (instead, you must look at the datasheet to understand what exact LUTs / logic elements they use, and not all details are even available).

If you were to manually control what gets synthesized, then you would need to work with LUTs and manual FPGA routing. Obviously no one does that, so trying to do something similar in VHDL is just totally erratic behavior: these people are writing low-level constructs for imaginary hardware, which do not map to their actual hardware, without even realizing their mistake. With higher level code, compiler would have better chances to produce optimum result, and you have the same level of control over specifics: you can add constraints such as clock speed or logic element usage limit for the compiler. Hand-optimizing, when truly needed (which is very rare) would then have to go through the compiler constraints, instead of trying random modifications in VHDL and hoping they give the desired result. There is too much randomness in the compilers for this to work, you don't even need to change the compiler version, any change in the project may suddenly change the number of muxes / adders used.

[hans]

any change in the project may suddenly change the number of muxes / adders used.

For many modern synthesis tools, just clicking the "Resynthesize" button is enough. Not a single LOC change is necessary for a completely different output. Synthesis time is actually a big problem for large projects, and considering the synthesis problem is NP-hard complex, logically heusterics are used to cut down the massive tree of possible design implementations to something that can finish within a few minutes to hours time. I've had plenty of cases where the synthesizer complains it cannot synthesize the design for a given clock constraint, but if you hit the resynthesize button a few times, you may get lucky 1 out of 5 or 10 times. Obviously it's not a desirable workflow, but it is what it is. It shows how non-deterministic these tools can be, and with that, also good in (out)smarting or surprising the designer.

Perhaps the example I gave was too simple to show a real difference on a FPGA. On an ASIC workflow I've seen the difference for a similar example. In fact, going to low-level designs, I've spoken to plenty of colleagues that were so fed up with the RNG of the synthesizer that at some point they 'manually' copy-pasted synthesized subparts of their arithmetic algorithms directly on the PDK/cell library they were using. This way, if they needed to resimulate their design post-layout, atleast they got somewhat repeatable results.
But perhaps that's also working with isolated implementations and pushing boundaries on them within academia.

[nctnico]

In my experience a design that doesn't place & route 'reliably' has issues in the timing constraints which make it hard for the 'place & route' process to get to a result. But I concur that some of this stuff is like black magic. Sometimes changing the random seed to the place & route can make the difference between having to wait for 2 hours and still not meeting timing or having the design ready in 20 minutes with all constraints met.

And sometimes you do need to nudge the place & route in the right direction by anchoring some parts of the logic in certain places so it makes the right decission at some points during place & route. See it as setting the order of the place & route process.

[westfw]

Then you clearly have a mismatch for the chosen engineering team.

we WERE talking about problem spaces where people wanted "some FPGA fabric" in their microcontroller.
That's well past the point (IMO) where either HW or SW team would want to use a Linux controller for a low-power application.  I hope.

[Berni]

It is more that you don't always have programmers that actually can/want to mess with low level stuff.

Here schools that teach programing teach it it more around the high level stuff. Arguably most software developers end up developing things that run on powerful machines like PCs, servers, phones...etc so it does make sense. But they don't expose them very much to embedded software development where they need to work together with hardware. It is also easier to run a computer lab with a bunch of x86 PCs rather than MCU dev boards. So they will come in with an expectation that processing power and memory is essentially infinite and OS would take care of any low level nitty gritty.

From my experience most of these programmers absolutely hate dealing with hardware drivers. They can't single step trough what the peripheral is doing, all they see are some changing values in a register, it is a mysterious black box that sometimes does what they want and sometimes not. So no surprise they run from HDL like the devil from the cross. So you can see how adding FPGA like features into a MCU is not popular with them.

So circling back to the thread topic this is why lots of people will avoid crappy tiny microcontrollers. They can't even imagine how you could do anything useful using 256 bytes of RAM. All of the code they compile already takes up 10KB of flash when doing almost nothing because of all the standard libraries that are dragged in. Yes given how cheep ARM MCUs have gotten (well... before the chip shortage at least) it makes sense to use a overpowered MCU in a lot of cases, but there are still really dumb applications that are easily done by a crappy little 8bit MCU at much lower cost and even lower power.

[Kjelt]

It is more that you don't always have programmers that actually can/want to mess with low level stuff.

Those are completely different graduation trajects here.
You have electronics, technical electronic computer science, embedded systems and Informatics to name a few.f
Then you have two levels: college BSc and university MSc.

You have exceptions but the BSc have more down to earth practical experience and the MSc are more theoretical.
The pure hw electronics guys (generalisation) are also not very good in embedded software such that they have no knowledge about design patterns, state machine implementations, etc. The informatics people learn their stuff on high powered pc's so indeed unless they hobby at home don't know about the small stuff etc.

These are generalisations and it all boils down to the person and his/her interest and also hobbie knowledge.
You don't let an power electronics guy with the last five years of megawatt power installations design your emc sensitive 8 layer board.
The same you don't let an informatics guy handle microcontroller choice or even the HSI, HAL interfaces and implementations. However you can give him a task to optimize some higher level library or design / optimize the complete software architecture so it stays easily expandable in the future.

But it goes on, in my company you are also not a good embedded SW engineer if you have no mechanics, mechatronics and pneumatics knowledge. I do all at the momen5 for hobbie and it has given me big benefits at my job. System and domain knowledge become very important. Very simple example we had five valves to switch in some logical order. Nowhere the timeconstraints were men5ioned so my colleague sw engineer built a nice state machine that switched the valves correctly. In the review when I first saw the code I blocked the delivery since the switching was directly, there were no waiting times for releasing pressure and timing for the valve to switch (which takes multiple ms) and that is a simple example why people should always know more than what they initially were learnt at college. So in a job interview for an assignment how do you pick those people? Or are they given bij your employer?

[Berni]

Actually one of the best software developers here was someone who actually is educated as IT, but he is a smart guy that is determined to figure out whatever he needs to get the job done.

He would also come to the hardware department with the most useful problem descriptions. Not only do you get a very concise and to the point description of what exactly is not working, but also a list of things he tried that support his reasoning. In most cases he indeed has narrowed down the problem rather precisely to the point where all you need to do is replace the blown chip he was suspecting and it worked. In contrast to the usual vague problem descriptions of "the ADC doesn't work", only to find later on that they forgot to set the ADC chips reset line high or they tried to run it at some absurd clock speed.

It is hard to find the kind of rockstar software engineers that can make seemingly anything run on anything. So you also get a mix of good enough ones that get the job done, but need to be kept a bit more of an eye on in case they do something silly.

[hans]

Wanted to add that low-level != embedded. Sure embedded is mostly low-level, but computers can also include low-level stuff.

For example, in C#, if you have a function that returns an array with no results, then a statements like ` return new int[0]; ` or ` return Array.Empty; ` make a slight difference in memory allocation and GC objects. Array.Empty is preallocated so slightly faster (single-digit ns).
All those little bits on how data is passed around, memory allocated, and cache is massaged, can make a large difference in some computation running in 5s vs 5min. To do that kind of optimization properly, you need good understanding how a modern computer works, not just understand the syntax or talk to some API.

I'm sure some SW people really like the low-level stuff, and once they got their feet wet with hardware peripherals, will thrive as an embedded SW engineer.

But as @Berni says, there are also people that hate writing hardware drivers as it seems like 'magic'. My brother-in-law is a backend SW engineer that works almost exclusively in JVM languages and web stuff. If I show him the edge cases and 'odd' things of C/C++ (like UB), he always laughs and suddenly has no regrets or intentions to work on some Arduino/ESP32 firmware anymore . It's not for everyone .