It ended up in the mess of the 1980s which nobody wants to go back and fix, get some of it still exists.
Give me a break. It was the same in 80-ies. It was the same before then (read the description of Algol-68 for example). Programmers had, have, and will have OCD. That's in their very nature. That's why they wanted to be programmers in the first place. So, they try to make everything universal, structural, systemic, reusable. Then they use all these things which are swarming in their brains and try to shape reality to match their picture of the world, while reality resists. When they're not satisfied with the results they try to add more systemic stuff on top. Just like drug addicts, gamblers, or people in deep debt, believe that the solution to their problem is to use more drugs, gamble more, or borrow more money. And convincing them otherwise is utterly hopeless.
Instead, they need to strive for simplicity. Don't add complexity where it is not needed. This works just as Occam's razor in science. If you strive for simplicity, other things will fall in place automatically. One programmer said: "Novices ignore complexity. Professionals deal with complexity. Geniuses remove complexity." That's the best programming advice I've ever heard.
Absolutely. What do you do when you have more than one programmer or more than one team of programmers?
Programming != Software Engineering.
Take an example we are all familiar with. This forum. While dated it's still a substancial bit of software. If you were to go and investigate what exactly is involved from a "near baremetal" point of view you would probably require the rest of your life to do so and you still won't be an expert in much of it.
Put another way. If someone in their 20s were to start out with a bare-metal computer and write every line of code required to recreate a forum like this... they would not be able to complete it in their life time. Not to the same standards as it is here today.
Even if we take the largest component out, the RDBMS, and the second, the webserver, we are still left with something too large for one person to implement inside of a decade.
So what do we do? Do we hire a bunch of "coders" "programmers" and just let them loose? NO! There lies dragons.
What frustrates me with MCU dev is that basically it's solving the same god damn problems over and over and over again. MCU's are ulimtately simple. Yes they are full of detailed fiddly bits, but it just annoys me that these cannot be encapsulated into a reusable form. Arduino is about the closest.
It's not like every single project is 100% despoke and unique. Seriously look at the total code you have written for MCU and honestly say that at least 60% of it is duplicated from previous projects.
The challenge in writing such components and having them fit together in thousands of combinations is not easy. Maintaining the small foot print and performance even harder, but it should not be impossible. Sacrifices may need to be made. It will most likely not be a "one size fits all". Some frameworks will be better at somethings than others.
I think that cost is the biggest hurdle. With MCUs normally requiring software which is sold with the product and never independant of it the margins are very tight. If one or two MCU devs can write almost bug free and almost useful firmware for $40k, why would you invest in a senior software engineer to facilitate faster, more efficient development if it costs another $40k?
EDIT:
Probably the largest amount of work in terms of enterprise software is not in writing it, the first time. It's in dealing with requirements changing constantly. Managing these. Modifying existing software to deliver to these changing requirements.
With MCU code being so small the temptation is always to start from scratch and "do it better this time". However, if you are releasing new updates to your firmware on a weekly basis and your business requirements are changing weekly. Good luck.
So in Big Iron we don't just encapsulation and engineer out repeatitive code we also factor in flexibility, extensibility and change, state, transition isolation, isolation of concern/responsibility, such that when one component changes the others will continue to work or error out sensibly.