Software Woes

[BadeBhaiya]

People who write high level code (imagine python, javascript etc) and also low level code (imagine C for a Cortex M3 or AVR etc), how do you guys context switch?

Even though I have a programming background since my late teenage years, my professional history has mostly been writing C and designing hardware. I now find it really hard to write high level software, basically anything that isn't close to the metal. I get frustrated when writing python/javascript because its just very fragmented.

• Documentation for most open source projects feels lacking, if not incomplete.
• The industry wide accepted way of doing things is to read working examples and change them to suit your needs.
• A lot of code isn't explained and is "just there" in the files. Not explained very well or sometimes at all.
• Guides and tutorial, if they exist at all only include the absolute happy path, and no more. If the guide tells you how to make a button green, you will have trouble making it any other colour (an exaggeration but you get the point)

I was writing a HomeAssistant integration and even though its not difficult, its just very confusing. So many data types, so many classes that must be extended etc etc. To make it worse, the time spent learning the ins-and-outs feels like a waste because LLMs like ChatGPT can write the code for me, without having to understand anything. I consider that to be a very bad thing but from a purely time optimising outlook, its a good thing. It feels like so much of high level code is just babysitting the code itself and fighting the framework/library, instead of actually doing what you want to do.

I think what I'm lacking is a mental model of how to approach high level code, where I can't be sure what its doing under the hood (and also not mentally knowing the entire 'API' mentally, unlike is the case with hardware HALs). I seem to have lost it in the way somewhere. I used to program in QBASIC, VB6, and a bunch of high level languages in the past and I was fairly happy doing it.

[brucehoult]

The only things I use Python for are absolutely non performance-critical, just because some kind soul out on the net has reverse-engineered the protocol used by some device to communicate over the WIFI/internet. And some simple glue logic around that.

If I want to write "high level" code on a smallish device (well, big enough that other people might be tempted towards uPython, which is pretty big) then I write C/C++ using Boehm GC.

For similar things on a Linux-capable machine I used to (mostly 20 years ago) use Dylan, specifically Gwydion "d2c", but while that is not completely dead it is now pretty sick. However now Julia has most of the same strengths and a number of others besides. Either one is a far better high performance high level language than Python, though without the huge number of libraries to draw on.

[paulca]

I think what I'm lacking is a mental model of how to approach high level code, where I can't be sure what its doing under the hood (and also not mentally knowing the entire 'API' mentally, unlike is the case with hardware HALs). I seem to have lost it in the way somewhere. I used to program in QBASIC, VB6, and a bunch of high level languages in the past and I was fairly happy doing it.

I think you might consider reframing or moving to a different perspective.

The number one defining thing about a "higher" level language and why we call it "higher" is that somethings are done for you or taken from you and conventialised.

Consider moving from ASM to C.  When you did it ASM you dealt with things like "Calling convention, argument passing, return value passing, addressing modes, etc. etc."  When you do it in C, the C runtime decides all that with help of the linker.

When you go up another level to the likes of Java and some of the things you used to have to do in C, like memory management and may have even implemented a "pattern" in your own code base to provide minimal "management" over memory and allocations/deallocations.  Java wraps that for you, for example.  Now your pointers are managed and not direct memory references at all, but done in a way that you don't need to know.

Each of them C, C++, Java come with "Standard libraries" sometimes called the "The API".  In C its usually some form of CRT and stdlib.  C++ has similar and extends with the, streams, collection and container template libs etc.  Java has the JVM API.

To learn a new high-level language it is critical to start with the language itself.  To learn Java or learn Python start learning them in their "out of the box" form.  Avoid tutorials which "Use this with that to do..."  if "this" and "that" are not part of the languages default, core, API.

This will teach you the form, style, conventions and the interface to the layer below.  The later will usually be familiar coming up from lower down and alien going the other way.

When you can tell what "Vanilla" Java or Python looks like, then when you read someone's application you will immediately know to look for things like dependencies, frameworks, libraries, sub-systems.  The core language then becomes just the fabric the larger system is built with from big chunky fat libraries.

On HomeAssistant.  That is not an easy first grasp of "high level" software design.  I would not work in that code base by choice.  It does highlight something which is very common and can be very frustrating.  Large open source code bases work a bit like governments.  They aim to try and support everyone and make the software all things to all people and end up, ultimately failing miserably and becoming indebted to their own complexity and bureaucracy until they can no longer progress.  Unless they have a vicious dog like Linus Torvalds guarding the main line branches.

Do not worry if you fear the nomeclature and structure in home assistant, "Platform", Entity, Device are okay, but then it just gets weird.  I have been writing code for 40+ years, high and low and ... I would not work in the HA code base unless they paid me to... a lot.

I would say, as a learner project, don't try and "integrate" with HA at the code level.  Start with a lower level like the MQTT message bus or other "wire" protocol which HA already supports.  That component will be FAR simplier and teach you far more.

Also...  Python, whether python fanboys like it or not, was NEVER designed to handle large complex applications.  Python is to C what Javascript is to the browser APIs.  It is literally a utility script language so that C programmers didn't have to write "admin and utils" in C.

Python itself might be a bad choice to move up to.  Partly because a lot of people don't agree with my prior statement.  Why?  They only know python, only ever learnt python and have no other persepective or model to work from.

However, python, as a wrapper around C is fine.  It's extremely opinionated, quirky and has syntax and grammar not found in other languages such as "list comprehension" operators.   It's "white space" dependant.  Formating matters.

To learn upwards from C/C++ something like Java or C# might be a better intro to "Virtual Machine" modern highlevel eco systems.

EDIT:
On structure and design of complexity.  Again using the analogy that in ASM/C you made up your own patterns, utilities, wrappers or borrowed from others or from text books.  Consider the classic "C Module" which is a "struct" and a set of functions.  It's only a step or two or syntactic sugar from OOP and a class.

In a large swaith of higher level languages this pattern was wrapped and the "OOP" buzz word was slapped on it.  You will not avoid OOP.  Some of it is surface thin, like it was in C, in others it's dogmatic and crippling, Java/C++.  C# tries to find middle ground in some places less so in others.  The point is, OOP is a running theme.  Most langages can be "made to" do proceedural, but you'll get flamed for it.

Where this surfaces is in application layers where you have standard integrations like "HTTP".  It's so common, you lift it out of most standard libraries directly.  However.  Most modern software stacks are going to want...  HTTP+REST+SSL+OAUTH Authentication+Payload serdes.  So... you would not "normally" go to the language API's lower level HTTP and start building.  You would go and download a REST API layer which supports the full "standard" stack out of tne box.

Here you meet the abstractions and the frustrations, because that stack can be done in dozens of different ways and the component you choice supports all of them with complex dependency injection, aspect orientated / annotation programming and implied configuration from introspection.... all of those are "Intermediate to advanced" topics.

The only way to remain sane is to think at different layers with real ceilings and floors.  However, you can still get into the elevator and go a floor up/down.  Just try not to take any baggage with you.  When on floor 5, floor 4 and 6 are outside of your scope, except as "integrations".

[tggzzz]

I think what I'm lacking is a mental model of how to approach high level code, where I can't be sure what its doing under the hood (and also not mentally knowing the entire 'API' mentally, unlike is the case with hardware HALs). I seem to have lost it in the way somewhere. I used to program in QBASIC, VB6, and a bunch of high level languages in the past and I was fairly happy doing it.

Concentrate on what using an HLL means you don't have to do and don't have to worry about - because the HLL does it for you. That's what makes it "high" level in some sense or other.

Find the language's "white paper" indicating what and why certain features are in it, what is excluded and why. if you can't find that it might be because the HLL is little more than a few "things that look neat" munged together. Here's a good example: https://stroustrup.com/1995_Java_whitepaper.pdf

Only after that, start looking at specific APIs and libraries.

[paulca]

My God I agree with tggzzz 2026 is going to be a weird one isn't it?

On "Mental model" and approach.  The challenge is to learn to have more than one mental model.  No mental model, in a human mind, is capable of understanding something as complex as an international airline booking system from the transistor/jfet on the IC subtrate and quantum mechanics to the fancy bidirectional, usually consistent, distributed transaction models, to the webpage or console terminal UIs.... and all the supporting infra mingled in there in the telecoms networks.  Yet ... somehow... it "usually" works.

This is why you hear lots of terms like "Architecture", "Frameworks", "Platforms", "Patterns".

One of the first highlevel design steps at that level is to "block out" logical components, then through experience, point to blocks and say, "Well that's a Message Queue, we can use some JMX or RabbitMQueue/Solace".  So the components around it, depending on it, now become "message queue" clients.

"Do we need to develop this component or if we bend the solution a little can we use 'this' or 'that' off the shelve?" - decreasing cost and time to market.  Increasing quality at the same time re-uses peer reviewed and tested code.

The costing models are very different, but this might be the same in electronics as swapping out a bunch of discretes for an opamp.  Like in software that may have "knock on" effects, forcing the upstream/downstream of that opamp to change also.

So, over all the design process is about reduction of the new code needing to be written.

The downsides and the pains.  First, these "off the shelf" components tend to try and be all things to all people.  They don't just solve your specific problem, they solve everyone elses to, at a cost of complexity, abstraction and things your "low level inner programmer" is crying over inside, "No!  We don't need all that, can we remove it?" and then answer is.... no... not practically.  You just gotta swallow it.

As if that wasn't bad enough...  feeling how painful some integrations can be and the impact changing them has on up/downstream components... the normal practice became to... abstract the abstractions.  Literally each component creates a "Facade" layer for integrations which couples the application only to a thin layer which abstracts the actual abstraction of the integration.  This places a "firewall of change", a defensive line where change and variance cannot pass.... a layer where it can be fixed alone.

The result is literally Kbytes of code being executed and Mbytes of code being deployed when only 10% of it was "functionally required".

Where low-level programmers like you and I come in, is when the above process produces something which technically works, passes functional tests, but falls short on "Resource foot print", "Actual change resilience", "Robustness" and the big one when it matters, "Performance, through put, latency".

Low-level programmers can simply unleash that "Crying inner, can we remove all this junk we don't need" engineer, give him some red bull and start deleting code.

[paulca]

OP mentioned "Coding with LLMs".

I think it might be better to get the LLM to explain other peoples code to you.  Upload the whole python file, or the whole folder zip (if you have license to) and then ask it specific questions about what X and Y do, you can even tell it you are a C programmer trying to learn python and it will attempt to bridge.

Could it lead you astray, yes.  Could I? Yes.

[golden_labels]

It feels to me that BadeBhaiya is really asking not about high-level languages, but about using 3rd party libraries.

I don’t know, how to tackle this without writing an entire essay. Let’s try with a generic advice and then leaving a few clarifying remarks on OP’s observations. Intentionally not delving into the details and nuances, despite that means these remarks are incomplete and to some extent not correct.

The advice
Write your own simple code, not necessarily too useful on itself. Ask questions. What could go wrong with it? Where could it fail and how to address that? What are the edge cases? Then search answers for these questions on the web. Focus on dedicated blog posts, bug report systems, and forum threads. There is a ton of in-depth explanation and discussion on why and how. They let you peek into the model behind the libraries. After that you may also start exploring source code of the libraries. But I stress the “after” part. Otherwise it’s like being an ant trying to map a city while hardly seeing surroundings beyond a few inches.

Avoid any content aggregation sites. They only provide low quality or outright wrong information. A few rare exceptions are Stack Overflow & co., when it comes to very specific questions.

Clarifying remarks
FOSS libraries suffer from poor documentation as much as proprietary ones. The mechanism is generally different, but the end result — lacking documentation — is the same and to a similar extent.

That “industry standard” of showing working examples is the teaching standard since forever. Ignoring a minority tendency in academic teaching, I would not expect seeing anything more than that.

Guides and tutorials follow the above. However, if found on aforementioned content aggregation sites they additionally are of poor quality or not meant to really teach anything.

[dobsonr741]

You’re treating high-level code as if it’s a different universe, but it’s the same object model you already know — just a different point on the abstraction stack. A Cortex-M HAL gives you registers, Python gives you objects, but both are structured interfaces to state and behavior.

The frustration comes from assuming you must mentally model everything under the hood. You don’t. Higher layers intentionally hide details. Instead of knowing every register, you only need to know the contract of the abstraction you’re using.

All those frameworks and “mysterious” classes aren’t magic — they’re just the same encapsulation patterns you use in embedded, multiplied across a deeper stack. The code that feels “just there” is the plumbing you would normally write yourself in C, except now someone else did it first.

So the approach isn’t a context switch — it’s the same reasoning applied higher up:

• Identify the boundaries
• Trust the abstraction layer
• Only drill down when needed

It’s all one model: just pick which layer you want to operate at.

[paulca]

This is important. When you hit a 3rd party dependency that you cannot understand.  DO NOT USE IT.

Pick a lower level one you do understand and build upwards. When you later find you need all those things you rejected from the "off the shelf", it's okay to return to use it.  This is perfectly normal sane project progress.

Not every one develops this way and the dividing points are individual.

Abstraction must reduce complexity not add to it and it must be opaque.  Rules too often forgotten.

[Siwastaja]

I write "high-level" tasks like I would write low-level code, in C, with minimum amount of library dependencies. This may require me to write something stupidly obvious like memory management or data structures that could be already solved in higher level languages, but now I don't have to "context switch", I can do things like include same headers in all parts of the project, and never worry about the pain points of modern-day high level programming like library/language version incompatibilities, "bootstrapping the workflow" for which you nowadays need an AI agent because apparently it has become so complicated, and so on...

[jfiresto]

I write "high-level" tasks like I would write low-level code, in C, with minimum amount of library dependencies....

I study a library's use history, issues and change logs, and will readily depend on one if it significantly simplifies my code, and appears very unlikely to be incompatibly changed, spoiled, or removed. At the moment, I am waiting for Python's importlib to reach steady state and am writing comments like these.

Code: [Select]

# Use run_module() to cache the .pyc compiled code: run_file() and exec()   
# won't. It appears that importlib, which should be useful, offers no way   
# to pass globals to the executing module. It's also been a bit of a WIP,   
# with a number of changed and deprecated objects since Python 3.7.
[EDIT: Added "readily" and "incompatibly".]