Familiar with programming, not prolific, any advice?

[RoGeorge]

Did an inventory of my unfinished tinkering projects last weekend, and most are stalled without firmware/software.

The goal is to write fast enough and good enough code, so to get functional projects, not to become a programmer able to work in a team.  Environment is Linux and FOSS tools.

I've wrote small programs occasionally, starting decades ago.  Fortran was probably the first I read a book about in the high school, but didn't practice, later practiced some Basic on home computers, then at university we did some Turbo Pascal, then read K&R which later applied to programming microcontrollers, with some other languages and algorithm books/classes in between (i.e. the Knuth's 'Algorithms' books, about half of the 4-5 volumes).  Latest language I've dealt with is Python.  I know the basics of OOP, but not able to take advantage of it.  I'm stuck with the procedural writing style.  Never tried functional programming or other fancier stuff.

My guess is I might have to unlearn the procedural programming style and move to OOP.  Or maybe I'm really bad at programming.  I don't know what I'm doing wrong, but I know my projects are eventually abandoned because the software part became overwhelming.

What to do to become more productive in writing code (for small/hobby projects)?  Some programming exercises/books/classes to follow?  Something else to learn? 

Most of the hobby projects require to write some MCU firmware, usually in C, and sometimes a PC interface, usually in Python.  Any directions of how to write that faster is welcome.

[Karel]

Just practise and learn from your mistakes. There's no way to speed up programming without introducing more bugs.
Regarding microcrontrollers, stick with C.

Let the flamewar begin 

[DavidAlfa]

It's all about practice, same like a language.
I was good at talking english until 18, went several times to Scotland, with 12 I could fully go on my own, book things at the library, go shopping... Stopped  going there and 15 years later, while it's not fully lost, your mind want to say something with perfect pronunciation and the output is, at best, like a phone call from an Indian scammer.

Programming needs everyday use to keep it fresh, stop for few months and you'll need at least several days to recover the agility.
10 years ago I would do a flowchart and implement the code lighting fast, buy I was on it 10hr daily.
Stopped for 8 years, it took quite some work, months, to get everything back.
It's overwhelming at first, you feel like you wasted the entire day for nothing, but your brain is slowly training up.
You might got to bed confused, but after a good sleeping, next day you'll notice it's clearer now, by magic, it's the brain settling things down.

Don't overload yourself too much, take a small nap when you feel it's too much, even sometimes it's better to let it sit until tomorrow.

[DiTBho]

Just practise and learn from your mistakes. There's no way to speed up programming without introducing more bugs.
Regarding microcrontrollers, stick with C.

I spent 10 years of my life forgetting the bad programming style I learned in college because it was all wrong.
As a personal opinion based on my personal experience, the biggest contribution here for me is my job, because you can both learn and be motivated.

[brucehoult]

My guess is I might have to unlearn the procedural programming style and move to OOP.  Or maybe I'm really bad at programming.  I don't know what I'm doing wrong, but I know my projects are eventually abandoned because the software part became overwhelming.

I don't think the language matters much, as long as it gives you access to what you need. Asm and C (and Rust, Pascal, and similar) by definition give you easy access to anything the machine can do.

Python doesn't. Not without calling out to C/asm, which you can do but it adds friction.

OOP or not doesn't matter.

You need to practice SOME form of generalisation and abstraction in your code, so you can design and write small modules that do one thing well, and test them in isolation to the rest of the program.

But it matter basically not at all whether that is structs with functions that do things with them, or "objects" with "methods". It's the same thing.  Also virtual functions vs a type enum and switch statements is basically the same thing and neither is better until you get into programs with tens of thousands of lines of code.

Don't ever write 1000 lines of code and hope it all works first try.

Write and test maybe 10 lines of code at a time. Maybe 20 or 30 if it's completely straightforward code. Maybe only 1 line if it's tricky.

[Nominal Animal]

Design and document.  Do not trust your thoughts are precise enough or practical; test.  Document your intent, what you want the code to achieve, and not what the code does.

When writing code, write in small units that you can test.  The smaller the amount of code a problem can hide in, the easier it is to find.  Test often.  You probably should spend at least as much time testing your code, as you spend writing it.

I often write tiny test programs to test a single data structure and its associated function interfaces, just so I can check it really works.  I don't make them into libraries, because I don't need generic stuff; I need stuff that does the task I want it to.  Perhaps learn to write such test cases so efficiently that you won't wonder if spending the time writing one is worth it.  (I'm tempted to claim they always are worth it, you see.)

Investigate the different ways of accomplishing things.  This means knowing about different abstract data structures (hash tables in particular, hash algorithms), even though you are not absolutely certain how to implement one, so that you can consider the different ways you can solve a given task.
I see programmers ask questions like "How do I do X with Y?" very often, with Y the completely wrong tool for X.  So, you must also keep asking yourself whether you are trying to solve a problem you need to solve, or are you trying to solve the problem only because a previous tentative decision made you arrive at this.  Very often, such possible solutions form a tree, and you need to consider many alternatives, possibly several steps ahead (or more often, backwards into things you assumed but are not actually axiomatically true), to find the good solutions.
Rarely is there only a single answer or way, and "the best" always depends on the context.

Always consider what the code does if something unexpected happens.  For example, you might have tested your potentiometer readings are always between 15 and 997, except that when the ambient temperature and humidity changes you get an occasional 14 or 998 or something, and your careful scaling routine shits itself.  (My particular favourite of this is when using floating-point numbers, and ranges that exclude one or both boundaries: 99999999.0f/100000000.0f = 1.0f, for example, because 99999999.0f = 100000000.0f using IEEE-754 single precision floating point, float AKA Binary32.)

Learn to read others' code.  At one level, this is understanding what the code does.  At the next level, it is deriving the intent of the programmer from that.  At the next level, how the code interacts with other code.  (Locking, mutexes, and parallel execution interactions tend to be hard on us humans, but fortunately on microcontroller targets we only need to consider interrupts: no locking, only atomicity or detecting interrupted access to non-atomic data.)

Learn to ask good questions, and ask them.  Even if you only ask the question from yourself, do put it into words.  This works, because it forces your mind to organize the idea/question, and format it in a linear fashion.  I use English for all my software development, including my experiment/test codes and their documentation, although my native language is Finnish.  For some odd reason, this helps.  A good question digs into the problem itself, describing the underlying problem, and does not impose unnecessary restrictions (like already selected solution methods).

[RoGeorge]

Probably my complaining was too generic.  Let's take an example.  Say the project is to make a datalogger.  I know there are many data loggers already, will only use this theme as a learning example, to identify the pitfalls.

My approach would be like this:

1.  Write some goals/specs for the logger:
    1.1.  Analog inputs:  4
    1.2.  Digital inputs:  8
        1.2.1  (nice to have) Digital outputs (e.g. for turning on/off the DUT, or to trigger a timelaps for a standalone photo camera, etc.):  4
    1.4.  Use a common MCU for its ADC and I/O pins, say an Arduino-like devboard, MCU firmware written in C

    1.5.  Sampling speed - from 1 sample/second to 1 sample/hour
    1.6.  Data is timestamped, then pushed to a COM port at the specified sampling interval
        1.6.1 (nice to have) - store data on a local SD card
    1.7.  Apart from interval sampling, samples can be taken at arbitrary time by a command through the COM port
        1.7.1 (nice to have) - wireless COM port, e.g. a nRF24 or alike

    1.8.  On the PC side, store the timestamped samples in a CSV file, plain text, preferably using Python
    1.9.  A live view window on the PC, to display each sample(s) after reading them from the COM port, text only, no plots
        1.9.1 (nice to have) - a command window able to send a request for an extra sampling commands to the same COM port, while keep logging the incoming periodic samples


The idea is to tell what/how I am tempted to do, then to learn if/what is wrong, and what should be done instead.  This is not code yet, didn't write it yet, but will try to in the next steps, then ask for corrections.

Please keep in mind that I'm an electronist by profession, so I'm tempted to (over)think it as if a program must fully work from the first try.  I'm tempted to design all before writing the first line, while professional programmers tend to rather grow their design (or at least that is how it looks to me), meaning many iterations with more and more complete code at each version.  Not requesting to make it in one go, just telling that it is not about coding advice only, more about software design, or how to write some code to get a job done (don't know if "software design" is the proper term).

- Is this first step a reasonable start?  What should be changed?
- How would you proceed to make/write a logger?

Most of all, how many hours should take to write this?  More like 2, or more like 200 hours?


To highlight again, this is a request for online debugging of what am I doing wrong.  Can not pinpoint that alone, need your help with it.  The intent with this topic is to say at first how I would usually do each step (including my own written code later), then to get feedback for what wrong habits do I have, and how to fix that, or how/what would you do differently.

[MK14]

More like 2, or more like 200 hours?

You'll get the software project, around 90% complete in the first 2 hours.  It only needs the other 198 hours, to finish the last 10%.

Edit: Not a joke.
I can't easily find a good source, but here is one example:

https://www.reddit.com/r/webdev/comments/gxcq78/ah_after_you_have_completed_almost_90_percent_of/

[DiTBho]

after you have completed almost 90 percent of/

Yup, it's true!

[dietert1]

For embedded the learning doesn't only happen inside the person but also in their equipment. To start with something simple but useful, one may get a MCU development kit/setup to become familiar with. Then try to find existing software examples and libraries related to what you want to do and modify/supplement with your own inventions.
Some basic tools for embedded projects are:
- Implement a serial, USB or ethernet link with your linux host.
- Implement a LCD or TFT display
- Implement a sensor with SPI or I2C
- Implement a RTOS
- others..
People in a hurry might prefer Arduino.

Regards, Dieter

[RoGeorge]

Might have done similar projects, the problem is they were painfully slow to write.  For each of these I've put online (somehow incomplete), there are countless other unfinished.  Even the published ones are in an unbaked state, such that not even myself can make a use of them after I forget what I've put there.  A few examples:

- Implement a serial, USB or ethernet link with your linux host.

Oscilloscope screen capture over LAN (either Linux or Windows)
https://github.com/RoGeorge/DS1054Z_screen_capture

- Implement a LCD or TFT display

LCD display packed as an Arduino library (only attached as a zip in EEVblog, not pushed to the online Arduino libs repositories):  https://www.eevblog.com/forum/projects/a-2-wires-lcd-with-hd44780-and-3-x-74ls74-(put-2-shift-registers-on-same-2-pins)/

- Implement a sensor with SPI or I2C

https://hackaday.io/project/2823-how-are-you-errr-are-you-alive

- Implement a RTOS

Not exactly an RTOS, the closest I might have wrote is a CP/M BIOS, written in Z80 assembler long time ago, it was about 4kB of assembled code, and took a couple of years to finish, didn't count the hours:

- others..

Some Python simulation of an inflating universe:

Code posted in the YouTube video description:  https://youtu.be/BjbNZ7SLzwM

Another Python example  (in Jupyter Notebooks) to control a power supply to make a MOSFET curve tracer:
https://www.eevblog.com/forum/testgear/rigol-dp832-power-supply-as-automated-curve-tracer/


Some multi channel PDM (pulse density modulation) demo in C:
https://hackaday.io/project/6356-delta-sigma-versus-pwm



Please note that they are all very low complexity programs.  Anything more elaborated ends up as never finished.

Can not write small programs fast enough, can not write larger programs at all. 
How to fix this?

[DiTBho]

Well, my-fs took several years just to be usable.

Don't worry about your difficulties with complexity, I can assure you are not alone.

Just, mind that complexity needs pianification, abstraction and design.

[dietert1]

https://hackaday.io/project/2823-how-are-you-errr-are-you-alive

Isn't that a Kinetis board? I would try to run that Kinetis project on the bluepill in order to not lose the effort. Porting things you already know is a good exercise. When i do these things i usually write a work protocol into a text file, so others (or me) can later understand and recover what happened.
Did you get the USB data link between the bluepill and your host running? You could try and use STM32CubeMX and the STM32 HAL libraries to make it happen. If you learn to reuse software components it will be less tedious.

Regards, Dieter

[cfbsoftware]

What to do to become more productive in writing code (for small/hobby projects)?  Some programming exercises/books/classes to follow?

Read the book Programming Pearls by Jon Bentley.

[james_s]

It depends on your style of learning, there are some decent programming courses on Udemy although the quality varies, they regularly have sales so you can get them pretty cheap sometimes.

The language doesn't really matter, and neither does OOP or not, the biggest challenge in programming, at least for me is just coming up with the optimal organization and algorithms. If you have a specific project you're having trouble with you might try posting the code on the forum along with a description of the difficulty and see if anyone has ideas. Most of the various building blocks you'll find in any program have been done by somebody before.

[tggzzz]

What to do to become more productive in writing code (for small/hobby projects)?  Some programming exercises/books/classes to follow?  Something else to learn? 

Choose simple tools that make it easy to create and understand your implementation. Avoid tools that are so complex that the "learning curve" for their applicability, "edge-cases" and "gotchas" is a problem in itself. Examples: C++, and to a slightly lesser extent, C

Create an abstract model of the problem and of your implementation's structure. Abstract => not technology specific. Example: use FSMs, which can be implemented in hardware or software.

Structure your implementation using layered abstractions, with the bottom level being very close to hardware, and the top level looking remarkably similar to your abstract model. Example: bottom level grabs an input when it arrives, top level has an event causing a state change. Implement and test each abstraction on its own.

Define the minimum possible set of functions/operations that allows you to test your design and implementation concepts. Delay elaborating all the variations until later, when convenient. Structure your implementation so that is possible.

And, of course, persevere.

[DiTBho]

I think you are asking for social encouragement 

if so: open a Youtube channel(1), and post projects there.

I think, it's Good for these points:

• you will be motivated to complete your projects
• you virtually always have your own "rubberducking" (what?  ) as introspect analysis backend (2)
• the duck also represents you and your audience
• you get a lot of social encouragement, which is good propeller for human beings
• you can develop method of meeting deadlines, therefore starting projects by steps, which are then the episodes on Youtube, where you show progress from time to time

(1) it's better and simpler than Hackaday, which can be used as "mirror"
(2) rubberducking

In software engineering, rubber duck debugging is a method of debugging code by articulating a problem in spoken or written natural language. The name is a reference to a story in the book The Pragmatic Programmer in which a programmer would carry around a rubber duck and debug their code by forcing themselves to explain it, line by line, to the duck

[RoGeorge]

Sorry, but no.  I'm asking for debugging/review of what I am doing wrong.  Not asking for encouragement.

Since others can write complex programs while I keep failing at it over and over, for decades, I guess I might be doing something wrong.  Or I might have developed bad programming habits.  Sure, there's always a possibility that I'm not suitable for programming, would not mind if so, though I would like to know why.

At some point complexity seems to explode in my programs, while other programmers seem to somehow keep that under control in their projects.  Or at least that's what I suspect as one of the causes.

Don't know what is it, and it's not the first time I'm asking.  So I thought this time to take a small project, the MCU data logger, and write here what I usually do, step by step (will also try to keep track of the time spent at each step), including the C and Python code, so others can point what I am doing wrong, and if something took me way longer than it should.

[dietert1]

Maybe it's about structured programming? Ever refactored something?
One way to attack this: Break down any source file larger than 400 lines into pieces. Invent meaningful interfaces between the components and a meaningful name for each component. Think about globals. 400 lines of C is the maximum i can write and debug on a good day. The number may depend on the person.
Same thing: Whenever modifying/extending a working application, o good way to modify it's code is to reference new interfaces implemented in additional source files.

Regards, Dieter

[DiTBho]

Sorry, but no.  I'm asking for debugging/review of what I am doing wrong.  Not asking for encouragement.

ok.

[tggzzz]

Since others can write complex programs while I keep failing at it over and over, for decades, I guess I might be doing something wrong.  Or I might have developed bad programming habits.  Sure, there's always a possibility that I'm not suitable for programming, would not mind if so, though I would like to know why.

Can you design and implement electronic hardware without such problems?

If so, use the same concepts and techniques on software. The details will differ, but the approach won't.

[RoGeorge]

Can you design and implement electronic hardware without such problems?

That's an interesting question, I never thought about programming that way.  I guess even complex hardware projects (i.e. when I designed a full Z80 Spectrum from scratch, with video and disk controller included https://hackaday.io/project/1411-xor-hobby-a-vintage-z80-computer-prototype ), are still low complexity when compared to software projects.

For the hardware projects, the most agonizing tasks to me are about how to design something using only parts I already had in the inventory, without ordering anything extra.  Another problem (with hardware) is to commit between something very basic, or something very generic.  For example, I need to measure the I/V curve for a lightbulb, but then I want to extend it a little over other ranges, or over other types of components, and so on, adding more, but then becomes too bloated and thinking again about a minimalist project, and so on.  So, for hardware is more of a problem about choice, than a problem about braking a big project into small blocks.
 
Your question made me think I might have a problem with decision making and committing to fixed specs.  Could be.  In hardware such problems are smaller, because the choices are limited by the available parts, and other limitations from the physical world, while in software the possibilities are endless.  In software it is always possible to want/add more, or to switch between a completely different programming ways.

A side effect to this is that even the nicest working program still feels unfinished.  There is always something more you could add, or you could improve to a program.  Very frustrating.

Ever refactored something?

Usually that's where I lose control, so I need to re-arrange/redesign some parts.  I've read a refactoring means better structured code, while retaining the same functionality from before.  I usually end up with buggy code after a refactoring attempt, or with a broken project that doesn't work any more.

When writing code, write in small units that you can test.  The smaller the amount of code a problem can hide in, the easier it is to find.  Test often.  You probably should spend at least as much time testing your code, as you spend writing it.

I often write tiny test programs to test a single data structure and its associated function interfaces, just so I can check it really works.

This is another thing I don't do.  Sometimes I test (manually), for example when I doubt about the limit cases of a loop, or of some calculation, never wrote automated tests, and never had regression tests associated with the code I write.  Testing only very brief, and the tests are always manual tests, with some inputs I suspect might be troublesome.

I've read once some book that was in fashion, about TDD (Test Driven Design).  Write a test for every line seems nonsense to me.  Who's testing the test, then?  I recall even from one of the working places where a software team tried TDD.  Very often the tests were failing because of bugs in the tests, and after struggling with the TDD paradigm for a couple of weeks, they ended up being less religious about TDD, and they only kept a few test that were making sense.  It was a huge software project, and those guys were top programmers at a multinational, not amateurs.

I'm confused what should I test, and how.  Many of my projs have some hardware, so I guess it will take considerable effort to mock the hardware signals.


Only replied here to a few specific question, but I want to thank you all for the help.

Sorry for the late reply, and didn't start the logger exercise because I've found a book with explained project examples, and avidly reading from it:  https://zjnu2017.github.io/OOAD/reading/Object.Oriented.Analysis.and.Design.with.Applications.3rd.Edition.by.Booch.pdf  So far this book seems very close to what I'm trying to learn.  Will see how much I'll be able to apply in practice.

Saying this because in teaching examples, programming seems always easy.  In practice it's never as easy as in the "hello world" seen in the books. 

[tggzzz]

Can you design and implement electronic hardware without such problems?

That's an interesting question, I never thought about programming that way.  I guess even complex hardware projects (i.e. when I designed a full Z80 Spectrum from scratch, with video and disk controller included https://hackaday.io/project/1411-xor-hobby-a-vintage-z80-computer-prototype ), are still low complexity when compared to software projects.

I disagree. There are many similarities between hardware and software at the requirements, design and testing levels, even if the implementation techniques are significantly different.

Key strategies in designing hardware or software or a combination of them are:

• keep the complexity under control. I know my software development is going well when the number of lines of code in the project reduces. (Explodes the brains of idiots that count LoC as a measure of productivity, though!)
• implement and test incrementally. That implies providing just enough "external scaffolding" for the unit currently being implemented and tested

For the hardware projects, the most agonizing tasks to me are about how to design something using only parts I already had in the inventory, without ordering anything extra.  Another problem (with hardware) is to commit between something very basic, or something very generic.  ... So, for hardware is more of a problem about choice, than a problem about braking a big project into small blocks.

The same is true for software:

• don't include libraries/frameworks/etc you don't need, and consider reusing something you already know
• implement the minimum viable functionality, if necessary elaborate it later

.

Your question made me think I might have a problem with decision making and committing to fixed specs.  Could be.  In hardware such problems are smaller, because the choices are limited by the available parts, and other limitations from the physical world, while in software the possibilities are endless.  In software it is always possible to want/add more, or to switch between a completely different programming ways.

A side effect to this is that even the nicest working program still feels unfinished.  There is always something more you could add, or you could improve to a program.  Very frustrating.

Again, the same points can be made about hardware!

When writing code, write in small units that you can test.  The smaller the amount of code a problem can hide in, the easier it is to find.  Test often.  You probably should spend at least as much time testing your code, as you spend writing it.

I often write tiny test programs to test a single data structure and its associated function interfaces, just so I can check it really works.

The same is true of hardware. It is a natural fit with manhattan prototyping styles.

I've read once some book that was in fashion, about TDD (Test Driven Design).  Write a test for every line seems nonsense to me.  Who's testing the test, then?  I recall even from one of the working places where a software team tried TDD.  Very often the tests were failing because of bugs in the tests, and after struggling with the TDD paradigm for a couple of weeks, they ended up being less religious about TDD, and they only kept a few test that were making sense.  It was a huge software project, and those guys were top programmers at a multinational, not amateurs.

I'm confused what should I test, and how.  Many of my projs have some hardware, so I guess it will take considerable effort to mock the hardware signals.

TDD has its virtues, but as is often the case in the software world, zealots rapidly expand good ideas so far they become stupid.

The quality of the tests is something zealous softies often ignore. Too often they think something works because it passes their tests.

Works for hardware and software:

• define what is necessary, as in "must do", "want to do", "would be nice if it can be done"
• structure your design so that you can see which component will implement the "musts", and how the component could be expanded to "wants" and "nice ifs"
• implement one component. Provide any inputs and measure  any outputs to see if it does what you need (i.e. test it!)
• repeat with other components, coupling them together as necessary
• after all components are working together, start elaborating their functionality to include the "wants" and "nice ifs". Reuse the previous tests to check you haven't messed things up

[madires]

Can not write small programs fast enough, can not write larger programs at all. 
How to fix this?

A large part of programming is the architecture of the software, i.e. how to split a big problem into smaller ones and how to orchestrate everything. Embedded software adds another level of complexity as the software has to match the resources available and make the best use of the hardware (in the ideal case, side note: the quick and dirty way is to use libs available and go for an oversized MCU). You can find a ton of literature on that, but for hobbyist softdev I'd recommend to get just an overview of common methods and to find out which approach works best for you. It can take some time and effort to find your way.

Another large part is about algorithms, i.e. how to solve the small problems from above. Again, tons of literature available. Reading code written by others helps when it's good code (most isn't). Be careful with coding speed. Sometimes a few lines of a bespoke algorithm can take hours and is worth the effort, instead of typing down a standard solution in minutes but running much slower. Especially with MCUs this can make a difference (or you'll have to go for the oversized MCU again).

And the most inportant part is talent. If you have it things will be much easier and faster. Otherwise it will be a tough ride.

[dietert1]

Formal/automated testing is important. Otherwise you lose a lot of time doing similar manual tests over and over again. There will be new bugs whenever working on a (almost) mature application. Think of tests as redundant implementation of the same application. Sometimes the application itself gets written twice in order to discover errors..

Regards, Dieter