Hitting a wall while writing software

[madires]

Basically it's about finding your way to code, i.e. the way working best for you. There are tons of programing concepts, methods and processes, and every few years there's a hype about some new concept which is promoted to be better and solving all issues. However, we still get a lot of crappy software.

[rstofer]

And the only good advice is John Carmack's: don't get attached to your code.  Be ready to throw everything out and rewrite it, multiple times, the moment it starts to seem like that might be a good idea. 

Of course, John Carmack isn't going to pay your mortgage in the meantime...

Way back around '70 and the days of FORTRAN and card decks, we used to joke that the best programs were written after the programmer dropped the box of cards.

[KE5FX]

Way back around '70 and the days of FORTRAN and card decks, we used to joke that the best programs were written after the programmer dropped the box of cards.

Reminds me of the story of the Zen monks who were renowned for their elaborate mandalas.  They would spend all year crafting a beautiful sand painting on the monastery floor, only to sweep it all away an instant after completion.  Visitors from all over the world would flock to the monastery on that day to admire the mandala and join the monks in their contemplation of impermanence.

One year, a toddler broke loose from the surrounding crowd at the last minute before the ceremony began.  The child darted across the floor, giggling and whooping, gleefully flinging sand everywhere. 

The monks were pissed.

Don't be like the monks. 

[RoGeorge]

Me monk?!  No.
Me monkey!

So far I've managed to install that QM tool, and find out how to compile those HSM diagrams on Linux (the examples and tutorials are mostly for Windows).  Made a HSM based blinky to run on a Kubuntu desktop.  Looks cumbersome, mostly because the QM tool uses classes and objects like it would be C++, while the generated code is plain C.

Next step is to understand how to do the same, but using specific board support file, so the compiled state machines will run on the given microcontroller.  There is a demo for Arduino from many years ago, will see if it still works.

Just to be sure everything will still run 10 years from now (in case I decide to keep using QM), the toolchain setup is made in a Kubuntu 20.04 LTS virtual machine kept always offline.

[CatalinaWOW]

And the only good advice is John Carmack's: don't get attached to your code.  Be ready to throw everything out and rewrite it, multiple times, the moment it starts to seem like that might be a good idea. 

Of course, John Carmack isn't going to pay your mortgage in the meantime...

Way back around '70 and the days of FORTRAN and card decks, we used to joke that the best programs were written after the programmer dropped the box of cards.

Must have been an inexperienced programmer.  Those who had already dropped a card deck used columns 76-80 for sequence numbers.  Initially stepped by five or more to allow for insertion of cards.  Then a few passes through a collator would put a dropped deck back in order.  This was also useful for debugging because compile errors could flag the card number in which the error was identified. 

[hamster_nz]

I find that unless I am familiar with the project from past experience that most projects need to be written three times.

Once to understand the problem

Again to understsnd a workable solution

And a third time to do it implement it properly.

[james_s]

For diagrams, I recommend trying out Dia as well as Inkscape.  Both are available for Windows, macOS, and Linux at least.

I've always preferred a pencil and a pad of paper. I find it easier to focus on the problem when I don't have the distractions of software and trying to make a perfect presentable diagram. I scribble up a flowchart and decide on how I want it all to work, then I start building one part at a time.

[MIS42N]

Start with a pile of blank sheets, a pencil and an ERASER (the most important tool). Map out the logic. I am still a "lozenge is a decision (with only Y or N exit), a rectangle is something to do, a squashed circle is an exit or entry" type person. Keep it simple. It can end up with a whole lot of IF ... THEN but that will clean up when writing code. Rectangles might be (usually are at the top level) complex and end up being another piece of paper with entry and exit and internal logic. This process can take quite a while and end up with lots of pieces of paper, but it makes one think of the process rather than the problem. Express the logic in general terms, "count until complete" rather than "count up to 9", focus on what to achieve rather than the detail of how to achieve it.

Some of the logic will lend itself to interrupts. They become little programs all of their own with hardware inputs and logical outputs.

Once the logic is worked out, consider the variables needed to make it work. Maybe rough out algorithms to achieve the purpose. Can it be integer, boolean, fractional etc. I quite often start a list of variables like they will be in the program, before I start to cut code.

Where to go from here might be personal preference. I look at hardware to see how to get those variables (or if it is even possible). That may lead to researching other peoples' solutions, maybe pinch some code snippets. Hardware interfaces are usually bricks of code with a few shared variables. I have many enlarged prints of Integrated Circuits with the signals they are going to deal with. Processors are particularly annoying as they have pins that can do many things, so the diagram pins down what is the particular function in this project.

Where a bit of code looks like it will be "difficult" I resort to writing a test bed in VBS (visual basic scripting) that can be run at the command line (cscript). When it works in VBS, it can be recoded in the target language. But that's me.

Time to start writing. Quite often I go for quick and dirty methods, rather than looping four times I'll cut and paste four copies. Or a string of if.. else instead of case. The aim is to get something that does something (anything). The first attempt is usually a steaming heap of ... but if it works in any fashion, it can be cleaned up. I find this stage usually results in a lot of "it could be done better this way" ideas. Maybe they can be incorporated, maybe it's back to the paper. But keep working versions until a better version eventuates.

At some stage during development, the whole thing may need a rewrite. It usually is restructuring logic, putting the code for similar functions together (or even combining them), cleaning up variable names, setting up #define for variables hard coded in version 0, etc. I expect to do this as a matter of course, it allows me to be very dirty in getting a working version.

Even V0.1 may not be the end although it will be close. Consider the 'nice to haves' and add if convenient.

And above all, DOCUMENT. Assume you will come back to this code in a year with amnesia. I was told many years ago the program IS the documentation. One should be able to strip out the comments and find what each part of the program is doing and how it is doing it. If there's more lines of code than comments, it probably isn't documented well enough.

That's my 2¢ worth.

[T3sl4co1l]

Another thing that careful design helps with, is excessive internal state.

It's relatively easy to sit down and start writing code, implementing various edges of the problem, and tacking on more bits until it meets or exceeds the full scope.

The problem is that "exceeds".  This is... maybe hard to give a good motivating example for, but uh, say you have one state machine, it has its state variable, maybe it has 27 states, stored in a byte variable (256 possible states).  What happens in the other 229 states?  Do they do weird things, like are some of the states done by matching numerical value and some by bitmasking and so on?  It's the software version of the Karnaugh map "do not care" value -- maybe you get nothing in the undefined range, maybe it's random, maybe it does strange and interesting (or even exploitable) stuff?

And then say you grew the program organically, and now it has two state machines.  Maybe they affect each other in weird ways, sometimes twiddling bits, sometimes adding or subtracting values from each others' state variables, or other state in the program.  What happens in all combinations?  It's a combinatorial explosion...

The more responsible approach might be to integrate both state machines into one; this doesn't solve the combinatorial problem, but fixing one part (the basic skeleton of the combined state machine) may lead to a more responsible design that abstracts away the combinatorial problem (e.g. offloading the additional state to dedicated counter registers, which are only accessed via arithmetic, so the range is well defined and easily checked).

Another general (if even more abstract, or difficult to analyze?) approach, is to write down the edge cases explicitly, and work to those.  Like, it's trivial to multiply two numbers together -- your average case is hardly any thought or effort at all.  But when does that process fail?  If that product needs to be larger than a few ten thousand, and you forgot that your platform does 16-bit arithmetic (because you were lazy and assumed C's "int" was big enough -- it's 64 bits on modern desktops and phones, 32 bits on much anything else, but 16 bits on 8086(?), AVR, etc.!), you're going to have a bad time!  Easy enough to check the edges here: put in min/max values of arguments, maybe signs as well, only a handful of tests are needed to verify result.  Now, obviously this gets much trickier in a nontrivial function.

And overflow, the result is undefined in C; though there are few platforms where anything but a modular result happens.  Maybe modular overflow is desired anyway, but maybe it should be saturating arithmetic instead -- who knows?  Does the program specification say anything about it?  If not, it's probably worth escalating that question...

And yet another very general approach: testing all possible inputs.  You can take a function as a complete black box, and in principle, check its behavior for all possible inputs.  Sure, it's not much, functions with hundreds of bits of arguments are common; but it's helpful when it can be done.  These days, this is very tractable (if rather slow in the upper range) for inputs up to 40, even 48 bits.  And random sampling of that space, may give useful insights for even more bits in less time (but easily misses things, too -- see Pentium FDIV bug!).  Obviously, forbidden or redundant states (like random pointers on a protected-mode architecture) can be excluded from search, easily enough, and pointer targets can be controlled, at least when the pointer is testing within allocated memory.  Guided random sampling can produce interesting results even faster, i.e., fuzzing.  For example, this is powerful enough to elucidate undocumented CPU instructions.

And related, one may be able to control the combinatorial explosion of parameter space, by carefully minding the type of that (collective) parameter.
https://en.wikipedia.org/wiki/Type_system#Specialized_type_systems
Supplying parameters independently, is a product type; the number of states to check is exponential with the number of parameters.  A sum type, uses the same bits for different purposes, by context.  The typed C implementation is, a function which takes a union as parameter; there may be unused bits in some of the subtypes, while the number of states only goes as the largest subtype.

Tim

[Nominal Animal]

For diagrams, I recommend trying out Dia as well as Inkscape.  Both are available for Windows, macOS, and Linux at least.

I've always preferred a pencil and a pad of paper. I find it easier to focus on the problem when I don't have the distractions of software and trying to make a perfect presentable diagram. I scribble up a flowchart and decide on how I want it all to work, then I start building one part at a time.

Oh, I also sketch out everything on paper first.  I have several pads, and even one fancy pad with thicker rougher paper for gel ink pen sketching when I feel fancy or particularly snazzy idea bubbling.  Probably TMI, but I don't bring an electronic device to the toilet, I have a pen and sketchpad there.

As is my wont, I usually sketch those small details I need to attack before integrating the whole.  Looking at my previously filled sketchpads, it looks like I also use them as temporary swap space, jotting up important realizations and conclusions of unit tests/simulations/thought models.  For me, a simple glance seems to be enough to recall the important parts, but that is secondary; the primary function is the writing down process itself, which forces me to organize my thoughts, similar to the rubber duck debugging method: the act of properly expressing the problem as speech or written text or diagram is often enough to organize ones thoughts enough to solve the problem.

A good example that I recently laughed at, it being basically a mix between a chicken walking on the page and madman's rants, was in an earlier full sketchpad, about escape character methods.  Basically, if the escape sequence does not contain the escaped character itself – meaning to escape | you use \!, to escape < you use \(, and to escape \ you use \/, and so on – you avoid most problems associated with escaping; my notes being a scattered collection of reasons and conclusions.  For example, you can process and de-escape such both forwards and backwards, and reprocess/re-escape such input without adding additional de-escaping iterations.

(If you consider the C string "/\\\\\t", you cannot determine what it represents unless you find the leftmost backslash first.  The equivalent "/\/\/\t" on the other hand is unambiguous regardless of which direction you parse it in, because a backslash character is always the first character in a two-character escape sequence.)

That is, my initial on-paper sketches are a working tool for myself; when I arrive at understanding a subproblem, I document the conclusion, basis, and if mathematical the key expressions in Maple/Maxima/Sagemath notation for further manipulation and verification, in text files; and graphs and 2D and 3D illustrations in SVG files or DOT language descriptions.

I have literally hundreds of directories with small example programs in C/Bash/Awk/Python, descriptive text files, SVG diagrams, and Maple/Maxima/Sagemath "workbooks", testing/simulating/brute-forcing a solution to some subproblem.  The programs, when run with --help, always describe the program usage and a simple outline of the problem it works on.  When I'm trying to recall a past subproblem, I often use find and grep to find the subset of candidates, then run the programs in those to find the exact one I'm looking for.  I'm not even sure of the exact number, because when a directory sits for a long time unmodified and unaccessed and therefore no longer that interesting, I tend to move it to backup storage.  Otherwise I'd have to start making a topic hierarchy for the directories, and I don't want to do that yet.

[RoGeorge]

Jeroen3
- try TDD

grumpydoc
- take an example of a typical project
- use C/C++ for embedded

NANDBlog
- try a higher level language e.g. Micropython
- use the soldering iron rather than the JTAG
- maybe offload more software work into hardware and analog
- maybe partner up with someone who doesn't do hardware and likes software

madires
- avoid feature creep

golden_labels
- avoid perfectionism
- understand the problem first, with only pen and paper
- write the primary logic first, even if it's just sketchy at first
- note down issues and solve them later
- note down corner cases, leave them as bugs for later
- note down feature requests, put them in the won't fix list
- set time limits

capt bullshot
- drew a state machine diagram before start coding

Caliaxy
- “Hardware eventually fails. Software eventually works.”

CatalinaWOW
- avoid the "software is simple" mindset
- hardware in the example project might be too simple, don't try to do everything in software
- software/hardware trades, e.g. add a pushbutton to signal a new battery
- polling style might lead to simpler software than interrupts
- break down the project into smaller parts for both hardware and software

T3sl4co1l
- draw diagrams, top down first
- implement meaningful standalone functions and test them from a custom shell-like environment
- leave optimizations and fine tuning at the end
- avoid excessive internal state
- treat all states in a state machine, even the invalid ones
- do not skip data validation and treating errors
- test for invalid inputs, too

rstofer
- Top Down Design, Bottom Up Coding
- draw top blocks and describe their interactions
- drill down with the blocks until you hit the hardware
- write the hardware code first then build up to the final interface, e.g. get the UART running at first
- software is usually the hard and expensive part to develop, not the hardware
- the best programs were written after the programmer dropped the box of cards

SiliconWizard
- set the proper mindset/expectations:  software ALWAYS takes longer than hardware to develop

KE5FX
- don't get attached to your code
- be ready to throw everything out and rewrite it, multiple times

nctnico
- start with a good description and diagrams / flow charts
- software is build upon a chassis structure, at some point that chassis can no longer support all the addons
- eventually let other people rewrite or finish what put you down, someone else might enjoy doing that

Nominal Animal
- treat the projects as tools to solve problems
- solve the tedious sub-problems first, e.g. define a comm format between the MCU and the PC
- refactor (rewrite in parts) easily, without dreading it
- code is never "finished", never "perfect";  only "usable"
- sketch out everything on paper first
- use the rubber duck debugging method

Syntax Error
- treat software as a breadboard, not as a final PCB
- add lots of comments for understanding the code years later

DiTBho
- it's normal to rewrite 5 times at least
- software is a hard job to manage, design and keep revisions

retiredfeline
- software is hard
- improve it until it's not worth the effort anymore

madires
- it's about finding your way to code, i.e. the way working best for you, there are tons of programing concepts

hamster_nz
- most projects need 3 rewrites:  first to understand the problem, the solution, then to implement it properly

james_s
- use pencil and paper for diagrams, rather than a drawing software

MIS42N
- start with pencil and paper and an ERASER
- map out the logic
- keep it simple
- use hierarchical blocks with described inputs and outputs
- use plain English rather than i++ < 9, focus on what to achieve and not on how to achieve
- clean up spaghetti logic when writing code
- write difficult pieces of code in a higher language first, test then translate into desired language
- at some stage it may need to rewrite
- document, usually need to have more comment lines than code lines

Very useful inputs for me so far, thank you all!
Will try to apply them by re-tinkering that discharger project this weekend.   

[SilverSolder]

Adopt minimalism as a philosophy when writing software.  As in, "What does it actually have to do, at its essence?" - and get that working first, before even considering any embellishments.

Pause what you are doing every once in a while, and ask yourself - "WTF am I doing right now?"  - if the answer is:  "I'm getting to the most minimal possible solution that will demonstrate it works", keep going.  If the answer is anything else...  backtrack, and get back on the holy path!

This way, you always end up with a project that at least works...  even if it isn't pretty, or lacks convenience features -  in a minimum amount of time.

Most of the time, I find the minimal solution is "good enough for Australia" and live with that - life is short, cool ideas are many! 

[radar_macgyver]

My 2c:

Make a list of all the features you want the project to have, and then check off each one as you finish implementing them. I find it helps me avoid the feature creep/perfectionism mentioned earlier, as otherwise I've found myself forever tinkering with the software and never finishing it.

Thank you to everyone who posted on here, these are good answers to something I've had trouble with too, recently.

[T3sl4co1l]

Adopt minimalism as a philosophy when writing software.  As in, "What does it actually have to do, at its essence?" - and get that working first, before even considering any embellishments.

The counterargument to this is: take the time to make something general enough that it can be expanded later.

Well, when you can afford to do that, and when there is a strong expectation (or primary need) to have it expandable -- it's good, yes.

The counter-counterargument to this is: making the simplest working model (the essence) probably isn't all that much work, and even if you have to rewrite the whole damn thing to make meaningful expansions to it -- it's not a big cost on the whole.

In a similar vein, I think there is value in discounting sunk costs -- obviously, the "sunk cost fallacy" is a pretty strong statement by itself.  In this situation specifically, the value may not be so much in the direct implementation (which is being thrown away!), as in the experience gained from having done it once.  Taking a more zen attitude, allows you to discard earlier stepping stones, while crafting a better, more elaborate final version.

Consider what a lot of computing consists of: testing a bajillion things, essentially all of which fail; then one passes, and you win.  Do you consider all the bajillion-minus-one cases a sunk cost, or productive output?

This probably isn't great advice to managers, who sooner or later need to balance the direct cost of that labor, with actual productivity.  Or for paid projects, where the same applies to oneself of course.


I have this experience, when I'm ripping up a PCB layout for example.  It's helpful to start with known-working component placement and routing.  Rearranging everything is faster the second time.  Mind, it's still more time -- this isn't economical as a primary design path, the ripup is still added cost -- but it does affect subsequent rework, say for major component or circuit changes, or size optimization.

Now, software is considerably more complex than PCB layout; it may well prove economical to use this kind of strategy, where the first attempts end up too rough and special-cased and should be discarded in favor of a cleaner (and more optimal to boot) final version.  That is to say: when the early versions are relatively cheap, and when fixing all the edge cases of those versions, would end up more expensive than redoing the whole thing (or substantial parts) from scratch.

Presumably, the number of cases where this happens, will go down over time, as one gains familiarity with programming in general, with the languages and features used, and the contours of the problem itself.  But I can see it might still pay off from time to time.  (Indeed, very productive programmers, tend to be very prolific.  Being able to conceive of a solution, and immediately -- within say a few hours -- bang out an implementation of it, allows one to do this very often indeed!)

Tim

[rstofer]

Taking a more zen attitude, allows you to discard earlier stepping stones, while crafting a better, more elaborate final version.

A more elegant way of saying "drop the box of Fortran punched cards".

[T3sl4co1l]

[bobcat2000]

These were the books they taught in my school.  Maybe you can pick up couple chapters so you can have a little bit of ideas for how to organize your projects.

This one teaches you how to organize your project.
https://www.amazon.com/Software-Engineering-Practitioners-Approach-Pressman-ebook/dp/B00HZ3BD5G/ref=sr_1_1?crid=3OCI9T8H8IHV2&dchild=1&keywords=software+engineering+roger+s.+pressman&qid=1615499979&sprefix=software+engineering+roger%2Caps%2C260&sr=8-1


This one teaches you how to make a blueprint for your application.
https://www.amazon.com/Unified-Modeling-Language-Addison-Wesley-Technology/dp/013485215X/ref=sr_1_2?crid=PCQC8TA5YT4Q&dchild=1&keywords=uml+user+guide&qid=1615500208&sprefix=uml+user%2Caps%2C291&sr=8-2