Hitting a wall while writing software

[RoGeorge]

It happens to me all the time when building something new.  The hardware is nice and fun to make, then writing the software turns the project into a chord.  Software is many times more time consuming, never finished, and tend to grow exponentially in complexity until it all turns into a mess, needs at least a refactoring if not rewriting everything, and so on.

Not sure how to deal with that.  I don't want to rewrite 2-3 times every program, end even when I rewrite something, I still hit a wall (at about a few hundreds lines) where it all became a huge mess and lose the control.  The language doesn't seem to be making much difference.  I'm not a software developer, but I had to write here and there, over many decades, all kind of software in all kinds of languages (mostly for personal use) i.e. BASIC, Z80 assembler, C, Pascal, VB, VBA, SQL, Python, etc.

Also familiar with techniques like versioning, automated testing, TDD, continuous integration, agile, etc. but I'm not really using any of that at home.  For example, I use git for home projects so I can easily share through github, but never roll back to previous versions, don't use branches, and when I try something new I rather make a manual "save as" before modifying.

I suspect my lack is in software design, maybe in OOP (I still use procedural programming style) or maybe it is because I was always "playing by ear".

TL;DR
What to learn, or what habit to change so my project will not end as abandoned because of unfinished software?
 



LATER EDIT:  summarize all answers before
2021-03-05 16:34, Fri
=====================
https://www.eevblog.com/forum/general-computing/hitting-a-wall-while-writing-software/msg3495656/#msg3495656

[Jeroen3]

Software is many times more time consuming, never finished, and tend to grow exponentially in complexity until it all turns into a mess, needs at least a refactoring if not rewriting everything, and so on.

You're going too fast  Slow down and design first. Try some TDD, it's very easy in Visual Studio, you can choose a new project with google test and start testing C/C++, very useful for embedded.
At least write tests for the little functions, eg: maths, base64encode. So you don't have to chase bugs in them on target, because that is unbelievable slow. And if you do think you found one, you create more tests to cover that specific scenario.

Begin here:
https://refactoring.guru/design-patterns/book
https://docs.microsoft.com/en-us/visualstudio/test/how-to-use-google-test-for-cpp?view=vs-2019
or some books/keynotes of Uncle Bob, they are an easy read/watch.

[grumpydoc]

Worth knowing how big a typical project of yours is, what hardware it targets and what sort of functionality you generally need to build in.

A half decent programmer can certainly knock together a few hundred lines of code which is reasonably clear and maintainable whether it is in assembler or C. C++ might be overkill for small platforms (the standard runtime libraries being rather large, but you can avoid using them if you need to do that) and I (no longer) like it much because it's too easy to produce "write only" code.

I don't think I'd recommend Pascal, BASIC or VB for an embedded project, but assembler, C or C++ would all be good choices depending on the exact functionality needed.

[tszaboo]

You are not doing anything wrong. Writing software IS boring, and most companies will hire specially bread people to do it. It only means that you don't find it interesting, just like me. I mean, yes, sometimes, to an extent. Usually, as you said, few hundred lines. Or few days.
You can achieve a lot more in that time by using higher level language. Micropython was more productive for me than C or C++.
Just move on, accept that you are a hardware guy. Nothing wrong with that. And then realize the power of soldering iron over the JTAG programmer.

[madires]

Note down the essential features and start small. Try to avoid feature creep while writing code. And don't force things.

[golden_labels]

Perfectionism will kill you in programming. I tell you that as both a perfectionist and a former software developer.

Unfortunately, one has to learn to stop worrying and love the bomb. That shouldn’t be treated as a permission to be lazy, to lack in knowledge or to avoid acquiring more exerience. But you have to understand that a perfect program can not be written and some sane limits of effort must be set.

A few tricks:

• Take your time understanding, what you have to write before even running your favourite code editor. Use paper+pen a lot, make some drafts on paper, spot major problems there.
• Write the primary logic first in a manner that it produces the desired effect for valid data. It may be sketchy, it may be imperfect, it may be ugly, it may be inefficient, but it must do what it is exected to do.
• If you see any obvious issues, that can’t be fixed within 2 minutes, mark them as bugs in comments and fix immediately after the sketchy version is in somewhat working. Not before, but also do not leave that for unspecified “later”.
• If you spot any potential problems that are unlikely, add them to some bugs list, describe them and leave them for being fixed later.

And now your main problem becomes not that you can’t write a program, but the ever-growing backlog of the “fix-later” issues . It is useful, though. First, the number of issues WONTFIX’d with no good reason is a measure of your laziness. Second: it tells you how many feature requests you may accept from other people (or yourself). Of course ideas are always welcome and you should always write them down, but often you can’t use them all. Third, probably most important, the number of issues WONTFIX’d with the reason “I prefer to spend some time with friend/family” or similar is a measure of your sane approach to developing programs. Really: set time limits.

[capt bullshot]

I know pretty well what you describe.
For one particular project, it helped a lot I drew this diagram: http://wunderkis.de/pvbat/sm.pdf before I started to write code.
So the state machine worked nearly out of the box, still were enough nasty interesting and challenging details to take care of (esp. HW interaction at certain state transitions).

[RoGeorge]

Worth knowing how big a typical project of yours is, what hardware it targets and what sort of functionality you generally need to build in.

Good point, without an example it will all be just talking words and generic advice.

It just happens that I was cleaning around, and stumbled upon a box with yet another unfinished project, a discharger for AA batteries.  It is a simple project, cobbled a minimal hardware in a couple of hours, with lots of corner cutting because "it should be possible to do that in software".

The goal was to sort out the health of AA rechargeable batteries by discharging them with a constant current:
- main function - measure the remaining capacity of the battery, C (mAh), stop when the voltage become too low
- must have - 4 independent discharging channels, and plot of voltage over time on the PC
- nice to have - measure internal resistance by changing the discharge current while observing the battery voltage variations
- nice to have - autodetect when another battery is plugged, and restart a new measuring process

The hardware was just a MOSFET with a shunt resistor for each battery, and a microcontroller to read each battery current and each battery voltage (ATmega48 from an Arduino nano).  Each MOSFET is controlled by a linear voltage (not by PWM) in order to set the desired current.  The constant current is stabilized by a PID loop for each battery.  The analogue voltage for MOSFETs is obtain by improvising a DAC from a PDM (Pulse Density Modulation) followed by an RC filter to average the pulses into an analogue voltage that will control the MOSFET gate.

Found a pic of it, this is channel 2 only, an AA battery socket with a MOSFET, an RC filter and a shunt resistor.  The rest is just wires connected to Arduino pins.  Will search for a schematic drawn, too.



I don't recall the exact state of the software.  The interrupts driven PDM DAC was working, oversampling ADC, too, digital PID kind of working but with some winding problems IIRC.  Battery change detection was a mess, and no serial commands were yet implemented (to control the whole thing for PC).  No PC side program for voltage plot or mAh calculation either.

Will search for the sources, and attach them as an example of unfinished software.

[Caliaxy]

“Hardware eventually fails. Software eventually works.”   

[RoGeorge]

Found the schematic and the software:



BattDisch.ino (main)

Code: [Select]

//------------------------------------------------------------------------------
// Hardware related definitions
//------------------------------------------------------------------------------
#define N_CH  4  // Number of PDM channels (Pulse Density Modulation)

#define PDOUT_MASK  B01100000 // Bits PD6 and PD5 are PDM out for CH4 and CH3
#define PBOUT_MASK  B00000110 // Bits PB2 and PB1 are PDM out for CH2 and CH1



//------------------------------------------------------------------------------
// Delta-Sigma related definitions
//------------------------------------------------------------------------------
#define MAX_ALL 2048         // PDM resolution

#define MAX_CH_1    MAX_ALL  // Distinct possible steps between 0-100%
#define MAX_CH_2    MAX_ALL  // Distinct possible steps between 0-100%
#define MAX_CH_3    MAX_ALL  // Distinct possible steps between 0-100%
#define MAX_CH_4    MAX_ALL  // Distinct possible steps between 0-100%



//------------------------------------------------------------------------------
// Global variables used for each software channel of a Delta-Sigma modulator
//------------------------------------------------------------------------------
volatile uint16_t  max[N_CH];    // Maxim level (resolution) for each channel
volatile uint16_t  req[N_CH];    // Requested levels, 0 <= req <= max
volatile uint16_t  sum[N_CH];    // Integrators value, 0 <= sum < 2*sum

volatile uint8_t   outBits;      // Each bit store the output value of one modulator



//------------------------------------------------------------------------------
// Global variables used for each battery channel
//------------------------------------------------------------------------------
volatile uint16_t battRead[N_CH];    // Store oversampled V_batt ADC value
volatile uint16_t sensRead[N_CH];    // Store oversampled I_batt ADC value

volatile uint16_t min[N_CH];           // Undervoltage level
volatile int8_t protStat[N_CH];     // Undervoltage protection status



//------------------------------------------------------------------------------
// Protection values
//------------------------------------------------------------------------------
#define Imin  3     // 1024 is for 1.1V on 0.15ohms,  1 unit ~ 7 mA
#define Vmin  100   // 1024 is for 5.0V,  1 unit ~ 5 mV



//------------------------------------------------------------------------------
// Desired discharging current
//------------------------------------------------------------------------------
volatile uint16_t dischI[N_CH];



//------------------------------------------------------------------------------
// Generic loops index reused variables
//------------------------------------------------------------------------------
int8_t i, n;



//------------------------------------------------------------------------------
// Function prototypes
//------------------------------------------------------------------------------
void init_all_CH_arrays();
void init_pins_dir();
void init_interrupts();

void init_prot();
void init_control();

void read_all_ADC();
void check_prot();
void control_I();



void init_all_CH_arrays() {
//------------------------------------------------------------------------------
// Initialize all 10 channels arrays with different periods and initial values
//------------------------------------------------------------------------------
  max[0] = MAX_CH_1;  // Set maxim value (resolution) for each PDM channel
  max[1] = MAX_CH_2;
  max[2] = MAX_CH_3;
  max[3] = MAX_CH_4;
 

  //  #define REQUEST MAX_ALL
  //#define REQUEST   3 * MAX_ALL / 4   // Initial PDM value after a HW reset
  #define REQUEST   1500   // Initial PDM value after a HW reset

  req[0] = REQUEST;      // Initial channels start values
  req[1] = REQUEST;
  req[2] = REQUEST;
  req[3] = REQUEST;

  // req[0] = MAX_ALL;      // Initial channels start values
  // req[1] = MAX_ALL;
  // req[2] = MAX_ALL;
  // req[3] = MAX_ALL;
 

  #define DISCH 70        // desired discharge current, 70 for aprox 0.5A

  dischI[0] = DISCH;
  dischI[1] = DISCH;
  dischI[2] = DISCH;
  dischI[3] = DISCH;
}



void init_pins_dir() {
//------------------------------------------------------------------------------
// Initialize pins direction
//------------------------------------------------------------------------------
  DDRB |= PBOUT_MASK;  // set PB2, PB1 as outputs
  DDRD |= PDOUT_MASK;  // set PD6, PD5 as outputs

  DDRC = 0;  // all port C as inputs for ADC
  PORTC = 0; // no initial pullups for ADC inputs
}



void setup() {
  // put your setup code here, to run once:

  // initialize serial communications at 9600 bps:
  Serial.begin(9600);
 
  init_all_CH_arrays();
  init_pins_dir();
  init_interrupts();

  init_prot();
  init_control();

  Serial.println("Initialized");

}

void loop() {
  // put your main code here, to run repeatedly:

  read_all_ADC();
  delay(10); // wait at least 5ms for the Vref to have time \
                to stabilize to 1.1V for the next measurement

  //check_prot(); // undervoltage protection

  control_I();
  delay(100);   // wait for Vgs to stabilize after PDM adjust



  //print only if a second has passed
  unsigned long time_s = 0;
  unsigned static long prev_s;

  time_s = millis() / 976;
  //time_s = millis() / (976 + 976);
  if (time_s == 0) prev_s = 0;

  // Serial.print(time_s);
  // Serial.print(" ");
  // Serial.println(prev_s);
 
  if (time_s > prev_s) {
    if (true) {
      send_to_serial_battRead();
      send_to_serial_sensRead();
      send_to_serial_req();
      send_to_serial_proStat();
      send_to_serial_dischI();
      Serial.println();
    }

    prev_s = time_s;
  }



  //convert_to_human_units();

  //Serial.println("loop ended");
}



ADCreadings.ino

Code: [Select]

void read_all_ADC() {
//------------------------------------------------------------------------------
// Read 2**OVERSAMPLINGs of ADC for each channel's V_batt and I_batt
//------------------------------------------------------------------------------

// !!! Switching Vref from 1.1V to 5V takes only 10us (Vcc has low impedance)
// !!! Switching Vref from 5V to 1.1V takes about 5ms because of the C on Aref
//     For default OVERSAMPLE 3 and DISCARDED 8, read_all_adc() takes ~10ms
//     For default OVERSAMPLE 5 and DISCARDED 8, read_all_adc() takes ~33ms

#define OVERSAMPLING    1   // must be 0...6, will measure 2^^OVERSAMPLING times
#define DISCARDED       4   // number of discarded conversions after Vref switch



    // set the ADC range to 1.1V and read all channels for I_batt
    analogReference(INTERNAL);              // set ADC Vref = 1.1V
    //delay(10);

    for (i = 1; i <= DISCARDED; ++i)           
        sensRead[0] += analogRead(N_CH);    // discarded batch of readings after changing the Vref
   
    for (n = N_CH - 1; n >= 0; n--) {

        sensRead[n] = 0;
        for (i = 1; i <= (1 << OVERSAMPLING); ++i)   {
            sensRead[n] += analogRead(N_CH + n);
        }

        sensRead[n] >>= OVERSAMPLING;   
    }



    // set the ADC range to 5V and read all channels for V_batt
    analogReference(DEFAULT);               // set ADC Vref = 5V
    //delay(10);

    for (i = 1; i <= DISCARDED; ++i)           
        battRead[0] += analogRead(0);       // discarded batch of readings after changing the Vref

    for (n = N_CH - 1; n >= 0; n--) {

        battRead[n] = 0;
        for (i = 1; i <= (1 << OVERSAMPLING); ++i) {
            battRead[n] += analogRead(n);
        }

        battRead[n] >>= OVERSAMPLING;
    }

    analogReference(INTERNAL);              // to have it stabilized on 1.1V next time
    analogRead(0);                          // dummy read to switch Vref to INTERNAL
}



Icontrol.ino

Code: [Select]

//------------------------------------------------------------------------------
// Stabilize the discharging current
//------------------------------------------------------------------------------
volatile static uint16_t start[N_CH];
volatile static uint16_t stop[N_CH];

void control_I() {
    for (int8_t n = 0; n < N_CH; n++) {
        if (dischI[n] > 0)          // if channel on (I desired > 0)
            control(n);
        else {
            noInterrupts();
                req[n] = 0;             // power off the channel
            interrupts();
            init_control_n(n);      // init control for future random value of I desired > 0
        }
    }
}



void init_control() {
    for (int8_t n; n < N_CH; n++) {
        init_control_n(n);
    }
}



void init_control_n(int8_t n) {
    start[n] = 0;
    stop[n] = max[n];
}



void control(int8_t n) {
    if ((stop[n] - start[n]) < 2) {         // if segment very small, enter in follower mode
        if (sensRead[n] < dischI[n]) {         // I DS too small, increment
            if (req[n] < max[n]) {
                noInterrupts();
                    req[n]++;
                interrupts();
            }
        };
        if (sensRead[n] > dischI[n]) {         // I DS too big, decrement
            if (req[n] > 1) {
                noInterrupts();
                    req[n]--;
                interrupts();
            }
        };
        start[n] = req[n];                      // follow
        stop[n] = req[n];
    }
    else {                                  // in search mode, half the searching segment
        if (sensRead[n] < dischI[n]) {         // I DS too small, pick the bigger half
            start[n] = req[n];
        }
        else if (sensRead[n] > dischI[n]) {                            // I DS too big, pick the smaller half
            stop[n] = req[n];
        }
        else {                                  // right on spot, switch to follower mode
            start[n] = req[n];
            stop[n] = req[n];
        }
        noInterrupts();
            req[n] = (start[n] + stop[n]) >> 1;
        interrupts();
    }
}



PDMinterupt.ino

Code: [Select]

void init_interrupts() {
//------------------------------------------------------------------------------
// Set Timer2 to generate interrupts at each 20us (TCCR2B = 2, OCR2A = 39)
//------------------------------------------------------------------------------
  TCCR2A = (1 << WGM21);    // CTC mode
  TCCR2B = (2 << CS20);     // 1..7 for 16MHz div 1, 8, 32, 64, 128, 256 or 1024
  OCR2A = 39;               // 0..255

  TIMSK2 |= (1 << OCIE2A);  // Set interrupt mask bit
}



ISR(TIMER2_COMPA_vect) {
//------------------------------------------------------------------------------
// Timer2 CTC interrupt rutine
//------------------------------------------------------------------------------
  // output last calculated
  PORTB = outBits & PBOUT_MASK;  // update value PB bits 1, 2
  PORTD = outBits & PDOUT_MASK;  // update value PD bits 5, 6
 

  // calculate the outputs status to be written at next interrupt
  //    mapping outBits bits order to output pins/channels
  //    bit 0 -> CH1 of the discharger (PB1)
  //    bit 1 -> CH2 of the discharger (PB2)
  //    bit 2 -> CH3 of the discharger (PD5)
  //    bit 3 -> CH4 of the discharger (PD6)

  // Sigma delta modulation algorithm using "synthetic division"
  outBits = 0;                  // Initialize all outBits for recalculation
  sum[3] += req[3];               // Update integrator value
  if (sum[3] < max[3])
    outBits++;                    // LSB = 1
  else
    sum[3] -= max[3];             // LSB = 0 (untouched) and adjust integrator

   
  // Sigma delta modulation algorithm using "synthetic division"
  outBits <<= 1;                  // Shift other bits and reset current LSB
  sum[2] += req[2];               // Update integrator value
  if (sum[2] < max[2])
    outBits++;                    // LSB = 1
  else
    sum[2] -= max[2];             // LSB = 0 (untouched) and adjust integrator


  // Sigma delta modulation algorithm using "synthetic division"
  outBits <<= 3;                  // Shift other bits and reset current LSB
  sum[1] += req[1];               // Update integrator value
  if (sum[1] < max[1])
    outBits++;                    // LSB = 1
  else
    sum[1] -= max[1];             // LSB = 0 (untouched) and adjust integrator

     
  // Sigma delta modulation algorithm using "synthetic division"
  outBits <<= 1;                  // Shift other bits and reset current LSB
  sum[0] += req[0];               // Update integrator value
  if (sum[0] < max[0])
    outBits++;                    // LSB = 1
  else
    sum[0] -= max[0];             // LSB = 0 (untouched) and adjust integrator


  // Align then invert the 4 calculated rezult bits
  outBits <<= 1;
  outBits ^= PDOUT_MASK | PBOUT_MASK;
}



printOut.ino

Code: [Select]

void send_to_serial_req() {
    for (n = N_CH - 1; n >= 0; --n) {
        Serial.print("req[");
        Serial.print(n);
        Serial.print("]=");
        Serial.print(req[n]);
        Serial.print(" ");
    }
    Serial.println();
}



void send_to_serial_battRead() {
        for (n = N_CH - 1; n >= 0; --n) {
        Serial.print("battRead[");
        Serial.print(n);
        Serial.print("]=");
        Serial.print(battRead[n]);
        Serial.print(" ");
    }
    Serial.println();
}



void send_to_serial_sensRead() {
        for (n = N_CH - 1; n >= 0; --n) {
        Serial.print("sensRead[");
        Serial.print(n);
        Serial.print("]=");
        Serial.print(sensRead[n]);
        Serial.print(" ");
    }
    Serial.println();
}



void send_to_serial_proStat() {
        for (n = N_CH - 1; n >= 0; --n) {
        Serial.print("protStat[");
        Serial.print(n);
        Serial.print("]=");
        Serial.print(protStat[n]);
        Serial.print(" ");
    }
    Serial.println();
}

   
   
void send_to_serial_dischI() {
        for (n = N_CH - 1; n >= 0; --n) {
        Serial.print("dischI[");
        Serial.print(n);
        Serial.print("]=");
        Serial.print(dischI[n]);
        Serial.print(" ");
    }
    Serial.println();
}




protection.ino

Code: [Select]

//void overdischarge_protection() {
//------------------------------------------------------------------------------
// Protection to trigger when V < Vmin or I < Imin, rearms when battery removed
// TO DO - reimplement it as a state machuine, this is just a D R A F T
//       - use PORTC = 255 (enable pullups for analog inputs)
//          to determine afsent battery (battRead wil be ~ 1020, sensRead ~ 70)
//------------------------------------------------------------------------------
//     for (n = N_CH - 1; n >= 0; --n) {
//         // if V <= Vprot thresh for long enough, disable channel
//         // if I is > 0 and V > Vprot for long enough, enable channel

//         if ((protStat[n] > 0) & (protStat[n] < 255))
//             protStat[n] += 1;
       
//         if (((battRead[n] <= Vmin) | (sensRead[n] < Imin)) & (protStat[n] > 0))
//             protStat[n] -= 1;

//         if ((protStat[n] == 0) & (battRead[n] < 10))
//             protStat[n] = 127;

//         if (protStat[n] == 0)
//             req[n] = 0;         // disable channel when protection has triggered
//     }
// }



#define INITPROTCOUNTER 10
#define LOWESTVOLTAGE   150 // 150 for minimum Vbatt = 0.73V



void arm_prot_ch(int8_t n) {
    protStat[n] = INITPROTCOUNTER;
    dischI[n] = DISCH;
}



void init_prot() {
    for (int8_t n = 0; n < N_CH; n++) {
        min[n] = LOWESTVOLTAGE;
        arm_prot_ch[n];
    }
}



void check_prot() {
    for (int8_t n = 0; n < N_CH; n++) {
        if ((req[n] > 0) && (dischI[n] > 0)) {                   // if channel is ON
            if (battRead[n] < min[n]) {
                protStat[n]--;
                if (protStat[n] < 0) protStat[n] = 0;
            }
            else {
                arm_prot_ch(n);
            }
        }
        if (protStat[n] == 0) {
            // req[n] = 0;                     // Undervoltage, turn channel OFF

            Serial.print("Channel ");
            Serial.print(n + 1);
            Serial.print(" turned OFF for battery voltage lower than ");
            Serial.println(min[n]);

            req[n] = 0;                     // Undervoltage, turn channel OFF
            dischI[n] = 0;


        }

        if (battRead[n] == 0) {
            arm_prot_ch(n);
        }       
    }
}

[CatalinaWOW]

While this falls in the area of general words, I think one of your problems is a "software is simple" mindset.  My history is very similar to yours so my thoughts may provide you some insight.

1.  You have a very simple hardware implementation.  All of the hard parts have been dumped into software.  Think of how difficult your feature set would be if all of the functions on your list had to be done in hardware.  That should provide some reference for how hard the software is.  In this case I think software is easy is a correct assessment in the sense that once you figure out the correct physics and electronics required to detect the status and operation you are asking it is easier to implement in software than in hardware, but there is a lot of work involved.  You may be assigning this learning and design work to software while it would be there however you implemented the project.  If you had done all that work prior to the project start it probably would have made the actual software seem much simpler.

2.  Based on that thought there are some opportunities for software/hardware trades.  For example adding a pushbutton at each station for the operator to signal a new battery insertion would make the software side trivial (sort of, see next comment).

3.  You may want to revise some of your software architecture.  Interrupts are an incredibly useful tool.  But do introduce some problems in software.  I see no particular benefit to using them in your basic design.  A polling structure would meet all of the needs of a slowly changing system such as charging/discharging.  Generating nice time stamps with known intervals can even be handled using the Arduino provided ecosystem, or if you want to use the bare processor just by calibrating your loop speed.  But here is the feedback to comment 2.  An interrupt isn't a bad way to deal with a pushbutton.  But another trade presents itself - software or hardware debounce.  With an appropriate hardware debounce switch detection will also work in a polling environment.

4.  All of this comes down to doing a lot of thinking before you start, breaking the total project down to hardware and software chunks and making sure that each chunk is within your capabilities, or at least within what you will be willing to learn.  This last point may be the core of the problem.  I have a large project that is still not done after more than ten years of off and on progress.  One of the "hold ups" was that the project controls some potentially dangerous machinery and needs an emergency stop button that will shut everything down.  The software is done in VB and several subpanels are used, and this stop needs to be effective no matter where you are in the software.  It requires ability to write multi-threaded code in VB, which I can't, and haven't been able to learn.  I have implemented the emergency shutdown in hardware and foregone the software version.  I would still like to have the software version as a secondary path, but it just isn't happening. 

[T3sl4co1l]

Agree, diagramming helps.  Sketch out the high level function, as tasks/events in a loop, or a flowchart, etc.  This can be implemented as a massive if-else in the main loop, or jumps between functions (make sure they don't loop forever of course...), or a more formal state machine structure (which in turn can be a big switch statement, or say, precalculated elsewhere using a state transition table).

Maybe you need to do something special-case, interrupting overall flow of the loop, and it's just not feasible to fold it into the loop normally -- that's fine, you can jump into another lengthy function and do your work there.  Don't do this lightly: you don't have access to all the clauses and functionality of the main loop, outside of it; more than a few special cases like this, will quickly lead to spaghetti.  See #1, diagram it out, try to integrate it better, maybe there's a pattern you're missing that would allow both operating modes to coexist, etc.

May also help to implement simple yet meaningful functions, and run them from a sort of command shell.  I've been doing this the last little while, and find it useful.  At least, because I don't have a debugger for my favored device family...  Example, figuring out a peripheral: well, start with the basics, write a command to PEEK/POKE its control registers.  This can be as basic as the MCU's own registers; which if they're memory mapped, simultaneously gives you insight into internal RAM, handy for inspecting the live program -- albeit from the shell's main loop, not at just any point along any particular function.  Maybe it's an SPI device, so you boot up the MCU's SPI peripheral, and implement a short data/command R/W function.  Maybe the next level up is an init command, or something that runs a sequence of values or commands or whatever.  Then commands to use specific internal functions, like uh, for an LCD display you'd certainly want some GRAM address and data commands so you can start drawing pixels.  Then maybe basic shapes, images, etc.

So you can start at a very low level, implementing bits and pieces, which don't go anywhere else in your program -- but you can still test them in this way (and yes, preferably with more detailed/formalized tests, as in TDD), meanwhile you can diagram the very highest of levels, and mark out what the overall program state goes through, what the highest level functions will be, etc.

And then finally, you can add in optimizations or customizations or whatever.  Maybe it's rather slow, so you need to reconsider some algorithms (accidentally quadratic?), or implement them in ASM.  Maybe you've been using it a while, and yeah it's usable, but it's still a bit of a pain and wouldn't it be great if it did X automatically?

Reminds me, I still need to add a filter calculator to my reverb effects box... It's usable, yes, but the filter coefficients are literally just right there on the menu, and good luck tweaking them just by eye!  That should be interesting by itself actually, I might do it in floating point (needs several trig functions; library is probably bloated and slow), or I might look into shaving a fixed-point-math-shaped yak...


Oh, anyway, the source to that project is here if you're curious,
https://github.com/T3sl4co1l/Reverb
console.c and commands.c may be of interest (the command shell, and the commands which plug into it), or main.c and menu.c for the UI loop.

Tim

[RoGeorge]

Try some TDD

TBH, I've tried TDD once after reading a book about it, and didn't like it.  Might worth using TDD for big projects and big teams of software developers, IDK, but does it worth for small single person projects?

I found TDD to be very impractical, if not impossible to follow.  Especially for embedded or some other software that controls real world machinery/hardware, I don't even know how to approach automatic testing when physical devices are involved, and even if it were to do that, it will probably take me 100 times more time writing the test cases then writing+debugging the program.

Is TDD still beneficial for small programs, like the example posted above?

https://refactoring.guru/design-patterns/book

Looks interesting, thank you.

Just move on, accept that you are a hardware guy. Nothing wrong with that.

Oh well, I move on, but then I found myself dedusting junk finished 2 years ago, yet still not usable because the software is not ready, like this battery discharger here (and most if not all of my other projects  ).

Perfectionism will kill you in programming.
...
Really: set time limits.

I'll write down especially these two, to remember and apply in the future, thank you.

About the time limits, what would be a reasonable time limit for the goals in that discharger project?

drew this diagram: http://wunderkis.de/pvbat/sm.pdf before
...
state machine

Indeed, at some point I felt I should have drawn a state machine first for that discharger project (and for most of the other projects, too).  Usually I don't lay down a state machine before proceeding, and almost always end up with unwanted behavior of a device, or hard to spot bugs.  Will write down this one, too, thank you.

What did you used to make the one from your PDF?

3.  You may want to revise some of your software architecture.  Interrupts are an incredibly useful tool.  But do introduce some problems in software.  I see no particular benefit to using them in your basic design.

Indeed, I need to revise the project, and next time better use an opamp to stabilize the discharging current rather than dealing with that in a digital PID.

As a side note, the interrupts are a must have here.  They are used to generate PDM (Pulse Density Modulation) instead of PWM, so the design would be scalable to as many channels as I/Os and ADCs are available.

Software PDM is cheaper to implement, and works at much higher frequency than it is possible with software PWM.  For example, the yellow PDM trace shows much denser pulses than the blue trace of PWM pulses.



This is beneficial, for example, to avoid flickering at very low duty factors, when compared to PWM.  I've compared once PDM vs. PWM, and wrote a software PDM implementation for a MSP430, where all its 10 existing I/O pins were used to dim LEDs:



https://hackaday.io/project/6356-delta-sigma-versus-pwm

[rstofer]

I ran into the idea back in the early '80s:  Top Down Design, Bottom Up Coding.

Draw pictures of the blocks starting at the very top level and describe their interaction.  Keep drilling down with blocks until you hit the hardware.  In my case, the first bit of code I write will be a UART handler and the next bit will be conversion routines for outputting debug values.  To get the UART to run, I may need to add code for the various clock enables as well as any pre-scalers.

In theory, I would write all the hardware code first and then build up to the final user interface.  But the top level interface would already be designed so I know where I am heading as I work up.

[tszaboo]

Just move on, accept that you are a hardware guy. Nothing wrong with that.

Oh well, I move on, but then I found myself dedusting junk finished 2 years ago, yet still not usable because the software is not ready, like this battery discharger here (and most if not all of my other projects  ).

So partner up with someone who doesnt do hardware and likes software.
Or make the hardware, open source it and then, if it is interesting someone will write the software for you.
Do projects, where the most work is in the hardware. Analog stuff, volt nut, time nut and so on. You want to discharge a battery? Replace the microcontoller with a ten turn potentiometer.

All that code that you wrote there. All you need is a current setpoint, a voltage setpoint, and one or two off the self current and voltage panel meter. You can make a PI controller from an opamp.

[SiliconWizard]

Well, the OP may not be very fond of writing software... but even if they were. My humble experience in industrial settings is that software ALWAYS takes longer than hardware to develop, except in very specific cases with very involved analog design. I've almost never seen the opposite. Except with the design of complex integrated circuits...

Now of course liking it will make the process less painful (except for those that are waiting for the software to be done!)
Good methodology will also help. But still, don't expect software to be done in less time than the hardware counterpart in a typical project when software is required.

[rstofer]

Software is hard.  Many years ago I worked for one of the mainframe manufacturers.  It was said that it cost us more money to ship the OS/compilers than it cost to ship the machine.  And we're talking big iron, not those Itty Bitty Machines.

Not only is software hard, it is expensive.

[KE5FX]

It happens to me all the time when building something new.  The hardware is nice and fun to make, then writing the software turns the project into a chord.  Software is many times more time consuming, never finished, and tend to grow exponentially in complexity until it all turns into a mess, needs at least a refactoring if not rewriting everything, and so on.

Not sure how to deal with that.  I don't want to rewrite 2-3 times every program, end even when I rewrite something, I still hit a wall (at about a few hundreds lines) where it all became a huge mess and lose the control.  The language doesn't seem to be making much difference.  I'm not a software developer, but I had to write here and there, over many decades, all kind of software in all kinds of languages (mostly for personal use) i.e. BASIC, Z80 assembler, C, Pascal, VB, VBA, SQL, Python, etc.

Also familiar with techniques like versioning, automated testing, TDD, continuous integration, agile, etc. but I'm not really using any of that at home.  For example, I use git for home projects so I can easily share through github, but never roll back to previous versions, don't use branches, and when I try something new I rather make a manual "save as" before modifying.

I suspect my lack is in software design, maybe in OOP (I still use procedural programming style) or maybe it is because I was always "playing by ear".

TL;DR
What to learn, or what habit to change so my project will not end as abandoned because of unfinished software?
 

After 10 years of experience, you will be able to maintain 10,000 lines of crufty C code piled up in one place, and adapt it successfully when new requirements emerge.

After 15 years of experience, 20000 lines.

After 20 years of experience, 30000 lines.

After 25 years of experience, you will be able to (barely) maintain 40000 lines of insanity in one C file.  That's pretty much where I am. 

The only good news is that your competitors are having just as hard a time as you are.  And the only good advice is John Carmack's: don't get attached to your code.  Be ready to throw everything out and rewrite it, multiple times, the moment it starts to seem like that might be a good idea. 

Of course, John Carmack isn't going to pay your mortgage in the meantime...

[nctnico]

In general when writing software: start with a good description and diagrams / flow charts. I can spend weeks on just figuring out how a (complicated) piece of software needs to be structured and put it on paper. Even if it is a re-incarnation of an existing piece of software.

The only good news is that your competitors are having just as hard a time as you are.  And the only good advice is John Carmack's: don't get attached to your code.  Be ready to throw everything out and rewrite it, multiple times, the moment it starts to seem like that might be a good idea. 

He is right. Software is build upon a base structure -call it a chassis- and at some point that chassis can no longer support all the addons. But I have an addition to John's statement: let someone else do the re-write and work on something more interesting yourself! You'll see that other people find the uninteresting stuff very interesting.

[capt bullshot]

drew this diagram: http://wunderkis.de/pvbat/sm.pdf before
...
state machine

Indeed, at some point I felt I should have drawn a state machine first for that discharger project (and for most of the other projects, too).  Usually I don't lay down a state machine before proceeding, and almost always end up with unwanted behavior of a device, or hard to spot bugs.  Will write down this one, too, thank you.

What did you used to make the one from your PDF?

So your question is a bit ambiguous to me, may one part of my answer is superflous:
I've got no special tools, neither to draw the state machine nor to convert it to code.
I've used inkscape to draw the diagram, and my brains and fingers to convert it to a big switch  () { case } statement in plain C, like this:

Code: [Select]

while (1) {
switch (spwm) {
case SPWM_OFF : do_pwm_off(); break;
case SPWM_START: do_pwm_start(); break;
case SPWM_INV: do_pwm_inv(); break;
case SPWM_INVON: do_pwm_invon(); break;
case SPWM_SOLSTART: do_pwm_solstart(); break;
case SPWM_SOLMPP: do_pwm_solmpp(); break;
case SPWM_SOLREG: do_pwm_solreg(); break;
case SPWM_SOLMPPC: do_pwm_solmppchg(); break;
case SPWM_SOLREGC: do_pwm_solregchg(); break;
case SPWM_SOLRION: do_pwm_solregion(); break;
case SPWM_SOLREGI: do_pwm_solreginv(); break;
case SPWM_SOLCION: do_pwm_solchgion(); break;
case SPWM_SOLREGCI: do_pwm_solregchginv(); break;
case SPWM_DISC: do_pwm_disc(); break;
case SPWM_DISCSOL: do_pwm_discsol(); break;
case SPWM_CHGSTART: do_pwm_chgstart(); break;
case SPWM_CHGMPP: do_pwm_chgmpp(); break;
case SPWM_BKUP: do_pwm_bkup(); break;
case SPWM_BKUPSOL: do_pwm_bkupsol(); break;
case SPWM_TB: do_pwm_tb(); break;
case SPWM_FAIL: do_pwm_fail(); break;
default: stoprq=1; break;
};
...


I've attached the source file containing this particular statement, just in case you're interested. There's still a lot of other stuff around this state machine.

[Nominal Animal]

For diagrams, I recommend trying out Dia as well as Inkscape.  Both are available for Windows, macOS, and Linux at least.

For state diagrams and graphs where you already know (that is, can list as text) the relationships, I recommend trying out Graphviz.  It is a tool that takes in simple textual descriptions of graphs (using the DOT language), and creates the graph images based on that description.  One of the supported output formats is SVG, so you can further finesse the output in e.g. Inkscape – although I've found that it is faster for me to just redraw the graph in Dia, looking at the Graphviz-generated graph.

The Graphviz Gallery page on Bazel Build System shows a pretty good example of both the output and the DOT language input you need for this kind of graphs.

As to the topic at hand, I'm afraid I'm so oddball my suggestions would probably not work for others.  I'm addicted to problem-solving, and because many problems are better solved by solving a different, underlying problem that causes the problem at hand to vanish, not all solutions I create involve me writing code.  I treat the projects as tools to solve problems, deconstructing it into sub-problems I can attack in any order.  I'm most productive if I solve the tedious sub-problems first; like what kind of communications format would work best between a computer program and a microcontroller, by implementing both ends in a simulated real-world scenario.  I learned rewrites are my friend, and nowadays by default try to create code I can refactor (rewrite in parts) easily whenever I like, without dreading it: it is par for the course.  I do write more code than I end up using, but every part I write tells me something about the problems I'm solving so is fundamentally useful.  Code is never "finished", never "perfect"; it is only "usable" or fragile agglomerated crap that needs a rewrite or at least a refactoring.

[Syntax Error]

As a 'softie', all I can add is that I agree with @NominalAnimal. Software is never done. At some magic state/threshold a "stable release" or "release candidate" is achieved. Which is immediately followed with "minor upgrades", "major upgrades" or "service packs" to correct some "stability issue" or a "zero day security flaw".

So then you're forever tweaking the codebase before the hardware becomes obsolete. And then beyond that, when the code achieves "legacy support".

Imagine being able to upgrade a PCB by adding and subtracting parts and tracks? But then every product would be made of those white plastic breadboards. Such is the beauty of software.

Contrasting SW to HW, HW is about fix it, sell it. But far too many SW developers have grown up with the luxury culture of  sell it, fix it. HW guys have "development prototypes", SW guys have "recursive normalization", "bug fixes" and a state of "perpetual beta".

Code: [Select]

namespace RefactorFactory
while(true) { /* add more code here */ }
Tip: Just add lots of comments. It will make your life so much easier... in (20)five years time

[DiTBho]

Over the past 10 years I have rewritten my C source something like 5 times at least.
Use to add a suffix to the folder to manage it and avoid confusion.

What I find really difficult is syncing sources between multiple machines when I have to travel, for instance when I clone a repo on my personal laptop, and then forget it for a while, with some probability of forking the same code revision on a workstation when I get home but without a push. I use Git Merge + Vim Diff, but I sometimes make mistakes during the merge process.

Managing software is an hard job, not only it's difficult to design it, but also to manage revisions.

[retiredfeline]

Hitting a wall while writing software

Well there's your problem. If you use one or both hands to hit then you won't be able to type. If you use your head to hit you won't be able to see the screen. 

No, I don't have any suggestions because it's all been said above. I've been writing software for decades and I'm still dissatisfied with my results, even though I get most of my projects working. Software is hard, and the best I can do is improve it until it's not worth my effort anymore.

[RoGeorge]

It seems I was falling for most (if not all) of the mistakes highlighted in this thread.  Very useful advises so far, thank you all.   

My biggest mistake, I guess, is the habit of start writing code without designing a state machine first.

I've stumbled upon this tool, QM, that not only can draw the diagram of a HSM (Hierarchical State Machine), but can also generate C/C++ code out of the drawing.





Will try to rewrite the battery discharger software using QM, and see how it goes.