Unit testing embedded software

[Remark]

Hello everyone!

For some time now I have been interested in how I could test software (or its individual modules) using dedicated unit testing frameworks. I've seen a few of them in use online: gtest, unity, CppUTest, MinUnit, Unity and so on... Maybe you have tried something and could suggest which one to choose and why? I'm going to test software for ARM processors, m0-m4.

As far as I understand, for embedded targets, code 'modules' should be compiled on my desktop target. And I should pull in the modules into my tests and stub out/mock hardware specific ones. I probably still need a separate build system for tests?

Thanks for the suggestions

Remark.

[ataradov]

Well, spammers got their hands on ChatGPT. This sucks.

On topic - yes, ideally you would need a separate build environment unless your actual CPU has enough memory to support the test code as well.

Setting up a custom test environment suited for your project is not that hard, I don't see why you would want to use a standard environment/framework and adapt to its requirements.

The details depend on what your software is doing and how it is structured. The number of cases where unit tests make sense is very limited in reality. You can try to use them everywhere, but you are likely to suffer for not a lot of gain.

Automated functional tests of the complete system are way better in most cases.

[Remark]

Well, spammers got their hands on ChatGPT. This sucks.

On topic - yes, ideally you would need a separate build environment unless your actual CPU has enough memory to support the test code as well.

Setting up a custom test environment suited for your project is not that hard, I don't see why you would want to use a standard environment/framework and adapt to its requirements.

The details depend on what your software is doing and how it is structured. The number of cases where unit tests make sense is very limited in reality. You can try to use them everywhere, but you are likely to suffer for not a lot of gain.

Automated functional tests of the complete system are way better in most cases.

Yeah, exactly.

Thanks for the answer!

And what about the portability of project? The main project will be compiled using f.e. STM32 Cube IDE, and the unit tests f.e. Cmake with ninja?

[nctnico]

Well, spammers got their hands on ChatGPT. This sucks.

On topic - yes, ideally you would need a separate build environment unless your actual CPU has enough memory to support the test code as well.

Setting up a custom test environment suited for your project is not that hard, I don't see why you would want to use a standard environment/framework and adapt to its requirements.

The details depend on what your software is doing and how it is structured. The number of cases where unit tests make sense is very limited in reality. You can try to use them everywhere, but you are likely to suffer for not a lot of gain.

Automated functional tests of the complete system are way better in most cases.

Yeah, exactly.

Thanks for the answer!

And what about the portability of project? The main project will be compiled using f.e. STM32 Cube IDE, and the unit tests f.e. Cmake with ninja?

You can create desktop C projects with STM32 Cude IDE just fine (and any other project in a language that is supported by Eclipse which Cube IDE is based on). There might even be an Eclipse plugin that can help doing the tests.

[ataradov]

And what about the portability of project? The main project will be compiled using f.e. STM32 Cube IDE, and the unit tests f.e. Cmake with ninja?

It does not matter, use whatever works.

At the same time, I personally prefer testing on the hardware. Rig up your code to accept automated inputs and provide outputs. And then run the tests scripts on the PC that generate test outputs and accept the results and compare them to the reference.

Majority of stuff that causes real issues in the embedded system can't be accurately replicated outside of the device.

There are exceptions, of course. If you have some heavily algorithmic code, then it is possible to test its functionality outside, but the code like this usually does not depend on the hardware in a first place.

[pcprogrammer]

Is unit testing not a last step in a development process to see if the manufactured devices work as intended? And therefore can only be done on the actual hardware?

All other testing should be part of the development phase, where parts of the code could be tested on a simulation of the device, but timing related code like interrupt handlers basically need the actual hardware, especially if mechanical interfacing is done. Like motors with feedback on speed, position etc.

The hardest bugs to catch are of course glitches. Things that only happen once in a while under difficult to determine circumstances. Things that are missed in simulations.

[ataradov]

Is unit testing not a last step in a development process to see if the manufactured devices work as intended? And therefore can only be done on the actual hardware?

Unit testing is testing of individual software "units" in isolation.

The whole test driven development methodology is based on writing the unit tests firsts and then writing the code to fulfill those tests. TDD is BS, of course.

[eutectique]

Is unit testing not a last step in a development process to see if the manufactured devices work as intended?

That last one would be validation testing, where you prove that the manufactured device meets the requirements. And not just the software requirements, for that matter.

[gpr]

Is unit testing not a last step in a development process to see if the manufactured devices work as intended? And therefore can only be done on the actual hardware?

Unit testing is testing of individual software "units" in isolation.

The whole test driven development methodology is based on writing the unit tests firsts and then writing the code to fulfill those tests. TDD is BS, of course.

This is highly subjective. Pure TDD (when you write tests first before writing any code) is questionable, but can be useful sometimes, and I certainly would not call it BS.

[tellurium]

May I recommend to read two sections of my tutorial:

https://github.com/cpq/bare-metal-programming-guide#automated-firmware-builds-software-ci - how to do automated builds
https://github.com/cpq/bare-metal-programming-guide#automated-firmware-tests-hardware-ci - how to do automated tests on a real target hardware

[cantata.tech]

Hello everyone!

For some time now I have been interested in how I could test software (or its individual modules) using dedicated unit testing frameworks. I've seen a few of them in use online: gtest, unity, CppUTest, MinUnit, Unity and so on...

Try CppUTest first and see how you go.

I probably still need a separate build system for tests?

Not really.

Just modify your Makefile to include tests as targets, ie

> make tests

and then run them from the command line on your development system.

[Nominal Animal]

Here's what I do.  Apologies for the wall of text, feel free to skip if TLDR.

I implement and test algorithms in fully hosted environments (i.e. running under an OS on full-featured computers, almost always Linux in my case), individually ("in units").  I archive them one per directory, with a README describing the thing, and any test/example executable showing short usage when run with a single -h or --help command-line argument.  For more complicated stuff, I often have images and data in the same directory too.

For complex data structures like trees, heaps, directed graphs, etc., these test and example programs often output data structure information in Graphviz format (DOT language), for easy visual verification.

The ones I write in C or C++ for use in microcontrollers, are never in library form, because I always end up adapting them for the use case at hand.  So, instead of focusing on the same code being able to handle all situations I can imagine, I write them to do one thing, in as simple (but robust/secure/non-buggy) form as possible, so that it is maximally adaptable to my future needs, whatever they may be.

When integrating various pieces into a functional firmware, I use a mixed approach: not purely top-down or bottom-up, but a mix of the two.  At this phase, I consider the various pieces via their requirements, inputs, and outputs, and piece it together like an interwoven puzzle, and often develop the underlying structural "scaffold" code in tandem or slightly ahead of the functional units I add – by "scaffold" I mean things like bus interfaces, interrupt handlers, with temporary timing and memory use measurements to make sure I do not unexpectedly run out of computing resources.  When there are details I am unsure of, for example using a single timer for two different purposes, or maybe examining the computational cost of a specific approach, I do write test firmwares (often in the Arduino environment for rapid testing) to examine that case and that case only, without anything else.  These too go into the same archive of tests and examples.  (I do have thousands of these already, but only keep the most interesting/useful few hundred on my work machine, and move the rest to offline storage.)

Whenever I have added a functional piece, I do test the whole in practice, concentrating on the edge cases.  For example, with button interfaces I test contact bounce (brushing two ends of multistrand wires is a pretty good stress test) and multiple simultaneous keypresses.  I never proceed while there are misbehaving edge cases, but some of my colleagues and former colleagues disagree with that, because they believe getting the product on the market is more important than having it behave exactly as desired.  I understand their viewpoint, but that is just not how I develop stuff.

I have also created simulation test benches I can run on fully-features OSes, before implementing it on the hardware.  This is particularly useful for GUI development, where one can use GTK or QT (or any similar toolkit, or even SDL) for the UI, with mouse clicks as touches on the touch screen (with randomness added to model limited resolution), and key presses to simulate physical buttons.  (For pure mockups, to illustrate intended final product interface without reusing any of the code, I recommend HTML + CSS + JS, so that it can be demonstrated on any device with a standard web browser.  This is especially useful for touch-based interfaces, since you can get a real physical feel for the UI by using e.g. a phone for the demonstration.  I also like to use local/serverless HTML+CSS+JS "tool pages", because JS is pretty darned well optimized in current browsers, and such tool pages work everywhere.)

During the development, I like to keep a diary of sorts, listing any interesting or frustrating bits.  This is a very recent addition to my workflow, in the last year or two, after almost three decades of paid professional software development.  I "stole" this from the diaries of historical explorers, as I saw it an excellent way of self-reflection and analysis of my own performance afterwards, something I've always struggled with.  You don't need to mention it to anybody else, especially your employers, if you don't want to; but it is an excellent tool to work on ones time estimates, for example.  (My own time estimates have always been off the mark: most often underestimating the time needed, but sometimes overestimating [when I forget checking my test case archive before making the estimate].)

Finally, one of my favourite real world tests is just describing a problem, then telling an unsuspecting victim tester that this gadget can solve it, and then see how they (try to) use the gadget to solve the issue.  I try not to intervene, although sometimes a couple of sentences outlining the idea of how I'd find the solution may be necessary.  Theory gives us possibilities, and is therefore very useful (and I do still read peer-reviewed papers in physics and cs, even though I'm no longer in academia), but practical reality rules.

[tggzzz]

Here's what I do.  Apologies for the wall of text, feel free to skip if TLDR.

It would be a mistake for other readers to skip that.

It is a refreshing change to see a practical and pragmatic approach that is free of cargo-cult engineering and religious practices.

[Smokey]

https://resources.altium.com/p/test-driven-development-embedded-systems

https://www.amazon.com/Driven-Development-Embedded-Pragmatic-Programmers/dp/193435662X

[SiliconWizard]

Is unit testing not a last step in a development process to see if the manufactured devices work as intended? And therefore can only be done on the actual hardware?

Unit testing is testing of individual software "units" in isolation.

The whole test driven development methodology is based on writing the unit tests firsts and then writing the code to fulfill those tests. TDD is BS, of course.

This is highly subjective. Pure TDD (when you write tests first before writing any code) is questionable, but can be useful sometimes, and I certainly would not call it BS.

This certainly is a controversial approach. "Questionable" is about right, but "subjective"? Not sure there should be anything much subjective about it.
"BS" is a bit strong, sure, something I could have said myself.

One problem with TDD is that it promotes bottom-up development almost exclusively, because 1/ writing tests for small units is easier than for larger ones, and 2/ if writing tests for larger units first, then people would have to do more work, like mocking up the smaller bits, etc. So in practice, that's almost 100% bottom-up.

Which sucks. Bottom-up can be done concurrently to top-down for some parts of the development, but should not be used exclusively IMHO. It leads to horrible piles of code stitched together with almost no architecture.

So, here is why I think TDD is a mess. Not that thinking of testing before coding is not a valid idea. But it just doesn't work, because we are lazy (/have no time/have no budget to do better.)

TDD would be more interesting in a top-down approach as I mentioned above. That would mean designing larger blocks first, then the required tests, then coding - which essentially means having to mock up a lot of stuff at the beginning, and refining as development progresses. That's a lot of work. So people rarely do that. They just use whatever method is trendy and will look cool on their resume.

[tggzzz]

Is unit testing not a last step in a development process to see if the manufactured devices work as intended? And therefore can only be done on the actual hardware?

Unit testing is testing of individual software "units" in isolation.

The whole test driven development methodology is based on writing the unit tests firsts and then writing the code to fulfill those tests. TDD is BS, of course.

This is highly subjective. Pure TDD (when you write tests first before writing any code) is questionable, but can be useful sometimes, and I certainly would not call it BS.

This certainly is a controversial approach. "Questionable" is about right, but "subjective"? Not sure there should be anything much subjective about it.
"BS" is a bit strong, sure, something I could have said myself.

One problem with TDD is that it promotes bottom-up development almost exclusively, because 1/ writing tests for small units is easier than for larger ones, and 2/ if writing tests for larger units first, then people would have to do more work, like mocking up the smaller bits, etc. So in practice, that's almost 100% bottom-up.

Which sucks. Bottom-up can be done concurrently to top-down for some parts of the development, but should not be used exclusively IMHO. It leads to horrible piles of code stitched together with almost no architecture.

So, here is why I think TDD is a mess. Not that thinking of testing before coding is not a valid idea. But it just doesn't work, because we are lazy (/have no time/have no budget to do better.)

TDD would be more interesting in a top-down approach as I mentioned above. That would mean designing larger blocks first, then the required tests, then coding - which essentially means having to mock up a lot of stuff at the beginning, and refining as development progresses. That's a lot of work. So people rarely do that. They just use whatever method is trendy and will look cool on their resume.

My opinion is that TDD is good for minor variations on what has been done before, and elaboration of an existing skeleton. That's pretty similar to your statements.

TDD has some major blind spots...
 
Obviously it is dependent on the quality of the specifications and tests, and they can easily be skimped.

Slightly less obviously, some thing simply cannot be tested. One classic case is demonstrating database ACID properties. Another is graceful degradation in the presence of random failures. Yet another is worst case latencies.

Nonetheless, used wisely TDD can help avoid problems.

[SiliconWizard]

Obviously it is dependent on the quality of the specifications and tests, and they can easily be skimped.

That's one important point I didn't mention. But it goes beyond TDD anyway - whether you start by writing tests, or code directly, specifications are what you're going to base development on.

And it has been shown again and again that - at least for teams of reasonable competency - the main source of software flaws was incorrect or incomplete specifications, well above implementation errors.

[pcbcrew]

In automobile industry, TDD is commonly used.
(This might be like off-topic of this thread, as I am considering the hardware development, not embedded software only)

Differences between software TDD and hardware TDD are:

1. Unlike in software where unit testing is a big portion, the TDD I experienced in automobile industry was something like component-testing and BDD. It may be a problem of how we define a "unit" in systems. But in my view, each component was too big and complex to call it as a "unit". A electrical and control system of a car consists of many components. The system designers write test specifications. Actual hardware and testing instruments are designed and manufactured by different suppliers. Both DUT and Testing equipment are mixture of hardware and software, that are, components.

2. TDD in software uses common libs and tools. For example, PyTest in Python, JUnit in Java, Mocha in JS, etc. But for the automobile TDD, we had to design and deliver a new testing instruements for the given specification. It differs per company, per model, per DUT (device under test)

3. TDD in software is mainly done during development process. And some during continuous delivery process. But TDD in automobile industry is both for development and production. Engineers use specialized testing equipments during development, workers also use it in production line to test each care before release.

As my hardware TDD experience is limited to a automobile testing equipment provider (ART-Logics - http://www.art-logics.com), these are some key differences I can tell now.
I think medical, defence, aerospace industries will apply similar processes that we can consider as TDD in hardware development.

[NorthGuy]

If you really do something with hardware, it's hardly a good idea to try to do it on PC.

You need to observe how your hardware works. Visually, by effects in the real world, or by analyzing signals with a scope. In theory, you can simulate all that, but using real hardware seems much easier to me.

Also, when you test a unit, it doesn't guarantee that the code will work when put together with other units - interrupts may clash, timing may be different, data may not arrive when needed.

Therefore, the flow is different for hardware. I design everything as a whole, making sure that all the requirements can be met, mostly timing, then start from empty program, to the blinking led, and so on, in small increments, testing what I wrote frequently.

[BadeBhaiya]

Many excellent points in this thread. I would just like to elaborate on a topic which may be valuable, one which I learned far too late.

The term "unit testing" is used a lot, but always remember that the "unit" part comes first. If the code is not properly partitioned, testing can be a massive chore, and something that is dreaded.

To avoid this, compartmentalise the code as soon as possible. No matter if you're starting a project from ground up, or taking over someone's old spaghetti code (the latter may be difficult, you may need to convince multiple team members but it is worth it). Make everything into a logical unit, and try to have as many units as possible (within reason, of course).

The sign of good modular code is, say if the code needs to be ported over to another hardware, the process only requires changing a few functions/HAL bindings in a single file. ie very little code that is hardware dependant. Abstract the code to the extent where anything above the drivers doesn't really care what hardware you're running on. Business logic, config etc should be just that, logic, stuff that should be able to compile for any architecture in the world

[tggzzz]

Many excellent points in this thread. I would just like to elaborate on a topic which may be valuable, one which I learned far too late.

The term "unit testing" is used a lot, but always remember that the "unit" part comes first. If the code is not properly partitioned, testing can be a massive chore, and something that is dreaded.

To avoid this, compartmentalise the code as soon as possible. No matter if you're starting a project from ground up, or taking over someone's old spaghetti code (the latter may be difficult, you may need to convince multiple team members but it is worth it). Make everything into a logical unit, and try to have as many units as possible (within reason, of course).

The sign of good modular code is, say if the code needs to be ported over to another hardware, the process only requires changing a few functions/HAL bindings in a single file. ie very little code that is hardware dependant. Abstract the code to the extent where anything above the drivers doesn't really care what hardware you're running on. Business logic, config etc should be just that, logic, stuff that should be able to compile for any architecture in the world

Agreed.

And to emphasise...

The "unit" can be any size up to the complete application, not just a couple of lines of code.

Event driven applications, especially FSMs, can be tested and explored on a PC using all the debugging facilities and techniques available. The events and actions associated with hardware are simulated, or "mocked" in TDD parlance. That's the benefit of good HAL partitioning and interface.

[Nominal Animal]

Simulations are most useful during the design phase, less so during the implementation phase.

That is, I do not use simulations or mockups as an implementation tool, but as a design tool – and experimental test setup, when I have multiple possible approaches and I want to find out the effects of selecting one versus another.

The exception is algorithms.  For example an FFT routine, or frequency responses of FIR or IIR filters, or anything math-intensive, like sensor fusion.  For these, I see no real difference between simulation and an unit test.  I very, very often experiment with various algorithms, with usually a naïve implementation, plus one or more "optimized"/"tricky" implementations, for example using C implementation on x86-64 for Cortex-M7 VFP4 SIMD (two 16 bit integers in each 32 bit register), often looking at what Clang and GCC actually generate for the target ARM architecture (see ACLE and #include <acle.h>), and "simulating" the instructions, even down to theoretical cycle counts (not taking things like cache effects into account, so theoretical minimum cycle counts or theoretical maximum performace).  Mostly, I'm worried about correctness and edge cases, rather than maximum performance, though; performance really only when the question is "can this hardware do this".

[tggzzz]

I will simulate events during design and implementation and maintenance of high-level FSMs. Sometimes they are quick and dirty poke it and see tests, sometimes more formal and repeatable unit tests.

Thus, if implementing a lung ventilator, I will verify if the FSM behaves as expected when there is and isn't an inhalation of sufficient volume with appropriate timing. For example, if the patient is assumed to be capable and is being passively monitored, but doesn't inhale, then the FSM should revert to a more pessimistic mode where the patient is actively helped to inspire (so they don't expire ).

Depending on the complexity of creating an event such as "start of inhalation" or "volume of last breath", I may or may not simulate hardware inputs.

[Nominal Animal]

I will simulate events during design and implementation and maintenance of high-level FSMs.

Yup, I classify FSM's in the same bin I do algorithms, and do simulate those as well; especially if I want to check for end-states (where the FSM would get "stuck") and state chains that are not accessible, or are accessible in undesired situations.  Granted, I use FSMs mostly for parsing structured data and for user interfaces.  Again, I consider these simulations closer to unit tests, as I'm usually worrying about correctness here.

I wonder if I'm the only one who mumbles and waves their fingers in the air when working out particularly complex FSMs.  I do like to use pen and paper sketches as a swap file, too; those "notes" become gibberish as soon as I move on, even for myself: I literally sketch something to represent a detail whose inner complexity I cannot hold in my mind while working on higher-level structure...  I also like to walk in circles (increases blood flow) when thinking hard.  These make me utterly incompatible with cubicle farms

[dobsonr741]

Back to the original question:

I probably still need a separate build system for tests?

No, the same build system can do. This guys below shows a great starter example with cmocka and cmake. I adopted it to one of my projects, the test build takes up a sub second and breaks the build if the test is failing.

https://admantium.com/blog/pico03_testing_with_cmocka/

[tggzzz]

I will simulate events during design and implementation and maintenance of high-level FSMs.

Yup, I classify FSM's in the same bin I do algorithms, and do simulate those as well; especially if I want to check for end-states (where the FSM would get "stuck") and state chains that are not accessible, or are accessible in undesired situations. 

There has been a lot of academic work on FSMs, attempting to formally prove/disprove useful properties such as being deadlock and livelock free. A useful amount of progress has been made, but there are two stumbling blocks:

• complexity and the combinatorial explosion, particularly if the FSM is the fixed starting point. It isn't clear to me to what extent refactoring an FSM can help
• notation, and ensuring that different representations (e.g. a digraph and code) are equivalent. That's a common problem with formal methods

Granted, I use FSMs mostly for parsing structured data and for user interfaces.  Again, I consider these simulations closer to unit tests, as I'm usually worrying about correctness here.

Ugh. Run away. Compilers and anything related have negative interest for me!

The FSMs that interest me represent "real-world" behaviour, frequently multiple cooperating FSMs with different FSM implemented by different parties.

I wonder if I'm the only one who mumbles and waves their fingers in the air when working out particularly complex FSMs.  I do like to use pen and paper sketches as a swap file, too; those "notes" become gibberish as soon as I move on, even for myself: I literally sketch something to represent a detail whose inner complexity I cannot hold in my mind while working on higher-level structure...  I also like to walk in circles (increases blood flow) when thinking hard.  These make me utterly incompatible with cubicle farms

FSMs seem to be begging to be expressed as hierarchical diagrams. Harel Statecharts are a good start, but that "language" has always struck me as having unbeneficial complexity.

[Nominal Animal]

There has been a lot of academic work on FSMs, attempting to formally prove/disprove useful properties such as being deadlock and livelock free.

Yes, I've seen some interesting papers at ACM, but my own real-world experience is from quite a narrow field of applications.  Heck, the most common FSMs I use are POSIX regular expressions.

Granted, I use FSMs mostly for parsing structured data and for user interfaces.  Again, I consider these simulations closer to unit tests, as I'm usually worrying about correctness here.

Ugh. Run away. Compilers and anything related have negative interest for me!

Ha! Not compilers, not even domain-specific languages, but more like large amounts of configuration data, and easy (test and experiment) microcontroller USB Serial text interfaces.

FSMs seem to be begging to be expressed as hierarchical diagrams. Harel Statecharts are a good start, but that "language" has always struck me as having unbeneficial complexity.

I like having a general understanding first, and a bit of an advance warning about where the complicated bits are.  So, I tend to think about abstract models a lot.  I often end up pushing against my own limits, and have found the behaviours I described helps a bit.  Walking in particular, and linearizing (by conversion into a human-language description) help a lot.  Proper charting and structural definitions comes later. 

I'm one of those people who can spend a day walking in a circle, mumbling to myself, and not write a single line of code; and the next day, after reviewing my notes and having my subconscious work on/reorganize it while I sleep, write the first implementation in one go.  Because I usually end up learning/noticing useful/important details, I typically end up rewriting the implementation to fit my needs better later on.

I do recommend developers find out how their own mind works, so they can use it to their advantage.  I cannot even count the number of times I've woken up with a better solution in mind than the one I worked on the previous day, simply because my subconscious reorganizes things when I sleep.
And not to be ashamed of things like keeping documentation constantly at hand: I always have a browser or terminal window open for checking details at man pages.  Some colleagues used to laugh at me initially, but they soon found out that instead of trying to memorize unimportant details like the order of arguments to memcpy(), verifying such details and becoming efficient at that, yields much better code with a lot less bugs.  They ended up adopting the exact same habit.  My own memory is limited, so I reserve it for understanding the principles, approaches, limitations and so on, and don't bother memorizing API details at all.

[wek]

I cannot even count the number of times I've woken up with a better solution in mind than the one I worked on the previous day, simply because my subconscious reorganizes things when I sleep.

I don't think it's the subconscious, IMO you quite consciously work the problem, but the sleep-induced interruption allows you to abandon the safe paths your mind kept walking during the previous day and approach it from a surprising angle.

I found that those better solutions come to me while driving home from work (usually in the night, so there's not much traffic; the interruption is provided by the procedure of leaving workplace and getting car to road, plus the initial cold).

But problems requiring such interruptions, telling to the teddybear, the staring-to-the-code/walking cycles, are mostly the non-"unit-testable" or -splittable problems.

JW

[Nominal Animal]

I cannot even count the number of times I've woken up with a better solution in mind than the one I worked on the previous day, simply because my subconscious reorganizes things when I sleep.

I don't think it's the subconscious, IMO you quite consciously work the problem, but the sleep-induced interruption allows you to abandon the safe paths your mind kept walking during the previous day and approach it from a surprising angle.

I do believe it is my "subconscious", specifically related to what happens in REM sleep, where the mind organizes the memories; does the proverbial "housekeeping and cleaning" tasks.  The discovery is then the effect of the mind removing the unnecessary clutter, if you will, with my conscious mind discovering the now easier to see solution.

While it sounds very similar, I do consider it separate from what you described, because they feel different to me.  I do take regular pauses (especially hot tea or coffee, maybe a chat over one about something else with colleagues), and that does help switch to a different viewpoint, and it sometimes does reveal a better solution.  However, I do find it relatively easy to switch between viewpoints, even opposite ones.

Of course, the two could be the exact same mechanism, and my own brain just slower (in the "more meticulous" sense, not in the "firing of synapses" sense) than typical, so that it often takes a pause of several hours for me to get there.

But problems requiring such interruptions, telling to the teddybear, the staring-to-the-code/walking cycles, are mostly the non-"unit-testable" or -splittable problems.

Definitely.

For me, most commonly in the stage where I have a problem but haven't decided on a solution yet, and need to consider the pros and cons of the various approaches.  So many facets and details to consider, to make a realistic, informed choice...

[paulca]

The hardest part about unit testing is writing testable code.

The amount of effort you would need to go to, to write testable code in MCU world would probably make a "wide coverage" impractical.

Note.  I say "testable code" with specific meaning, which is what unit testing is about.  It says nothing, absolutely nothing about whether the code 'functions' as per requirements.  The idea is just to test the code does what the developer intended it to do.

When you come to testing functionality, there is hardware involved, so it becomes an integration test.  When everything is connected up together that is a system test.  When real users are using it, it's a user acceptance test.  Then alpha, beta, gamma etc. etc. etc.  ad.inf.

Trying to pick a suitable MCU world example... say DMA'ing UART buffers for Tx and Rx.  You do some detection, pattern matching and formulate some response and send it.

In most MCU projects people just start writing hardware UART code and then the "business logic" just grows like moss around it.  Completely un-unittestable.  At best you can integration test bits of it with a pyschical test harness maybe, but you are more likely to just skip all testing all the way through to full system test.  This is fine until you have 2 or more engineers working on the thing, or even one developer with multiple change lists open.  Then it ruins you day and your week.

In the Big Iron world, assuming we have no internet connection to go download a framework that does this already, already fully tested and in a full RELEASE stable state...  assuming we don't have the memory either. 

I/We would start by splitting it into layers. 
* Code which does the analysis, pattern matching and produces the correct responses - this code knows nothing about buffering or UARTs.
* Code which manages buffers.  Provides them, cleans them, queues them, blah, blah.
* Code which sends and receives buffers via UART.

Not only does these 3 layers allow you to test the top two in "code only test harness", aka unit test, but you can swap any layer component for any other which complies to the same contract (header file).

The bottom one can also be tested much lower level.  You just need to feed it buffers, so a proper integration test of the UART hardware access layer is possible.

The "isolation of concern" applies not just to the code, but the programmer and which component he is working in.   As long as it meets it's contract it is said to be good.  A unit test that proves it meets it's contract makes it formally tested to do so.  You are only testing that one bit of code, not the whole application.  If the whole appliction is broken, verifying if a component is function correctly in isolation is very advantagous.

You do have to keep remembering where you are though, what memory footprint you can afford, how much of the extra reference hops and some unnecessary protections cost in terms of cycles where that is important.

The biggest, by far, advantage of unit tests, which is so often forgotten is not that they prove the code works.  It's that in 6 months time, when someone just did a 3 week long refactoring... that the code STILL works.

Analogy with electronics....   your "units" are ICs.  Except that you can make your own.  So while scattering a bunch of passives and discretes all over the PCB might be required in RF stuff, most people would prefer to get an IC which encapsulates that functionality.  If you were in a lab capable of making fab'd prototype ICs, you could see the appeal of pushing functionality into ICs where they can be tested in a repeatable way and then manufactured in a pre-tested, quality form and re-use in other prjects.  Your ICs are your units and the test hardness for them the unit tests.  Having those ICs run with the minimum amount of external support is the art of writing unit testable code.

[paulca]

In brass tac's "How to?"

Code: [Select]

#ifdef TEST
#include "test/testsuite1.h"
//#include "test/testsuite2.h"
test();
#endif


[NorthGuy]

I do believe it is my "subconscious", specifically related to what happens in REM sleep, where the mind organizes the memories; does the proverbial "housekeeping and cleaning" tasks.  The discovery is then the effect of the mind removing the unnecessary clutter, if you will, with my conscious mind discovering the now easier to see solution.

When you think about something often, sort of a buzz is going in the brain returning back to the problem even when you do other task. The same may go on during the night perhaps giving you bad dreams. Thinking the same sequence of thoughts over and over again fortifies neural pathways, makes them much faster and more energy efficient. After certain time, often in the morning, you feel how your new fast end efficient pathways simplify the thinking - what was difficult suddenly becomes easy, what was hard becomes enjoyable, what was unclear and foggy becomes obvious, you now understand the solution better and can now easily implement it and improve on it.

But this takes time. If you start implementing the idea prematurely, your neural pathways will be weak, the implementation will be harder, will take longer, and you will make more mistakes. So, by giving your brain time to fortify the required neural pathways, you may actually save time.

On the other hand, when you already have fast neural pathways, it becomes difficult to give up your idea. Thinking outside the box requires using different pathways which are not as fast and not as efficient, therefore your brain tries hard to push your thinking back into the area of well-developed pathways where thinking is fast and effortless. It takes enormous conscious effort to go against this and wonder into a foggy area of undeveloped neural pathways.

Thus, before you start thinking about the solution and develop the neural pathways necessary to implement it, there must be a quiet time, when you observe the problem, collect information without attempting to make any decisions. That's the time where you can easily pick up any road, any solution, without giving up much. Clean slate where you don't need to jump to conclusions. But even then, the dominance of pathways you have developed solving similar problems before (your experience), will try to force you to "rather bear those ills we have than fly to others that we know not of".

[tggzzz]

The hardest part about unit testing is writing testable code.

The amount of effort you would need to go to, to write testable code in MCU world would probably make a "wide coverage" impractical.

IMNSHO the hardest part about writing testable code is getting a suitably well defined specification.

It is practical and useful to write (unit) testable code in the MCU world. I noted one technique above
https://www.eevblog.com/forum/microcontrollers/unit-testing-embedded-software/msg4718465/#msg4718465 and
https://www.eevblog.com/forum/microcontrollers/unit-testing-embedded-software/msg4718762/#msg4718762
so the "big iron" technique you mention below is possible and useful.

Note.  I say "testable code" with specific meaning, which is what unit testing is about.  It says nothing, absolutely nothing about whether the code 'functions' as per requirements.  The idea is just to test the code does what the developer intended it to do.

When you come to testing functionality, there is hardware involved, so it becomes an integration test.  When everything is connected up together that is a system test.  When real users are using it, it's a user acceptance test.  Then alpha, beta, gamma etc. etc. etc.  ad.inf.

Trying to pick a suitable MCU world example... say DMA'ing UART buffers for Tx and Rx.  You do some detection, pattern matching and formulate some response and send it.

In most MCU projects people just start writing hardware UART code and then the "business logic" just grows like moss around it.  Completely un-unittestable.  At best you can integration test bits of it with a pyschical test harness maybe, but you are more likely to just skip all testing all the way through to full system test.  This is fine until you have 2 or more engineers working on the thing, or even one developer with multiple change lists open.  Then it ruins you day and your week.

In the Big Iron world, assuming we have no internet connection to go download a framework that does this already, already fully tested and in a full RELEASE stable state...  assuming we don't have the memory either. 

I/We would start by splitting it into layers. 
* Code which does the analysis, pattern matching and produces the correct responses - this code knows nothing about buffering or UARTs.
* Code which manages buffers.  Provides them, cleans them, queues them, blah, blah.
* Code which sends and receives buffers via UART.

Not only does these 3 layers allow you to test the top two in "code only test harness", aka unit test, but you can swap any layer component for any other which complies to the same contract (header file).

Even those top two layers are almost certainly integration tests, i.e. integration of your code with any library code written by other people. Yes, the distinction between "unit" and "integration" test is arbitrary and not particularly useful.

What is useful is to define the unit. In the cases above the unit would be "my code + libraries".

The bottom one can also be tested much lower level.  You just need to feed it buffers, so a proper integration test of the UART hardware access layer is possible.

The "isolation of concern" applies not just to the code, but the programmer and which component he is working in.   As long as it meets it's contract it is said to be good.  A unit test that proves it meets it's contract makes it formally tested to do so.  You are only testing that one bit of code, not the whole application.  If the whole appliction is broken, verifying if a component is function correctly in isolation is very advantagous.

You do have to keep remembering where you are though, what memory footprint you can afford, how much of the extra reference hops and some unnecessary protections cost in terms of cycles where that is important.

The biggest, by far, advantage of unit tests, which is so often forgotten is not that they prove the code works.  It's that in 6 months time, when someone just did a 3 week long refactoring... that the code STILL works.

Strictly speaking it demonstrates that the refactoring hasn't changed the operation in any way detectable by the (unit) tests. That emphasises that the quality of the (unit) tests is vital.

Analogy with electronics....   your "units" are ICs.  Except that you can make your own.  So while scattering a bunch of passives and discretes all over the PCB might be required in RF stuff, most people would prefer to get an IC which encapsulates that functionality.  If you were in a lab capable of making fab'd prototype ICs, you could see the appeal of pushing functionality into ICs where they can be tested in a repeatable way and then manufactured in a pre-tested, quality form and re-use in other prjects.  Your ICs are your units and the test hardness for them the unit tests.  Having those ICs run with the minimum amount of external support is the art of writing unit testable code.

Analogies are dangerous; they often generate more heat than illumination.

Most people can fab their own prototype and production ICs, and have been able to do so for 30 or 40 years (FPGAs and PALs respectively).

For unit testing purposes, it is accepted practice to regard all of these as units: discrete semiconductors and passives, ICs, populated PCBs, modules (e.g. motherboard containing RAM), racks containing multiple PCBs, subsystem containing (say) one or more racks, etc.

[cantata.tech]

ok peeps, I have a real project..

How do I do it?

I have an uC running under PlatformIO. It's an Arduino/ESP32 hardware in C++.

It's reading a waveform on some pins and the requirement is to react to that digital waveform when it sees it.

Back when I was young we never had TDD in my neck of the woods. Not enough memory and other excuses.

Here's a description of the scenario. A pin on the uC is low, it's sitting there waiting to go high. When it goes high the next routine needs to read that there was a waveform detected and go do something else on another pin.

It's sounds very simple and the code to write it without TDD i could write with my eyes closed. Doing TDD with Hardware wasn't something I spent a lot of time doing.

How do I write a Unit test for that?

I have Pulseview and I can see my signal on the line. It's not a signal with a library and in my case I don't want to use a library - let's say there's no library. I want to rough my way in and if i see the line go high, fire off a code event which sends a signal somewhere else.

pseudo code is fine but c++ is the language being used.

How do I mock a wire/gpio going high and my code being fired?

[ataradov]

How do I write a Unit test for that?

Short answer - you don't.

This is exactly the scenario where unit testing makes no sense. If there is some non-trivial algorithmic code being executed when GPIO interrupt fires, then that is the code you test assuming that GPIO interrupt happened at the right time.

What you are describing sounds like there is trivial amount of processing done. There is no point in unit testing that, you will waste a lot more time because this is not where errors in the code like this come from. They come from integration and running on the real hardware asynchronously.

The way to test things like this is to do functional testing using real hardware. You will need a test setup that  can generate external inputs on command from the PC and read back the state of the firmware. Then PC runs a series of tests where it generates valid and invalid sequences and then looks for the reported results.

This is where you get the most out of your tests. You can leave it running for days just generating random noise on the pins and checking if something crashed or misbehaved.

[paulca]

The pin change is the trigger.

What you do about it is the event handler.

Test the event handler. 

Just create a function called "test_eventHandler()" and exercise it in code without needing the GPIO stuff.

In case it's not obvious.  Elsewhere, poll/interrupt on the GPIO states, when you see what you want, call the event handler you previously tested.

[cantata.tech]

This is exactly the scenario where unit testing makes no sense.

Well it's part of a bigger system that will get much more complicated later.

What you are describing sounds like there is trivial amount of processing done. There is no point in unit testing that, you will waste a lot more time because this is not where errors in the code like this come from. They come from integration and running on the real hardware asynchronously.

I wouldn't say that.

It's audio gear btw and I'm a lazy software engineer who wants to run code in the night without the neighbours or wife complaining. I have to limit time on the real equipment.

The way to test things like this is to do functional testing using real hardware. You will need a test setup that  can generate external inputs on command from the PC and read back the state of the firmware. Then PC runs a series of tests where it generates valid and invalid sequences and then looks for the reported results.

This is where you get the most out of your tests. You can leave it running for days just generating random noise on the pins and checking if something crashed or misbehaved.

It's audio equipment with front panel buttons. I don't have anything to simulate pressing the front panel buttons. Plus it makes noise - it's an untuned radio station.

What I'm actually doing is capturing a 'Mode' change on a radio, going between AM to FM, capturing the bus data and then running a different handler downstream. Whilst I do have some datasheets for the radio the level of detail is not great. Also, there are fields in the data stream that I know that are there but I just don't wish to decode them.

I was reading on the net between posts that somebody else uses a spare GPIO pin on the dev board to generate the input signal as an output and then wire pin to the desired input pin. That is workable for me. From the commandline, I could just toggle whether I wanted the simulator or the real input running.

Thanks for the answer 

[ataradov]

Well it's part of a bigger system that will get much more complicated later.

But testing trivial units on their own will not improve overall system that much.

Unit test make sense if there is a piece of code that is hardware-independent and has fixed inputs and outputs. A good example would be if you have an ADC reading a buffer of data and then some DSP algorithm runs on this buffer. In this case you would unit test just the algorithm. You can generate test buffers and expected outputs. You don't simulate the ADC itself.

It's audio equipment with front panel buttons. I don't have anything to simulate pressing the front panel buttons. Plus it makes noise.

Well, this is what you have to work on. Either firmware must have alternative control method to the front panel (take commands form UART), or you need a rig to press the buttons, or if you are really serious about your testing, the buttons should be exposed on some test connector so that they could be "pressed" by the test software.

It is hard to tell without looking into the details of the exact equipment, but unit tests for things with user input like this are rarely useful.

[paulca]

It's audio equipment with front panel buttons. I don't have anything to simulate pressing the front panel buttons. Plus it makes noise - it's an untuned radio station.

Again.  The button event is hardware/gpio.  How that event is handled, is unit testable.

So you don't need to push buttons to test IT.

You will need button pushers to test the hardware handler.

I would say there is enough there to consider that approach.

EDIT:  Consider, as I am, right now, that SPI, I2C and UART are all options for MCU<->MCU communications.  My application wants to send stats one way and config the other.  The actually required data protocol is independent of the transmission protocol.   Save for "ended-ness" of the events.  It makes sense to separate the code along that line.  Code which is called from the hardware event handlers, whatever they are.  You have the application using the abstraction and the abstraction being used by the hardware.  The bit that remains mostly constant is how that particular event is handled in your application.

It should come down to functions which do not take, nor reference hardware at all.

HardwareISR -> HAL Callback -> Application code -> HAL function -> Hardware calls

The "application code" is unit testable.  As long as you call it's "API" correctly, it should not require access to hardware and still produce adequate results.

[paulca]

Flipping my hobby hats...

What MCU and realtime developer do do (hehe) that big iron people tend not to, is fail fast, fail early, fail hard.

The technique is often referred to as "Assertion".   Analogy.  Having to press the clutch to start a car.  It simply won't let you do it without pressing it... in some cars.  It's to prevent you from starting the car in gear.  In many circumstances its in neutral and it still won't let you.  An assertion.  Not an error, not a warning, not a negociable event, an assertion.  Shall not proceed.  Low level is FAR more full of them.

You will find yourself in many, "What if?" scenarios, if you program defensively.  Checking limits, boundaries etc.  A lot of them will be a case of "Shouldn't ever happen".   Don't ignore these.  Force them into the error handling routine.  Within there...  halt everything.  Capture all the information you can.  Got while(1).  Do not tolerate sloppiness.  If an error is an error it's an error.  Treat it as such.  Fail immediately.  It will force in in development to simply not tolerate shit code.

Or your team.

We can't do that in big iron.  We tend to play a game of "hot potato" and throw the exception up and up and up until either something can do something about it, or the whole transaction gets aborted back at source.  Many layers of Not my problem and sorry can't help.  In this scenario, 24/7 running enterprise applications, serving millions of requests a second, one can simply not have a single worker thread "assert" and abort the whole virtual machine.  I have seen people put asserts in their code like this, in enterprise in big iron.... I want to slap them.  "Thou shall not abort the VM from a library!  You are servant, not master, you shall NOT control my runtime!"

[cantata.tech]

But testing trivial units on their own will not improve overall system that much.

Unit test make sense if there is a piece of code that is hardware-independent and has fixed inputs and outputs. A good example would be if you have an ADC reading a buffer of data and then some DSP algorithm runs on this buffer. In this case you would unit test just the algorithm. You can generate test buffers and expected outputs. You don't simulate the ADC itself.

In this case I'm trying to replace an existing module with a new module. So the exact task is to simulate an old piece of hardware with a new piece of hardware going through a convertor.

One example is a radio module, replacing an AM module with a DAB+ or Internet Radio.

A second example is replacing an MD (minidisc) with a USB or SD drive.

It is hard to tell without looking into the details of the exact equipment,

That will take too long. My setup is not entirely dissimilar to this --> http://gizmosnack.blogspot.com/2015/11/aux-in-volvo-hu-xxxx-radio.html

Also, the code that's shown down the page is certainly indicative of what I have.

[cantata.tech]

You will need button pushers to test the hardware handler.

As strange as it may seem, I don't. The mcu on the radio decodes the keypress and sends a decoded 'instruction' downstream to a device as a command to instruct it to do something.

Think of the system as running on a CAN/LIN or I2C bus although they use their own bus system.

EDIT:  Consider, as I am, right now, that SPI, I2C and UART are all options for MCU<->MCU communications. 

These units have their own proprietary bus, such as Melbus. So those won't work.

I want the TDD to help me over time so that I can run regressive tests.

[nctnico]

It is hard to tell without looking into the details of the exact equipment, but unit tests for things with user input like this are rarely useful.

IMHO there is a blurry line between unit testing & verification testing. In the past I have been simulating user input by using relays instead of buttons (this was for a vending machine-ish application). See if a million button pushes also result in a million transactions recorded by the system.

[paulca]

I suppose, as we already knew, we have to be careful of being "training mules" for the AI rather than responding to trolls.

[John Coloccia]

I have seen people put asserts in their code like this, in enterprise in big iron.... I want to slap them.  "Thou shall not abort the VM from a library!  You are servant, not master, you shall NOT control my runtime!"

I can't even begin to tell you how annoyed I get at libraries/APIs that throw instead of returning an error.