Author Topic: Unit testing embedded software  (Read 4706 times)

0 Members and 1 Guest are viewing this topic.

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19281
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Unit testing embedded software
« Reply #25 on: February 23, 2023, 03:35:34 pm »
I will simulate events during design and implementation and maintenance of high-level FSMs.
Yup, I classify FSM's in the same bin I do algorithms, and do simulate those as well; especially if I want to check for end-states (where the FSM would get "stuck") and state chains that are not accessible, or are accessible in undesired situations. 

There has been a lot of academic work on FSMs, attempting to formally prove/disprove useful properties such as being deadlock and livelock free. A useful amount of progress has been made, but there are two stumbling blocks:
  • complexity and the combinatorial explosion, particularly if the FSM is the fixed starting point. It isn't clear to me to what extent refactoring an FSM can help
  • notation, and ensuring that different representations (e.g. a digraph and code) are equivalent. That's a common problem with formal methods

Quote
Granted, I use FSMs mostly for parsing structured data and for user interfaces.  Again, I consider these simulations closer to unit tests, as I'm usually worrying about correctness here.

Ugh. Run away. Compilers and anything related have negative interest for me!

The FSMs that interest me represent "real-world" behaviour, frequently multiple cooperating FSMs with different FSM implemented by different parties.

Quote
I wonder if I'm the only one who mumbles and waves their fingers in the air when working out particularly complex FSMs.  I do like to use pen and paper sketches as a swap file, too; those "notes" become gibberish as soon as I move on, even for myself: I literally sketch something to represent a detail whose inner complexity I cannot hold in my mind while working on higher-level structure...  I also like to walk in circles (increases blood flow) when thinking hard.  These make me utterly incompatible with cubicle farms ;D

FSMs seem to be begging to be expressed as hierarchical diagrams. Harel Statecharts are a good start, but that "language" has always struck me as having unbeneficial complexity.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6173
  • Country: fi
    • My home page and email address
Re: Unit testing embedded software
« Reply #26 on: February 23, 2023, 04:06:33 pm »
There has been a lot of academic work on FSMs, attempting to formally prove/disprove useful properties such as being deadlock and livelock free.
Yes, I've seen some interesting papers at ACM, but my own real-world experience is from quite a narrow field of applications.  Heck, the most common FSMs I use are POSIX regular expressions.

Quote
Granted, I use FSMs mostly for parsing structured data and for user interfaces.  Again, I consider these simulations closer to unit tests, as I'm usually worrying about correctness here.
Ugh. Run away. Compilers and anything related have negative interest for me!
Ha! Not compilers, not even domain-specific languages, but more like large amounts of configuration data, and easy (test and experiment) microcontroller USB Serial text interfaces.

FSMs seem to be begging to be expressed as hierarchical diagrams. Harel Statecharts are a good start, but that "language" has always struck me as having unbeneficial complexity.
I like having a general understanding first, and a bit of an advance warning about where the complicated bits are.  So, I tend to think about abstract models a lot.  I often end up pushing against my own limits, and have found the behaviours I described helps a bit.  Walking in particular, and linearizing (by conversion into a human-language description) help a lot.  Proper charting and structural definitions comes later. 

I'm one of those people who can spend a day walking in a circle, mumbling to myself, and not write a single line of code; and the next day, after reviewing my notes and having my subconscious work on/reorganize it while I sleep, write the first implementation in one go.  Because I usually end up learning/noticing useful/important details, I typically end up rewriting the implementation to fit my needs better later on.

I do recommend developers find out how their own mind works, so they can use it to their advantage.  I cannot even count the number of times I've woken up with a better solution in mind than the one I worked on the previous day, simply because my subconscious reorganizes things when I sleep.
And not to be ashamed of things like keeping documentation constantly at hand: I always have a browser or terminal window open for checking details at man pages.  Some colleagues used to laugh at me initially, but they soon found out that instead of trying to memorize unimportant details like the order of arguments to memcpy(), verifying such details and becoming efficient at that, yields much better code with a lot less bugs.  They ended up adopting the exact same habit.  My own memory is limited, so I reserve it for understanding the principles, approaches, limitations and so on, and don't bother memorizing API details at all.
 
The following users thanked this post: nctnico

Offline wek

  • Frequent Contributor
  • **
  • Posts: 486
  • Country: sk
Re: Unit testing embedded software
« Reply #27 on: February 23, 2023, 11:29:01 pm »
Quote from: Nominal Animal
I cannot even count the number of times I've woken up with a better solution in mind than the one I worked on the previous day, simply because my subconscious reorganizes things when I sleep.

I don't think it's the subconscious, IMO you quite consciously work the problem, but the sleep-induced interruption allows you to abandon the safe paths your mind kept walking during the previous day and approach it from a surprising angle.

I found that those better solutions come to me while driving home from work (usually in the night, so there's not much traffic; the interruption is provided by the procedure of leaving workplace and getting car to road, plus the initial cold).

But problems requiring such interruptions, telling to the teddybear, the staring-to-the-code/walking cycles, are mostly the non-"unit-testable" or -splittable problems.

JW
 

Offline Nominal Animal

  • Super Contributor
  • ***
  • Posts: 6173
  • Country: fi
    • My home page and email address
Re: Unit testing embedded software
« Reply #28 on: February 24, 2023, 09:30:55 am »
Quote from: Nominal Animal
I cannot even count the number of times I've woken up with a better solution in mind than the one I worked on the previous day, simply because my subconscious reorganizes things when I sleep.

I don't think it's the subconscious, IMO you quite consciously work the problem, but the sleep-induced interruption allows you to abandon the safe paths your mind kept walking during the previous day and approach it from a surprising angle.

I do believe it is my "subconscious", specifically related to what happens in REM sleep, where the mind organizes the memories; does the proverbial "housekeeping and cleaning" tasks.  The discovery is then the effect of the mind removing the unnecessary clutter, if you will, with my conscious mind discovering the now easier to see solution.

While it sounds very similar, I do consider it separate from what you described, because they feel different to me.  I do take regular pauses (especially hot tea or coffee, maybe a chat over one about something else with colleagues), and that does help switch to a different viewpoint, and it sometimes does reveal a better solution.  However, I do find it relatively easy to switch between viewpoints, even opposite ones.

Of course, the two could be the exact same mechanism, and my own brain just slower (in the "more meticulous" sense, not in the "firing of synapses" sense) than typical, so that it often takes a pause of several hours for me to get there. :-//

But problems requiring such interruptions, telling to the teddybear, the staring-to-the-code/walking cycles, are mostly the non-"unit-testable" or -splittable problems.
Definitely.

For me, most commonly in the stage where I have a problem but haven't decided on a solution yet, and need to consider the pros and cons of the various approaches.  So many facets and details to consider, to make a realistic, informed choice...
 

Offline paulca

  • Super Contributor
  • ***
  • Posts: 4003
  • Country: gb
Re: Unit testing embedded software
« Reply #29 on: February 24, 2023, 11:19:57 am »
The hardest part about unit testing is writing testable code.

The amount of effort you would need to go to, to write testable code in MCU world would probably make a "wide coverage" impractical.

Note.  I say "testable code" with specific meaning, which is what unit testing is about.  It says nothing, absolutely nothing about whether the code 'functions' as per requirements.  The idea is just to test the code does what the developer intended it to do.

When you come to testing functionality, there is hardware involved, so it becomes an integration test.  When everything is connected up together that is a system test.  When real users are using it, it's a user acceptance test.  Then alpha, beta, gamma etc. etc. etc.  ad.inf.

Trying to pick a suitable MCU world example... say DMA'ing UART buffers for Tx and Rx.  You do some detection, pattern matching and formulate some response and send it.

In most MCU projects people just start writing hardware UART code and then the "business logic" just grows like moss around it.  Completely un-unittestable.  At best you can integration test bits of it with a pyschical test harness maybe, but you are more likely to just skip all testing all the way through to full system test.  This is fine until you have 2 or more engineers working on the thing, or even one developer with multiple change lists open.  Then it ruins you day and your week.

In the Big Iron world, assuming we have no internet connection to go download a framework that does this already, already fully tested and in a full RELEASE stable state...  assuming we don't have the memory either. 

I/We would start by splitting it into layers. 
* Code which does the analysis, pattern matching and produces the correct responses - this code knows nothing about buffering or UARTs.
* Code which manages buffers.  Provides them, cleans them, queues them, blah, blah.
* Code which sends and receives buffers via UART.

Not only does these 3 layers allow you to test the top two in "code only test harness", aka unit test, but you can swap any layer component for any other which complies to the same contract (header file).

The bottom one can also be tested much lower level.  You just need to feed it buffers, so a proper integration test of the UART hardware access layer is possible.

The "isolation of concern" applies not just to the code, but the programmer and which component he is working in.   As long as it meets it's contract it is said to be good.  A unit test that proves it meets it's contract makes it formally tested to do so.  You are only testing that one bit of code, not the whole application.  If the whole appliction is broken, verifying if a component is function correctly in isolation is very advantagous.

You do have to keep remembering where you are though, what memory footprint you can afford, how much of the extra reference hops and some unnecessary protections cost in terms of cycles where that is important.

The biggest, by far, advantage of unit tests, which is so often forgotten is not that they prove the code works.  It's that in 6 months time, when someone just did a 3 week long refactoring... that the code STILL works.

Analogy with electronics....   your "units" are ICs.  Except that you can make your own.  So while scattering a bunch of passives and discretes all over the PCB might be required in RF stuff, most people would prefer to get an IC which encapsulates that functionality.  If you were in a lab capable of making fab'd prototype ICs, you could see the appeal of pushing functionality into ICs where they can be tested in a repeatable way and then manufactured in a pre-tested, quality form and re-use in other prjects.  Your ICs are your units and the test hardness for them the unit tests.  Having those ICs run with the minimum amount of external support is the art of writing unit testable code.
« Last Edit: February 24, 2023, 11:32:02 am by paulca »
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline paulca

  • Super Contributor
  • ***
  • Posts: 4003
  • Country: gb
Re: Unit testing embedded software
« Reply #30 on: February 24, 2023, 11:36:39 am »
In brass tac's "How to?"

Code: [Select]
#ifdef TEST
#include "test/testsuite1.h"
//#include "test/testsuite2.h"
test();
#endif
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline NorthGuy

  • Super Contributor
  • ***
  • Posts: 3137
  • Country: ca
Re: Unit testing embedded software
« Reply #31 on: February 24, 2023, 03:54:51 pm »
I do believe it is my "subconscious", specifically related to what happens in REM sleep, where the mind organizes the memories; does the proverbial "housekeeping and cleaning" tasks.  The discovery is then the effect of the mind removing the unnecessary clutter, if you will, with my conscious mind discovering the now easier to see solution.

When you think about something often, sort of a buzz is going in the brain returning back to the problem even when you do other task. The same may go on during the night perhaps giving you bad dreams. Thinking the same sequence of thoughts over and over again fortifies neural pathways, makes them much faster and more energy efficient. After certain time, often in the morning, you feel how your new fast end efficient pathways simplify the thinking - what was difficult suddenly becomes easy, what was hard becomes enjoyable, what was unclear and foggy becomes obvious, you now understand the solution better and can now easily implement it and improve on it.

But this takes time. If you start implementing the idea prematurely, your neural pathways will be weak, the implementation will be harder, will take longer, and you will make more mistakes. So, by giving your brain time to fortify the required neural pathways, you may actually save time.

On the other hand, when you already have fast neural pathways, it becomes difficult to give up your idea. Thinking outside the box requires using different pathways which are not as fast and not as efficient, therefore your brain tries hard to push your thinking back into the area of well-developed pathways where thinking is fast and effortless. It takes enormous conscious effort to go against this and wonder into a foggy area of undeveloped neural pathways.

Thus, before you start thinking about the solution and develop the neural pathways necessary to implement it, there must be a quiet time, when you observe the problem, collect information without attempting to make any decisions. That's the time where you can easily pick up any road, any solution, without giving up much. Clean slate where you don't need to jump to conclusions. But even then, the dominance of pathways you have developed solving similar problems before (your experience), will try to force you to "rather bear those ills we have than fly to others that we know not of".
 
The following users thanked this post: Nominal Animal, eutectique

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19281
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Unit testing embedded software
« Reply #32 on: February 24, 2023, 04:12:20 pm »
The hardest part about unit testing is writing testable code.

The amount of effort you would need to go to, to write testable code in MCU world would probably make a "wide coverage" impractical.

IMNSHO the hardest part about writing testable code is getting a suitably well defined specification.

It is practical and useful to write (unit) testable code in the MCU world. I noted one technique above
https://www.eevblog.com/forum/microcontrollers/unit-testing-embedded-software/msg4718465/#msg4718465 and
https://www.eevblog.com/forum/microcontrollers/unit-testing-embedded-software/msg4718762/#msg4718762
so the "big iron" technique you mention below is possible and useful.

Quote
Note.  I say "testable code" with specific meaning, which is what unit testing is about.  It says nothing, absolutely nothing about whether the code 'functions' as per requirements.  The idea is just to test the code does what the developer intended it to do.

When you come to testing functionality, there is hardware involved, so it becomes an integration test.  When everything is connected up together that is a system test.  When real users are using it, it's a user acceptance test.  Then alpha, beta, gamma etc. etc. etc.  ad.inf.

Trying to pick a suitable MCU world example... say DMA'ing UART buffers for Tx and Rx.  You do some detection, pattern matching and formulate some response and send it.

In most MCU projects people just start writing hardware UART code and then the "business logic" just grows like moss around it.  Completely un-unittestable.  At best you can integration test bits of it with a pyschical test harness maybe, but you are more likely to just skip all testing all the way through to full system test.  This is fine until you have 2 or more engineers working on the thing, or even one developer with multiple change lists open.  Then it ruins you day and your week.

In the Big Iron world, assuming we have no internet connection to go download a framework that does this already, already fully tested and in a full RELEASE stable state...  assuming we don't have the memory either. 

I/We would start by splitting it into layers. 
* Code which does the analysis, pattern matching and produces the correct responses - this code knows nothing about buffering or UARTs.
* Code which manages buffers.  Provides them, cleans them, queues them, blah, blah.
* Code which sends and receives buffers via UART.

Not only does these 3 layers allow you to test the top two in "code only test harness", aka unit test, but you can swap any layer component for any other which complies to the same contract (header file).

Even those top two layers are almost certainly integration tests, i.e. integration of your code with any library code written by other people. Yes, the distinction between "unit" and "integration" test is arbitrary and not particularly useful.

What is useful is to define the unit. In the cases above the unit would be "my code + libraries".

Quote
The bottom one can also be tested much lower level.  You just need to feed it buffers, so a proper integration test of the UART hardware access layer is possible.

The "isolation of concern" applies not just to the code, but the programmer and which component he is working in.   As long as it meets it's contract it is said to be good.  A unit test that proves it meets it's contract makes it formally tested to do so.  You are only testing that one bit of code, not the whole application.  If the whole appliction is broken, verifying if a component is function correctly in isolation is very advantagous.

You do have to keep remembering where you are though, what memory footprint you can afford, how much of the extra reference hops and some unnecessary protections cost in terms of cycles where that is important.

The biggest, by far, advantage of unit tests, which is so often forgotten is not that they prove the code works.  It's that in 6 months time, when someone just did a 3 week long refactoring... that the code STILL works.

Strictly speaking it demonstrates that the refactoring hasn't changed the operation in any way detectable by the (unit) tests. That emphasises that the quality of the (unit) tests is vital.

Quote
Analogy with electronics....   your "units" are ICs.  Except that you can make your own.  So while scattering a bunch of passives and discretes all over the PCB might be required in RF stuff, most people would prefer to get an IC which encapsulates that functionality.  If you were in a lab capable of making fab'd prototype ICs, you could see the appeal of pushing functionality into ICs where they can be tested in a repeatable way and then manufactured in a pre-tested, quality form and re-use in other prjects.  Your ICs are your units and the test hardness for them the unit tests.  Having those ICs run with the minimum amount of external support is the art of writing unit testable code.

Analogies are dangerous; they often generate more heat than illumination.

Most people can fab their own prototype and production ICs, and have been able to do so for 30 or 40 years (FPGAs and PALs respectively).

For unit testing purposes, it is accepted practice to regard all of these as units: discrete semiconductors and passives, ICs, populated PCBs, modules (e.g. motherboard containing RAM), racks containing multiple PCBs, subsystem containing (say) one or more racks, etc.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline cantata.tech

  • Regular Contributor
  • *
  • Posts: 75
  • Country: au
Re: Unit testing embedded software
« Reply #33 on: February 24, 2023, 05:06:22 pm »
ok peeps, I have a real project..

How do I do it?

I have an uC running under PlatformIO. It's an Arduino/ESP32 hardware in C++.

It's reading a waveform on some pins and the requirement is to react to that digital waveform when it sees it.

Back when I was young we never had TDD in my neck of the woods. Not enough memory and other excuses.

Here's a description of the scenario. A pin on the uC is low, it's sitting there waiting to go high. When it goes high the next routine needs to read that there was a waveform detected and go do something else on another pin.

It's sounds very simple and the code to write it without TDD i could write with my eyes closed. Doing TDD with Hardware wasn't something I spent a lot of time doing.

How do I write a Unit test for that?

I have Pulseview and I can see my signal on the line. It's not a signal with a library and in my case I don't want to use a library - let's say there's no library. I want to rough my way in and if i see the line go high, fire off a code event which sends a signal somewhere else.

pseudo code is fine but c++ is the language being used.

How do I mock a wire/gpio going high and my code being fired?

 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: Unit testing embedded software
« Reply #34 on: February 24, 2023, 05:37:42 pm »
How do I write a Unit test for that?
Short answer - you don't.

This is exactly the scenario where unit testing makes no sense. If there is some non-trivial algorithmic code being executed when GPIO interrupt fires, then that is the code you test assuming that GPIO interrupt happened at the right time.

What you are describing sounds like there is trivial amount of processing done. There is no point in unit testing that, you will waste a lot more time because this is not where errors in the code like this come from. They come from integration and running on the real hardware asynchronously.

The way to test things like this is to do functional testing using real hardware. You will need a test setup that  can generate external inputs on command from the PC and read back the state of the firmware. Then PC runs a series of tests where it generates valid and invalid sequences and then looks for the reported results.

This is where you get the most out of your tests. You can leave it running for days just generating random noise on the pins and checking if something crashed or misbehaved.
Alex
 

Offline paulca

  • Super Contributor
  • ***
  • Posts: 4003
  • Country: gb
Re: Unit testing embedded software
« Reply #35 on: February 24, 2023, 05:44:20 pm »
The pin change is the trigger.

What you do about it is the event handler.

Test the event handler. 

Just create a function called "test_eventHandler()" and exercise it in code without needing the GPIO stuff.

In case it's not obvious.  Elsewhere, poll/interrupt on the GPIO states, when you see what you want, call the event handler you previously tested.
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 
The following users thanked this post: cantata.tech

Offline cantata.tech

  • Regular Contributor
  • *
  • Posts: 75
  • Country: au
Re: Unit testing embedded software
« Reply #36 on: February 24, 2023, 06:13:28 pm »

This is exactly the scenario where unit testing makes no sense.

Well it's part of a bigger system that will get much more complicated later.

Quote
What you are describing sounds like there is trivial amount of processing done. There is no point in unit testing that, you will waste a lot more time because this is not where errors in the code like this come from. They come from integration and running on the real hardware asynchronously.

I wouldn't say that.

It's audio gear btw and I'm a lazy software engineer who wants to run code in the night without the neighbours or wife complaining. I have to limit time on the real equipment.

Quote
The way to test things like this is to do functional testing using real hardware. You will need a test setup that  can generate external inputs on command from the PC and read back the state of the firmware. Then PC runs a series of tests where it generates valid and invalid sequences and then looks for the reported results.

This is where you get the most out of your tests. You can leave it running for days just generating random noise on the pins and checking if something crashed or misbehaved.

It's audio equipment with front panel buttons. I don't have anything to simulate pressing the front panel buttons. Plus it makes noise - it's an untuned radio station.

What I'm actually doing is capturing a 'Mode' change on a radio, going between AM to FM, capturing the bus data and then running a different handler downstream. Whilst I do have some datasheets for the radio the level of detail is not great. Also, there are fields in the data stream that I know that are there but I just don't wish to decode them.

I was reading on the net between posts that somebody else uses a spare GPIO pin on the dev board to generate the input signal as an output and then wire pin to the desired input pin. That is workable for me. From the commandline, I could just toggle whether I wanted the simulator or the real input running.

Thanks for the answer  :)
« Last Edit: February 24, 2023, 06:21:11 pm by cantata.tech »
 

Online ataradov

  • Super Contributor
  • ***
  • Posts: 11228
  • Country: us
    • Personal site
Re: Unit testing embedded software
« Reply #37 on: February 24, 2023, 06:25:05 pm »
Well it's part of a bigger system that will get much more complicated later.
But testing trivial units on their own will not improve overall system that much.

Unit test make sense if there is a piece of code that is hardware-independent and has fixed inputs and outputs. A good example would be if you have an ADC reading a buffer of data and then some DSP algorithm runs on this buffer. In this case you would unit test just the algorithm. You can generate test buffers and expected outputs. You don't simulate the ADC itself.

It's audio equipment with front panel buttons. I don't have anything to simulate pressing the front panel buttons. Plus it makes noise.
Well, this is what you have to work on. Either firmware must have alternative control method to the front panel (take commands form UART), or you need a rig to press the buttons, or if you are really serious about your testing, the buttons should be exposed on some test connector so that they could be "pressed" by the test software.

It is hard to tell without looking into the details of the exact equipment, but unit tests for things with user input like this are rarely useful.
Alex
 

Offline paulca

  • Super Contributor
  • ***
  • Posts: 4003
  • Country: gb
Re: Unit testing embedded software
« Reply #38 on: February 24, 2023, 07:01:19 pm »
It's audio equipment with front panel buttons. I don't have anything to simulate pressing the front panel buttons. Plus it makes noise - it's an untuned radio station.

Again.  The button event is hardware/gpio.  How that event is handled, is unit testable.

So you don't need to push buttons to test IT.

You will need button pushers to test the hardware handler.

I would say there is enough there to consider that approach.

EDIT:  Consider, as I am, right now, that SPI, I2C and UART are all options for MCU<->MCU communications.  My application wants to send stats one way and config the other.  The actually required data protocol is independent of the transmission protocol.   Save for "ended-ness" of the events.  It makes sense to separate the code along that line.  Code which is called from the hardware event handlers, whatever they are.  You have the application using the abstraction and the abstraction being used by the hardware.  The bit that remains mostly constant is how that particular event is handled in your application.

It should come down to functions which do not take, nor reference hardware at all.

HardwareISR -> HAL Callback -> Application code -> HAL function -> Hardware calls

The "application code" is unit testable.  As long as you call it's "API" correctly, it should not require access to hardware and still produce adequate results.
« Last Edit: February 24, 2023, 07:08:47 pm by paulca »
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline paulca

  • Super Contributor
  • ***
  • Posts: 4003
  • Country: gb
Re: Unit testing embedded software
« Reply #39 on: February 24, 2023, 07:16:12 pm »
Flipping my hobby hats...

What MCU and realtime developer do do (hehe) that big iron people tend not to, is fail fast, fail early, fail hard.

The technique is often referred to as "Assertion".   Analogy.  Having to press the clutch to start a car.  It simply won't let you do it without pressing it... in some cars.  It's to prevent you from starting the car in gear.  In many circumstances its in neutral and it still won't let you.  An assertion.  Not an error, not a warning, not a negociable event, an assertion.  Shall not proceed.  Low level is FAR more full of them.

You will find yourself in many, "What if?" scenarios, if you program defensively.  Checking limits, boundaries etc.  A lot of them will be a case of "Shouldn't ever happen".   Don't ignore these.  Force them into the error handling routine.  Within there...  halt everything.  Capture all the information you can.  Got while(1).  Do not tolerate sloppiness.  If an error is an error it's an error.  Treat it as such.  Fail immediately.  It will force in in development to simply not tolerate shit code.

Or your team.

We can't do that in big iron.  We tend to play a game of "hot potato" and throw the exception up and up and up until either something can do something about it, or the whole transaction gets aborted back at source.  Many layers of Not my problem and sorry can't help.  In this scenario, 24/7 running enterprise applications, serving millions of requests a second, one can simply not have a single worker thread "assert" and abort the whole virtual machine.  I have seen people put asserts in their code like this, in enterprise in big iron.... I want to slap them.  "Thou shall not abort the VM from a library!  You are servant, not master, you shall NOT control my runtime!"
« Last Edit: February 24, 2023, 07:22:53 pm by paulca »
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline cantata.tech

  • Regular Contributor
  • *
  • Posts: 75
  • Country: au
Re: Unit testing embedded software
« Reply #40 on: February 24, 2023, 07:18:46 pm »
But testing trivial units on their own will not improve overall system that much.

Unit test make sense if there is a piece of code that is hardware-independent and has fixed inputs and outputs. A good example would be if you have an ADC reading a buffer of data and then some DSP algorithm runs on this buffer. In this case you would unit test just the algorithm. You can generate test buffers and expected outputs. You don't simulate the ADC itself.

In this case I'm trying to replace an existing module with a new module. So the exact task is to simulate an old piece of hardware with a new piece of hardware going through a convertor.

One example is a radio module, replacing an AM module with a DAB+ or Internet Radio.

A second example is replacing an MD (minidisc) with a USB or SD drive.

Quote
It is hard to tell without looking into the details of the exact equipment,

That will take too long. My setup is not entirely dissimilar to this --> http://gizmosnack.blogspot.com/2015/11/aux-in-volvo-hu-xxxx-radio.html

Also, the code that's shown down the page is certainly indicative of what I have.
« Last Edit: February 24, 2023, 08:25:21 pm by cantata.tech »
 

Offline cantata.tech

  • Regular Contributor
  • *
  • Posts: 75
  • Country: au
Re: Unit testing embedded software
« Reply #41 on: February 24, 2023, 07:25:52 pm »
You will need button pushers to test the hardware handler.

As strange as it may seem, I don't. The mcu on the radio decodes the keypress and sends a decoded 'instruction' downstream to a device as a command to instruct it to do something.

Think of the system as running on a CAN/LIN or I2C bus although they use their own bus system.

Quote
EDIT:  Consider, as I am, right now, that SPI, I2C and UART are all options for MCU<->MCU communications. 

These units have their own proprietary bus, such as Melbus. So those won't work.

I want the TDD to help me over time so that I can run regressive tests.

« Last Edit: February 24, 2023, 08:45:06 pm by cantata.tech »
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26755
  • Country: nl
    • NCT Developments
Re: Unit testing embedded software
« Reply #42 on: February 24, 2023, 08:16:53 pm »

It is hard to tell without looking into the details of the exact equipment, but unit tests for things with user input like this are rarely useful.
IMHO there is a blurry line between unit testing & verification testing. In the past I have been simulating user input by using relays instead of buttons (this was for a vending machine-ish application). See if a million button pushes also result in a million transactions recorded by the system.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline paulca

  • Super Contributor
  • ***
  • Posts: 4003
  • Country: gb
Re: Unit testing embedded software
« Reply #43 on: February 24, 2023, 08:28:40 pm »
I suppose, as we already knew, we have to be careful of being "training mules" for the AI rather than responding to trolls.
"What could possibly go wrong?"
Current Open Projects:  STM32F411RE+ESP32+TFT for home IoT (NoT) projects.  Child's advent xmas countdown toy.  Digital audio routing board.
 

Offline John Coloccia

  • Super Contributor
  • ***
  • Posts: 1208
  • Country: us
Re: Unit testing embedded software
« Reply #44 on: March 03, 2023, 09:38:30 am »
I have seen people put asserts in their code like this, in enterprise in big iron.... I want to slap them.  "Thou shall not abort the VM from a library!  You are servant, not master, you shall NOT control my runtime!"

I can't even begin to tell you how annoyed I get at libraries/APIs that throw instead of returning an error.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf