Author Topic: Unit testing for embedded system  (Read 2666 times)

0 Members and 1 Guest are viewing this topic.

Offline Kittu20Topic starter

  • Regular Contributor
  • *
  • Posts: 96
  • Country: in
Unit testing for embedded system
« on: February 10, 2024, 04:14:24 pm »

I've been working with the PIC18F45K80 microcontroller using MPLAB XC8, C language and recently I came across the concept of unit testing. From what I gather, it involves checking individual components of a program, such as functions or modules, to verify their correctness.  I think there are two possible methods: utilizing a third-party library or developing our own custom library

However, I'm still unclear about its practical utility and how to implement it effectively in embedded systems I'd appreciate hearing about your experiences with unit testing in embedded systems. Do you incorporate unit testing into your development process?
 

Online PlainName

  • Super Contributor
  • ***
  • Posts: 6848
  • Country: va
Re: Unit testing for embedded system
« Reply #1 on: February 10, 2024, 04:57:30 pm »
The idea with unit testing is that you can verify that some code works as expected (and not as not expected). Useful when you are writing the code but also later when you make changes to a system. Essentially, you design the specification of something (perhaps binary to hex character) and then write a test that would prove an implementation completely works (or not! - give it max and min values, edge cases, data you know is problematic, etc), and then you write the actual implementation which is exercised by your test in all those ways. Your project would have many of these tests and the passing of them all would show that some change you've made hasn't screwed things up.

At least, that's the blurb. It's a useful means for writing code for hardware that doesn't exist (yet, perhaps), although that can be  problematic if your virtual hardware test assumes it works in a way it actually doesn't.

Anyway, it can be hard to get into and it can be a pain to keep working to, but I think it is a plus overall. Since you write in C you're already limiting available options, but the one I use is CppUTest which is available on github. It is C++ based, but the target system is C so you don't need to either learn C++ (much) or change your language.

Ultimately, even if you decide it's not for you, realising how to better partition you code, how to reduce dependencies and better modularise it, should be a win anyway.
« Last Edit: February 10, 2024, 04:59:22 pm by PlainName »
 
The following users thanked this post: Kittu20

Offline Jeroen3

  • Super Contributor
  • ***
  • Posts: 4078
  • Country: nl
  • Embedded Engineer
    • jeroen3.nl
Re: Unit testing for embedded system
« Reply #2 on: February 10, 2024, 05:08:41 pm »
Yes, testing is great! However, if all you've done now is interacting with compilers and code through an IDE like MPLAB you're in for a steep learning curve.

You use unit testing to validate your software units. Eg: functions or libraries.
Most people run this automatically on one of the git hosters and a cloud or their own runner.

C makes unit testing tricky, because in order to test it must compile. And in embedded it's often very messy with dependencies. Especially when using vendor stuff.
And since you don't want to emulate the entire chip, you substitute what your code depend on, simulate/mock or stub it.
Simply put: you substitute what #include imports for test emulations.
This is the hardest part, because it exposes your software design flaws. And correcting this is often a lot of work.

Another tricky part is that you need to compile it to run in Linux/Windows, so you need to substitue compiler specific syntaxes.
You can also make it run on pic but then you need a host board for it and write scripting to compile/load/run and view the results. People do this, but it's for the advanced user.

A very easy start would be to download visual studio with c/c++ support and create a mstest/googletest project and include the files you want to test.
The frameworks googletest/mstest offer various tools for running tests. Pay attention, most testing suites assume you run Linux.

ci/cd
Automating stuff like this can be "as simple" as using GitLab pipeline default linux runner compiling it with your makefile and run the tests. Of course this requires you know how to compile your code from scratch with makefiles because there is no IDE....
Or, create a VM with gitlab runner, put it on a pc in the corner and run your own ci/cd tests. Basically every git commit will then be sent to the runner and your automations will run, eg: run the visual studio googletest project. And a nice checkmark will show next to your commits  :) And it will yell at anyone who commits something that fails tests.

example
A unit test for a library such as an eeprom wear leveling storage driver you can verify if it writes to the right addressed and converts datatypes correctly.
You can mock the HAL for eeprom access to access a big array and since you also have access to the big array you can check that the test function writes the right data to the right place.
« Last Edit: February 10, 2024, 05:11:49 pm by Jeroen3 »
 
The following users thanked this post: Kittu20

Offline Kittu20Topic starter

  • Regular Contributor
  • *
  • Posts: 96
  • Country: in
Re: Unit testing for embedded system
« Reply #3 on: February 10, 2024, 06:34:28 pm »
Thank you for the detailed explanation. I appreciate the clarity you provided.

If you've had experience with unit testing on the ARM platform, particularly with STM32 devices, I'm curious to know whether you utilized a third-party library or developed your own custom library for unit testing?
 

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26907
  • Country: nl
    • NCT Developments
Re: Unit testing for embedded system
« Reply #4 on: February 10, 2024, 06:54:11 pm »
A first step towards testing inside / outside an embedded platform is to use the same tools for the embedded platform and on the PC platform you use for testing. This is much easier to do for ARM while using gcc / clang compared to proprietary compilers where it comes to accuracy of the results. But you can still get bitten by unexpected results if your coding style is sloppy. Or put differently: your code is open to interpretation by the compiler. Mapping variables or structs onto byte buffers is one example where you can run into nasty surprises.
« Last Edit: February 10, 2024, 06:55:55 pm by nctnico »
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 3725
  • Country: us
Re: Unit testing for embedded system
« Reply #5 on: February 10, 2024, 09:08:42 pm »
It really helps if you design your code with testing in mind.  In particular making sure that hardware specific code is compartmentalized.  Then you can sub those out with code that e.g. logs the transactions or provides fake data that you can then verify is handled correctly.  This is of course good practice anyway for portability and debugging but doubly so when you want to run tests.  A random write directly to a GPIO peripheral will crash in the test environment if it doesn't get stubbed out because it bypasses your hardware abstraction layer.

It's also possible to do full system tests in an emulator like qemu.  Depending on how deep you want to go you can design virtual drivers for all your peripherals and run the unmodified binary in the emulator.  That's usually more work than it's worth but i have used it when I wanted to test that my program wrote to specific IO memory address in a certain order.
 

Online PlainName

  • Super Contributor
  • ***
  • Posts: 6848
  • Country: va
Re: Unit testing for embedded system
« Reply #6 on: February 10, 2024, 09:43:13 pm »
Quote
use the same tools for the embedded platform and on the PC platform you use for testing
... [because] your code is open to interpretation by the compiler

Actually, I think it's better to have different compilers for that case, to catch out compiler-specific code. It would be fine on the current product but what about if/when you port to a bigger and better processor (or smaller, cheaper)? Of course, one should be aware of that kind of thing and code appropriately anyway, but you'd never know until the fan starts spreading if you've actually managed that.
 

Online hans

  • Super Contributor
  • ***
  • Posts: 1641
  • Country: nl
Re: Unit testing for embedded system
« Reply #7 on: February 10, 2024, 11:20:52 pm »
Yes I use unit testing extensively.

TDD (Test Driven Development) is a paradigm where you first write a test, and then implement it. This forces you to think about how the code should behave externally.

However, do mind a issues with unit testing on embedded, as there a few additional criteria to regular software:
- Code may rely on hardware peripherals
- Code may rely on vendor libraries
- A project may be sensitive to time deadlines (realtime)

I want to emphasize 2 distinct approaches to unit testing: you could do some kind of structured/automated testing on hardware itself, or you could crosscompile your C code to another platform (x86 on your PC) and run the code there. The latter is just a behavioural verification.
Its usually quite a big step to properly set-up unit tests on a target platform itself. Its only useful for hardware tests, and if you want to test your hardware properly, you will need some oscilloscope or other external gear that can be remotely managed to exercise your firmware. It will require a specific setup which not every developer may have. So it has some use cases.. but for most projects I "manually" debug.
It can be useful for qualification purposes.. e.g. you make some automated tests for battery life measurements on daily builds of your firmware.

Everything that is above hardware is unit tested on PC though. Sometimes I may even go to the effort to make an emulation/simulation environment so I can actually run parts of the embedded application on my PC. For example for networked devices, it can be useful to test them with more powerful PC tools.

Unit testing does not help in verifying if code is "always" correct though. For example, you can still write 'bad' code that in isolation works, but starts to break when timing deadlines are missed. Simiarily, code can also break on race conditions that are not reproducible on PC. Code can also break on non-portable code, which may work on 64-bit x86, but breaks on any other platform.
« Last Edit: February 10, 2024, 11:24:02 pm by hans »
 

Offline Jeroen3

  • Super Contributor
  • ***
  • Posts: 4078
  • Country: nl
  • Embedded Engineer
    • jeroen3.nl
Re: Unit testing for embedded system
« Reply #8 on: February 12, 2024, 01:50:52 pm »
Unit testing is especially valuable in giving your a sense of trust in your code building blocks. In a way that you will not be hunting bugs in communication protocols cause by a glitch in the CRC function.
The CRC function is an excellent example of code that is easily tested.
 

Offline wek

  • Frequent Contributor
  • **
  • Posts: 495
  • Country: sk
Re: Unit testing for embedded system
« Reply #9 on: February 12, 2024, 02:13:11 pm »
Give me your test, and I will write you a glitch.

JW
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14488
  • Country: fr
Re: Unit testing for embedded system
« Reply #10 on: February 12, 2024, 08:21:15 pm »
Give me your test, and I will write you a glitch.

JW

 ;D
 

Online djsb

  • Frequent Contributor
  • **
  • Posts: 893
  • Country: gb
Re: Unit testing for embedded system
« Reply #11 on: February 12, 2024, 08:44:48 pm »
So would something as simple as flashing an LED or using a serial output for each function entered or exited be classed as a unit test? Or is this something much more complicated?
David
Hertfordshire,UK
University Electronics Technician, London PIC,CCS C,Arduino,Kicad, Altium Designer,LPKF S103,S62 Operator, Electronics instructor. Give a man a fish and you feed him for a day. Teach a man to fish and you feed him for a lifetime. Credited Kicad French to English translator.
 

Offline Cavhat

  • Contributor
  • Posts: 11
  • Country: us
Re: Unit testing for embedded system
« Reply #12 on: February 13, 2024, 01:23:17 am »
James Grenning wrote an excellent book on the topic: Test Driven Development for Embedded C (https://books.google.com/books/about/Test_Driven_Development_for_Embedded_C.html?id=dA9QDwAAQBAJ&source=kp_book_description).  I found it very helpful in my efforts.
 

Offline ejeffrey

  • Super Contributor
  • ***
  • Posts: 3725
  • Country: us
Re: Unit testing for embedded system
« Reply #13 on: February 13, 2024, 02:29:55 am »
So would something as simple as flashing an LED or using a serial output for each function entered or exited be classed as a unit test? Or is this something much more complicated?

Unit testing means to take a unit (a piece of the whole with well defined boundaries and behavior), make a request, and check that the response is what the specification says. That can be "call the sqrt function with many values and assert that the return values are correct" or "ask to read a sensor and verify that the correct SPI register reads are performed."

Just flashing an LED when you enter a function could be part of a unittest, but its not really a test by itself.  At least you need more description of what behavior you are testing and how flashing the LED accomplished it.  Just showing that a function runs is usually not a unit test although it can be if the thing you are testing is an event handling framework.
 
The following users thanked this post: djsb

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14488
  • Country: fr
Re: Unit testing for embedded system
« Reply #14 on: February 13, 2024, 03:38:22 am »
The best part of unit testing in an agile environment is the green light icons when all tests pass. In some settings, that's a shining sun. Invaluable.
 

Online dobsonr741

  • Frequent Contributor
  • **
  • Posts: 674
  • Country: us
Re: Unit testing for embedded system
« Reply #15 on: February 13, 2024, 05:25:02 am »
 

Online hans

  • Super Contributor
  • ***
  • Posts: 1641
  • Country: nl
Re: Unit testing for embedded system
« Reply #16 on: February 13, 2024, 11:53:51 am »
So would something as simple as flashing an LED or using a serial output for each function entered or exited be classed as a unit test? Or is this something much more complicated?

No it doesn't have to be. Most unit tests are just calling functions and checking whether it computes the right values. This can be a "pure" function like CRC, data goes in, checksum goes out, and there is only 1 correct answer.
But it could also be a whole module with statemachines that is checked and exercised.

Just checking the calling of functions is only a subset of what an unit test can be. It doesn't tell the whole story, although it is sometimes used in mocking. Mocking is a method to isolate your unit test code from other modules. You then "mock" whatever other code would be called. Many test frameworks allows you to set this up conveniently, and also test whether the mocked calls are correct (e.g. correct number of calls, correct arguments, etctera).

For example: say you write a piece of code that stores and retrieves configuration parameters for your application. The settings are stored in EEPROM that is handled by some other code. If you want to test your configuration code in isolation, you would mock the EEPROM module You could then write unit tests for reading settings... e.g. you mock the EEPROM read function and in the test handcode some values that correspond to some expected configuration state. Likewise for writing you could pass some configuration state, and then mock the EEPROM write function.. at which you check whether the correct bytes are written to memory.
These trivial examples may not be that interesting to test; the nice thing starts when you need to handle exceptional cases which are hard to exercise by hand or don't occur frequently. In manual testing these tend to be less well tested. Unit tests are just an automation of those tests. For example, migration of parameters between firmware upgrades might be one of those things you want to test. Etc.

It just so happens a LED blinky would be quite hard to test because
A) It interfaces directly to hardware code.. and unit tests on your workstation won't capture that unless you would go through the effort to setup a whole emulation environment etc. Or perhaps test on real hardware and have equipment that externally verify.. but that is more often only worth it as integration test. Unit tests should aid development and quality, not create a bunch of useless bureaucracy before you can start writing code.

B) The requirements for a "correct" blinky should include timing, and that is one the hardest parts of unit testing.. Just the function calls themselves don't reveal anything about how long something will take, and whether that has any interactions with the rest of your program. Maybe you could write a test where you mock e.g. delay_ms.. but that assumes that delay_ms on your platform works OK.

It does lead to another issue in writing "solid" code: programs that depend on timing are in my experience the worst to test and verify. However, this can be an essential part of embedded, where we have to deal with real-time deadlines for something as simple as an UART. Unit tests won't reveal whether your code will miss interrupts or have race conditions on the target platform.
At best, you can test what happens in some emulated failure cases. E.g for an UART Rx interrupt.. getting the byte from hardware has a hard deadline. However, if that byte is pushed into a FIFO for the application to read, then that can still fail (e.g. your main loop is stalled for CPU time). You could write an unit test that checks what happens in cae of a FIFO overrun (e.g. you emulate the arrival of many UART Rx events). You could verify that your application won't hang, whether partial packets are discarded, etc. This can help in some robustness, but it won't magically make all problems go away.
« Last Edit: February 13, 2024, 11:56:45 am by hans »
 
The following users thanked this post: djsb

Offline Doctorandus_P

  • Super Contributor
  • ***
  • Posts: 3367
  • Country: nl
Re: Unit testing for embedded system
« Reply #17 on: February 13, 2024, 12:37:54 pm »
C is a pretty much standardized and universal language. When I want to write a program for a microcontroller, I often start by writing some commandline PC programs first. Often quite big sections of a program for a microcontroller can be developed and debugged on a PC. Such as for example a loop for a PID controller, formatting of packets (inclusive checksum) for some network stack, string operations, calculations etc.

There are also some differences. A PC does not have your uC hardware. You can create the (special function) registers as simple variables. This lets you at least compile the code code without complaints. Once the code is compiled, it's easy to write some test routines (put them in another file) to exercise the library code you have written in the first file. And with a little bit of extra care, your test routines are unit tests too. And you can use them if later a bug is found in your uC code, or just to verify the code still works after you have made modifications later.

TDD (Test driven development) is good, but it is not a catch-all method, and it also does not always make sense to use it. Browse around and experiment with some other (more or less formal) methods of code design and writing, and when you have knowledge of a bunch of different development techniques, you can use the method that best fits (the part of) a program you are working on.
 

Offline std

  • Contributor
  • Posts: 14
  • Country: ru
Re: Unit testing for embedded system
« Reply #18 on: February 14, 2024, 03:13:58 am »
James Grenning wrote an excellent book on the topic: Test Driven Development for Embedded C (https://books.google.com/books/about/Test_Driven_Development_for_Embedded_C.html?id=dA9QDwAAQBAJ&source=kp_book_description).  I found it very helpful in my efforts.
I have read it, still hoping to see solution offered for hardware testing. Nothing like this. This is software testing. The word "embedded" is misleading here, author does not concern hardware. His only example with “Led Driver” is very artificial. So, +/- informative book about software testing, nothing for embedders.
 
The following users thanked this post: dobsonr741

Online PlainName

  • Super Contributor
  • ***
  • Posts: 6848
  • Country: va
Re: Unit testing for embedded system
« Reply #19 on: February 14, 2024, 10:17:12 am »
Quote
author does not concern hardware

How would you deal with hardware in the context of unit testing?
 

Offline AVI-crak

  • Regular Contributor
  • *
  • Posts: 125
  • Country: ru
    • Rtos
Re: Unit testing for embedded system
« Reply #20 on: February 14, 2024, 10:20:35 am »
I have read it, still hoping to see solution offered for hardware testing.
The hardware test for embedded systems is a separate one-time firmware test. At the assembly stage it is necessary to find out the serviceability of all electronic components on the PCB. However, the test itself requires reaction to all system states, it is a lot of lines of code, too many. In addition, the speed of execution of the algorithms is critically slow. You will have a hard time explaining the merits of your product to a customer, even harder to sell.
Besides, error detection in a serial product does not make sense. Let's say a sensor fails, does the customer have to re-solder it himself?

For a serial product there are two states available: working / requires repair in a service center. For this purpose it is possible to leave a very shortened test in the serial firmware, without going through all the states. Literally perform one variant of the test, and in case of failure de-energize the device (while there is still something to do).

Translated with DeepL.com (free version)
 

Offline std

  • Contributor
  • Posts: 14
  • Country: ru
Re: Unit testing for embedded system
« Reply #21 on: February 14, 2024, 11:36:30 am »
The hardware test for embedded systems is a separate one-time firmware test.
...
For a serial product there are two states available: working / requires repair in a service center.

Analog poweron self-test  - is the primary reason, why you see this relays in our product.
The leds also idicating health / problems, including self-test failure code.
 

Offline SiliconWizard

  • Super Contributor
  • ***
  • Posts: 14488
  • Country: fr
Re: Unit testing for embedded system
« Reply #22 on: February 14, 2024, 07:18:34 pm »
That's a lot of relays. But I like the green LEDs.
 

Offline std

  • Contributor
  • Posts: 14
  • Country: ru
Re: Unit testing for embedded system
« Reply #23 on: February 16, 2024, 07:44:11 am »
That's a lot of relays. But I like the green LEDs.

I exaggerated for the sake of saying :)  its not "primary" reason.  We need all this relays because in power-off state realys switched all signals to go-through (and in the same time circuitry loopbacks for self-test).  It's actually quite an old design, I just had it handy on my desk, newer one has small smd relays like small IC's.
« Last Edit: February 16, 2024, 07:49:13 am by std »
 

Online hans

  • Super Contributor
  • ***
  • Posts: 1641
  • Country: nl
Re: Unit testing for embedded system
« Reply #24 on: February 16, 2024, 09:14:43 am »
James Grenning wrote an excellent book on the topic: Test Driven Development for Embedded C (https://books.google.com/books/about/Test_Driven_Development_for_Embedded_C.html?id=dA9QDwAAQBAJ&source=kp_book_description).  I found it very helpful in my efforts.
I have read it, still hoping to see solution offered for hardware testing. Nothing like this. This is software testing. The word "embedded" is misleading here, author does not concern hardware. His only example with “Led Driver” is very artificial. So, +/- informative book about software testing, nothing for embedders.

That's true.. but there are a few points that align with embedded:

I would argue that a majority of modern firmware is already at the "software domain". We do have some specific requirements like no malloc, mindful of codesize, etc. Only a minority of the code I write actually interfaces with RTL.

Thus by utilizing unit testing and other modern software development techniques, I'm able to remove the highest necessity of debugging code on MCUs with slow debug cables and minimize time spent in lab while occupying expensive test equipment.

I see unit tests as SPICE is for analog.. you run the design with 1st order models to get 90% of the result, and the last 10% requires manual tweaking later. You could go the extra mile with 2nd, 3rd order models to get to 99% or 99.9%, but it becomes harder to do.

Even for firmware tailored towards a PIC16, I can see some benefits. But I think these would be mostly theoretical, since the PICs (and some other cores) use ancient compilers which cannot tremendously deal well with abstractions. Those abstractions would be needed for e.g. a BSP layer to support mocking for unit tests. If we value devtime over anything else, obviously we wouldn't use a PIC16, and it also frees up a path to a workflow with (atleast initially) more overhead.

Automated tests with hardware-in-the-loop is certainly possible.. but its a question of what is worth you investment. You could hook up a MCU to a 16, 32, 64ch logic analyzer (or AWG, scope, etc.), write scripts for all interfaces you may ever want to exercise, write DUT code you want to test in isolation, and then verify all BSP-level functionality works as expected. But do you really want to occupy potentially k$'s worth of equipment for that? Or would be more specific setups in production be a better bet?

Another issue with such hardware-in-the-loop tests, is that they will be relatively slow. Crosscompiling, uploading, equipment control.. its all very slow. Unit tests should be fast. This is not a technological requirement at all, rather a psychological/process requirement: by keeping them fast, we as devs will keep on running them for every change we make. This keeps the loop tight at which we can catch accidently broken stuff. One compromise would be to move slow tests to a continuous integration server.. so for hardware-in-the-loop setup that would certainly be an option. But still.. those setups are expensive, and I don't think it's worth it to test some very basic SPI or UART code.
In contrast, testing more product-level properties may be much more interesting. E.g. something like this: https://medium.com/tado-product-development-blog/building-an-iot-product-continuous-battery-lifetime-testing-3245b6cdfa40
« Last Edit: February 16, 2024, 09:18:00 am by hans »
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf