Unit testing C embedded code - systems and frameworks

[ricko_uk]

Hi,
when I needed to do unit testing in C for embedded systems I did either one of the following:
- did it in simulator
- used the hardware and either have a breakpoint at the end of the unit testing code and see the results or output the results to the PC

A couple of questions:

1) How do you do it, especially when the hardware does play a role so it or something emulating its actions is involved? I am just wondering if there are either other preferred systems and/or frameworks that make it easier and/or faster?

2) same question as above but in the case of running an RTOS?

Thank you

[SiliconWizard]

Hi,
when I needed to do unit testing in C for embedded systems I did either one of the following:
- did it in simulator
- used the hardware and either have a breakpoint at the end of the unit testing code and see the results or output the results to the PC

A couple of questions:

1) How do you do it, especially when the hardware does play a role so it or something emulating its actions is involved? I am just wondering if there are either other preferred systems and/or frameworks that make it easier and/or faster?

If you absolutely need accurate hardware behavior, doing it on target can be your better option. You don't necessarily use a debugger - which may make automating tests difficult - there are other ways, usually by "instrumenting" code. Downside is you need to be careful about your instrumentation (additional code) having minimal effect on the result.

You may have simulators for specific devices, but that's not necessarily available. IIRC, there was a simulator for some PIC MCUs provided by Microchip that we used for this.

But I've most often done unit testing for embedded devices purely on a software level, "mocking" device-dependant parts, such as peripherals, when required. It will test the code, but (usually) not timings. I personally think this is appropriate as far as unit testing is concerned. You use other forms of testing for anything timing-related, and more generally hardware-related.

2) same question as above but in the case of running an RTOS?

Same usually.
But keep in mind, as I said, that I use unit testing mostly for testing *software* behavior, not hardware behavior, and think that's where it's most appropriate.
As soon as there is some hardware interaction, then it looks more like forms of "integration testing" than unit testing anyway. At least that's my take on this.

Actually, for some higher-level functions in code - for instance things that implement a bit complex algorithms - testing them separately on a different target, typically a computer, can be useful and much easier too, especially when you need to test a lot of possible inputs/configurations/etc. Sure in this case you're not testing the code on the end target, and you need to be careful about any limitation and most of all any non-portable part of your code that may behave differently on a different target, but if your code is portable - which can often be the case for the higher-level parts of the code - this is a workable approach that works quite well.

[ricko_uk]

Thank you all!

[SiliconWizard]

No probs. That can be an interesting discussion in the "embedded development" context, because, as I pointed out, I don't think you can strictly consider doing "unit testing" as soon as you have several parts interacting. The whole point of unit testing is the opposite of that: testing blocks of code (the granularity remaining up to you...) independently. Which is why "mocking" anything the particular block you're testing may be accessing outside of it, is a common practice.

You're not really doing unit testing if the piece of code involved in a particular test needs to interact with hardware, IMHO. But that's a typically controversial topic, and YMMV. Can be a tricky one too, to determine what parts of hardware you consider not part of the "unit" you're testing, and what parts would be.

But what I've commonly witnessed is that management will just tell you to do unit testing, because that's good practice, after which you're on your own.

[Jeroen3]

The most difficult part about unit testing your C code is that your C code need to be unit-testable.

It's quite easy to get loads of dependencies you need in your code that make testing very hard. Especially if you have to mock all those dependencies by dummy functions and defines.
An abstraction layer you can help you. For example it's quite trivial to write a module using ST's I2C HAL functions and then emulator being an EEPROM by some very simple array indexing in your mock functions. So you can test and develop your wear leveling and checksum algorithms for various data corruption and memory failure situations. Without actually using an chip, eeprom and logic analyzer.

The second most difficult thing is actually compiling and running your unit tests with frameworks on windows. I found the easiest way to go about it to install Visual Studio Community and to create a Google Test Project. It will present you with an empty (C++) project you can include your code in. Click compile, and you can debug your unit test programs.
Two drawbacks on this: 1. you have to repeatedly disable "precompiled headers" for some reason. 2: your c code need some #ifdef cpluspus extern "c"....

I think this is the most portable and beginner friendly way. Since you can use visual studio which fixes all the dependencies and compilation for you.
And actually compiling your code in a different compiler (not gcc) is beneficial to test your code portability as well. May even throw you some new warnings.

[BBBbbb]

But I've most often done unit testing for embedded devices purely on a software level, "mocking" device-dependant parts, such as peripherals, when required. It will test the code, but (usually) not timings. I personally think this is appropriate as far as unit testing is concerned. You use other forms of testing for anything timing-related, and more generally hardware-related.

Sums up my experience as well, but as you noted in embedded environment this is debatable, and I do understand especially in low level projects the need to have "all in". Embedded SW is a broad term going from a few KB uC to SBCs running embedded Linux. More complex the SW is, the arguments for abstracting HW in unit tests grow stronger IMO.

But what I've commonly witnessed is that management will just tell you to do unit testing, because that's good practice, after which you're on your own.

yeah, there are some metrics showing unit tests decrease significantly bugs in production - and that's basically what they want to know. What and how not so much... (of course this can vary depending if management is coming from a technical background)

[SiliconWizard]

But I've most often done unit testing for embedded devices purely on a software level, "mocking" device-dependant parts, such as peripherals, when required. It will test the code, but (usually) not timings. I personally think this is appropriate as far as unit testing is concerned. You use other forms of testing for anything timing-related, and more generally hardware-related.

Sums up my experience as well, but as you noted in embedded environment this is debatable, and I do understand especially in low level projects the need to have "all in".

Yes it's certainly not a simple matter in this context. But as I said, IMHO anyway, if you have "all in", then it's just not unit testing. It's something else.

[PlainName]

For embedded C projects I use CppUTest:

http://cpputest.github.io/index.html

It's C++ but you don't need to know that . I looked at Unity and this and did a couple of projects in both, and landed on this one. There's a book on how to do unit testing with Unity/CppUTest (Test-Driven Development for Embedded C [Grenning]) and it's worth running through. The author is a contributor to both Unity and CppUTest (but mostly the latter).

As Jeroen3 says above, the difficult part is making your code unit testable, and there are several approaches. I found the book and some other sources invaluable in detailing the capabilites of the testing tool, which allows you to design your code to suit.

[PlainName]

To answer your question in a tool-agnostic way:

1) How do you do it, especially when the hardware does play a role so it or something emulating its actions is involved?

Essentially you fake the hardware with mocks. You want to split your code into the bit that doesn't know about the hardware details, and to which you can send or receive fake data, and the hardware-specific stuff that you could mock in quite complete ways but probably is too much hassle. For instance, if you receive a fixed-size frame of an invalid size, that something that's easy to test in the code that doesn't know about the hardware detail and just handles the data. Missing an interrupt is something I'd rather sort out with actual hardware since my interpretation of what it should do is likely embedded in the code as well as the mock.

2) same question as above but in the case of running an RTOS?

Generally it's nothing special - you can test the code making up a task as if it were a single-tasking system. Queues and the like are just function calls returning status and data, and those are easy to fake. Essentially, you're testing your code, not the RTOS.

There are potential issues with shared resources, but unit testing can only test that your solution to that works in the way intended. That is, if you use a mutex it can test that the mutex is correctly obtained and released, but not that the mutex is the right way to protect the resource.

- did it in simulator
- used the hardware and either have a breakpoint at the end of the unit testing code and see the results or output the results to the PC

These are great for debugging, but debugging isn't testing - it is fixing once a test has been found to fail (i.e. if crashes or gives the wrong output).

Unit testing is automated verification, which means you can run it every time you do a compile and catch issues there and then, even (and perhaps especially) ones you're not looking for. With simulators or an ICE you have to figure out at that time what you want to see and then check that you see it, and it's very time consuming. You wouldn't want to test a hundred or so different things like that every time you make a change to your code.

[radar_macgyver]

Are there some good examples of FOSS embedded projects that use unit testing? It's one of those things that I understand the concept but often don't see how it can be applied. Some good examples might help me understand that part.

[mfro]

Unit testing embedded code most often can't run on "the real thing" in real time, but requires you to simulate the hardware somehow.

In my experience, that alone already improves my understanding what I'm (and the device is) doing in a way that helps finding misconception and potential bugs.

[BBBbbb]

In my experience, that alone already improves my understanding what I'm (and the device is) doing in a way that helps finding misconception and potential bugs.

Yes if the same people are doing the unit tests and the code, but I've seen projects with full split, and a specific team focusing on unit tests and reaching full coverage (code completely isolated from HW) and it's just pointless, but KPIs are there...
If completely isolated and done at function level, I don't see much benefit of somebody else writing up the tests. At least with the code developers writing them up it's like an additional code review.

But it really depends what you consider unit tests, and I've seen a lot of different interpretations, that I'm not sure anymore...

[PlainName]

Unit testing embedded code most often can't run on "the real thing" in real time, but requires you to simulate the hardware somehow.

Well, kind of. I mean, yes, if you want to run the actual code en block as if it were the final thing you need to simulate the hardware, but that's like saying you can't taste test a meal without eating the entire thing after it's cooked. But you can, say, cook a few peas and check they look and taste OK, and cook a potato an check the time required, and have several goes at gravy until you get the desired consistency. Then you know how to do the real thing.

With unit testing you're not testing the entire thing as it would run - that's simulation. You're testing each unit, the little bits that make up the whole. A unit is probably a single function, perhaps a couple if you can't get the dependencies right. You test that unit in isolation to everything else - all you're testing is that this unit does what it says on the tin. Do that for all the units and when you put them together the chance of them working is much higher.

But unit testing can be a drag when you start, and it adds considerable time to a project. It can be tempting to just test particularly complex or obscure units, and doing it like that might be right for you - it's better than not doing it at all.

Possibly a big difficulty is getting the tool to run. I would steer away from tools embedded into IDEs because that's lock-in to that IDE, unless it's a generic took that works away from the IDE too. Getting started with your project can involve some aggro too, in faking OS calls and the like, but once set up it's pretty simple to use, so being able to re-use the same tool on any project and with any IDE is an attractive bonus.

[Kalvin]

But unit testing can be a drag when you start, and it adds considerable time to a project. It can be tempting to just test particularly complex or obscure units, and doing it like that might be right for you - it's better than not doing it at all.

Yes, and unit testing starts paying off right at the moment when someone changes something in the existing code. After any changes, it is typically required to test whether the new changes has introduced some regression in the existing code base (ie. did the new changes brake anything in the existing unit test cases). Adding new features requires writing and maintaining the unit test cases, of course. But the payoff is that after each change there is a way to run unit tests automatically, and find any problems (which are covered by the unit tests, of course) at the very earliest state of development. Alternative method for automated unit testing is manual testing for each unit after modifications. And this will be a real drag in the development cycle.

[gmb42]

ChibiOS has a lot of unit tests for itself and its HAL "drivers".  Not very well documented so you have to read the code.

[SiliconWizard]

But unit testing can be a drag when you start, and it adds considerable time to a project. It can be tempting to just test particularly complex or obscure units, and doing it like that might be right for you - it's better than not doing it at all.

Yes, and unit testing starts paying off right at the moment when someone changes something in the existing code. After any changes, it is typically required to test whether the new changes has introduced some regression in the existing code base (ie. did the new changes brake anything in the existing unit test cases). Adding new features requires writing and maintaining the unit test cases, of course. But the payoff is that after each change there is a way to run unit tests automatically, and find any problems (which are covered by the unit tests, of course) at the very earliest state of development. Alternative method for automated unit testing is manual testing for each unit after modifications. And this will be a real drag in the development cycle.

Yes. Now the question of course is who will write the unit tests. Ideally not the same person as the one who wrote the code being tested. In that regard, unit testing for regression testing is good, but if the persons making modifications to the code are allowed to make modifications to the tests themselves, that's a bit pointless. And yes, I've seen that happen.

[enz]

If you want to get a good explanantion, including practical examples, of unit testing in the embedded context, i can highly recommend James Grennings "Test Driven Development for Embedded C".
Very good read.

[SiliconWizard]

If you want to get a good explanantion, including practical examples, of unit testing in the embedded context, i can highly recommend James Grennings "Test Driven Development for Embedded C".
Very good read.

And you can get the code repo here: https://github.com/jwgrenning/tddec-code

[ricko_uk]

Thank you all!

[Kalvin]

But unit testing can be a drag when you start, and it adds considerable time to a project. It can be tempting to just test particularly complex or obscure units, and doing it like that might be right for you - it's better than not doing it at all.

Yes, and unit testing starts paying off right at the moment when someone changes something in the existing code. After any changes, it is typically required to test whether the new changes has introduced some regression in the existing code base (ie. did the new changes brake anything in the existing unit test cases). Adding new features requires writing and maintaining the unit test cases, of course. But the payoff is that after each change there is a way to run unit tests automatically, and find any problems (which are covered by the unit tests, of course) at the very earliest state of development. Alternative method for automated unit testing is manual testing for each unit after modifications. And this will be a real drag in the development cycle.

Yes. Now the question of course is who will write the unit tests. Ideally not the same person as the one who wrote the code being tested. In that regard, unit testing for regression testing is good, but if the persons making modifications to the code are allowed to make modifications to the tests themselves, that's a bit pointless. And yes, I've seen that happen.

As I have understood, in some high-reliability projects it is required that someone else than the original author is writing the unit tests. In normal, less-critical projects it is typical that the original author/authors are writing and maintaining the unit tests as the code is evolving.

There are two aspects in unit testing: white box testing where you can see inside the code and its implementation, and black box testing in which you just see the interfaces and test the code using these interfaces only. In some cases it is beneficial to use external developers for writing unit tests for the interfaces, as this will provide good feedback how well the interfaces are designed and documented, and how robust your code is for illegal parameters and abuse.

Well-written unit tests are always welcome, because they help with regression and sanity tests. Poorly written unit tests should be removed altogether from any project because they just introduce false sense of code quality.

Maintaining the unit tests is another story. As the code evolves, some features are added or changed, the unit tests need to be updated accordingly. If the unit of code to be tested is a monolithic blob of code or very complicated, the maintenance may become an issue. Thus, it is recommended to keep the units of code as small and independent as possible, so that it is practical to maintain unit tests during product's lifetime. It is much easier to test a well defined code with clear interface, functionality and intention, than some obscure blob of code.

If writing unit tests for a particular module looks very hard, it is typically an indication that the code needs refactoring in the first place. In this regard unit tests may also help with improving the overall design & architecture. If developers know that they need to be able to write unit tests for the code they produce, they probably tend to write code that is simple, clean and to easy to understand and maintain.

About unit test frameworks: Try to use any of the well-known, popular, successful unit test framework that suit your needs. In this way the new developers are able to find documentation for the unit test framework, and you do not have to spend time for tutoring, supporting and writing the documentation for the framework per se.

Personally I am having a habit of adding lots of compile-time static asserts, run-time asserts, verify-macros and precondition/postcondition-macros in my code [where possible]. The run-time checks can be enabled/disabled from the project's Makefile(s) per project or per unit. These checks are always enabled during unit testing. Of course, it is important to understand what side-effects mean in this context, and write these checks so that they do not introduce any side-effects. These asserts and other run-time macros have helped with catching hard to find run-time issues in the code, ie. thos wtf-moments. They also help in catching illegal function argument values or corrupted data structures etc. Using these macros it is possible to implement programming by contract quite easily.

[gmb42]

Some standards, e.g. IEC 62443-4-1:2018 Security for industrial automation and control systems - Part 4-1: Secure product development lifecycle requirements require that some aspects of testing is performed by testers who are not implementers and they may not report to the same manager as that of the implementers to give a "departmental" independence.

[SiliconWizard]

But unit testing can be a drag when you start, and it adds considerable time to a project. It can be tempting to just test particularly complex or obscure units, and doing it like that might be right for you - it's better than not doing it at all.

Yes, and unit testing starts paying off right at the moment when someone changes something in the existing code. After any changes, it is typically required to test whether the new changes has introduced some regression in the existing code base (ie. did the new changes brake anything in the existing unit test cases). Adding new features requires writing and maintaining the unit test cases, of course. But the payoff is that after each change there is a way to run unit tests automatically, and find any problems (which are covered by the unit tests, of course) at the very earliest state of development. Alternative method for automated unit testing is manual testing for each unit after modifications. And this will be a real drag in the development cycle.

Yes. Now the question of course is who will write the unit tests. Ideally not the same person as the one who wrote the code being tested. In that regard, unit testing for regression testing is good, but if the persons making modifications to the code are allowed to make modifications to the tests themselves, that's a bit pointless. And yes, I've seen that happen.

As I have understood, in some high-reliability projects it is required that someone else than the original author is writing the unit tests.

That's right. For good reasons.

In normal, less-critical projects it is typical that the original author/authors are writing and maintaining the unit tests as the code is evolving.

Yep it's typical, but not a good thing IMO.

Of course in very small teams (extreme case: a lone developer), it's hard or even impossible to do otherwise.
But to be effective in this case, it requires a lot of self-discipline. You may reply that even so, doing unit testing even if by the same developer is better than not doing it. I don't know if that's necessarily always the case.

Some thought about this: of course, if there is only one developer, there is no question. You may question how useful unit testing is in this case, but assuming the developer has reasonable self-discipline, it's workable. Now in a team of several - even a small team - it's another matter. One "interesting" fact is that some dev methods favor "pair programming" and don't see it as a waste of resource - and yet, given a similar-sized team, they will have more problem with a dev method separating development from testing. That's as "interesting" as it is concerning, but let's not start with hype software dev methods - it's an endless source of fun.

[PlainName]

doing unit testing even if by the same developer is better than not doing it. I don't know if that's necessarily always the case

I think it is. If we take a simple example of Simon's recent desire to convert ADC register value to mV, we might have a function that returns mV given a 16-bit register value. The register will, as per the norm, be a left-shifted 12-bit value.

So, you could just write the code, run it up on the hardware and it looks OK so job's a good 'un. Until you find, some while later out in the field, that certain ADC values cause an integer overflow because of multiplication, or lack of resolution because you divide before multiply. Bum.

Obviously, writing a dummy app to test that function would have been jolly useful, but it's time consuming and you only do it once anyway before it drops off your horizon and the source gets lost.

Unit testing, even just partial testing, and even by the same developer that wrote the code, would likely fix all that. I don't think having the same developer is necessarily bad because the usual problem with that is he doesn't want to break his code. That is, it compiles and runs but there's something at the back of his mind suggesting not to press that button quite that quickly, don't feed it really crap data, etc. But with unit testing the test might be aimed at doing exactly that. You might think the developer will automatically not write breaking tests, but I think it's different at that point because it's still being created and one aim is to make it robust. Once it's done, dusted and running you don't want to have to undo all that work and get pretty protective of it.

[SiliconWizard]

You might think the developer will automatically not write breaking tests,

That's obviously one of the concerns. And they are likely not to even do it fully consciously. But that's not the only problem. The other issue is that if a given developer is not able to correctly implement a given piece of code, what makes you think they'll be able to correctly implement code testing the same functionality? Because - and the example you gave, I think, was telling - if issues in the implementation come from a poor understanding of the problem itself (which happens a lot more often that one may think), or of ways to tackle it, it's also likely that the same person will have difficulties properly testing it.

As I said, unless you have a team of exactly one, I'll always favor splitting tests and development. Of course people can exchange roles from project to project or even within a given project. Not only does it avoid or at least limit the bias factor (not willing to "break" one own's code), but it's also likely that two different persons will have slightly different skill sets and backgrounds and thus will be somewhat complementary.

Sure, with good self-discipline and a rigorous approach, doing it all at the same time is possible. But IME, the end result is almost always disappointing.

Fun fact, one position against this approach (people who implement and people who test are different) I've witnessed a number of times is the following kind of attitude - usually coming from stubborn management: "you code it, you test it". Of course you should test your own code, and when handing it to others for testing, it should at least "work" to the extent of your own abilities (I'm not advocating to just write shit and hand it to test teams so they can sort your bugs out). But then, an outside view will ALWAYS be beneficial, and having other people writing tests for others' code is complementary, but all in all a lot more effective than "static" code reviews.

[Kalvin]

A person who writes or maintains the code needs to be able to perform at least some testing and validation before shipping/committing the code. It is not feasible to expect that "Ok, now it compiles, now you test it", and then hand it over to someone else for testing, will work in real world. The author of the code has to be able to perform testing for the code before the code is eligible to be integrated into a project. Unit testing is one such tool.

An experienced designer knows already typical cases where the code tends to break (off-by-one errors, underflows, overflows, invalid data, unopened resources, allocated resources left behind etc.). Unit tests can be brainstormed with other designers, as they will typically have a good ideas where the code will break and how the interfaces will be abused. Unit tests are good place for testing these kind of typical issues. And the list of typical cases where code will fail will increase as people will gain more experience.

Manual testing vs unit testing? I would say that use unit tests whenever it is possible and feasible. Unit testing will be automatic, so the tests are cheap to run over and over again after each modification during project lifetime. Those tests will be there even after many years, even the original people have left the project. Manual testing is slow, tedious, superficial, expensive, and error-prone [due to human factors].

Developing software requires discipline and good practices. Implementing unit testing requires discipline and good practices, too. Without good discipline and practices, there will not be good software.

Thinking that using exhaustive test suites will increase the quality of the software is just plain wrong. Testing is not a substitute for a good design, good implementation, or good practices. The quality of the software comes from good design, good implementation and good practices. The quality needs to built in from the start of the project, and the quality needs to be maintained through out the project's life-time. Testing is needed to verify that the system or a piece of code is really behaving in a way that it was design to do, and it will fail gracefully when something unexpected will happen. Unit testing is one method for making sure that the software will remain healthy and consistent through out of the project.