Author Topic: Test Driven Development  (Read 22129 times)

0 Members and 1 Guest are viewing this topic.

Offline krish2487Topic starter

  • Frequent Contributor
  • **
  • Posts: 500
  • Country: dk
Test Driven Development
« on: November 20, 2014, 07:35:08 am »
Hello.


I am just starting out with TDD in embedded world.
I am a noob when it comes to TDD - embedded or otherwise.




The book TDD in embedded C by James W Grenning is a big help to understand the importance behind TDD but it has it WTF moments where most of the stuff he talks about goes over the head. Fortunately it is more to do with the frameworks he has chosen and not with the principle behind it.


My question is - How do you guys manage to do TDD - partially or completely. Or is it just a waste of time??


If yes, then what frameworks do you guys use. I ask because because I am leaning towards CUnit (mostly because it is easy to follow along along with a decent tutorials on the interweb).
If god made us in his image,
and we are this stupid
then....
 

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #1 on: November 20, 2014, 11:08:27 am »
TDD is valuable providing you understand its limitations - which many religious zealots don't.

There's an old engineering adage: "you can't inspect/test quality into a product".

TDD is critically dependent on the quality and completeness of the tests. Writing good tests is often more difficult and time consuming than the design itself - and can be practically impossible. Consider, for example, trying to test that your design copes with metastability in synchronizers, or that the correct database ACID properties are being used.

TDD can help you to build the "product right", but cannot ensure that you are building the "right product". (Validation vs verification)

TDD only helps with the things you test; guess what happens if you don't test important aspects of your product!
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 
The following users thanked this post: FlyingDutch, cobusve

Offline andersm

  • Super Contributor
  • ***
  • Posts: 1198
  • Country: fi
Re: Test Driven Development
« Reply #2 on: November 20, 2014, 11:15:19 am »
At my work we use Unity and CMock (I believe they hired Grenning to help set the system up, but that was before I joined). We rarely do TDD "by the book", but we are big users of unit tests. They are run both on workstations while developing, and on our CI servers after committing to version control.

The best advice I can give is to write your code with testing in mind. Our test suite has been retrofitted onto an existing codebase, and the dependency spaghetti has made the tests both more difficult to understand and slower to execute. Our complete unit test suite (currently about 5900 tests, for a family of products) takes almost fifteen minutes to build and run on pretty beefy quad-core machines. We have set up macros in our IDEs that can run a single unit test file, or all the tests for one project, which helps a lot.

Offline krish2487Topic starter

  • Frequent Contributor
  • **
  • Posts: 500
  • Country: dk
Re: Test Driven Development
« Reply #3 on: November 21, 2014, 03:48:16 am »
Sound advice from both!
More interesting to see that not many have actually replied to the question.


Thank you.

[/size]So for your typical run-of-mill small projects, as a percentage, how much time do you spend visualizing/arriving at the tests for the project. [size=78%][/size]I have a more specific question I wanted to ask. If you have a given module that has to be tested, whats the best compromise on the spectrum of test inputs that you typically run. I ll elaborate, if a particular function takes 1 argument as input and 1 as a return value, and the input can have a 100 "correct" values, do you sweep the test across the 100 values and record the data or is it more of a common practice to test say, boundary values, a typical value and a exception value to see that the function and test pair function correctly. Thank you for your time!!![size=78%]
If god made us in his image,
and we are this stupid
then....
 

Offline andersm

  • Super Contributor
  • ***
  • Posts: 1198
  • Country: fi
Re: Test Driven Development
« Reply #4 on: November 21, 2014, 09:05:31 am »
Exhaustive tests would be far too slow. A good guideline would be a representative case, plus any corner cases. While there is a bit of redundancy there, test cases also serve as documentation which makes it worthwhile to explicitly include the "normal" case. Also keep in mind that the refactoring step is an integral part of the TDD loop, so while developing a feature you can have as complex tests as you like, as long as you clean them up before moving on.

We measure the code coverage of our unit tests, and while that says nothing about the quality of the tests, it does clearly show if there are some code paths that aren't excercised at all.

Offline krish2487Topic starter

  • Frequent Contributor
  • **
  • Posts: 500
  • Country: dk
Re: Test Driven Development
« Reply #5 on: November 21, 2014, 09:34:40 am »
Thank you!!!
That clears up a lot of doubts I ve had for sometime now.


Again, thank you for your time and patience explaining how it is done in the industry.
 :)
If god made us in his image,
and we are this stupid
then....
 

Offline krish2487Topic starter

  • Frequent Contributor
  • **
  • Posts: 500
  • Country: dk
Re: Test Driven Development
« Reply #6 on: November 21, 2014, 11:51:49 am »
You have to be very careful with TDD. There is a tendency to code to pass the tests, not to produce a robust system. The idea is that the tests are well designed enough to prevent that, but in practice they never are.


I can see what you are pointing at. I caught myself more than once trying to design a framework that would pass the test. Not be inherently robust as you said.


So how do you manage to avoid such issues while coding??


I am asking because I would like to learn the attitude behind coding safely. I am being deliberately vague and open ended because I would like to know the members experiences and coding process. Some, if not all, will be helpful.
If god made us in his image,
and we are this stupid
then....
 

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #7 on: November 21, 2014, 05:02:53 pm »
We measure the code coverage of our unit tests, and while that says nothing about the quality of the tests, it does clearly show if there are some code paths that aren't excercised at all.

I wish everybody understood that; I've worked on a project where people seemed not to understand/care about that point.

Of course, excercising a path is only effective if the results of exercising the path are visible to the test harness. And that can be extremely difficult to achieve for code that is designed to trap the "should rarely occur" exceptional conditions. Which raises the question: "should you break encapsulation to enable testing?".
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #8 on: November 21, 2014, 05:05:36 pm »
You have to be very careful with TDD. There is a tendency to code to pass the tests, not to produce a robust system. The idea is that the tests are well designed enough to prevent that, but in practice they never are.

I've seen people seriously argue that the code works because the code passes the tests. Nonsense of course, but it can be a difficult to persuade people otherwise!
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline Alexei.Polkhanov

  • Frequent Contributor
  • **
  • Posts: 684
  • Country: ca
Re: Test Driven Development
« Reply #9 on: November 21, 2014, 06:24:50 pm »
I am a little famous for saying something like that in forums about 15 years ago:

"If test fails don't fix the software - fix the developers, and more inspections, code reviews, training etc."

I have not heard of TDD when I said that. Nothing changed since then and in many cases situation is much worth.

You cannot fix software after defects are found. Proof of that is not so obvious and lies in statistics. First of all adding more tests only gives you benefits in terms of Noise to Signal ratio NOT - number of defects found maybe just fine even in first test. Let's say your tests gives yield of 10% so that you know you will find 1 defect out of 10. Knowing it is enough to estimate that in 1000 lines of code you have 10 bugs if your tests reveal one. Now if you add more tests you will find 2 - but why not simply multiply number of defects from previous test by 2?

Second point is easier to understand, because all developers intuitively know it already. When you attempt to fix defects in a software after they were found by tests you make "modification" as oppose to "writing new code". Every modification will result in more defects than initial coding. At some point these two curves intersect and you adding exactly same number of defects as you trying to remove. So you going in direction of replacing easy to find bugs by those that your tests don't catch but otherwise you are not improving anything.

Why TDD is accepted in some places? Well first of all many managers are familiar with this maths but because it is part of curriculum. They often refer to it as "clean room" approach or Toyota method - but they always add that it is not practical HERE. Second reason is economics - software is made by people, not machines. If you invest into new plasma etching machine that reduces your defects by 5% you will get your money back in 1 year for example. If you invest same money into training your developers, improving process etc - they will leave to work for another shop and your investment is wasted.




 

Offline andersm

  • Super Contributor
  • ***
  • Posts: 1198
  • Country: fi
Re: Test Driven Development
« Reply #10 on: November 21, 2014, 07:40:15 pm »
Of course, excercising a path is only effective if the results of exercising the path are visible to the test harness. And that can be extremely difficult to achieve for code that is designed to trap the "should rarely occur" exceptional conditions. Which raises the question: "should you break encapsulation to enable testing?".
I am currently reading "Modern C++ Programming with Test-Driven Development" by Jeff Langr, and he takes the position that it is better to be able to test than maintaining some academic sense of purity of design, and that it usually isn't a problem. There are of course many ways to handle this. For example, we use macros that can make a static function global when building the tests.

Offline nctnico

  • Super Contributor
  • ***
  • Posts: 26751
  • Country: nl
    • NCT Developments
Re: Test Driven Development
« Reply #11 on: November 21, 2014, 07:52:58 pm »
Macros for function definitions.... yuk...

What I have learned in the past is to write code which is tolerant to getting wrong parameters from other parts of the program. IOW trying to compartmentalize the software. Especially on boundaries between objects and APIs I do checks on the parameters which are passed to a function. That way a wrong value causes the program to do nothing or produce an error instead of something which is (slightly) wrong or cause something else/unrelated to fail because data gets overwritten. Either way it is easier to spot during testing and because the failure doesn't cascade the description of the failure is most likely to point to the offending piece of code. And even if the problem goes unnoticed the bug doesn't cascade into unexpected behaviour or crashes.
There are small lies, big lies and then there is what is on the screen of your oscilloscope.
 

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #12 on: November 21, 2014, 08:21:40 pm »
Of course, excercising a path is only effective if the results of exercising the path are visible to the test harness. And that can be extremely difficult to achieve for code that is designed to trap the "should rarely occur" exceptional conditions. Which raises the question: "should you break encapsulation to enable testing?".
I am currently reading "Modern C++ Programming with Test-Driven Development" by Jeff Langr, and he takes the position that it is better to be able to test than maintaining some academic sense of purity of design, and that it usually isn't a problem. There are of course many ways to handle this. For example, we use macros that can make a static function global when building the tests.

In which case such tests will have little value, and should be scrapped before you lull yourself into a false sense of security. Why? You simply aren't testing the software product - you are testing  a different (albeit similar at one level) product.

It can be guaranteed that with C/C++ the compiler will produce significantly different code if you change the visibility. Root cause: the pessimisations the compiler must insert when it can't prove there is no aliasing.

I suggest you read books by someone that knows how C/C++ compilers generate code for real-world non-toy programs.

Bugger "academic purity". His is the kind of sentiment I have previously heard from XP/Agile/TDD religious zealots. Unless you are only considering tiny toy applications, encapsulation is the key to large programs/systems which are reliable, maintainable, and high-performance.

I suggest you read books by someone that has been at the sharp end of writing commercially important code that has to work reliably and be extended for years.

Have a look at Jeff Langr's website, and see if you can find what programs he has written (as opposed to at which companies he was a mentor).
« Last Edit: November 21, 2014, 08:29:47 pm by tggzzz »
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline rollatorwieltje

  • Supporter
  • ****
  • Posts: 571
  • Country: nl
  • I brick your boards.
Re: Test Driven Development
« Reply #13 on: November 21, 2014, 08:26:58 pm »
One fundamental issue I have with TDD is that the tests are usually written by the same person who does the implementation. The unit tests often are exact implementations of how the object / function should be used. This is acceptable if the complexity of the function is low, but usually bugs are not found in low-complexity functions. Bugs usually come from assumptions about the system, and the programmer would make the exact same assumptions when writing the test.
You could get rid of all assumptions by designing a solid software architecture, but at some point you have to make a trade-off between writing code without assumptions (causing lots of boilerplate code) and writing simple software that just does exactly what you want.

A small example:
Code: [Select]
void MyMachine::openDoor()
{
   hardwareLayer->openDoor();
}

/**
  Open the door!
  @pre pin 1 must be set to output.
**/
HardwareLayer::openDoor()
{
  ASSERT(pinDirection(1) == OUTPUT);
  setPin(1, high);
}

TEST(MyMachine, openDoor)
{
  EXPECT_CALL(hardwareLayerTest.openDoor());
  myMachine.openDoor(); 
}

TEST(HardwareLayer, openDoor)
{
  //Test when pin is input instead of output
  hardwareLayerTest.setPinDirection(1, INPUT);
  EXPECT_ASSERT(myMachine->openDoor());

  //Test when pin is output
  hardwareLayerTest.setPinDirection(1, OUTPUT);

  EXPECT_CALL(hardwareLayerTest, setPin(1, HIGH));
  myMachine->openDoor();
}


The above code yields 100% coverage and 100% test pass, but it still isn't very robust. There won't be a bug in this code, the bug wil manifest itself when somebody during the execution of the program abusively calls setPinDirection(). Even then it still works as intended.
 You could design away the setPindirection() function with some clever OO stuff, but how far do you want to go? Do you really want 100 lines of code to open a damn door? This case is relatively easy to make more robust, but things get more complex really fast... Also be aware that adding code to cover a potential bug also may introduce new potential bugs.

I'm not saying unit tests are completely useless, because they aren't. You often write test functions anyway when writing software. But don't consider unit tests with code coverage the holy grail of writing bug-free software.
 

Offline andersm

  • Super Contributor
  • ***
  • Posts: 1198
  • Country: fi
Re: Test Driven Development
« Reply #14 on: November 21, 2014, 09:07:56 pm »
Macros for function definitions.... yuk...
Well, instead of "static" you write "STATIC". Hardly the end of the world.

In which case such tests will have little value, and should be scrapped before you lull yourself into a false sense of security. Why? You simply aren't testing the software product - you are testing  a different (albeit similar at one level) product.
These are unit tests. By definition you are not testing "the software product." There are other tests for that.

This is acceptable if the complexity of the function is low, but usually bugs are not found in low-complexity functions.
The obvious solution to that is not to write very complex functions.

Offline rollatorwieltje

  • Supporter
  • ****
  • Posts: 571
  • Country: nl
  • I brick your boards.
Re: Test Driven Development
« Reply #15 on: November 21, 2014, 09:28:27 pm »
Macros for function definitions.... yuk...
Well, instead of "static" you write "STATIC". Hardly the end of the world.

Except it completely changes what the compiler does.

Quote
In which case such tests will have little value, and should be scrapped before you lull yourself into a false sense of security. Why? You simply aren't testing the software product - you are testing  a different (albeit similar at one level) product.
These are unit tests. By definition you are not testing "the software product." There are other tests for that.
The usual thing to to is build the software as a static library and link it either against the unit test executable or the release executable. Changing preprocessor options between your test and release just invalidates the whole concept of testing what you're going to release.

edit: This is obviously not really possible for platforms that cannot run a unit test framework. But still, avoid the use of preprocessor definitions. It just changes the program without making it obvious.

This is acceptable if the complexity of the function is low, but usually bugs are not found in low-complexity functions.
The obvious solution to that is not to write very complex functions.
[/quote]
Many low-complexity functions still yield a complex piece of software, just split up more. That doesn't necessarily make it less prone to bugs. Just what I said, adding boilerplate introduces new opportunities for making bugs. It's a delicate balance.
« Last Edit: November 21, 2014, 09:45:10 pm by rollatorwieltje »
 

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #16 on: November 21, 2014, 11:15:38 pm »
In which case such tests will have little value, and should be scrapped before you lull yourself into a false sense of security. Why? You simply aren't testing the software product - you are testing  a different (albeit similar at one level) product.
These are unit tests. By definition you are not testing "the software product." There are other tests for that.

Don't be silly.

Read what rollatorwieltje writes in his next post - he correctly expands what I merely hinted at, viz:
Quote
Except it completely changes what the compiler does.
Changing preprocessor options between your test and release just invalidates the whole concept of testing what you're going to release.
But still, avoid the use of preprocessor definitions. It just changes the program without making it obvious.
Just what I said, adding boilerplate introduces new opportunities for making bugs. It's a delicate balance.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline IanB

  • Super Contributor
  • ***
  • Posts: 11790
  • Country: us
Re: Test Driven Development
« Reply #17 on: November 21, 2014, 11:29:14 pm »
is it more of a common practice to test say, boundary values, a typical value and a exception value to see that the function and test pair function correctly

This. Choose the smallest number of tests that will exercise all of the code paths and all the permutations of logical tests in the code unit being tested. Expect this number to be fewer than ten.

If you have too many permutations to test it is likely your code is too complex and you should try to simplify it.

Which highlights an important aspect of TDD. TDD is to help you write better code. It makes you keep your eye on the ball and guards against creating the "stream of consciousness, big ball of yarn" that so often is the outcome of coding in the moment without a plan.
 

Offline krish2487Topic starter

  • Frequent Contributor
  • **
  • Posts: 500
  • Country: dk
Re: Test Driven Development
« Reply #18 on: November 22, 2014, 04:52:56 am »
Thank you Ian.


Interesting to see the discussion and experiences of others
If god made us in his image,
and we are this stupid
then....
 

Offline vvanders

  • Regular Contributor
  • *
  • Posts: 124
Re: Test Driven Development
« Reply #19 on: November 22, 2014, 05:10:24 am »
TDD is great for building APIs that make sense and are approachable. As with everything it can be taken to extremes.

I'm a big fan of introducing a new unit test each time a regression is found that exercises that regression. It means your tests are coming from real-world, potentially complex scenarios and aren't "tests for test sake".
 

Offline mrflibble

  • Super Contributor
  • ***
  • Posts: 2051
  • Country: nl
Re: Test Driven Development
« Reply #20 on: November 22, 2014, 04:24:07 pm »
What tools do you use to do unit testing etc when the development platform is not the same as the target platform? As in development is on ye olde x86 PC, and target is an MCU.

Compile for different target, and hope for the best? Compile for correct target and run in an emulator? Run on the actual MCU? If on the MCU what kind of communications? UART? ITM?

As for macro or not ... why not? I know macro's have their drawbacks, but I've also seen some alternatives that are C++ templates galore. Doesn't that bring the risk that you get to feel all happy and purist about it, but not get anything done? And I don't mean STATIC versus static qualifier. I mean use macro's for your asserts and such. Nothing wrong with that IMO inside the unit test code (not the production code). Or maybe I'm missing something...

 

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #21 on: November 22, 2014, 08:26:23 pm »
While acknowledging that TDD can have some advantages, I disagree with your statements below.

If you have too many permutations to test it is likely your code is too complex and you should try to simplify it.

I don't think that's helpful, except in the limited cases where you test trivially simple segments of code. The consequent canonical example, which I have seen too often, is unit tests for getter and setter methods - which is especially dangerous/nonsensical if the getter/setter method's visibility has been widened simply to allow those unit tests!

In addition, some algorithms simply cannot be decomposed to trivial units - an FFT function, for example.

Quote
Which highlights an important aspect of TDD. TDD is to help you write better code. It makes you keep your eye on the ball and guards against creating the "stream of consciousness, big ball of yarn" that so often is the outcome of coding in the moment without a plan.

The downside is that it encourages focussing on tiny segments of code at the expense of seeing/understanding/testing larger cohesive units. Effectively throwing the "trees" into focus and the "wood" out of focus.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline IanB

  • Super Contributor
  • ***
  • Posts: 11790
  • Country: us
Re: Test Driven Development
« Reply #22 on: November 22, 2014, 09:40:33 pm »
While acknowledging that TDD can have some advantages, I disagree with your statements below.

Well the world is too complicated to be described entirely by simple rules. Note the qualifiers "likely too complex" and "try to simplify". I have seen many cases where the code really is too complex and really does need simplifying. So it's a thought to keep in mind.

My personal area of specialism is with numerical code dealing with floating point and complex algorithms (not unlike FFT). It certainly is very challenging to provide adequate test coverage that will assure such code never fails. On the other hand, decomposition and modularization helps enormously towards that goal.

To your last comment, many would argue that TDD should help you to focus on the forest and achieve a good overall structure. But there is no substitute for experience. Even with TDD, lack of experience will show through.
 

Offline mrflibble

  • Super Contributor
  • ***
  • Posts: 2051
  • Country: nl
Re: Test Driven Development
« Reply #23 on: November 23, 2014, 05:11:59 am »
My personal area of specialism is with numerical code dealing with floating point and complex algorithms (not unlike FFT). It certainly is very challenging to provide adequate test coverage that will assure such code never fails. On the other hand, decomposition and modularization helps enormously towards that goal.

I was wondering about that. Suppose you have a function that spits out a non-LTI time-series, how do you unit test that? Create a bunch of test cases, simulate them in matlab so that you know the input and outputs, and then generate test C code using those inputs and outputs?
 

Offline tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #24 on: November 23, 2014, 09:55:28 am »
It feels like we are in violent agreement.

I have encountered too many zealots that think and teach that following a set of incantations process will lead to a good result. Clearly you don't fall into that category.

Mature engineering judgement rules!

While acknowledging that TDD can have some advantages, I disagree with your statements below.

Well the world is too complicated to be described entirely by simple rules. Note the qualifiers "likely too complex" and "try to simplify". I have seen many cases where the code really is too complex and really does need simplifying. So it's a thought to keep in mind.

My personal area of specialism is with numerical code dealing with floating point and complex algorithms (not unlike FFT). It certainly is very challenging to provide adequate test coverage that will assure such code never fails. On the other hand, decomposition and modularization helps enormously towards that goal.

To your last comment, many would argue that TDD should help you to focus on the forest and achieve a good overall structure. But there is no substitute for experience. Even with TDD, lack of experience will show through.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf