Author Topic: Test Driven Development  (Read 22127 times)

0 Members and 1 Guest are viewing this topic.

Offline rollatorwieltje

  • Supporter
  • ****
  • Posts: 571
  • Country: nl
  • I brick your boards.
Re: Test Driven Development
« Reply #25 on: November 23, 2014, 11:19:24 am »
What tools do you use to do unit testing etc when the development platform is not the same as the target platform? As in development is on ye olde x86 PC, and target is an MCU.

Compile for different target, and hope for the best? Compile for correct target and run in an emulator? Run on the actual MCU? If on the MCU what kind of communications? UART? ITM?

You can't do much when your target really cannot run a unit test framework. In those cases we run the unit tests on Windows. Of course you have to write code as platform-independent as possible (mostly by using predefined inttypes and making sure external communications are done in the correct byte order).

Quote
As for macro or not ... why not? I know macro's have their drawbacks, but I've also seen some alternatives that are C++ templates galore. Doesn't that bring the risk that you get to feel all happy and purist about it, but not get anything done? And I don't mean STATIC versus static qualifier. I mean use macro's for your asserts and such. Nothing wrong with that IMO inside the unit test code (not the production code). Or maybe I'm missing something...
When you need to get information from the preprocessor there's no way around it, like the file and line number of an ASSERT or your logging function.
When you don't need information from the preprocessor there's almost always a better way. Macros are often abused as functions or variables, but then you lose every useful protection functions and variables provide (scope, type safety).
One exception I could think of is generating an enum + matching string map, something like this: http://stackoverflow.com/a/147582
 

Offline dannyf

  • Super Contributor
  • ***
  • Posts: 8221
  • Country: 00
Re: Test Driven Development
« Reply #26 on: November 23, 2014, 12:27:11 pm »
Quote
My question is ...

I thought the only reason you develop a piece of code is for customers. Writing a piece of code for testing seems odd, :)
================================
https://dannyelectronics.wordpress.com/
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #27 on: November 23, 2014, 03:16:21 pm »
What tools do you use to do unit testing etc when the development platform is not the same as the target platform? As in development is on ye olde x86 PC, and target is an MCU.

Compile for different target, and hope for the best? Compile for correct target and run in an emulator? Run on the actual MCU? If on the MCU what kind of communications? UART? ITM?

There are a whole range of different techniques for that.

The "high-level" machine independent code (e.g an FSM that accepts events and generates events/actions) can and should be tested in a separate environment, with the events/actions mocked out. Then hope that the cross-compiled code is equivalent.

The "low-level" machine dependent code (e.g. turning event/action to/from pin voltage changes etc) can partially be tested in a separate environment, but in reality has to be tested in the target environment with real pin voltage changes.

It is extremely useful to symbolically log each event/action plus sequence of states:
  • helps debugging: waggle a pin, see the correct event is generated
  • helps in-field fault-finding,
  • helps deflect blame and lawsuits since it can be used to demonstrate that the problem is with other companies' equipment, not your own code

Don't make a fetish out of the tests - if the code visibly works then does it actually need formal tests? A good example of that is a logging interface to a remote computer; if the correct messages are plainly getting to the other end, is exhaustive testing of the comms really beneficial? Can be argued either way, but don't lose sight of the objectives.

Quote
As for macro or not ... why not? I know macro's have their drawbacks, but I've also seen some alternatives that are C++ templates galore. Doesn't that bring the risk that you get to feel all happy and purist about it, but not get anything done? And I don't mean STATIC versus static qualifier. I mean use macro's for your asserts and such. Nothing wrong with that IMO inside the unit test code (not the production code). Or maybe I'm missing something...

C++ templates are the devil's invention which have to be used to as a sticking plaster over the gaps in C++ :)
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline vvanders

  • Regular Contributor
  • *
  • Posts: 124
Re: Test Driven Development
« Reply #28 on: November 23, 2014, 08:56:09 pm »
...

C++ templates are the devil's invention which have to be used to as a sticking plaster over the gaps in C++ :)

That's a bit strong, like everything else in C++ knowing when and where to use/not use a feature is just as important as knowing the feature its self.
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #29 on: November 23, 2014, 10:35:29 pm »
...

C++ templates are the devil's invention which have to be used to as a sticking plaster over the gaps in C++ :)

That's a bit strong, like everything else in C++ knowing when and where to use/not use a feature is just as important as knowing the feature its self.

True. But while developing the template specification, the designers didn't comprehend what they were creating. In particular they refused to believe that the specification was itself a Turing complete language. Until, that is, someone presented a short and valid C++ program that caused the compiler to, very slowly, emit the sequence of prime numbers during compilation.

That was, to me, a clear indication that C++ templates were becoming an uncontrolled and mis-understood monster. I've seen nothing to make me change my mind since then. Start by considering compilation and run-time error messages, then consider the foul interctions with exceptions.

Sure, they are helpful and arguably necessary for C++ programs - but that's because C++ is a bad base for an object-oriented language.

But this shouldn't degenerate into a C++ thread; it is about TDD.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline hamster_nz

  • Super Contributor
  • ***
  • Posts: 2803
  • Country: nz
Re: Test Driven Development
« Reply #30 on: November 23, 2014, 11:06:12 pm »
True. But while developing the template specification, the designers didn't comprehend what they were creating. In particular they refused to believe that the specification was itself a Turing complete language. Until, that is, someone presented a short and valid C++ program that caused the compiler to, very slowly, emit the sequence of prime numbers during compilation.

I just had to find more about that ... the paper is available at http://ubietylab.net/ubigraph/content/Papers/pdf/CppTuring.pdf
Gaze not into the abyss, lest you become recognized as an abyss domain expert, and they expect you keep gazing into the damn thing.
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 19279
  • Country: gb
  • Numbers, not adjectives
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #31 on: November 24, 2014, 10:21:58 am »
True. But while developing the template specification, the designers didn't comprehend what they were creating. In particular they refused to believe that the specification was itself a Turing complete language. Until, that is, someone presented a short and valid C++ program that caused the compiler to, very slowly, emit the sequence of prime numbers during compilation.

I just had to find more about that ... the paper is available at http://ubietylab.net/ubigraph/content/Papers/pdf/CppTuring.pdf

Thanks for that reference, I hadn't seen it before. There are other less formal documents around that I haven't bookmarked. It does drive home the point that C++ templates are "undecidable"!

I wonder why they settled on the recommended recursion depth of 17, not 19? Far too magic for my comfort.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline coppice

  • Super Contributor
  • ***
  • Posts: 8605
  • Country: gb
Re: Test Driven Development
« Reply #32 on: November 27, 2014, 03:34:02 am »
I am just starting out with TDD in embedded world.
I am a noob when it comes to TDD - embedded or otherwise.
Why are you starting out with this? What draws you to it? Programming paradigms are 10 a penny. Why are you interested in this one?
The book TDD in embedded C by James W Grenning is a big help to understand the importance behind TDD but it has it WTF moments where most of the stuff he talks about goes over the head. Fortunately it is more to do with the frameworks he has chosen and not with the principle behind it.
Isn't the TDD principle just a restatement of the infinite monkey principle?
Quote
My question is - How do you guys manage to do TDD - partially or completely. Or is it just a waste of time??
You missed "badly". When most people try most of the software development paradigms they do them badly, so they never really act as a reference for whether the paradigm has any merit. The main reasons for doing them badly are:
  • They don't really grasp the idea, so they fail to implement the spirit of what the methodology is trying to achieve.
  • They treat the tool as their master rather than their servant (the zealot problem).
A lot of methodologies are bunk. A lot have some merit. Most have books and courses behind them, and are making people money, so they massively oversell their benefits. None are actually a panacea, and they need to be seen in that light. There are times when stumbling around with test cases and code is an excellent way to get things clear in your mind. This is usually at the algorithm development stage, rather than the implementation stage. It doesn't seem like a particularly sound general principle, though.
 

Offline krish2487Topic starter

  • Frequent Contributor
  • **
  • Posts: 500
  • Country: dk
Re: Test Driven Development
« Reply #33 on: November 27, 2014, 04:28:52 am »
Quote
Why are you starting out with this? What draws you to it? Programming paradigms are 10 a penny. Why are you interested in this one?


Because it is as good as any. I do not have any formal SW development training. My course of pursuit is purely a outcome of my reading on the net and what I choose to pursue. It is purely a lack of experience. The validity of a course of action is only useful in hindsight. However even for hindsight, I still need to pursue a course.


[/size]Isn't the TDD principle just a restatement of the infinite monkey principle?

[/size]
Quote

[/size]
[/size][size=78%]As I ve said, I really do not have enough experience nor maturity to comment on it. I am just learning. [/size]
[/size]
[/size]
[/size]You missed "badly". When most people try most of the software development paradigms they do them badly, so they never really act as a reference for whether the paradigm has any merit. The main reasons for doing them badly are:[/size]
    • They don't really grasp the idea, so they fail to implement the spirit of what the methodology is trying to achieve.
    • They treat the tool as their master rather than their servant (the zealot problem).

    • [/size]A lot of methodologies are bunk. A lot have some merit. Most have books and courses behind them, and are making people money, so they massively oversell their benefits. None are actually a panacea, and they need to be seen in that light. There are times when stumbling around with test cases and code is an excellent way to get things clear in your mind. This is usually at the algorithm development stage, rather than the implementation stage. It doesn't seem like a particularly sound general principle, though.[/size]
      Quote

      [/size]
      [/size][size=78%]Correct, It is not really very useful when the point of following a paradigm is  missed. I am a noob as I ve stated above. I would like to understand the what, where, why, how correctly rather than blindly following them. I ve started this post to ask more experienced developers their insights and experiences. It will certainly help me avoid some of those issues you ve mentioned. [/size]
      [/size]
      [/size]I do understand what you wrote about verifying an algorithm. I concur. [size=78%]

      [/size]It is, afterall, a tool. How we use it to achieve what we want is upto us. Someone else might see the useful in verifying some specific use cases. Some might want to run the entire range of acceptable and unacceptable values. It is purely dependant on the user. [size=78%]

      [/size]I also think that with some effort, TDD can a extremely useful tool to map out codeflow and code coverage. [size=78%]

      [/q]
    If god made us in his image,
    and we are this stupid
    then....
     

    Offline jlangr

    • Newbie
    • Posts: 1
    Re: Test Driven Development
    « Reply #34 on: November 27, 2014, 07:47:58 am »
    Of course, excercising a path is only effective if the results of exercising the path are visible to the test harness. And that can be extremely difficult to achieve for code that is designed to trap the "should rarely occur" exceptional conditions. Which raises the question: "should you break encapsulation to enable testing?".
    I am currently reading "Modern C++ Programming with Test-Driven Development" by Jeff Langr, and he takes the position that it is better to be able to test than maintaining some academic sense of purity of design, and that it usually isn't a problem. There are of course many ways to handle this. For example, we use macros that can make a static function global when building the tests.

    Bugger "academic purity". His is the kind of sentiment I have previously heard from XP/Agile/TDD religious zealots. Unless you are only considering tiny toy applications, encapsulation is the key to large programs/systems which are reliable, maintainable, and high-performance.

    I suggest you read books by someone that has been at the sharp end of writing commercially important code that has to work reliably and be extended for years.

    Have a look at Jeff Langr's website, and see if you can find what programs he has written (as opposed to at which companies he was a mentor).

    I've gotten this sort of critique before, and I understand where it comes from. I know a number of my peers to whom I might even subject the same condemnation. Personally I wouldn't expect someone to believe me unless I'd been in the trenches--and so I do for a couple years, do consulting for a couple years, back to a dev team, etc. The stuff I write is based on practical application. See http://langrsoft.com/jeff/2012/09/the-consulting-legitimacy-cycle/.

    So yeah, about the last time that someone dinged me for spouting consulting BS, I knew it was time to move on. You'll notice no blog entries since August 2013; I've been too busy working full-time as a lead developer at Outpace Systems since. I'm writing Clojure code with TDD/unit testing (we're not quite so dogmatic as to insist it's all test-driven). Our product is largely invisible--it's an offer engine that processes 100,000s of events per day and crunches through that plus customer stats to produce better offers. It's been deployed to a very happy customer for about a year. The unit testing has made a big difference (although an insufficient amount of good acceptance testing has cost us a bit--which says that, of course, unit testing is only a portion of what you need to do to deliver quality systems).

    As far as the TDD is concerned, it's paid off more often than not, though it can be a bit challenging at time (and we punt sometimes). It's not magic and doesn't solve all problems--that's some of the take on TDD I try to present in the book, how to approach it from a pragmatic stance. I've done plenty of non-TDD, too, and while I can survive that way, it's simply not as effective--or fun--for me and my colleagues, *particularly* on larger systems. The goal is to build a system that's maintainable and keeps the cost of change to a minimum. The ability to know that you can make changes without unwittingly breaking stuff (too easy to do) matters a lot, and is the main reason I do TDD. (You can get there with test-after, but I find it to be harder and less effective.)

    You're correct, encapsulation is important, and core design principles matter a lot. That too is a personal emphasis I have on development. Some of the TDD folks buy into a heavily mock-based approach; I think mocks need to be used carefully, otherwise you get into some nasty dependencies of the tests on private details, and that can really squash your ability to refactor your code when you must. My recommendation is to minimize and isolate any such exposures--but it's still more important to know that the code works. So I allow certain elements to be inspected and possibly overridden. It's yet to bite me.

    Regards,
    Jeff

    PS sorry about the delayed response, but I happened to stumble across this forum while doing a once-a-month-or-two search for my name. If you have a burning concern about any of this, feel free to email me in case I don't find my way back here soon. If you don't buy into any of what I'm claiming, that's ok too; we're happy with it, and it's not for everyone.
     

    Offline IanB

    • Super Contributor
    • ***
    • Posts: 11790
    • Country: us
    Re: Test Driven Development
    « Reply #35 on: November 27, 2014, 08:36:11 am »
    ...

    Wow. I think this is what's known in the vernacular as "click to summon"  :)
     

    Online tggzzz

    • Super Contributor
    • ***
    • Posts: 19279
    • Country: gb
    • Numbers, not adjectives
      • Having fun doing more, with less
    Re: Test Driven Development
    « Reply #36 on: November 27, 2014, 11:07:41 am »
    Much snipped, except where I've a comment...

    The unit testing has made a big difference (although an insufficient amount of good acceptance testing has cost us a bit--which says that, of course, unit testing is only a portion of what you need to do to deliver quality systems).

    Very true.

    I've always been surprised that the "unit testing" zealots only have one, very limited, concept of what constitutes a "unit", and that they don't understand the value of integration and acceptance testing.

    Their "unit" is only a method or class, not the interface to a group of closely coupled classes, nor a library, nor a complete subsystem. Neither do they understand that every non-trivial "unit" test is actually an "integration" test, e.g. where they are integrating their code with standard run-time libraries.

    As for the concept of unit testing the interface to their complete product - well, there's no need since the individual unit tests "prove the whole thing works"! Yeah, right. Especially when you are in a finger-pointing argument with something that is connected to your product!

    Quote
    As far as the TDD is concerned, it's paid off more often than not, though it can be a bit challenging at time (and we punt sometimes). It's not magic and doesn't solve all problems--that's some of the take on TDD I try to present in the book, how to approach it from a pragmatic stance. I've done plenty of non-TDD, too, and while I can survive that way, it's simply not as effective--or fun--for me and my colleagues, *particularly* on larger systems. The goal is to build a system that's maintainable and keeps the cost of change to a minimum. The ability to know that you can make changes without unwittingly breaking stuff (too easy to do) matters a lot, and is the main reason I do TDD. (You can get there with test-after, but I find it to be harder and less effective.)

    Yes, I completely agree with that.

    TDD isn't sufficient, but is sufficiently beneficial that it is almost necessary.

    TDD does, of course, presume that somebody on the team really does know everything that is required to produce a successful product. In most cases that's reasonable, but I've spent a fair part of my life in commercial R&D where the first objective is to determine the reasonable and useful objectives!

    Quote
    You're correct, encapsulation is important, and core design principles matter a lot. That too is a personal emphasis I have on development. Some of the TDD folks buy into a heavily mock-based approach; I think mocks need to be used carefully, otherwise you get into some nasty dependencies of the tests on private details, and that can really squash your ability to refactor your code when you must. My recommendation is to minimize and isolate any such exposures--but it's still more important to know that the code works. So I allow certain elements to be inspected and possibly overridden. It's yet to bite me.

    I've seen stupidly applied unit tests unnecessarily ossify a product. One argument says the only necessary tests are those that confirm the external behaviour - pretty much acceptance/integration tests. Another argument says "internal" unit tests are only scaffolding that helps ensure that the external behaviour hasn't accidentally been affected.
    There are lies, damned lies, statistics - and ADC/DAC specs.
    Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
    Having fun doing more, with less
     

    Offline andersm

    • Super Contributor
    • ***
    • Posts: 1198
    • Country: fi
    Re: Test Driven Development
    « Reply #37 on: December 01, 2014, 07:58:57 pm »
    Except it completely changes what the compiler does.
    Changing a function's linkage does not alter its logic, which is what's being tested. Never mind that the test is building a tiny part of the system in isolation, using a different compiler, on a different computer architecture, running a different OS.

    Quote
    Many low-complexity functions still yield a complex piece of software, just split up more. That doesn't necessarily make it less prone to bugs.
    But it makes the individual pieces easier to test.

    Online tggzzz

    • Super Contributor
    • ***
    • Posts: 19279
    • Country: gb
    • Numbers, not adjectives
      • Having fun doing more, with less
    Re: Test Driven Development
    « Reply #38 on: December 01, 2014, 09:04:06 pm »
    Except it completely changes what the compiler does.
    Changing a function's linkage does not alter its logic, which is what's being tested. Never mind that the test is building a tiny part of the system in isolation, using a different compiler, on a different computer architecture, running a different OS.

    That's only true in a ideal world that does not exist.

    Now consider compiler optimisations which make presumptions (that may or may not be correct) about multithreaded code, aliasing, and all the other poorly defined aspects of C. And then go on to consider the interaction with libraries supplied by other companies. And don't forget the myriad compiler errors that do occur.

    Quote
    Quote
    Many low-complexity functions still yield a complex piece of software, just split up more. That doesn't necessarily make it less prone to bugs.
    But it makes the individual pieces easier to test.

    But such tests are often very uninteresting and unilluminating - and therefore not a particularly good use of developer's time.

    Many, many problems arise from so-called "emergent behaviour" in complex systems. As a trivial example, consider testing all the properties of individual sand grains. Exactly which of those properties implies that piles of sand have a half-angle of 35 degrees?!

    Used intelligently, TDD can be helpful - but too often its benefits are overstated.
    There are lies, damned lies, statistics - and ADC/DAC specs.
    Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
    Having fun doing more, with less
     

    Offline andersm

    • Super Contributor
    • ***
    • Posts: 1198
    • Country: fi
    Re: Test Driven Development
    « Reply #39 on: December 01, 2014, 10:48:49 pm »
    Now consider compiler optimisations which make presumptions (that may or may not be correct) about multithreaded code, aliasing, and all the other poorly defined aspects of C. And then go on to consider the interaction with libraries supplied by other companies. And don't forget the myriad compiler errors that do occur.
    Again, since the focus of the tests is the business logic (in our setup anyway), that's kind of not relevant. And while compiler errors do happen, in practice they're rare enough that it's not much of a concern in day-to-day development.

    Quote
    But such tests are often very uninteresting and unilluminating - and therefore not a particularly good use of developer's time.
    As Jeff Langr posted, one of the purposes of tests is to alert you to unintended changes of behaviour. From that perspective uninteresting tests are also useful.

    Online tggzzz

    • Super Contributor
    • ***
    • Posts: 19279
    • Country: gb
    • Numbers, not adjectives
      • Having fun doing more, with less
    Re: Test Driven Development
    « Reply #40 on: December 02, 2014, 12:26:48 am »
    Now consider compiler optimisations which make presumptions (that may or may not be correct) about multithreaded code, aliasing, and all the other poorly defined aspects of C. And then go on to consider the interaction with libraries supplied by other companies. And don't forget the myriad compiler errors that do occur.
    Again, since the focus of the tests is the business logic (in our setup anyway), that's kind of not relevant. And while compiler errors do happen, in practice they're rare enough that it's not much of a concern in day-to-day development.

    Lucky you.

    Being involved in infrastructure/environment/stack or embedded code, I haven't had that luxury.

    Quote
    Quote
    But such tests are often very uninteresting and unilluminating - and therefore not a particularly good use of developer's time.
    As Jeff Langr posted, one of the purposes of tests is to alert you to unintended changes of behaviour. From that perspective uninteresting tests are also useful.

    Just so, provided the tests are maintained so they accurately reflect the changing requirements.

    And such maintenance is, IMNSHO, at least as difficult as the core logic - and if not done perceptively it can easily either fail or ossify the code.
    There are lies, damned lies, statistics - and ADC/DAC specs.
    Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
    Having fun doing more, with less
     

    Offline Alexei.Polkhanov

    • Frequent Contributor
    • **
    • Posts: 684
    • Country: ca
    Re: Test Driven Development
    « Reply #41 on: December 02, 2014, 05:10:15 am »
    Have you guys ever tried or even researched formal methods and tools? I work in software industry for 20+ years and I have seen some many fads like TDD so my cynicism and skepticism about all something like that is around level 11  :) (if you don't get the reference  here is video - ).

    Ever since beginning back in 1970s or 1980s formal methods were always somewhat impractical and academia toy to play with. However over past 5-6 years I see steady adoption of some tools in industry. When I read an article of John Carmack about application of static analysis tools in gaming then I said  "It is coming, I can hear it!". Here is little list of tools that I have seen used in real world:

    - sal.h in windows SDK. It is a collection of macros for C that allow Windows Driver developers to give hints to compiler and static analyzers like "PREFAST" about function contracts.
    - Coverity
    - JML - Java Modelling Language for Java
    - and then ACSL (ANSI C Specification Language) that used together with Frama-C toolchain - designed SPECIFICALLY FOR EMBEDDED DEVELOPMENT in C

    Last 3 are more formal than simple static checking tools. When you write your code you provide a spec for every function like this:

    Code: [Select]
    /*@ ensures \result >= x && \result >= y;
        ensures \result == x || \result == y;
    */
    int max (int x, int y) { return (x > y) ? x : y; }

    or like this

    Code: [Select]
    /*@ requires \valid(p) && \valid(q);
        ensures *p <= *q;
        ensures (*p == \old(*p) && *q == \old(*q)) ||
                (*p == \old(*q) && *q == \old(*p));
    */
    void max_ptr(int* p, int*q);

    or

    Code: [Select]
    /*@
          requires \valid(root);
          assigns \nothing;
          ensures
           \forall list* l;
             \valid(l) && reachable(root,l) ==>
               \result >= l?>element;
          ensures
           \exists list* l;
             \valid(l) && reachable(root,l) && \result == l?>element;
      */
      int max_list(list* root);

    and tool generates statements in functional language that passed down to another tool that does not test but tries to PROOVE that all  pre- and post conditions are satisfied. If it cannot prove it say - "I don't know". It is expected that you tweak the function till it can find prof in reasonable time. I have heard about good success of this tool for projects in range of 16-20K of C code.
    Frama-C automatically inserts statements that are implied by language - it knows when int can overflow and that you cannot dereference NULL and requires you to avoid undefined behaviors etc. Coverity does more of later for C and C++.



     

    Online tggzzz

    • Super Contributor
    • ***
    • Posts: 19279
    • Country: gb
    • Numbers, not adjectives
      • Having fun doing more, with less
    Re: Test Driven Development
    « Reply #42 on: December 02, 2014, 09:39:07 am »
    Have you guys ever tried or even researched formal methods and tools? I work in software industry for 20+ years and I have seen some many fads like TDD so my cynicism and skepticism about all something like that is around level 11  :)

    Oh, yes indeed. Just so!

    Quote

    Ever since beginning back in 1970s or 1980s formal methods were always somewhat impractical and academia toy to play with. However over past 5-6 years I see steady adoption of some tools in industry. When I read an article of John Carmack about application of static analysis tools in gaming then I said  "It is coming, I can hear it!". Here is little list of tools that I have seen used in real world:

    <snipped>

    Those look like little more than Eiffel-style pre and post conditions. They are nice for toy examples in an academic context, but useless in more industrial contexts. For example, how would you express them for an FFT or inverse FFT function, or even something as simple as calculating the cost of a phone call.

    I'll start taking them seriously when they can be used for something interesting and useful, for example proving that a set of communicating FSMs are deadlock-free, or proving the liveness of some real-time code. Background: the last time I showed a pure mathematician a real-life FSM (from network protocols), he recoiled in horror at the complexity.

    But yes, such techniques can be useful in limited circumstances, e.g. proving the correctness of floating point implementations.
    There are lies, damned lies, statistics - and ADC/DAC specs.
    Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
    Having fun doing more, with less
     

    Offline mrflibble

    • Super Contributor
    • ***
    • Posts: 2051
    • Country: nl
    Re: Test Driven Development
    « Reply #43 on: December 02, 2014, 04:57:50 pm »
    For example, how would you express them for an FFT or inverse FFT function, or even something as simple as calculating the cost of a phone call.
    Interesting question. How would you go about testing the correctness of an FFT implementation? And then the exact same question, but then for testing an FFT implementation of an other platform where you have some room for a limited test environment but by no means as much as on a modern PC. Lets say the target is a cortex M3.

    Quote
    I'll start taking them seriously when they can be used for something interesting and useful, for example proving that a set of communicating FSMs are deadlock-free, or proving the liveness of some real-time code. Background: the last time I showed a pure mathematician a real-life FSM (from network protocols), he recoiled in horror at the complexity.
    Again an interesting problem. I'm not a big fan of following the latest acronym soup, but rather a fan of mix & match. TDD (or rather my limited understanding of TDD) seems to have some nice ideas. And it's not as if these ideas are unique to TDD. The way I look at it is as a collection of ideas from which you grab the ones you like and ditch the ones you don't. All that introductory waffle for the following point: one of the ideas of TDD is that you let the tests drive the design. And lets not get into a whole debate about good or bad on that. Short version: who the fuck thought it a good idea to let tests dictate the product? However, having your tests inform your design seems like a good idea to me. You want a design that is testable. So when you find out during writing your test plan that the way you designed your product makes it really really difficult to test, then maybe it's time to rethink the design. Case in point the tangled FSM web. If you find that testing that sucker gives your local friendly math dude a migraine, then maybe a redesign that is less tangled? And I'm not saying you should have rewritten it, because what do I know. It's your design, and you know it way better than random internet person. I'm just thinking out loud here. Isn't one of the (IMO useful) ideas of TDD to have your tests inform the design? Assuming the aim is to end up with a design that is testable, then you want a design that is as test friendly as possible. Maybe the FSM isn't the best idea, because when implementing protocols, the protocol is a given. But hopefully it gets the point across.

    Incidentally, how do you prove the FSM doesn't get stuck in an unintended state?
     

    Offline Alexei.Polkhanov

    • Frequent Contributor
    • **
    • Posts: 684
    • Country: ca
    Re: Test Driven Development
    « Reply #44 on: December 02, 2014, 06:05:21 pm »
    Those look like little more than Eiffel-style pre and post conditions. They are nice for toy examples in an academic context, but useless in more industrial contexts. For example, how would you express them for an FFT or inverse FFT function, or even something as simple as calculating the cost of a phone call.

    I'll start taking them seriously when they can be used for something interesting and useful, for example proving that a set of communicating FSMs are deadlock-free, or proving the liveness of some real-time code. Background: the last time I showed a pure mathematician a real-life FSM (from network protocols), he recoiled in horror at the complexity.

    But yes, such techniques can be useful in limited circumstances, e.g. proving the correctness of floating point implementations.

    Well Eiffel has them as part of language - something I do in assert() in C/C++. I have very little experience with Eiffel - but it does compile into C code so it is not that impractical.

    Speaking of Frama-C - tool does support floating point, so it tries to check proof obligations for your IEEE 754 floating point code on bit-wise basis. I think if you split FFT into small functions and then prove them one by one - then you will arrive at your destination.

    For large FSM, deadlock checking etc. I had very good experience using PROMELA + SPIN. Promela is a language and SPIN is a tool that runs Promela scripts. It is very easy to use in my opinion. It does not, however, automatically generate C code. It is NOT an academia toy - it was designed by Bell Labs and used by them to check phone station/PBX software. Looks like C.

     

    Online tggzzz

    • Super Contributor
    • ***
    • Posts: 19279
    • Country: gb
    • Numbers, not adjectives
      • Having fun doing more, with less
    Re: Test Driven Development
    « Reply #45 on: December 02, 2014, 06:11:50 pm »
    For example, how would you express them for an FFT or inverse FFT function, or even something as simple as calculating the cost of a phone call.
    Interesting question. How would you go about testing the correctness of an FFT implementation? And then the exact same question, but then for testing an FFT implementation of an other platform where you have some room for a limited test environment but by no means as much as on a modern PC. Lets say the target is a cortex M3.

    Well, you can't fully test an FFT, of course. One technique would be to compare (a large number of) input and output with that from a "golden" implementation written by other people, e.g. Matlab.

    As for the cross-compiled version, you would have to "inject" the inputs and "extract" the outputs from the target environment.

    But neither of those techniques are TDD - which doesn't matter in the slightest except that they are a good example of the limitations of TDD.

    Quote
    Quote
    I'll start taking them seriously when they can be used for something interesting and useful, for example proving that a set of communicating FSMs are deadlock-free, or proving the liveness of some real-time code. Background: the last time I showed a pure mathematician a real-life FSM (from network protocols), he recoiled in horror at the complexity.
    Again an interesting problem. I'm not a big fan of following the latest acronym soup, but rather a fan of mix & match. TDD (or rather my limited understanding of TDD) seems to have some nice ideas. And it's not as if these ideas are unique to TDD. The way I look at it is as a collection of ideas from which you grab the ones you like and ditch the ones you don't. All that introductory waffle for the following point: one of the ideas of TDD is that you let the tests drive the design. And lets not get into a whole debate about good or bad on that. Short version: who the fuck thought it a good idea to let tests dictate the product? However, having your tests inform your design seems like a good idea to me.

    Agreed.

    Quote
    You want a design that is testable.

    No, you want a design that works as intended and solves the problem for which it is designed. Don't forget the old engineering aphorism that "You can't inspect/test quality into a design/product". As you say, it isn't a good idea to let the tests dictate the design.

    Quote
    So when you find out during writing your test plan that the way you designed your product makes it really really difficult to test, then maybe it's time to rethink the design. Case in point the tangled FSM web. If you find that testing that sucker gives your local friendly math dude a migraine, then maybe a redesign that is less tangled? And I'm not saying you should have rewritten it, because what do I know. It's your design, and you know it way better than random internet person. I'm just thinking out loud here. Isn't one of the (IMO useful) ideas of TDD to have your tests inform the design? Assuming the aim is to end up with a design that is testable, then you want a design that is as test friendly as possible. Maybe the FSM isn't the best idea, because when implementing protocols, the protocol is a given. But hopefully it gets the point across.

    I'm always in favour of simplification - indeed I often "joke" that I know things are going well when the number of lines of code reduces.  A beneficial side effect is that it drives naive managers and code counters up the wall :)

    In the case in question, however, the FSM was the protocol, as defined in the standard. The FSM was designed and specified independently of any implementation, and ensured that different equipment from different manufacturers interoperated with each other.

    Incidentally, that is normal practice for networking and telecom systems: they don't care how they are implemented, but they do care about the externally visible behaviour.

    Quote
    Incidentally, how do you prove the FSM doesn't get stuck in an unintended state?

    By careful thought. :(
    There are lies, damned lies, statistics - and ADC/DAC specs.
    Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
    Having fun doing more, with less
     

    Online tggzzz

    • Super Contributor
    • ***
    • Posts: 19279
    • Country: gb
    • Numbers, not adjectives
      • Having fun doing more, with less
    Re: Test Driven Development
    « Reply #46 on: December 02, 2014, 06:18:08 pm »
    I think if you split FFT into small functions and then prove them one by one - then you will arrive at your destination.

    I'm skeptical, and would require a demonstration that TDD can lead you to that nirvana. In other cases, I believe TDD can be very helpful; indeed I have helped introduce such disciplines into companies.

    Quote
    For large FSM, deadlock checking etc. I had very good experience using PROMELA + SPIN. Promela is a language and SPIN is a tool that runs Promela scripts. It is very easy to use in my opinion. It does not, however, automatically generate C code. It is NOT an academia toy - it was designed by Bell Labs and used by them to check phone station/PBX software. Looks like C.

    I'm not aware of those, but will look into them if I have suitable problems in future.

    There is, of course, the standard question of how to ensure/demonstrate that the implementation matches the promela/spin executable specification.
    There are lies, damned lies, statistics - and ADC/DAC specs.
    Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
    Having fun doing more, with less
     

    Offline Alexei.Polkhanov

    • Frequent Contributor
    • **
    • Posts: 684
    • Country: ca
    Re: Test Driven Development
    « Reply #47 on: December 02, 2014, 07:44:18 pm »
    For code generation maybe you can try SPIN "Refinement" - I never tried it but looks like it is pretty mature project. https://github.com/codelion/SpinR

     

    Offline andersm

    • Super Contributor
    • ***
    • Posts: 1198
    • Country: fi
    Re: Test Driven Development
    « Reply #48 on: December 03, 2014, 12:12:02 am »
    Being involved in infrastructure/environment/stack or embedded code, I haven't had that luxury.
    Are you really trying to claim that you run into so many compiler bugs that unit tests are useless? Because you're either overstating the problem or need to switch compiler vendor post-haste.

    Online tggzzz

    • Super Contributor
    • ***
    • Posts: 19279
    • Country: gb
    • Numbers, not adjectives
      • Having fun doing more, with less
    Re: Test Driven Development
    « Reply #49 on: December 03, 2014, 12:31:54 am »
    Being involved in infrastructure/environment/stack or embedded code, I haven't had that luxury.
    Are you really trying to claim that you run into so many compiler bugs that unit tests are useless? Because you're either overstating the problem or need to switch compiler vendor post-haste.

    ?!?

    I suggest you go back and (re)read the context in which I made my statement. Even then I don't see how you could infer that was the case.

    It would also help you (and therefore your audience) if you didn't delete the relevant context.
    There are lies, damned lies, statistics - and ADC/DAC specs.
    Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
    Having fun doing more, with less
     


    Share me

    Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
    Smf