Author Topic: Test Driven Development  (Read 12669 times)

0 Members and 1 Guest are viewing this topic.

Online rhb

  • Super Contributor
  • ***
  • Posts: 2386
  • Country: us
Re: Test Driven Development
« Reply #75 on: January 22, 2019, 05:17:13 am »
The function of testing is not to prove something is correct.

Many people believe that is the function of testing, and it isn't trivial to disabuse them of that notion. :(

A test can show that a bug exists.
[snip]

Most importantly it can show it *is* fixed and stays that way.   

Beyond rudimentary unit tests, that is what I consider the most important objective. Regression testing catches complementary errors which result in an error when one is fixed, but not the other.  I've seen that happen quite a lot.  It's the main reason I am a big fan of regression testing.

It was a real surprise the first time a small change in module B resulted in a unit test failing for a seemingly unrelated function in module Y.  If you've only written a few thousand lines of code between system build & test  cycles it's not too hard to find.  I don't want to think what it would be like if you had 100K lines of changes and new code.

Despite knowing better, I bought a bunch of books on proving correctness.  None actually proved useful for anything other than an example of a delusion.
 

Offline snarkysparky

  • Regular Contributor
  • *
  • Posts: 158
  • Country: us
Re: Test Driven Development
« Reply #76 on: January 22, 2019, 05:59:44 am »
So basically what can be done to test software is to check it for known issues like edge cases and invalid input ...etc, but bug free operation cannot be guaranteed unless the software is so simple that exhaustive test of all input sequences can be performed.

Lots of people don't like to hear that. 

 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 8601
  • Country: gb
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #77 on: January 22, 2019, 06:01:05 am »


You should start by understanding the differences between "designing" and "testing".

Once you have that under your belt, you can move on to understanding the difference between "inspection" and "formal proof".

I googled "designing vs testing" and nothing came up.  Is it a common topic ?

I left out a "no" in original post.  I meant.
 somewhere that there is no way to prove software correctness


It's of interest to me now because I am coding for some people that demand 'proof' for lack of errors.

It sounds like your role is "project scapegoat".

Demand they prove there are no errors in their specification, or run away.
« Last Edit: January 22, 2019, 06:12:28 am by tggzzz »
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 8601
  • Country: gb
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #78 on: January 22, 2019, 06:05:15 am »
So basically what can be done to test software is to check it for known issues like edge cases and invalid input ...etc, but bug free operation cannot be guaranteed unless the software is so simple that exhaustive test of all input sequences can be performed.

Lots of people don't like to hear that.

Getting a specification that is bug free is at least as hard as getting code that implements the specification.

"Verification" is not the same as "validation". One aims to "do the right thing", the other aims to "do the thing right".
« Last Edit: January 22, 2019, 06:07:00 am by tggzzz »
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Online rhb

  • Super Contributor
  • ***
  • Posts: 2386
  • Country: us
Re: Test Driven Development
« Reply #79 on: January 22, 2019, 08:09:51 am »
So basically what can be done to test software is to check it for known issues like edge cases and invalid input ...etc, but bug free operation cannot be guaranteed unless the software is so simple that exhaustive test of all input sequences can be performed.

Lots of people don't like to hear that.

Getting a specification that is bug free is at least as hard as getting code that implements the specification.


A substantial part of the programmer's job is to make sure the specification is correct.  I am really sick of programmers who blame the customer when the real problem was the programmer never bothered to find out what the actual problem was that the customer wanted solved.

Most of the times that people came in my office wanting a program written, after I asked them a few questions, they suddenly exclaimed, "I know how to do that!" and ran off. Typically I'd just solved a $30K problem for $20.
 

Offline dmills

  • Super Contributor
  • ***
  • Posts: 1545
Re: Test Driven Development
« Reply #80 on: January 22, 2019, 08:26:04 am »
A substantial part of the programmer's job is to make sure the specification is correct.  I am really sick of programmers who blame the customer when the real problem was the programmer never bothered to find out what the actual problem was that the customer wanted solved.
What **REALLY** helps is having at least some of the team have some serious task domain chops, it is MUCH easier to figure out what the customer actually needs (as well as where the problems in the spec are) if you know how to do the customers job to at least a reasonable level.
There is however a somewhat popular view among the 'hard of thinking' end of management that programmers are interchangeable, which makes achieving this harder then it should be. 

Regards, Dan.
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 8601
  • Country: gb
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #81 on: January 22, 2019, 08:31:15 am »
So basically what can be done to test software is to check it for known issues like edge cases and invalid input ...etc, but bug free operation cannot be guaranteed unless the software is so simple that exhaustive test of all input sequences can be performed.

Lots of people don't like to hear that.

Getting a specification that is bug free is at least as hard as getting code that implements the specification.


A substantial part of the programmer's job is to make sure the specification is correct.  I am really sick of programmers who blame the customer when the real problem was the programmer never bothered to find out what the actual problem was that the customer wanted solved.

Most of the times that people came in my office wanting a program written, after I asked them a few questions, they suddenly exclaimed, "I know how to do that!" and ran off. Typically I'd just solved a $30K problem for $20.

We are in violent agreement.

That attitude is valuable in many cases and in many ways. And that specifically includes this forum. There's a lot of "applied philosophy" in systems development ("what do you mean by 'fast'" and "does a vehicle include horses"?).

The only thing I'll note is that customers can become hostile when asked to explain themselves; such twats dislike their ignorance being exposed ("it is obvious; just do it") and/or having their wooly thinking exposed.
« Last Edit: January 22, 2019, 08:38:41 am by tggzzz »
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline coppice

  • Super Contributor
  • ***
  • Posts: 4173
  • Country: gb
Re: Test Driven Development
« Reply #82 on: January 22, 2019, 09:37:40 am »
A substantial part of the programmer's job is to make sure the specification is correct.  I am really sick of programmers who blame the customer when the real problem was the programmer never bothered to find out what the actual problem was that the customer wanted solved.
Programmers aren't usually in a good position to know what the quality of a requirement specification really is. They just don't know enough about the details of what the customer would consider correct behaviour. In the late 80s there was a burst of activity with specification by simulation techniques. I saw these result in some of the best specified projects I have seen. It went out of fashion, though. It isn't always easy to get a customer to treat a simulation seriously, and put effort into playing with it until it does what they expect in every circumstance they check. However, once they start finding things they didn't expect they tend to warm to the idea, and it becomes productive.
 

Online rhb

  • Super Contributor
  • ***
  • Posts: 2386
  • Country: us
Re: Test Driven Development
« Reply #83 on: January 22, 2019, 01:00:15 pm »

That attitude is valuable in many cases and in many ways. And that specifically includes this forum. There's a lot of "applied philosophy" in systems development ("what do you mean by 'fast'" and "does a vehicle include horses"?).

The only thing I'll note is that customers can become hostile when asked to explain themselves; such twats dislike their ignorance being exposed ("it is obvious; just do it") and/or having their wooly thinking exposed.

I had one instance where  per my standard practice I wrote up a 1 page requirement and gave it to the requester.  I went back a couple of days later to discuss it.  "I don't have time for that paper work."  Not something I generally tolerate, but I let it pass.  I had estimated it as 2 weeks work.  So I went ahead  with it.  Meanwhile another guy across the hall from me was doing a very much simpler version of the same thing. Probably 1/4 of the effort or less.  The math for what I'd been asked to do was complex and prone to instability.  The complexity is just an extra day or so, but unstable equations can take days to develop a reliable implementation.

So two weeks after his dismissive comment about the 1 page specification he wanted the program.  I told him, it's not finished.  He's all in a lather because his project will be late, so I told him about Wen Long's simplified version and suggested he might be able to get by with that.

A couple of days later we have a very tense meeting in his boss' office.  The two of them are all over my case.

"What's going on, You said two weeks work"   

"I said two weeks effort.  I didn't say it would be ready in 2 weeks.  I have a lot of other people I support."

"I don't know if we can continue to use you."

blah, blah , blah

When I got back to my office, there was a message from my boss' secretary that he and his boss wanted to see me in his office right away.  So I go down the hall expecting to be fired.

They wanted to see me to give me an award for the excellence of my work and how responsive I was to staff requests.  I was never able to find out what had initiated it. I just solved the problems I encountered and looked for problems that might arise.  I was a contractor, so the company award was rather unusual.  The first guy's project was never completed as he found Wen Long's simplified version adequate.

The first guy did not like me.  Previously he was buying a Linux cluster and I attended the meeting with the sales people and his boss.  During the discussion I raised the issue of power consumption.  This guy had a corner office with all glass walls in Houston.  The sales people said it's not a problem. It runs on a regular 20 A 120 circuit.  Problem was it dissipated about 1800 watts.  They bought it and put it in his office.  In 2-3 days they moved it to the computer room.  After he'd been made to sweat for a few days :)

The most ironic part of the entire tale is his boss, who told me to my face at my interview that he didn't want to hire an oil industry programmer and who had hired a Indian outfit to develop a system for running jobs on workstations evenings and weekends became my primary client.  I attended his weekly staff meetings.  At one of these, during a vendor presentation one of them remarked that they were considering hiring some Indians.  I was gob smacked when he said, "Well, you get what you pay for." or similar.  All the work the Indians had done was discarded after burning through $100K or so at their cheap hourly rates.

I was returning to my office after visiting a senior seismic processor when two of the Indians called out to me.  They had a problem.  They needed to divert stderr from the programs they were running off hours to a file but couldn't figure out how to do it.  The two of them had been trying all morning. 

They showed me their code.  They were calling dup(2), dup2(2) and a long list of other exotic Unix system calls.  I was familiar with the calls and what they were used for but to this day have never encountered a reason to use them.  Once they explained the task, I wrote a 5 line (lots of white space) program that wrote to stderr and another 5 line program that reopened stderr on a file and ran the first program.  Compiled them, demonstrated it worked and left. They'd spent 8 man hours going nowhere.  The entire demonstration took me 5 minutes.  Even though I was being paid 3-4x what they were, I was still the lowest cost solution.

My career was unusual as I had been trained as a scientist, not as a programmer.  I did not get my PhD because of personal conflicts with my supervisor and after 4 years at Austin, it was not worth going to Stanford and spending another 6.  I'd also given myself a common law computer science PhD by buying and reading 80 ft of books.  So even without my doctorate I still commanded first rate PhD rates. I also survived a lot of layoffs.

When I quit work to move to Arkansas to look after Mom & Dad I'd expected to get a job where I did most of my work at home and flew to Houston for 3-4 days once a month.  The crash in 2008 killed that and I had a really tough time for several years until I stumbled across this forum.  I find living in cyber space more than a bit weird, but the company here matches the company anywhere I ever worked professionally.  This forum has completely changed my outlook on life.  The trolls are annoying, but the only thing that comes close is Usenet back in the late 80's and early 90's when mention of a book or paper in a post was likely to get a response from the author.
« Last Edit: January 22, 2019, 09:36:58 pm by rhb »
 
The following users thanked this post: Siwastaja

Offline thermistor-guy

  • Regular Contributor
  • *
  • Posts: 188
  • Country: au
Re: Test Driven Development
« Reply #84 on: January 22, 2019, 01:28:47 pm »
...
Despite knowing better, I bought a bunch of books on proving correctness.  None actually proved useful for anything other than an example of a delusion.

To this day, I often use loop invariants and ghost variables, which I learned from David Gries' book:

https://www.springer.com/gp/book/9780387964805
 

Online rhb

  • Super Contributor
  • ***
  • Posts: 2386
  • Country: us
Re: Test Driven Development
« Reply #85 on: January 22, 2019, 02:39:03 pm »
One of the few books with useful information.
 

Offline andyturk

  • Frequent Contributor
  • **
  • Posts: 890
  • Country: us
Re: Test Driven Development
« Reply #86 on: January 22, 2019, 05:42:25 pm »
So basically what can be done to test software is to check it for known issues like edge cases and invalid input ...etc, but bug free operation cannot be guaranteed unless the software is so simple that exhaustive test of all input sequences can be performed.

Lots of people don't like to hear that.
Of course there are no guarantees, even with (unit) testing. But if you devote some effort to test development, you'll end up with a better result than without it.

In my (perhaps limited) experience, tests are still code. Tests have bugs. Tests are incomplete. Tests don't necessarily keep up when the spec or the implementation changes. Even so, if you're able to write test fixtures from a perspective that's orthogonal to how the actual implementation works, you're ahead.

E.g., in addition to checking specific input values, you can generate a "random" collection of input values and look at all the results statistically. Or perform some numerical transformation in the test harness that be used to measure the implementation.

That's one of the reasons I like using C++... I can use Googletest and bring the full power C++ to bear on the test harness even if the production code is straight C. And googlemock is a beautiful thing.
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 1581
  • Country: fi
Re: Test Driven Development
« Reply #87 on: January 22, 2019, 05:58:03 pm »
Programmers aren't usually in a good position to know what the quality of a requirement specification really is. They just don't know enough about the details of what the customer would consider correct behaviour.

I think this is a management disaster. IMHO, programmers need to know the details of the customer case. Keeping programmers "out of the loop" never works, because the high-level design and specification never is perfect. If the programmer has something to say, they are able to produce huge savings of time and money. OTOH, if they are treated as "implementation machines", even in best case, they will implement the specification perfectly, resulting in an expensive product with multiple flaws, because I have never seen or heard about a specification that's both exactly and actually what the customer wants, and also technologically implementable in a sane/efficient way.

Programmers at least tend to have the best ground-level understanding what can be easily done, and what is complex. They have best chances of understanding which data structure is a PITA to use, and which data structure works.

The idea that programmers are comparable to the manual workers building a house, or a bridge exactly according to blueprints, and the house/bridge is separately designed by engineers, is a total fallacy. Programmers are part of the design team, want it or not. The equivalent of the house-building manual workers would be people who manually punch out cards to feed the program into a computer, exactly according to the listing. Since we have totally automated the manual part, some management people have became confused and tend to assume we still have them, and incorrectly take the programmers as such.
« Last Edit: January 22, 2019, 06:07:38 pm by Siwastaja »
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 8601
  • Country: gb
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #88 on: January 22, 2019, 08:17:44 pm »
So basically what can be done to test software is to check it for known issues like edge cases and invalid input ...etc, but bug free operation cannot be guaranteed unless the software is so simple that exhaustive test of all input sequences can be performed.

Lots of people don't like to hear that.
Of course there are no guarantees, even with (unit) testing. But if you devote some effort to test development, you'll end up with a better result than without it.

You'll end up with more confidence in the result, but whether it is a sufficiently good result depends on...

Quote
In my (perhaps limited) experience, tests are still code. Tests have bugs. Tests are incomplete. Tests don't necessarily keep up when the spec or the implementation changes. Even so, if you're able to write test fixtures from a perspective that's orthogonal to how the actual implementation works, you're ahead.

Regrettably you will find that limited experience repeated throughout your career :( I've seen some truly dreadful unit tests that were "OK because the green light is on".

Quote
E.g., in addition to checking specific input values, you can generate a "random" collection of input values and look at all the results statistically. Or perform some numerical transformation in the test harness that be used to measure the implementation.

If you have a "golden" implementation that you can compare with your implementation, then it can increase confidence - but no more. Electronics: behavioural vs structural implementations. Software: executable specification vs your implementation.

Otherwise fuzzing can be useful for breaking black boxes.

Quote
That's one of the reasons I like using C++... I can use Googletest and bring the full power C++ to bear on the test harness even if the production code is straight C. And googlemock is a beautiful thing.

The phrase "lipstick on a pig" springs to mind :)
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline coppice

  • Super Contributor
  • ***
  • Posts: 4173
  • Country: gb
Re: Test Driven Development
« Reply #89 on: January 22, 2019, 09:26:35 pm »
Programmers aren't usually in a good position to know what the quality of a requirement specification really is. They just don't know enough about the details of what the customer would consider correct behaviour.

I think this is a management disaster. IMHO, programmers need to know the details of the customer case. Keeping programmers "out of the loop" never works, because the high-level design and specification never is perfect. If the programmer has something to say, they are able to produce huge savings of time and money. OTOH, if they are treated as "implementation machines", even in best case, they will implement the specification perfectly, resulting in an expensive product with multiple flaws, because I have never seen or heard about a specification that's both exactly and actually what the customer wants, and also technologically implementable in a sane/efficient way.

Programmers at least tend to have the best ground-level understanding what can be easily done, and what is complex. They have best chances of understanding which data structure is a PITA to use, and which data structure works.

The idea that programmers are comparable to the manual workers building a house, or a bridge exactly according to blueprints, and the house/bridge is separately designed by engineers, is a total fallacy. Programmers are part of the design team, want it or not. The equivalent of the house-building manual workers would be people who manually punch out cards to feed the program into a computer, exactly according to the listing. Since we have totally automated the manual part, some management people have became confused and tend to assume we still have them, and incorrectly take the programmers as such.
You keep jumping between things which are requirements gathering, and things which are implementation related. I suggest you try some books on the topic of requirements gathering. There are a few good ones around, although its some years since I consulted one, so I have no specific recommendation. There are some good short courses, too, where you can start to practice the art of flushing implied requirements out of customers.

Good requirements gathering is one of the toughest jobs in Engineering. You can never be sure you have gathered every requirement, even if you take the task really seriously. Any time you interview a customer, trying to flush out all their requirements, and they say something like "Isn't that obvious?", you've just hit something that was only obvious to them, and they they had no idea needed to be explicitly stated. Flushing out as many of these job knowledge related requirements, which probably took years to become part of the customer's working knowledge set, is usually the difference between a satisfied customer and one litigating with you. Since it took the customer years to learn what needs to be done, how much of this do you think the programmers will pick up during a development cycle? This is one of the wonderful things about developing long term relationships, as the implementers really can build up quite a detailed understanding of their customer's needs. Staff turnover in software development is usually quite high, though, so even long term relationships are no panacea.
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 8601
  • Country: gb
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #90 on: January 22, 2019, 09:42:57 pm »
You keep jumping between things which are requirements gathering, and things which are implementation related.

That's reasonable, since the implementation tests must include (but not be limited to) the points of value to the customer - i.e. their requirements.

I agree with your other points.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline coppice

  • Super Contributor
  • ***
  • Posts: 4173
  • Country: gb
Re: Test Driven Development
« Reply #91 on: January 22, 2019, 10:24:33 pm »
You keep jumping between things which are requirements gathering, and things which are implementation related.
That's reasonable, since the implementation tests must include (but not be limited to) the points of value to the customer - i.e. their requirements.
I have found it to be very unreasonable. As you start to gather requirements the implementation sometimes looks like its going to be a nightmare. Then as you continue to gather requirements you find something that makes the complexity fall away. More often you start gathering requirements and the implementation looks like it will be a breeze. Then as you continue to gather requirements you suddenly flush out something obscure that will be a huge roadblock. The best course is usually to focus on gathering requirements with little thought about how a solution might be implemented. When the requirements are well fleshed out, and you look at the implementation, you might find you have serious issues. At that point you might want/need to go back and negotiate on requirements which cause problems. However, the first priority is to flush them out.
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 1581
  • Country: fi
Re: Test Driven Development
« Reply #92 on: January 23, 2019, 12:29:42 am »
I don't believe that the implementers should be decoupled from the requirement gathering. Well, maybe in a massive project, not every coder needs to participate in everything, but in general, I don't believe in the paradigm of "collect all the information" -> "top-level design without coding" -> "implementation" - all by different people.

The team is likely to get too big and too decoupled, and "very important" requirements very hard to implement sneak in, despite the fact they might not be very important after all, or all completely wrong yet eat up 90% of the schedule. Communication is key here, not "abstracting away" the communication.

Quote
As you start to gather requirements the implementation sometimes looks like its going to be a nightmare. Then as you continue to gather requirements you find something that makes the complexity fall away. More often you start gathering requirements and the implementation looks like it will be a breeze. Then as you continue to gather requirements you suddenly flush out something obscure that will be a huge roadblock.

This exactly shows why the implementation aspect needs to be present while gathering requirements. Because the requirements are not only software functions; requirements also include price and schedule (and whether it is feasible at all). If the customer gets a feedback that a certain feature is almost impossible to implement and would blow the budget and schedule, they may want to compromise and think again. This is best found out early, before time is committed to the implementation. This is why the implementers should sit down together with the end customer.

And, the coders are not idiots; quite the opposite, they may have bright ideas to help the customer solve the problem more easily and save money.

If done in a hierarchical waterfall model, the requirement-gatherers and top-level designers need to have almost superhuman skills what comes to understanding implementation; they should be able to implement it themselves. If they are more like marketing people, it's a disaster waiting to happen - especially if the customer is represented by their marketing people, as well.
 

Offline coppice

  • Super Contributor
  • ***
  • Posts: 4173
  • Country: gb
Re: Test Driven Development
« Reply #93 on: January 23, 2019, 12:41:30 am »
I don't believe that the implementers should be decoupled from the requirement gathering.
Has anyone suggested that should be the case?
Well, maybe in a massive project, not every coder needs to participate in everything, but in general, I don't believe in the paradigm of "collect all the information" -> "top-level design without coding" -> "implementation" - all by different people.
I agree that paradigm is stupid. Has anyone suggested it?
 

Online tggzzz

  • Super Contributor
  • ***
  • Posts: 8601
  • Country: gb
    • Having fun doing more, with less
Re: Test Driven Development
« Reply #94 on: January 23, 2019, 01:25:53 am »
You keep jumping between things which are requirements gathering, and things which are implementation related.
That's reasonable, since the implementation tests must include (but not be limited to) the points of value to the customer - i.e. their requirements.
I have found it to be very unreasonable. As you start to gather requirements the implementation sometimes looks like its going to be a nightmare. Then as you continue to gather requirements you find something that makes the complexity fall away. More often you start gathering requirements and the implementation looks like it will be a breeze. Then as you continue to gather requirements you suddenly flush out something obscure that will be a huge roadblock. The best course is usually to focus on gathering requirements with little thought about how a solution might be implemented. When the requirements are well fleshed out, and you look at the implementation, you might find you have serious issues. At that point you might want/need to go back and negotiate on requirements which cause problems. However, the first priority is to flush them out.

Nobody has suggested anything that is in conflict with that.
There are lies, damned lies, statistics - and ADC/DAC specs.
Glider pilot's aphorism: "there is no substitute for span". Retort: "There is a substitute: skill+imagination. But you can buy span".
Having fun doing more, with less
 

Offline Siwastaja

  • Super Contributor
  • ***
  • Posts: 1581
  • Country: fi
Re: Test Driven Development
« Reply #95 on: January 23, 2019, 01:28:59 am »
Has anyone suggested that should be the case?
I agree that paradigm is stupid. Has anyone suggested it?

I was probably reading "between the lines" a tad too much. Thanks for clarification.
 


Share me

Digg  Facebook  SlashDot  Delicious  Technorati  Twitter  Google  Yahoo
Smf