Exploding Software-Engineering Myths

[Chris Brandsma]

Here is an interesting article that just came out of Microsoft Research.

http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx

Here is my favorite part:

What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer.

“Over a development cycle of 12 months, 35 percent is another four months, which is huge,” Nagappan says. “However, the tradeoff is that you reduce post-release maintenance costs significantly, since code quality is so much better. Again, these are decisions that managers have to make—where should they take the hit? But now, they actually have quantified data for making those decisions.”

Saw this quote on a Reddit thread:

yah, i like the reasoning here. you can get "done" faster without testing, if you change the value of "done".

--
--------------------------------
Christopher Brandsma
http://www.ChrisBrandsma.com
http://www.ElegantCode.com

[Jim McKeeth]

So if I am reading that correctly, developing software without TDD is like eating McDonalds (i.e. fast / junk food).  Sure it is quicker and it seems to meet your needs, but we all know that in the long run you are just killing yourself slowly.  You pay the price with needing to exercise more and work it off later.  In the long run it costs you more time. 

It only makes sense if you are not planning to maintain the code.

-Jim McKeeth
j...@mckeeth.org
www.Delphi.org - The Podcast at Delphi.org
www.McKeeth.org - Personal home page

[Steven Dahlin]

The real question is the total time taken for the TDD approach versus non-TDD one.  Will one get something to production/in-use faster by going TDD with a longer dev time but short test time as opposed to going non-TDD with a shorter dev time but longer test time?  And I presume it might vary considerably considering the type & complexity of the project.

[Jacob Munson]

The real question is the total time taken for the TDD approach versus non-TDD one.  Will one get something to production/in-use faster by going TDD with a longer dev time but short test time as opposed to going non-TDD with a shorter dev time but longer test time?  And I presume it might vary considerably considering the type & complexity of the project.

I agree.  On my team we don't do TDD (at least not the way I understand it), but we do have a comprehensive testing phase in all of our project schedules.  I'd be curious to see some metrics that compare TDD to QA testing.

[Aaron Backer]

It only makes sense if *you* are not planning to maintain the code.

aye, there's the rub. So for certain values (and value-judgements) of
"you", this makes sense.

Do you buy a Sauder entertainment center made out of pressboard and
plastic veneer or do you lovingly hand-construct one from antique
cherry, or find something in the middle?

Hmm...now I suppose I'd say "that depends..."

-a

On Oct 9, 12:37 pm, Jim McKeeth <j...@mckeeth.org> wrote:
> So if I am reading that correctly, developing software without TDD is like
> eating McDonalds (i.e. fast / junk food).  Sure it is quicker and it seems
> to meet your needs, but we all know that in the long run you are just
> killing yourself slowly.  You pay the price with needing to exercise more
> and work it off later.  In the long run it costs you more time.
>
> It only makes sense if you are not planning to maintain the code.
>
> -Jim McKeeth

> j...@mckeeth.orgwww.Delphi.org- The Podcast at Delphi.orgwww.McKeeth.org- Personal home page

[Tony Rasa]

that analogy has problems though.  would it change your decision process if randomly the sauder pressboard one collapsed in on itself (and all the stuff you put in it)?

[Dave Herron]

Did I miss something here?  I don't think there was enough information in the article alone to draw any conclusions.  Show me the facts.

-- DaveH

[Aaron Backer]

Well yes.  That wouldn't be sauder though.  That'd be anything I make after 11pm.

I'd think it'd be more: the stuff you buy from wally world that can hold 3-4 books without sagging and should last you a year or two vs. the heirloom "built like a tank" things that get passed down from generation to generation.

It's an imperfect analogy, but I'd say that for some businesses, all they need is a temporary bookshelf.  Some have businesses that rely on their bookshelves and stack 300lb legal docs on them.

However...*no* one needs bookshelves that spontaneously combust, flood, and chew through the floor all while simultaneously causing a singularity that sucks all rational thought into non-space.

-aaron

[Tony Rasa]

I think that analogy speaks more to what price for quality, rather than if unit testing makes sense or not.  

[Tony Rasa]

from the article:

http://research.microsoft.com/en-us/projects/esm/nagappan_tdd.pdf

[Andrew Hanson]

The pdf does do a better job of explaining their results. 

The biggest issue I've had with TDD in the past couple of years is that on every new project I have to spend time training 1 or more developers on just how to use TDD and often even a Continuous Build process. When I can keep a team togther for a couple of releases ( even on different products) the teams velocity always tends to pick up.

[Chris Brandsma]

I've problem I've seen with the comprehensive testing phase is that the comprehensive tests take forever to run though, are actually rarely done, and are never complete.

How many database changes are required to simulate each situation?   How well documented is are the tests?  How many screens?  Except in the simplest of applications this style of testing has a proven record of not working or anyone.

Basically, there is no single testing methodology that covers all of the code, and to only rely on one of them is insane.  If you want your code actually tested you need TDD and QA Testing. 

On Fri, Oct 9, 2009 at 1:04 PM, Jacob Munson <yaco...@gmail.com> wrote:

The real question is the total time taken for the TDD approach versus non-TDD one.  Will one get something to production/in-use faster by going TDD with a longer dev time but short test time as opposed to going non-TDD with a shorter dev time but longer test time?  And I presume it might vary considerably considering the type & complexity of the project.

I agree.  On my team we don't do TDD (at least not the way I understand it), but we do have a comprehensive testing phase in all of our project schedules.  I'd be curious to see some metrics that compare TDD to QA testing.

[Dave Herron]

I reviewed that also and still don't have enough "facts" to support his conclusions.  For instance, how did he determine that any additional time was needed for TDD.  Who decided that a project could have been completed any sooner with or without TDD.  As near as I can tell, the statistics he based his conclusions on were just pulled our of the air.  You get 10 different development teams, creating the same solution, all using TDD and you would probably have a huge variation in time to market for each team.  There are tool many variables.

-------- Original Message --------
Subject: [BSDG] Re: Exploding Software-Engineering Myths
From: Tony Rasa <tr...@meancat.com>
Date: Fri, October 09, 2009 3:58 pm
To: bs...@googlegroups.com

from the article:

http://research.microsoft.com/en-us/projects/esm/nagappan_tdd.pdf

On Fri, Oct 9, 2009 at 3:41 PM, Dave Herron <daveh...@boisetools.com> wrote:

Did I miss something here?  I don't think there was enough information in the article alone to draw any conclusions.  Show me the facts.

-- DaveH

-------- Original Message --------
Subject: [BSDG] Exploding Software-Engineering Myths
From: Chris Brandsma <chris.b...@gmail.com>
Date: Fri, October 09, 2009 12:22 pm
To: bsdg <bs...@googlegroups.com>

Here is an interesting article that just came out of Microsoft Research.

http://research.microsoft.com/en-us/news/features/nagappan-100609.aspx

Here is my favorite part:

What the research team found was that the TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer.

“Over a development cycle of 12 months, 35 percent is another four months, which is huge,” Nagappan says. “However, the tradeoff is that you reduce post-release maintenance costs significantly, since code quality is so much better. Again, these are decisions that managers have to make—where should they take the hit? But now, they actually have quantified data for making those decisions.”

Saw this quote on a Reddit thread:

yah, i like the reasoning here. you can get "done" faster without testing, if you change the value of "done".

[Chris Brandsma]

Even then you wouldn't get a real answer unless the programmers were robots.  Even with 10 different teams you still have to deal with varying skill levels and other design methodologies. 

Basically, there is no absolute answer to your question, there is no absolute way to measure it, nor is there an absolute way to define the question.  As soon as you start trying to define it the definition of what you need to be provable everything blows up.

My suggestion: look to the conclusions of various people and aggregate their answers. Near as I can tell, the "facts" from the linked study jive both with what I've seen personally, and with what I've heard from everyone else.

[Jim McKeeth]

You can do a lot with Function Points to determine how long something should take to develop.  There is a lot of materials to draw from to create accurate estimates.  Additionally if you have a good team that has worked with any discipline for a while, and a well organized catalog of previous projects, then you can typically draw on that to estimate future performance.  Sure, people get sick, etc. and things happen, but it isn't pure chaos. 

-Jim McKeeth
j...@mckeeth.org
www.Delphi.org - The Podcast at Delphi.org
www.McKeeth.org - Personal home page

[Mike Moore]

Sent from my iPhone

On Oct 9, 2009, at 1:04 PM, Jacob Munson <yaco...@gmail.com> wrote:

> I agree. On my team we don't do TDD (at least not the way I
> understand it), but we do have a comprehensive testing phase in all
> of our project schedules. I'd be curious to see some metrics that
> compare TDD to QA testing.

We do a full QA cycle at the end of each milestone and before we
release. The QA time is much more than the supposed high end cost of
35%. Theoretically we can start on the next milestone while in the QA
cycle, but I've never seen that happen. We typically work on fixing
bugs during the entire QA cycle, and somtimes even for weeks after we
release.

I haven't had the time to read the entire article, but were the TDD
teams doing all the XP practices, or just TDD? TDD by itself is only
part of the equation. Taking it completely out of the context of XP
will skew the results. You need full XP to see all the benefits.

[Mike Moore]

The analogy is flawed because software is not analogous to manufacturing. These comparisons just perpetuate this.

Sent from my iPhone

[Mike Moore]

TDD is not a testing practice, it is a design practice. Doing TDD will help you design code that is more easilly testable, which will show when you attempt to automate conprehansive tests. You don't just get these benefits for free, you have to design them.

Sent from my iPhone

[Chris Brandsma]

TDD = Test Driven Design.   And I do agree with that.   There is a notion in TDD that the fact that tested code falls out of the process is wonderful side benefit.  One thing you will not hear me harping on is test coverage.  I don't care actually.

[Richard Hundhausen]

Agreed. Some would take it a step further and claim that TDD replaces debugging!

 

Not sure if I’m ready to swallow that red pill just yet.

 

From: bs...@googlegroups.com [mailto:bs...@googlegroups.com] On Behalf Of Chris Brandsma
Sent: Friday, October 09, 2009 9:37 PM
To: bs...@googlegroups.com
Subject: [BSDG] Re: Exploding Software-Engineering Myths

 

TDD = Test Driven Design.   And I do agree with that.   There is a notion in TDD that the fact that tested code falls out of the process is wonderful side benefit.  One thing you will not hear me harping on is test coverage.  I don't care actually.

[Chris Brandsma]

Replace no.  But the less you can spend there the better.

[Tony Rasa]

It definately makes debugging time more effective, for me. Haven't managed to do away with the practice completely :)