This paper1 from Microsoft’s Empirical Software Measurement group reports that
Case studies were conducted with three development teams at Microsoft and one at IBM that have adopted TDD. The results of the case studies indicate that the pre-release defect density of the four products decreased between 40% and 90% relative to similar projects that did not use the TDD practice. Subjectively, the teams experienced a 15-35% increase in initial development time after adopting TDD.
Reading the small print, these were not “pure” TDD teams since they did a lot of the requirement and design up-front. Still, a nice data point that suggests that, taking the cost of fixes into account, it’s worth taking the time to get it right.
1) “Realizing quality improvement through test driven development: results and experiences of four industrial teams” Nachiappan Nagappan, Michael Maximilien, Thirumalesh Bhat, Laurie Williams. Empirical Software Engineering Journal, Volume 13, Number 3, pp. 289-302, Springer 2008
Interesting read – just wondering if you’d come across any similar articles regarding pair programming?
Interesting! Thanks for the link, Steve!
My suspicion is there would be a substantial divergence between subjective increased development time and actual development time. There are three factors I’d expect to cause overestimates:
First, people notice the new thing more. Second, they’d have a hard time noticing the time spent not debugging. And third, software projects are always under time pressure, so many managers habitually find and presents reasons why the team isn’t going as fast as people want.
The decrease in defect rates matches my experience, though. For one team where we mainly coached them on TDD, we cut bug rates by 50% on the first project, and by 50% again on the second project. For an experienced team doing full XP, I’d expect 90%+.
I can only conclude from this statistically inconclusive study that managers of teams who say they’re doing TDD tend to believe that development takes longer.
Like Agent Mulder, I want to believe – I really do – but I’m afraid my search for statistically compelling evidence continues…
@Jason, in which case the story would be better because they reduced the bug count and might not have taken longer after all…
BTW The link to the paper is broken.
@Mark. Fixed the link. They seem to have changed the capitalisation. Ta.
[...] terms of whether or not actual development time takes longer with TDD, Microsoft recently released a paper which suggested that code written using a TDD approach takes longer to write originally but puts [...]
steve.freeman > it’s worth taking the time to get it right
It’s a trade-off.
“Trade-offs between Productivity and Quality in Selecting Software Development Practices”
Also, see “Product-Development Practices That Work: How Internet Companies Build Software”
@Isaac. I read the first paper and it doesn’t clash with the MS paper at all. In particular, the trade-off is between “conventional” measures of quality such as a detailed spec. In practice, they say that “agile” practices such as repeated testing can compensate for a lack of preplanning, and that practices such as early prototypes provide a huge improvement.
I don’t have the full second paper but, again, I don’t see the case for it being a trade-off.
Maybe the resistance to take-up of TDD can be explained as follows:
If the initial value to the business is greater than the cost of the 15-35% increase in development time, then this is the real pressure forcing management and thereby IT teams to focus on delivery and thus push TDD out of the picture? ie: Business are happy to pay the higher defect cost, because they get value earlier (assuming the initial user acceptance testing phase meets their initial expectations).
I think the resistance is more to do with the counter-intuitive shift that most people would have to make. My call would be that getting the quality in early reduces the cost to delivery, unless you’re doing something extremely simple or extremely flaky.
There’s also another level of value that most of the studies don’t take into account which is the extent to which writing a test first encourages people to think about what they really want, rather than building in speculative or cool ideas.
Just to agree with steve.freeman. Yes of course it encourages Developers to think about what they need. And remember that TDD is really about designing classes and methods. It is not directly about testing. So my statement to business owners and Managers that resist TDD would be to say, ‘Do you really want code that has not been designed?’
Stumbled on this again while looking for a link to the paper.
steve.freeman > but, again, I don’t see the case for it being a trade-off
Given the authors state “only one practice—releasing an early prototype during development—was associated with both higher productivity and a lower defect rate” perhaps you should read the paper again.
“Integration or regression testing at check-in” DID NOT contribute to higher productivity.
“Integration or regression testing at check-in” DID contribute to lower defect rate.
@Isaac Are you sure you have the same paper? I can’t find the quotation.
On the other hand, the authors start their conclusions with, “Our experiences and distilled lessons learned, all point to the fact that TDD seems to be applicable in various domains and can significantly reduce the defect density of developed software without significant productivity reduction of the development team.” Which sounds like an overall win.
> I canât find the quotation.
âTrade-offs between Productivity and Quality in Selecting Software Development Practicesâ
http://www.pitt.edu/~ckemerer/CK%20research%20papers/TradeoffProductivityQuality_MacCormackKemererCusumanoCrandall03.pdf
p84 paragraph beginning “Overall, our results suggest…”