Some Software Engineering Myths Busted

Measuring programming progress by lines of code is like measuring aircraft building progress by weight. – Bill Gates

Nachi Nagappan, a senior researcher at Microsoft Research worked with Microsoft development teams to validate conventional software-engineering wisdom and came out with these findings (copied almost verbatim) -

* Code coverage measures how comprehensively a piece of code has been tested; if a program contains 100 lines of code and the quality-assurance process tests 95 lines, the effective code coverage is 95 percent. Higher code coverage is not the best measure of post-release failures.

The reason is that software quality depends on so many other factors and dynamics that no one metric can predict quality—and not all metrics apply to all projects. Two of the most obvious reasons why code coverage alone fails to predict error rates: usage and complexity.

..it is more beneficial to achieve higher code coverage of more complex code than to test less complex code at an equivalent level

* Test-Driven Development - TDD teams produced code that was 60 to 90 percent better in terms of defect density than non-TDD teams. They also discovered that TDD teams took longer to complete their projects—15 to 35 percent longer.

...the tradeoff is that you reduce post-release maintenance costs significantly, since code quality is so much better.

* Assertions are contracts or ingredients in code, often written as annotations in the source-code text, describing what the system should do rather than how to do it.

More assertions and code verifications means fewer bugs.

Looking behind the straight statistical evidence, they also found a contextual variable: experience. Software engineers who were able to make productive use of assertions in their code base tended to be well-trained and experienced, a factor that contributed to the end results.

* Organizational metrics, which are not related to the code, can predict software failure-proneness with a precision and recall of 85 percent. This is a significantly higher precision than traditional metrics such as churn, complexity, or coverage that have been used until now to predict failure-proneness.

* Distributed-Development - One of the most cherished beliefs in software project management is that a distributed-development model has a negative impact on software quality because of problems with communication, coordination, culture, and other factors.

Does distributed development affect software quality? ...the differences were statistically negligible.

Organizational cohesiveness plays a bigger role than geographical distance.