Software quality study - Finding 7: scores decline with more frequent releases

[podenski]

There is a new software quality study that just came out which states (see page 15 of the CRASH report, in section named: Finding 7 -- scores decline with more frequent releases): 

"The 319 applications that reported the number of releases per year were grouped into three categories: one to three releases (n=140), four to six releases (n=114), and more than six releases (n=59). As shown in Figure 17a, 17b, and 17c, scores for Robustness, Security, and Changeability declined as the number of releases grew, with the trend most pronounced for Security."

The article speculates that:

"In this sample most of the applications with six or more releases per year were reported to have been developed using custom methods, and the sharp decline for projects with more than six releases per year may be due in part to less effective development methods."

So my real curiosity is how proper and consistently conducted Continuous Delivery projects would fare in this study?

REFERENCES FOLLOW:

Slashdot article:

   http://developers.slashdot.org/story/11/12/09/1533252/java-apps-have-the-most-flaws-cobol-the-least

The CRASH report 2011/12

   http://www.castsoftware.com/resources/cast-research-labs

[Tim Coote]

You'd need to see these data in much more detail to form any sort of informed view of what is being measured and whether it makes sense to compare the numbers.

For instance, the performance metric seems fatuous as it doesn't measure performance at all, and although the type highlighted is J2EE, the measurement is of Java code, whereas most performance issues in J2EE systems are either in the connectors (and downstream systems) or in the ORM layer.

From a CD point of view, the expensive part, I'd expect is going to be building the tests. I've only (so far) seen business cases that can justify this cost if the immediate intent is to rip out an old piece of functionality that's too expensive/flakey to live with.

Personally, I think that there's a great product opportunity to produce testing tools for these technologies that drive/test at protocol and/or api interfaces, as the sheer cost of testing legacy often leads to large technical debt (eg unsupported COTS components and huge proliferation of configuration variation and suboptimal work-arounds).

Tim