The Pragmatic Capability Model Of Software Delivery (PCM-OSD)

[Philip Fitzsimons]

Thoughts on this maturity model? I’ve been working on this for a while and it has been influenced by the book...

 

http://blog.figmentengine.com/2010/08/pragmatic-capability-model-of-software.html

 

interested to see what % coverage people expect in TDD & Acceptance testing? Any good studies in this area?

 

Also any missing pieces?

 

Ta!

[Figment Engine]

broken url - short version http://bit.ly/dclvGj

On Aug 15, 7:14 pm, "Philip Fitzsimons" <phi...@hilltown.info> wrote:
> Thoughts on this maturity model? I've been working on this for a while and
> it has been influenced by the book...
>

> http://blog.figmentengine.com/2010/08/pragmatic-capability-model-of-s...
> .html

[Steve Freeman]

You might want to tell us who you are, your site is essentially anonymous.

For the record, greenfield projects I'm on usually hit 95% code coverage without really trying. The wonders of TDD.

S.

[Philip Fitzsimons]

Ah, certainly not intended to be secret squirrel stuff ;-)

I work as an architect for a publishing company, I really interested in
seeing if people think the model is generally applicable.

Lots of people shy away from a % coverage - I agree that if its TDD you
would expect high numbers, but even then people are nervous of targets for
targets sake..

Philip Fitzsimons.

[Steve Freeman]

On 15 Aug 2010, at 20:48, Philip Fitzsimons wrote:
> Ah, certainly not intended to be secret squirrel stuff ;-)

nice to meet you :)

> Lots of people shy away from a % coverage - I agree that if its TDD you
> would expect high numbers, but even then people are nervous of targets for
> targets sake..

rightly so, I've seen teams do stupid things to achieve coverage numbers. It should really be used an indication for possible action, or a symptom of a change in the team, rather than a hard commitment.

S.

[Gishu Pillai]

+1. My 2 cents...
Use a drop in coverage to highlight that source code has been added without corresponding tests - this should trigger an inquiry as to what happened.. and if possible, revert the check-in and TDD it before checking in again. Just a enforcing policy to keep the team following the rules.

Keeping coverage as a "goal" doesn't yield good results;  "the team must have X%+ coverage at all times" leads to situations where the team only investigates drops in coverage when it drops below the threshold. Ignores it otherwise. You need to keep an eye on any code that enters the codebase, without corresponding tests/specs.

That said, following TDD rigorously usually leads to the high 90% coverage numbers.
Gishu

[Figment Engine]

good point - should be possible in most pipelines to pickup a % drop
automatically.

nice one!

> > S.- Hide quoted text -
>
> - Show quoted text -

[Figment Engine]

I'm thinking that from reading the "continuous delivery" book that
there should be an attribute for poppendieck's cycle time...
so in descending order...
months,weeks,days,hours,< hour,JIT

[Peter Thomas]

(long time reader, first time poster)

Whilst I totally agree with the intent, I also agree that any metrics
can and will be gamed by the team, whether that be explicitly or
unintentionally. I am yet to see a use case where test coverage
metrics provide any meaningful information as to the quality of the
tests, except for the assertion that if test coverage is low then the
quality of tests is also correspondingly low; however the inverse
isn't always true, I have seen code bases with high test coverage but
these tests are at best poor (not really unit tests and with very
little assertions) and the coverage metric gave the indication to
management that things were better than they really where (although
you could state that they should have just gone and taken a look
themselves rather than rely on the metrics).

So, coverage metrics in my experience are really only beneficial to
the developers in helping identify areas which aren't covered by tests
and may need some tests added (typical examples are exception blocks
which are often missed) but using it for anything more than this
hasn't worked. We also attempted to break the build if the test
coverage went down, but there are valid use cases where this can
happen, such as deleting a class with high test coverage.

Sorry that this post has gone on a bit, but this is something I feel
very strongly about and have often argued for ages about this very
point. However the rest of the article is very good and I totally
agree with the main points, it reminds me a little of the Joel Test
(from Joel Spolksy's blog) which he recommended as a good check of a
prospective employer.

Cheers, Peter

[Graeme Foster]

I have the build fail if coverage drops below 100%, but allow code to be explicitly excluded from coverage by decorating it with an attribute. This means that no code is accidentally committed without test coverage, but if a developer decides to exclude some code then they can do so but will have to be able to defend their decision.

Works pretty well.

G.

[J. B. Rainsberger]

On Mon, Aug 16, 2010 at 04:56, Graeme Foster <gra...@gmail.com> wrote:

> I have the build fail if coverage drops below 100%, but allow code to be
> explicitly excluded from coverage by decorating it with an attribute. This
> means that no code is accidentally committed without test coverage, but if a
> developer decides to exclude some code then they can do so but will have to
> be able to defend their decision.
> Works pretty well.

That's just crazy enough to work, as evidenced by your making it work.
How have people reacted to the implicitly bureaucratic nature of this
system? Have they adjusted well?
--
J. B. (Joe) Rainsberger :: http://www.jbrains.ca ::
http://blog.thecodewhisperer.com
Diaspar Software Services :: http://www.diasparsoftware.com
Author, JUnit Recipes
2005 Gordon Pask Award for contribution to Agile practice :: Agile
2010: Learn. Practice. Explore.

[Graeme Foster]

On 19 August 2010 19:21, J. B. Rainsberger <m...@jbrains.ca> wrote:

On Mon, Aug 16, 2010 at 04:56, Graeme Foster <gra...@gmail.com> wrote:

> I have the build fail if coverage drops below 100%, but allow code to be
> explicitly excluded from coverage by decorating it with an attribute. This
> means that no code is accidentally committed without test coverage, but if a
> developer decides to exclude some code then they can do so but will have to
> be able to defend their decision.
> Works pretty well.

That's just crazy enough to work, as evidenced by your making it work.
How have people reacted to the implicitly bureaucratic nature of this
system? Have they adjusted well?

On my very small team of 2, yes, although to be fair I didn't give my colleague much of a say in the matter as he joined a little later on :)

This is the first project on which we've attempted TDD (it started just after I read GOOS the first time), and initially the "100% rule" helped make us add tests after writing code. Now that we're much more comfortable with writing the tests first it's rarely an issue and it just points out the odd line here and there that we've missed.

I think it's really paid off for us as we could have easily left swathes of code untested in the early days, but instead we were encouraged to test everything we could. I have no doubt that it's helped us learn to do TDD and we have much better code as a result.

On my last project we had a lot of existing code and tried to add tests afterwards; we used coverage to measure progress, and I found it quite demoralising to see the number slowly creep up and eventually grind to a halt. To have it at 100% from day one and then keep it there feels great!

Cheers,

G.

[Uberto Barbini]

problem is that test coverage doesn't tell you much...
what's the point in testing getter and setters? (ok this is a point to
remove them but... ;))
also testcoverage tools I know doesn't measure what has been asserted,
but only what is ran in tests... so they are so easy to fool (both
intentionally and unintentionally).

cheers

Uberto

[Dan Haywood]

On 20/08/2010 10:07, Uberto Barbini wrote:
> problem is that test coverage doesn't tell you much...
> what's the point in testing getter and setters? (ok this is a point to
> remove them but... ;))
>

On the other hand, if you test them, then you at least know that any
test coverage <100% is for stuff that isn't as trivial as getters/setters.

Testing getters and setters is also dead easy, and can even be done
reflectively if you want... see
http://danhaywood.com/2010/06/11/pojo-properties-auto-testing/

Flame away!
Dan