What kind of branching coverage is measured by Jacoco

[realtim...@gmail.com]

Hi,

I must say I'm quite amazed with Jacoco - heard about it for the first in spring, so how it was used in a CI environment.

Now I have started looking into the details of Jacoco and I stumbled across your definition of branching coverage.

I am used to the following definition (and you will find this on Wikipedia and also in something like the ISTQB):

branching coverage is covering all branches of a decision - i.e.

if ((x < 5) && ( y > 10))

has two branches: one for the expression true and one for the expression false.
So (x=4, y=11) and (x=6, y=11) would be test data to cover all branches.

To me it seems that what is called branching coverage in Jacoco is what is called multiple condition coverage in literatur - you need to cover all combinations of the logical conditions (predicates) of the decision.

So in the above example you would need 4 tests to get multiple condition coverage:

x < 5 | y > 10 | test date
T T 4, 11
T F 4, 9
F T 6, 11
F F 6, 9

Do I get it correctly that branching coverage in Jacoco is the latter?

And again, thanks for a nice tool.
/Carsten

[realtim...@gmail.com]

OK, I was jumping to conclusions. A little experiments with more than 2 conditions show that what you are checking for is mostly called "condition coverage" or "single condition coverage".

So with the two conditions you get 4 possibilities, and with three conditions you will get 6 possibilities.

[Evgeny Mandrikov]

Hi,

First of all - thank you for your warm words about our tool.

Now about counters - according to https://en.wikipedia.org/wiki/Code_coverage :

if (a && b) { /* ... */ } 

condition coverage can be satisfied by two tests "a = true, b = false", "a = false, b = true"

this seems a bit weird given the fact that && in Java is a short-circuit operator - second test can't trigger retrieval of value of "b".

Same applies for Wikipedia's description of "multiple condition coverage".

But let's forget about terminological mess - here is how JaCoCo works:

JaCoCo analyzes Java bytecode, this is stated on page describing counters - http://www.eclemma.org/jacoco/trunk/doc/counters.html

In bytecode all constructions such as

if (a && b && c) something();

are decomposed into something like

if (a) if (b) if (c) something();

And in a such case there seems to be no difference between Wikipedia's definition of "branch coverage", "condition coverage" and even with "multiple condition coverage".

You need 4 tests to satisfy them:

a = false

a = true, b = false

a = true, b = true, c = false

a = true, b = true, c = true

And there are 6 ways to go thru such control-flow-graph and 6 is exactly the value, which JaCoCo reports as branch coverage.

The same is going to happen in case of construction such as

if (a > 0 && b > 0 && c > 0) something();

Hope this helps and clarifies things.

[realtim...@gmail.com]

[Sorry Evgeny - didn't think about visibility when replying from gmail...so I repost here]

Thanks, Evgeny!

Yes, this clarifies it - I was guessing it comes from the bytecode.
Condition Coverage is different for languages with and without short circuit semantics, true.
I guess I will need to look at this in more detail, but my understanding is that this is decision coverage (a.k.a. branching coverage) on bytecode level.
And you might be right, that it is (single) condition coverage for short-circuit semantics.
It's probably not MC/DC - question would be how complicated it would be to add MC/DC - this is an important coverage in safety criticial systems.

[realtim...@gmail.com]

And so Evgeny asked me:

> Also I'm wondering - if not a secret, what are the reasons to dive so deep into terminology?

No it's not a secret: I work in research and industry in software testing, and especially I am an ISTQB (http://www.istqb.org/) trainer - the ISTQB is a body of knowledge for software testing, where we also teach white box testing methods.

I saw the first usage of Jacoco within an continuous integration framework, and was thinking about using Jacoco to illustrate statement and decision/branching coverage.

In our ISTQB course, we are using Myers' triangle test. When you use this example with Jacoco, you will get 100% statement coverage with the test cases from our textbooks, but you will not get 100% decision coverage with the text book solution. So now I need to think about an example, where theory and practice meet ;-)

> And Looking forward for your conclusion - maybe we should refresh our own knowledge of subject and update documentation.

Testing terminology differs a lot - one of the reasons why a lot of people push for the ISTQB, as it clarifies/unifies terminology.
Regarding branching/decision coverage, from googling you will see that most agree on the definition.
Probably a good idea to try to align terminology.

Is there an easy way to put up some example on the web and discuss the differences between the different coverage measurements and how Jacoco sees them?

/Carsten

[Marc R. Hoffmann]

Hi,

just for clarification: JaCoCo does not support MC/DC, which is defined as:

1. Each entry and exit point is invoked
2. Each decision takes every possible outcome
3. Each condition in a decision takes every possible outcome
4. Each condition in a decision is shown to independently affect the outcome of the decision.

JaCoCo does not support number 4.

Anyways JaCoCo targets the Java platform only which at least to my knowledge has not certified implementation anyways (for SIL 4 or the like).

Regards,
-marc