SonarQube Generic Test Coverage plugin : no overall coverage calculation

[Michel Pawlak]

Hello,

As discussed during the conference yesterday, I cross checked and confirm that it seems that the Generic Test Coverage plugin does not populate metrics for overall coverage. I searched for "overall" on the plugin's github repository and got no result.

In order to make sure that I'm not missing something, I had a look at java-jacoco module's code in sonar-java, and saw that there is a specific sensor for overall coverage reporting as well as a UT/IT report merger class. Similar classes are not available for the generic test coverage plugin.

Is the overall coverage supposed to be calculated by the platform or is it the plugin's job to do it ? If it is the plugin's job, it seems that something is missing. If it's the platform's job I'm missing something.

I'll make a test project available on github this evening, it would be great if you could have a look at this issue.

Cheers and thanks in advance,

Michel

[Julien HENRY]

Hi Michel,

In 5.2 we made some initial work in the direction to make overall coverage computed by the platform. We though this could be easily deduced from UT + IT. But then we came to an issue for branch coverage:

  - if UT reports 2/4 covered covered

  - if IT reports 2/4 covered branches

then we can't guess what is value of overall branch coverage (could be 2/4, 3/4 or 4/4).

That's why we put this project on hold, and that's still up to the plugin to provide overall branch coverage.

++

Julien

[Michel Pawlak]

Hi Julien,

This is indeed an issue. In my context I will then to have to merge UT/IT reports prior to feeding them to the generic test coverage plugin, however, currently this plugin is not able to receive an overall coverage as input. Can you please create a ticket in order to be able to feed the plugin with such an input ?

Thanks in advance,

See you,

Michel

[Julien HENRY]

Done:

https://jira.sonarsource.com/browse/SONARCOVRG-14

[Michel Pawlak]

By the way, in order to fix your issue, you should not keep the information "covered or not" for lines with branches, but rather "hit true count" and "hit false count" -> then you'll be able to merge UTs with ITs branch/condition coverage easily as well as line coverage :

With condition true hit count / condition false hit cont :

• ut 0/n, it 0/0 -> sum = 0/n -> line coverage = covered / condition coverage = 50% with n > 0
• ut 0/0, it 0/n -> sum = 0/n -> line coverage = covered / condition coverage = 50% with n > 0
• ut n/0, it 0/0 -> sum = n/0 -> line coverage = covered / condition coverage = 50% with n > 0
• ut 0/0, it n/0 -> sum = n/0 -> line coverage = covered / condition coverage = 50% with n > 0
• ut n/x, it y/m -> sum = n+y/m+x -> line coverage = covered / condition coverage = 50% with n > 0, m > 0, and any x,y
• ut x/n, it m/y -> sum = m+x/n+y -> line coverage = covered / condition coverage = 100% with n > 0, m > 0, and any x,y

Cheers,

Michel

On Friday, September 25, 2015 at 2:06:07 PM UTC+2, Julien HENRY wrote:

[Michel Pawlak]

Thanks for the ticket !

On Friday, September 25, 2015 at 4:10:47 PM UTC+2, Julien HENRY wrote:

Done:

https://jira.sonarsource.com/browse/SONARCOVRG-14

[Michel Pawlak]

Hi Julien,

I added the missing feature, here is the pull request:

https://github.com/SonarSource/sonar-generic-coverage/pull/5

You'll find an example project to generate the data in the following GitHub repository : https://github.com/pawlakm/qualinsight-mojo-cobertura-example/tree/master

It would be great if you could review this PR / comment it / do your own (backward compatibility) tests then hopefully accept it and release a new version of the plugin anytime soon :-)

Cheers, 

Michel

[Julien HENRY]

Hi Michel,

I'm sorry but I don't understand your proposal. Note that we should talk about "condition coverage" and not "branch coverage".

Let's say we have the code:

if (A && B) {

  // do something

} else {

  // do something else

}

There are 4 conditions to be covered:

  - A == true && B == true

  - A == true && B == false

  - A == false && B == true

  - A == false && B == false

If UT reports to cover 2 conditions and IT 2 conditions, I don't see how we can "merge" them without having a unique "id" for each condition.

[Michel Pawlak]

Hi Julien,

My bad, I based my answer on the fact that the generic test coverage plugin's xml format  uses the "branch" keyword instead of "condition". I should have thought about it a bit more prior to writing. Well forget about my previous post then and sorry for having wasted your time.

Indeed you will need an unique ID to be able to compute condition coverage as you cannot guess what condition has been evaluated to true or not. This would require this kind of input (where branches are now conditions, and where line can have a conditionsToCover sub node:

<?xml version="1.0" encoding="UTF-8"?>
<coverage version="2">
    <file path="src/main/java/example/AClass.java">
        <lineToCover lineNumber="10" covered="partial">
            <conditionsToCover total="4" covered="1">
                <condition id="1" covered="false"/>
                <condition id="2" covered="true"/>
                <condition id="3" covered="false"/>
                <condition id="4" covered="false"/>
            </conditionsToCover>
        </lineToCover>
        <lineToCover lineNumber="11" covered="false"/>
        <lineToCover lineNumber="13" covered="true"/>
        <lineToCover lineNumber="15" covered="true"/>
    </file>
    <file path="src/main/java/example/AnotherClass.java">
        <lineToCover lineNumber="6" covered="false"/>
        <lineToCover lineNumber="9" covered="true"/>
        <lineToCover lineNumber="12" covered="true"/>
        <lineToCover lineNumber="16" covered="false"/>
    </file>
</coverage>

Michel

[Julien HENRY]

No problem. I'm not aware of any tool able to preserve condition 'id' (and BTW it would first need to agree on what is a condition). All that to say for now we will continue to let to the responsibility of plugins to compute overall condition coverage.

[Michel Pawlak]

Hi, 

I made this pull request ~2 month ago, do you plan reviewing it ?

Michel

[Pierre-Yves Nicolas]

Hi Michel,

Yes, it's in the plans, but I cannot give you any date.

Sorry for that.

Regards,

Pierre-Yves

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/4b65098f-18b2-4178-9e46-6c3ea88141c2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Massimo Paladin]

Hi Michel,

the PR has been merged, thank you for your contribution.

Cheers,

Massimo PALADIN | SonarSource

Software Developer @ Language Team
http://sonarsource.com

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/CAE3muQr9%2BB9Pxh3fu1dFHgO6PgdWoBLvAu_etmGYq-Kr3VoZRw%40mail.gmail.com.