Coverage report showing very different numbers between operating systems

[smi...@googlemail.com]

Hi,

We have noticed that a *.exec file generated on windows / mac are producing reports with very different coverage numbers.

On windows the *.exec file is showing 90% whilst on mac it is showing 84%. If we take the *.exec file from the mac machine and put it onto the windows machine and then generate the report again from this file it shows 91% - so closer, but still not a match.

Wondering what we can do to try and investigate this further?

If it helps at all we are running multiple test tasks that we are then trying to combine into an aggregate report ...

val jacocoReportLocations = fileTree("/build/jacoco") { include("*.exec") }
jacocoReportLocations.files.forEach {
    println(it)
}

val jacocoTestReport by getting(JacocoReport::class) {
    executionData(jacocoReportLocations)
    reports {
        xml.isEnabled = true
    }
    afterEvaluate {
        classDirectories.setFrom(files(classDirectories.files.map {
            fileTree(it) {
                exclude(coverageExclusions)
            }
        }))
    }
}

and the dir that sources the files has the following in it ...

..\build\jacoco\contractTest.exec

..\build\jacoco\integrationTest.exec

..\build\jacoco\test.exec

Thanks in advance,

Phil.

[Marc Hoffmann]

Hi Phil,

are you moving exec files between builds and/or machines? Please note that JaCoCo requires the exact same class files at execution and analysis time. See our documentation for background information:

https://www.jacoco.org/jacoco/trunk/doc/classids.html

Regards,

-marc

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/8da52cc5-d650-4bf9-b392-27c0c8e50d9d%40googlegroups.com.

[Philip Smith]

Hi Marc,

The moving between machines was simply being done to try and understand the discrepancy we were seeing between our windows and unix based platforms; it is not something that we would normally be doing. 

This issue actually came to light as we had our coverage set to 90% in our jacocoTestCoverageVerification task (see belo). On windows this was fine as based on the 3 *.exec files we had a reported coverage of 90%, but when we ran the job on the CI server (linux) the build failed as the verification task was only reporting 84%.

So when either the jacocoTestReport or jacocoTestVerification task runs, based on what you have said above, does it also reference the local class files as well as the *.exec files to produce the answer. I had assumed perhaps wrongly that the only thing used at this point was the exec file?

Really just trying to understand why a build process that produces 3 *exec files, due to 3 separate test targets, can produce widely different answers between the operating systems. Before when we had all the tests running under the single target we had consistent results?

val jacocoReportLocations = fileTree("/build/jacoco") { include("*.exec") }

val jacocoTestReport by getting(JacocoReport::class) {
    executionData(jacocoReportLocations)
    reports {
        xml.isEnabled = true
    }
    afterEvaluate {
        classDirectories.setFrom(files(classDirectories.files.map {
            fileTree(it) {
                exclude(coverageExclusions)
            }
        }))
    }
}

val jacocoTestCoverageVerification by getting(JacocoCoverageVerification::class) {
    executionData(jacocoReportLocations)
    mustRunAfter("jacocoTestReport")
    violationRules {
        rule {
            limit {
                counter = "LINE"
                minimum = "0.90".toBigDecimal()

You received this message because you are subscribed to a topic in the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jacoco/gmFf8Q1YWL8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/02056E8B-3887-4E1D-8171-9279435CCC07%40mountainminds.com.

[Marc Hoffmann]

Wild guessing without knowing your exact setup: Maybe separation of modules eliminated "cross coverage". I.e. a.test only counts for a not for b.

To debug this you can:

* drill down both coverage reports and search for specific discrepancies

* Use our command line tool to inspect and diff the content of the exec files (execinfo, https://www.jacoco.org/jacoco/trunk/doc/cli.html)

Regards,
-marc

To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/CANmyrU3MQVKv01siHMbaVMhpRzaOUQMf3ZCg_je0VfL-40JX8A%40mail.gmail.com.