Unable to get coverage numbers to match between jacocoTestReport and jacocoTestCoverageVerification

Jim Showalter

unread,

Mar 1, 2020, 2:22:16 PM3/1/20

to JaCoCo and EclEmma Users

We have a Gradle project with JaCoCo coverage.

We have centralized our excludes in a function, so we can use it in both the reporting and verification tasks:

def coverageExcludes() {
	return [
		"com/ourcompany/*Config*",
		"com/ourcompany/ServiceApplication*",
		"com/ourcompany/Shutdown*",
		"com/ourcompany/ShutdownConnector*",
		"com/ourcompany/Tomcat*",
		"com/ourcompany/TomcatCustomConnectorCustomizer*",
		"com/ourcompany/endpoint/exception/**",
		"com/ourcompany/endpoint/util/ObjectMapperBuilder*",
		"com/ourcompany/framework/**",
		"com/ourcompany/helper/ApiHelper*",
		"com/ourcompany/helper/OffsetDateTimeDeserializer*",
		"com/ourcompany/persistence/entity/**",
		"com/ourcompany/persistence/exception/**",
		"com/ourcompany/service/exception/ServiceException*",
		"com/ourcompany/service/model/**",
		"com/ourcompany/v2/**",
		"com/ourcompany/v3/**"
	]
}

The reporting task is configured like this:

jacocoTestReport {
	reports {
		xml {
			enabled true
		}
		html {
			enabled true
		}
	}
	getExecutionData().setFrom(fileTree(buildDir).include("/jacoco/*.exec"));
	afterEvaluate {
		classDirectories.setFrom(files(classDirectories.files.collect {
			fileTree(dir: it,
				exclude: coverageExcludes()
			)
		}))
		for (Object c : classDirectories) { // Temporary loop, using it for debugging this problem.
			System.out.println(c);
		}
	}
}

The build generates coverage reports without a problem, and the numbers look correct:

But when we try to set coverage-percentage thresholds to fail the build if coverage dips too low, we have to set it to a much lower number than the report displays.

For example, instead of the report's 87% coverage, we have to set the failure threshold at 70%. Any higher results in:

Execution failed for task ':settings:jacocoTestCoverageVerification'. Rule violated for bundle settings: instructions covered ratio is 0.70, but expected minimum is 0.87

We are using the same excludes in both Gradle tasks:

jacocoTestCoverageVerification {
	afterEvaluate {
		classDirectories.setFrom(files(classDirectories.files.collect {
			fileTree(dir: it,
				exclude: coverageExcludes()
			)
		}))
		for (Object c : classDirectories) { // Temporary loop, using it for debugging this problem.
			System.out.println(c);
		}
	}
	violationRules {
		rule {
			element = 'BUNDLE'
			limit {
				counter = 'INSTRUCTION'
				value = 'COVEREDRATIO'
				minimum = 0.87
			}
		}
	}
} check.dependsOn jacocoTestCoverageVerification

The loops that dump the contents of classDirectories output the same exact lists in both tasks.

Comparing the un-excluded classes to the exclusions, we see that the excludes are working as expected.

It's just that the two tasks are seeing different coverage numbers.

We are pretty sure we configured the jacocoTestCoverageVerification violationRules with the same settings that the report task defaults to: BUNDLE, INSTRUCTION, COVEREDRATIO.

We're like the guy with two watches: we're not sure what time it is. Are the numbers in the report correct, or do we in fact only have 70% coverage, not 87%?

If we could get past this problem, we could retire an in-house custom test-coverage plugin that isn't multi-module aware and isn't compatible with Gradle past 5.5.1, and that would free us to upgrade to latest Gradle, so there is pent-up demand for a solution!

A possible related question: why is the variable called "classDirectories", instead of something like "classes"? Is there some other variable we should be using? We got the above task configurations from multiple online examples that said that's the way to configure excludes.

P.S. I reported this on stackoverflow in https://stackoverflow.com/q/60438452/1461450 and nobody responded. It now has a bounty. If someone here knows the answer, you could answer there and here, and pick up 50 reputation points.

Evgeny Mandrikov

unread,

Mar 2, 2020, 4:51:54 AM3/2/20

to JaCoCo and EclEmma Users

Hi,

Partial snippets are not debuggable. Without a complete runnable example one can only make random guesses:

you configure "ExecutionData" in your "jacocoTestReport" task, but not in "jacocoTestCoverageVerification" task.

On Sunday, March 1, 2020 at 8:22:16 PM UTC+1, Jim Showalter wrote:

A possible related question: why is the variable called "classDirectories", instead of something like "classes"?

Gradle plugin is not developed by JaCoCo team, you'd better ask developers of Gradle plugin about their choice of name.

Regards,

Evgeny

jameslee...@gmail.com

unread,

Mar 3, 2020, 10:48:35 AM3/3/20

to JaCoCo and EclEmma Users

It was the execution data. Feel free to answer over on https://stackoverflow.com/questions/60438452/gradle-jacoco-report-coverage-percent-doesnt-match-verification-percent and collect the bounty.

As for the name classDirectories, I looked at the Gradle source code and it really does look like a poor choice of name, because it's just a file collection.

Reply all

Reply to author

Forward