Jacoco Coverage issue with Junit Parallel Execution

[causa...@gmail.com]

Hi Guys,
I am seeing significant variations in jacoco coverage report for the same set of integration tests run in "parallel" vs "sequential". Coverage percentages below.

Maven : 2.x

Sequential:
INSTRUCTION 7%
BRANCH 3%
COMPLEXITY 6%
LINE 7%
METHOD 10%
CLASS 18%

Parallel:
INSTRUCTION 2%
BRANCH 1%
COMPLEXITY 1%
LINE 2%
METHOD 2%
CLASS 8%

<plugin>
<artifactId>maven-failsafe-plugin</artifactId>
<version>2.20.1</version>
<dependencies>
<dependency>
<groupId>org.apache.maven.surefire</groupId>
<artifactId>surefire-junit47</artifactId>
<version>2.20.1</version>
</dependency>
</dependencies>
<executions>
<execution>
<id>default-test</id>
<phase>integration-test</phase>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
<argLine>-Xms1024m -Xmx3072m</argLine>
<trimStackTrace>false</trimStackTrace>
<skipTests>false</skipTests>
<parallel>classes</parallel>
<threadCount>30</threadCount>
<perCoreThreadCount>true</perCoreThreadCount>
<includes>
<include>**/*Test.java</include>
</includes>
....
</plugin>

jacoco config (using offline instrumentation for JMockit based unit tests and java agent for integration tests)

<plugin>
<!-- Create/generate the application.html using xsltproc -->
<groupId>org.jacoco</groupId>
<artifactId>jacoco-maven-plugin</artifactId>
<version>0.7.9</version>
<executions>
<execution>
<id>instrument</id>
<phase>process-classes</phase>
<goals>
<goal>instrument</goal>
</goals>
</execution>
<execution>
<id>restore</id>
<phase>prepare-package</phase>
<goals>
<goal>restore-instrumented-classes</goal>
</goals>
</execution>
<execution>
<id>it-bind-agent</id>
<goals>
<goal>restore-instrumented-classes</goal>
<goal>prepare-agent-integration</goal>
</goals>
</execution>
<execution>
<id>ut-report</id>
<goals>
<goal>report</goal>
</goals>
</execution>
<execution>
<id>it-report</id>
<goals>
<goal>report-integration</goal>
</goals>
</execution>
</executions>
</plugin>

[Evgeny Mandrikov]

On Sunday, January 14, 2018 at 6:12:41 PM UTC+1, Causalbody wrote:

Hi Guys,
I am seeing significant variations in jacoco coverage report for the same set of integration tests run in "parallel" vs "sequential".

As far as we know and according to our tests JaCoCo perfectly works in case of parallel execution of tests.

IMO the only advices that can be given in absence of complete runnable reproducer:

• make sure that behavior of your tests is deterministic and that your own components do not have concurrency problems
  
• JaCoCo writes execution data from JVM shutdown hook, so make sure that your JVM(s) terminate gracefully and that shutdown hooks are executed, in particular that you don't leave main threads in interrupted state - see https://bugs.openjdk.java.net/browse/JDK-8154017
  

[Causalbody]

Thank you. I figured out the issue. It wasn't really parallel vs sequential issue. I was running jmockit for unit test and that coverage wasn't included in the jacoco report because I was on jacoco version 0.7.9, the issue described in https://github.com/jmockit/jmockit1/issues/125 .

 After i switched to 0.8.0 the jmockit unit test coverage was also accounted for.