Code coverage for integration tests with Cucumber

[Matthew Harrison]

Hi,

I've been having a little look around, and wonder if someone could give me some guidance.  

We're using SonarQube 6.3.1, looking to gather data from a maven project.

We have tests at the integration level which run through Cucumber.  I've successfully got coverage working for my Unit tests using what I think is the recommended way:

<plugin>

        <groupId>org.apache.maven.plugins</groupId>

        <artifactId>maven-surefire-plugin</artifactId>

<!-- Minimal supported version is 2.4 -->

<version>2.13</version>

<configuration>

<properties>

<property>

<name>listener</name>

<value>org.sonar.java.jacoco.JUnitListener</value>

</property>

</properties>

</configuration>

</plugin>

But if I try something similar with our cucumber tests:

<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-failsafe-plugin</artifactId>

<version>2.20</version>

<executions>

<execution>

<goals>

<goal>integration-test</goal>

<goal>verify</goal>

</goals>

<configuration>

<properties>

<property>

<name>listener</name>

<value>org.sonar.java.jacoco.JUnitListener</value>

</property>

</properties>

</configuration>

</execution>

</executions>

</plugin>

I get the error:

[ERROR] Feature: Transform  Time elapsed: 0.007 s  <<< ERROR!

org.sonar.java.jacoco.JacocoController$JacocoControllerError: Looks like several tests executed in parallel in the same JVM, thus coverage per test can't be recorded correctly.

I think this is because of the way that Cucumber uses the unit tests to run the steps that make up the features (I'm guessing a bit here - but hopefully I'm not too far off the mark), in that the main unit test will start, and then a step (which is also a JUnit test) will start.

Looking at  org.sonar.java.jacoco.JacocoController this then causes an issue.  So two questions:

1.  Is there a different setup I can use to gather coverage for the integration tests?  I guess we won't be able to get per test coverage, but could you point me to an example around getting overall coverage?

2.  The test listener stops any usage of parallel testing I think(?), is this inherent in the usage of the listener or Jacoco, or just a choice at the time?  (my, very vague, understanding is that Jacoco can deal with being written to in parallel - but I really haven't dug into it)

Any pointers/advice or information would be great.

Thanks,

Matt

[Nicolas Peru]

Hi, 

Sorry for the late reply.

This is a part that does not trigger a lot of questions nor traction so it takes a bit of time to switch back into it. 

So your guess about the two start of junit test for one jacoco agent seems right.

So I don't think you can actually support coverage per test for cumcumber but that would have to be confirmed properly as I am no cucumber expert.

So regarding question 1 : This has nothing to do with coverage per tests but with coverage. 

So to gather coverage from all your tests you should generate the reports from your different test tools/suite and provide it to SonarJava through sonar.java.reportPaths (see documentation : https://docs.sonarqube.org/display/PLUG/Code+Coverage+by+Unit+Tests+for+Java+Project ) and those should be picked up. 

But the key understanding here is that you have to generate the reports during your build.

Regarding question 2: The problem is to associate coverage to a specific test. If tests are run in parallel there is no easy way to know which data is associated to which tests.

Cheers, 

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/a074dcb8-1609-4018-a089-1b1676ffdc3b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Nicolas Peru | SonarSource

Senior Developer

http://twitter.com/benzonico

http://sonarsource.com