[JVM] Using JUnit categories to control Cucumber tests

[George Ferris]

I'm having trouble using JUnit categories to control which Cucumber scenarios get executed.   I created a JUnit category called "IntegrationTest".  When I run my CI build, I expect everything marked "@IntegrationTest" to run during a certain step of the build.  Normally, it is the JUnit runner that respects the JUnit categories, but since I'm using the Cucumber runner (@RunWith) instead, it doesn't seem to care about the JUnit category.  I can think of some workarounds, but I'm just wondering if there is any plan to make the Cucumber runner aware of JUnit categories.  Or am I missing something basic here altogether?

My runner looks like this:

package com.mycompany.shared.cucumber;

import org.junit.experimental.categories.Category;

import org.junit.runner.RunWith;

import com.mycompany.test.categories.IntegrationTest;

import cucumber.api.CucumberOptions;

import cucumber.api.junit.Cucumber;

@Category(IntegrationTest.class)

@RunWith(Cucumber.class)

@CucumberOptions(tags = {"@IntegrationTest" })

public class SampleRunner {

}

My thought was that JUnit would pick up this runner because of the @Category annotation and invoke cucumber with the corresponding "@Integrationtest" cucumber tag.  What actually happens is that it just always runs this Cucumber runner no matter what the JUnit category is set to.

[George Ferris]

Let me correct that last sentence... I said "What actually happens is that it just always runs this Cucumber runner no matter what the JUnit category is set to.". In reality, if I use maven-surefire-runner <excludedGroups> tag, no cucumber tests get excluded.  If I use the <groups> tag, no cucumber tests get included.

Here is what I have in my pom file. In this case, cucumber will not even attempt to run the Cucumber test class that has @IntegrationTest specified.  It will however run a plain Junit test with that category.

<build>

    <plugins>

        <plugin>

            <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-surefire-plugin</artifactId>

            <version>2.18.1</version>

            <configuration>

                <groups>com.mycompany.test.categories.IntegrationTest</groups>

            </configuration>

        </plugin>

   <plugins>

<build>

 

[Roberto Lo Giacco]

Why going for the extra step of using the JUnit categories when you already have the tags? Can't you just use the tag and use Maven to pass the tags option? 

[George Ferris]

I want three distinct testing steps in the build process: unit, integration and end-to-end. For each of these build steps 
I am doing a "mvn test -Pxxx" passing it a profile for unit, integration or end to end. In that profile, as you say, I am 
setting up the cucumber tags as you described. For integration testing, by way of example, I expect all cucumber tests 
marked @IntegrationTest to run and all plain Junit tests marked @IntegrationTest to also run as a result of that single 
maven command. They should generate surefire and cucumber metrics and also total jacoco code coverage metrics for all the 
code covered by either kind of test. Because of the fact that Cucumber does not respect JUnit category tags I have to do some 
messy things. For instance, remember the effect of this "bug" is this: if I use surefire <groups> to include categories no cucumber
tests will run at all. If I use surefire <excludeGroups> then ALL the surefire tests in the project will run. So the (terrible) hack
is to use <excludeGroups> to exclude every JUnit category I know about except for the one I want and make sure there is only 
one cucumber runner class in the project (or filter out any unwanted ones). Thus, I run "mvn test -Pintegration-tests" 
with a pom that includes the profile shown below and it kinda works. But it isn't pretty.

<profile>
   <id>integration-tests</id>
   <build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.18.1version>
            <configuration>
               <systemPropertyVariables>
                  <cucumber.options>src/test/resources/features --tags @IntegrationTest --tags ~@Ignore --strict --monochrome --plugin pretty --plugin html:target/cucumber-results/integration-test-reports --plugin json:target/cucumber-results/integration-test-report.json --plugin junit:target/cucumber-results/integration-test-report.xml</cucumber.options>
               </systemPropertyVariables>
               <argLine>${jacocoJavaAgent} -Xms512m -Xmx3072m</argLine>
               <excludedGroups>test.categories.UnitTest,test.categories.EndToEndTest</excludedGroups>
            </configuration>
         </plugin>

 

[Roberto Lo Giacco]

,

First of all I wouldn't use a profile to run ITs, but I'd rather use the integration-test Maven phase, so that when I run the test phase I get the unit tests only. There are a few benefits by doing it this way, like the pre-integration-test and post-integration-test phases being executed.

Second I would use maven-failsafe-plugin rather than surefire to execute the ITs and the E2E, so to have a more reliable build.

With such confit I've achieved to separate the tests into two well defined 'categories' and I would use **/*Test and **/*IT class name inclusion patterns, with the assumption your E2E tests are not cucumber based. Now, I would add cucumber execution by using a class named CucumberWrapper (not picked by the two patterns above) and use a profile or, my preference, a separate failsafe execution to run the Cucumber tests during the integration-test phase or the test phase, wathever best fits your needs.

It's a very different approach, you might not like it, but it has its own advantages over your solution, one being having the unit tests and the others running during separate phases, which mean you don't have to run maven multiple times to collect coverage data for the different tests types.

 

Thus, I run "mvn test -Pintegration-tests" 
with a pom that includes the profile shown below and it kinda works. But it isn't pretty.

<profile>
   <id>integration-tests</id>
   <build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.18.1version>
            <configuration>
               <systemPropertyVariables>
                  <cucumber.options>src/test/resources/features --tags @IntegrationTest --tags ~@Ignore --strict --monochrome --plugin pretty --plugin html:target/cucumber-results/integration-test-reports --plugin json:target/cucumber-results/integration-test-report.json --plugin junit:target/cucumber-results/integration-test-report.xml</cucumber.options>
               </systemPropertyVariables>
               <argLine>${jacocoJavaAgent} -Xms512m -Xmx3072m</argLine>
               <excludedGroups>test.categories.UnitTest,test.categories.EndToEndTest</excludedGroups>
            </configuration>
         </plugin>

 

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to a topic in the Google Groups "Cukes" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/cukes/jVqbTdwoAPo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[George Ferris]

I'll make a note to look into the surefire versus failsafe plugin question.  My impression was that going forward test categories are the way to go because they are more flexible and allow for combinations of tags, etc as opposed to depending on file naming conventions.  I guess the point is JUnit provides the category mechanism but Cucumber does not honor it.  It would be nicer if it did. 

[Roberto Lo Giacco]

I can't speak for the development team, but I believe it would be nice if you can add that and submit a pull request for it...