pitest failed with no green test suite when execute a maven module . But passed when I specify the failed test in <targetTests> .

[richard yuwen]

  Hi all:

     I happened to meet the following situation .

     1. mvn test is passed 

     2. pitest failed with no green test suite.

                 logs:

                  2:43:24 PM PIT >> INFO : Sending 77 test classes to slave

2:43:24 PM PIT >> INFO : Sent tests to slave

2:43:25 PM PIT >> INFO : SLAVE : 2:43:25 PM PIT >> INFO : Checking environment

2:43:28 PM PIT >> INFO : SLAVE : 2:43:28 PM PIT >> INFO : Found  353 tests

2:43:28 PM PIT >> INFO : SLAVE : 2:43:28 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0

2:43:28 PM PIT >> INFO : SLAVE : 2:43:28 PM PIT >> INFO : 353 tests received

                   ...................... 

                  [INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 01:29 min

[INFO] Finished at: 2015-08-06T14:44:49+08:00

[INFO] Final Memory: 14M/603M

[INFO] ------------------------------------------------------------------------

[WARNING] The requested profile "with-test-dependency" could not be activated because it does not exist.

[ERROR] Failed to execute goal org.pitest:pitest-maven:1.1.5:mutationCoverage (default-cli) on project app-mdi: Execution default-cli of goal org.pitest:pitest-maven:1.1.5:mutationCoverage failed: All tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.

[ERROR] See http://pitest.org for more details.

[ERROR] -> [Help 1]

[ERROR] 

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

 

     3. pitest passed when I specify the test failed in step 2 using <targetTests>

           logs:

2:47:24 PM PIT >> INFO : Verbose logging is disabled. If you encounter an problem please enable it before reporting an issue.

2:47:24 PM PIT >> INFO : Sending 8 test classes to slave

2:47:24 PM PIT >> INFO : Sent tests to slave

2:47:25 PM PIT >> INFO : SLAVE : 2:47:25 PM PIT >> INFO : Checking environment

2:47:26 PM PIT >> INFO : SLAVE : 2:47:26 PM PIT >> INFO : Found  8 tests

2:47:26 PM PIT >> INFO : SLAVE : 2:47:26 PM PIT >> INFO : Dependency analysis reduced number of potential tests by 0

2:47:26 PM PIT >> INFO : 8 tests received

|2:47:30 PM PIT >> INFO : Calculated coverage in 6 seconds.

2:47:34 PM PIT >> INFO : Created  1618 mutation test units

                   .....

                   [INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 01:14 min

[INFO] Finished at: 2015-08-06T14:48:32+08:00

[INFO] Final Memory: 38M/1751M

[INFO] ------------------------------------------------------------------------

  

     my configuration:

               <jvmArgs>

    <value>-server</value>

    <value>-ea</value>

    <value>-XX:PermSize=521m</value>

    <value>-XX:MaxPermSize=1g</value>

    <value>-Xmx2g</value>

    <value>-XX:ReservedCodeCacheSize=512m</value>

    <value>-XX:-UseSplitVerifier</value>

</jvmArgs>

     mvn org.pitest:pitest-maven:mutationCoverage -Dthreads=32 -DmaxMutationsPerClass=20 -DmutationUnitSize=1 -DtimeoutConstant=50 -DtimeoutFactor=0.5 -DdetectInlinedCode=true  -DfailWhenNoMutations=false -DmaxDependencyDistance=1  

     

  

thanks!

 

[richard yuwen]

BTW, I use the verbose to see the details.

some test will have "3:03:03 PM PIT >> FINE : SLAVE : FAIL Description [testClass=xxxx, name=xxx)] -> java.lang.NullPointerException" 

[Henry Coles]

On 6 August 2015 at 08:25, richard yuwen <richar...@gmail.com> wrote:

BTW, I use the verbose to see the details.

some test will have "3:03:03 PM PIT >> FINE : SLAVE : FAIL Description [testClass=xxxx, name=xxx)] -> java.lang.NullPointerException" 

The most likely explanation is that you have a test order dependency in your suite - i.e one or more tests modify some sort of state that persists between tests (often in a singleton or static variable but other types are also possible). This poisoned state then causes a different tests to fail sometime later.

The problem might not be revealed during a normal test run, but if pitest runs the tests in a different order from surefire the issue is revealed.

Maven surefire supports a runOrder of "random". Try running with this set 10 times or so and see if any tests fail. If they do you definitely have an order dependency. If they don't then you may or may not have one. If you have the tests set to fork each time in the surefire you'll need to disable this first - this used to be a common workaround for testing code with mutable static state.

Henry

[richard yuwen]

Hi Henry:

   I have tried what you have said . 

    set below configuration in sure-fire plugin

    1. reuseForks=true

    2. forkCount=1

    3. runOrder=random

 

 the junit test passed . 

 we will investigate some test dependency.

Richard