Unit tests pass ok, but pitest is not able to start

[akashihi]

Hello!

I'm trying pitest with maven and have some strange problem with it.
Unit tests, that run before pitest and without pitest are green. When
i run them with pitest it complains that "Mutation testing requires a
green suite." and, obviously, pitest is not able to pass the suite.

Here are some details:

Running just a unit test:

Running com.satgate.invoice.conversion.JSONIdSerializerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115
sec

Running same test with a pitest:

java.lang.NullPointerException
at
org.unitils.UnitilsJUnit4TestClassRunner.run(UnitilsJUnit4TestClassRunner.java:
67)
at org.pitest.junit.adapter.Custom
stderr : RunnerExecutor.run(CustomRunnerExecutor.java:42)
at
org.pitest.junit.adapter.AdaptedJUnitTestUnit.execute(AdaptedJUnitTestUnit.java:
85)
at
org.pitest.coverage.execute.CoverageDecorator.execute(CoverageDecorator.java:
45)
at org.pitest.containers.UnContai
stderr : ner.submit(UnContainer.java:46)
at org.pitest.Pitest$3.run(Pitest.java:148)
at java.lang.Thread.run(Thread.java:662)


11:56:54 PIT >> WARNING : Description
[testClass=com.satgate.invoice.conversion.JSONIdSerializerTest,
name=testSerialize(com.satgate.invoice.conversion.JSONIdSerializerTest)]
did not pass without mutation.


I'm running pitest at 'integration-test' phase and usual tests are
started at 'test' phase:



<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<configuration>
<excludes>
<exclude>**/integration/*Test.java</exclude>
</excludes>
<classesDirectory>${project.build.directory}/
generated-classes/emma/classes</classesDirectory>
</configuration>
<executions>
<execution>
<id>integration-tests</id>
<phase>integration-test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>false</skip>
<excludes>
<exclude>none</exclude>
</excludes>
<includes>
<include>**/integration/*Test.java</
include>
</includes>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>0.26</version>
<configuration>
<inScopeClasses>
<param>com.satgate.*</param>
</inScopeClasses>
<targetClasses>
<param>com.satgate.*</param>
</targetClasses>
<excludedClasses>
<param>**/integration/*Test.java</param>
</excludedClasses>
</configuration>
<executions>
<execution>
<goals>
<goal>mutationCoverage</goal>
</goals>
<phase>integration-test</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.sonatype.maven.plugin</groupId>
<artifactId>emma-maven-plugin</artifactId>
<version>1.2</version>
<configuration>
<filters>
<filter>+com.satgate.*</filter>
</filters>
</configuration>
<executions>
<execution>
<phase>process-classes</phase>
<goals>
<goal>instrument</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.sonatype.maven.plugin</groupId>
<artifactId>emma4it-maven-plugin</artifactId>
<version>1.3</version>
<executions>
<execution>
<id>report</id>
<phase>test</phase>
<goals>
<goal>report</goal>
</goals>
<configuration>
<sourceSets>
<sourceSet>
<directory>$
{project.build.sourceDirectory}</directory>
</sourceSet>
</sourceSets>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>


As you may see, i'm also using Emma coverage tool, but i've tried to
remove it from the build and it didn't helped.

So, the question is what i'm doing wrong and why it doesn't works? :)

[Henry Coles]

The only thing that I can see that doesn't look quite right is the filtering of integration tests. The filters supplied to PIT are run against the compiled byte code, not the source so

     <param>**/integration/*Test.java</param>

Will not exclude anything. Assuming your integration tests are all within packages called integration, the following ought to exclude them

   <param>*.integration.*Test</param>

I don't think this is related to your problem however. At some point I'll try an improve the interoperability between surefire and PIT to get rid of this type of confusion.

Are you able to provide a minimal project that reproduces the problem? A cut down version of JSONIdSerializerTest and mocked up version of the class it's testing + pom.xml ought to be all that's required.

Thanks

Henry

[Denis Chapligin]

Hello!

2012/6/18 Henry Coles <henry...@googlemail.com>

I don't think this is related to your problem however. At some point I'll try an improve the interoperability between surefire and PIT to get rid of this type of confusion.

Fixed :) Thanks for advice.

 

Are you able to provide a minimal project that reproduces the problem? A cut down version of JSONIdSerializerTest and mocked up version of the class it's testing + pom.xml ought to be all that's required.

Test, class under test and couple of entities. 

http://dl.dropbox.com/u/17691684/pitest.tar.gz 

Here is the execution log:

http://pastebin.com/cFEh9uDq

[henry]

Hi akashihi,

This looks to be an issue with classpath order and slf4j.

Just running surefire I get the following

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/henry/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/home/henry/.m2/repository/org/glassfish/extras/glassfish-embedded-all/3.0/glassfish-embedded-all-3.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

I think when pit runs it is using some sl4j settings from glassfish.

0.26 didn't guarantee that the classpath was in the same order as when tests were run in surefire. This was fixed in 0.27and seems to resolve this issue, so if you update your pom the sl4j issue should go away.

I think this might be diversion however - the original stack trace you posted showed a null pointer unrelated to sl4j. Can you confirm if you still have this problem using 0.27?

Thanks

Henry


On Tuesday, June 19, 2012 9:30:24 AM UTC+1, akashihi wrote:

Hello!

2012/6/18 Henry Coles <>

[Denis Chapligin]

Hello!

2012/6/19 henry <henry...@googlemail.com>:

> This looks to be an issue with classpath order and slf4j.
>
>

> 0.26 didn't guarantee that the classpath was in the same order as when tests
> were run in surefire. This was fixed in 0.27and seems to resolve this issue,
> so if you update your pom the sl4j issue should go away.

Upgrade to 0.27 helped with my example, but tests for app are still
failing with slf4j error:

15:24:40 PIT >> INFO : Found 53 classes that might define tests
15:24:44 PIT >> INFO : Sent tests to slave
stderr : 15:24:45 PIT >> INFO : Found 303 tests
15:24:45 PIT >> INFO : Dependency analysis reduced number of potential
tests by 0
15:24:45 PIT >> INFO : 303 tests received
15:24:45 PIT >> INFO : Running 303 units

SLF4J: Class path contains multiple SLF4J bindings.

stderr :
SLF4J: Found binding in
[jar:file:/home/chollya/.m2/repository/org/slf4j/slf4j-log4j12/1.6.1/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/home/chollya/.m2/repository/org/glassfish/extras/glassfish-em
stderr : bedded-all/3.0/glassfish-embedded-all-3.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

15:24:48 PIT >> WARNING : Description
[testClass=junit.framework.TestSuite$1,
name=warning(junit.framework.TestSuite$1)] did not pass without
mutation.
stderr : 15:24:48 PIT >> INFO : Finished

15:24:48 PIT >> INFO : Calculated coverage in 8 seconds.
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] fail

Embedded error: All tests did not pass without mutation when
calculating line coverage. Mutation testing requires a green suite.
See http://pitest.org for more details

> I think this might be diversion however - the original stack trace you
> posted showed a null pointer unrelated to sl4j. Can you confirm if you still
> have this problem using 0.27?

I believe it was because unitils uses slf4j internally, so it failed
with wrong slf4j initialization order.

Anyway, it's quite obvious that i have broken slf4j setup in my test
environment, but i'm not interested in working slf4j in tests at all
;) glassfish-embedded is only used for some integration test and as a
implementation of some servlet related classes, so it's not an issue
in production. Surefire is quite tolerate to this issue. I believe
pitest could forgive me slf4j errors too:)

[Henry Coles]

Could you enable pit's verbose option and post the console output? From the snippet you've posted

15:24:48 PIT >> WARNING : Description
[testClass=junit.framework.

TestSuite$1,
name=warning(junit.framework.TestSuite$1)] did not pass without

I don't think slf4j is necessarily causing the problem - pit looks to have pulled a test class or suite apart incorrectly and is trying to run a test that does not exist.

[Denis Chapligin]

Hello!

2012/6/19 Henry Coles <henry...@googlemail.com>:

> Could you enable pit's verbose option and post the console output? From the
> snippet you've posted

Sorry for the delay.

Most of the output from pitest in verbose mode looks good(couple
hundreds of tests skipped):

stderr : 9:57:57 PIT >> FINE : Gathering coverage for test
Description [testClass=com.satgate.invoice.conversion.InflaterServiceImplTest,
name=testInflateInvoiceFailUser(com.satgate.invoice.conversion.InflaterServiceImplTest)]

stderr : 9:57:57 PIT >> FINE : Gathering coverage for test
Description [testClass=com.satgate.invoice.conversion.JSONIdSerializerTest,
name=testSerialize(com.satgate.invoice.conversion.JSONIdSerializerTest)]

but i see at least one error:

stdout : FAIL Description [testClass=junit.framework.TestSuite$1,
name=warning(junit.framework.TestSuite$1)] ->
junit.framework.AssertionFailedError: Class
junit.framework.TestSuite$1 has no public constructor TestCase(String
name) or TestCase()

[henry]

Something somewhere in your test suite is tripping PIT up - it's trying
to split a test into individual units, but has ended up trying to run a
test called "warning" (i.e. the message that JUnit generates when you
attempt to run a non test).

Could you see if you can create a minimal project that re-creates this?
Likely candidates for the root cause would be custom runners etc.

Thanks

Henry

On Fri, 2012-06-22 at 10:05 +0300, Denis Chapligin wrote:
> Hello!
>
> 2012/6/19 Henry Coles <>:

[Stefan Penndorf]

Hi there,

> stdout : FAIL Description [testClass=junit.framework.TestSuite$1,
> name=warning(junit.framework.TestSuite$1)] ->
> junit.framework.AssertionFailedError: Class
> junit.framework.TestSuite$1 has no public constructor TestCase(String
> name) or TestCase()

Seems to me you are using JUnit 3? Looks like JUnit finds itself as a
test to execute or PIT also scans JUnit internal classes for tests to
run. There is an anonymous class in junit.framework.TestSuite which
extends TestCase ("junit.framework.TestSuite$1"). And as an annonymous
class it does not have one of the required public constructors. So this
annonymous class may not be passed to JUnit as test class. Might be a
class path issue or a scan-for-tests issue.
All the interesting stuff can be found here:
http://grepcode.com/file/repo1.maven.org/maven2/junit/junit/4.8.2/junit/framework/TestSuite.java
- The class can be found at line 94, the code throwing the
AssertionFailedError/ creating the error message can be found at line 56.

Regards
Stefan

[Denis Chapligin]

Hello!

2012/6/26 Stefan Penndorf <ste...@cyphoria.net>:

>> stdout  : FAIL Description [testClass=junit.framework.TestSuite$1,
>> name=warning(junit.framework.TestSuite$1)] ->
>> junit.framework.AssertionFailedError: Class
>> junit.framework.TestSuite$1 has no public constructor TestCase(String
>> name) or TestCase()
>
>
> Seems to me you are using JUnit 3?

I'm using junit4, actually 4.10.

Unfortunately i still haven't found an easy way to reproduce this
issue and i'm working on it :)

[henry]

Hi Denis,

Are you still able to replicate this using the latest 0.28 snapshot? It
contains some fixes to PIT's JUnit3 handling.

0.27 tries to run things that extend TestCase that JUnit won't.

Thanks

Henry

[Denis Chapligin]

Hi Henry,

2012/7/20 henry <henry...@googlemail.com>

Are you still able to replicate this using the latest 0.28 snapshot? It
contains some fixes to PIT's JUnit3 handling.

0.27 tries to run things that extend TestCase that JUnit won't.

Unfortunately i'm in business trip right now so i'm not able to test anything atm :(

I'll check it next week wednesday. Sorry for the delays.

[Denis Chapligin]

Hi Henry,

2012/7/20 henry <henry...@googlemail.com>

Hi Denis,

Are you still able to replicate this using the latest 0.28 snapshot? It
contains some fixes to PIT's JUnit3 handling.

I'm not able to install 0.28-SNAPSHOT (latest hg actually):

 Tests in error: 

  shouldSetUserDirToArtefactWorkingDirectory(org.pitest.PitMojoIT): Exit code was non-zero: 1; command line and log = (..)

Check attached surefire report for details.

[henry]

On Tue, 2012-07-24 at 09:55 +0300, Denis Chapligin wrote:

>
> Tests in error:
> shouldSetUserDirToArtefactWorkingDirectory(org.pitest.PitMojoIT):
> Exit code was non-zero: 1; command line and log = (..)

Things are running ok on the CI server

https://hjc.ci.cloudbees.com/

So I'm guessing that either you're running on windows and this
integration test is unintentionally platform specific, or you have a
local issue.

If you can confirm you're running windows I'll see if I can find a
windows box and reproduce.

There's no need to build from source however - snapshots are published
to the sonatype oss repo

https://oss.sonatype.org/content/repositories/snapshots/org/pitest/

Thanks

Henry

[Denis Chapligin]

Hello!

2012/7/24 henry <henry...@googlemail.com>

If you can confirm you're running windows I'll see if I can find a

windows box and reproduce.

Unfortunately no :( 

Linux biker 2.6.32-5-amd64 #1 SMP Mon Mar 26 07:00:19 UTC 2012 x86_64 GNU/Linux

cat /etc/debian_version

6.0.5

The only unusual thing is that i'm running 64-bit kernel with 32-bit userspace.

 

There's no need to build from source however - snapshots are published
to the sonatype oss repo

https://oss.sonatype.org/content/repositories/snapshots/org/pitest/

Well, it stops with the same error:

9:49:10 PIT >> INFO : Mutating from /home/.../target/classes

[INFO] ------------------------------------------------------------------------

[ERROR] FATAL ERROR

[INFO] ------------------------------------------------------------------------

[INFO] null

[INFO] ------------------------------------------------------------------------

[INFO] Trace

java.lang.NullPointerException

        at org.pitest.PitMojo.execute(PitMojo.java:235)

        at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)

        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)

        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalWithLifecycle(DefaultLifecycleExecutor.java:556)

        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:535)

        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)

        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:348)

        at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)

        at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)

        at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)

        at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)

        at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

        at java.lang.reflect.Method.invoke(Method.java:597)

        at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)

        at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)

        at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)

        at org.codehaus.classworlds.Launcher.main(Launcher.java:375)

[henry]

On Wed, 2012-07-25 at 09:53 +0300, Denis Chapligin wrote:

>
> The only unusual thing is that i'm running 64-bit kernel with 32-bit
> userspace.
>

I take it 0.27 runs fine on the same box?

The test that was failing for you suggests that this could relate to a
change made in 0.28 to where pit writes a temporary jar file. This used
to be to the project workspace, but is now to java.io.tmpdir.

This is however also currently the only test that will try and run pit
end to end via maven - using the one installed on the build box. So I
think the problem is more likely to be something to do with your maven
setup.

What version are you running?

Thanks

Henry

[Denis Chapligin]

Hello!

2012/7/25 henry <henry...@googlemail.com>

On Wed, 2012-07-25 at 09:53 +0300, Denis Chapligin wrote:

>
> The only unusual thing is that i'm running 64-bit kernel with 32-bit
> userspace.
>

I take it 0.27 runs fine on the same box?

Well, tests for 0.27 fail too :) 

Tests run: 939, Failures: 0, Errors: 30, Skipped: 2

I've checked out parent-pitest-0.27 tag and rn mvn test on it. All erroneous surefire reports are in the attachment.

 

The test that was failing for you suggests that this could relate to a
change made in 0.28 to where pit writes a temporary jar file. This used
to be to the project workspace, but is now to java.io.tmpdir.

I clearly see that directory in my /tmp so it shouldn't be a permissions/etc issue.

 

This is however also currently the only test that will try and run pit
end to end via maven - using the one installed on the build box. So I
think the problem is more likely to be something to do with your maven
setup.

What version are you running?

chollya@biker:/tmp/pitestrunner$ mvn --version

Apache Maven 2.2.1 (rdebian-4)

Java version: 1.6.0_26

Java home: /usr/lib/jvm/java-6-sun-1.6.0.26/jre

Default locale: ru_RU, platform encoding: UTF-8

OS name: "linux" version: "2.6.32-5-amd64" arch: "i386" Family: "unix"

[henry]

On Thu, 2012-07-26 at 09:31 +0300, Denis Chapligin wrote:

>
> chollya@biker:/tmp/pitestrunner$ mvn --version
> Apache Maven 2.2.1 (rdebian-4)
> Java version: 1.6.0_26

I'm mixing up my changes. The failing test isn't for the tmp dir change,
it was for a change to set the working directory to match surefire's
behaviour. The change required a change to the very line which is null
pointering for you.

Sounds like the change might be broken for maven 2 - I'll need to go and
set it up somewhere to confirm.

If you replace

this.goalStrategy.execute(project.getExecutionProject().getBasedir(),data);

with

this.goalStrategy.execute(null,data);

and disable the test that was failing I think you should be able to
build and try out the junit3 fix.

* I'll see if I can find a proper fix for the null pointer.

Thanks


Henry

[Denis Chapligin]

Hello!

2012/7/26 henry <henry...@googlemail.com>

On Thu, 2012-07-26 at 09:31 +0300, Denis Chapligin wrote:If you replace

this.goalStrategy.execute(project.getExecutionProject().getBasedir(),data);

with

this.goalStrategy.execute(null,data);

and disable the test that was failing I think you should be able to
build and try out the junit3 fix.

      * I'll see if I can find a proper fix for the null pointer.

Well, with this change and latest hg (defaut branch) i get new test failure:

 Failed tests:   shouldRegisterProbesWithCodeCoverageStore(org.pitest.coverage.codeassist.CoverageClassVisitorTest): expected:<[34359738380]> but was:<[0, 0, 0, lots of zeroes here]

See attachmanet with surefire report.

[henry]

On Thu, 2012-07-26 at 15:28 +0300, Denis Chapligin wrote:

> Failed tests:
> shouldRegisterProbesWithCodeCoverageStore(org.pitest.coverage.codeassist.CoverageClassVisitorTest): expected:<[34359738380]> but was:<[0, 0, 0, lots of zeroes here]

We're having lots of fun here aren't we?

The CI server agrees with you on this one - possibly this is a 32 vs 64
bit issue as things are building fine for me.

I'll take a look.

Cheers

Henry

[Denis Chapligin]

2012/7/26 henry <henry...@googlemail.com>

On Thu, 2012-07-26 at 15:28 +0300, Denis Chapligin wrote:

>  Failed tests:
> shouldRegisterProbesWithCodeCoverageStore(org.pitest.coverage.codeassist.CoverageClassVisitorTest): expected:<[34359738380]> but was:<[0, 0, 0, lots of zeroes here]

We're having lots of fun here aren't we?

Sure :)

 

The CI server agrees with you on this one - possibly this is a 32 vs 64
bit issue as things are building fine for me.

I have mixed 32/64 bit installations and clear 32-bit only/64-bit only installations, so i could try to check this issue on all platforms if needed. 

[henry]

On Thu, 2012-07-26 at 16:48 +0300, Denis Chapligin wrote:


>
> I have mixed 32/64 bit installations and clear 32-bit only/64-bit only
> installations, so i could try to check this issue on all platforms if
> needed.

Thanks - I might take you up on that, but I think will most likely turn
out to be a problem of an order dependency in the tests (there's some
static state in play), possibly combined with bit of bad code.

[henry]

I've pushed out a new build to the snapshots repository. I've not had chance to setup maven 2, so I'm not sure if the null pointer you were seeing is an issue with maven 2, or something else. I've put a null guard in so you hopefully you will no longer be blocked.

The other failing test looks to be due to a multi-threaded test elsewhere in the suite that sometimes leaves some static state in a condition that breaks the test. This test now cleans up after itself, so the build will now hopefully be stable.

All being well you'll now be able to try out the latest snapshot and see if the original issue is fixed.

Cheers

Henry

[Denis Chapligin]

Hello!

All being well you'll now be able to try out the latest snapshot and see if the original issue is fixed.

Everything works just great now! Thank you very much!