PIT giving line coverage 15% for the project while JaCoCo giving 44% using same tests

[Jarkko Rantavuori]

Hi,

I'm out of ideas as to why I'm getting so low line coverage, and thus also mutation coverage, while JaCoCo gives a lot bigger.

I've run PIT command line, and turned on verbose logging but there seems to be no issues reported.

Logging shows that these tests are used to account for coverage:

> java -cp "%ANT_HOME%/lib/ant.jar;%JTEST_LIB%;%PIT_PATH%;%JARJAR_PATH%/lib/asm-4.0.jar;%JARJAR_PATH%/lib/asm-commons-4.0.jar;%JARJAR_PATH%/lib/junit-4.8.1.jar;%JARJAR_PATH%/lib/maven-plugin-api.jar;%JARJAR_PATH%/testlib;%JARJAR_PATH%/testlib/enumtest.jar;%JARJAR_PATH%/target/classes;/bin" org.pitest.mutationtest.commandline.MutationCoverageReport --reportDir %PIT_REPORT_DIR% --targetClasses com.tonicsystems* --targetTests com.tonicsystems* --verbose --sourceDirs %JARJAR_PATH%/src/main,%JARJAR_PATH%/src/test,main 2>&1 | tee -a %PIT_LOG_FILE%

[snip]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.ClassPathEntryTest, name=com.tonicsystems.jarjar.util.ClassPathEntryTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.ClassPathEntryTest, name=com.tonicsystems.jarjar.util.ClassPathEntryTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.ClassPathEntryTest, name=com.tonicsystems.jarjar.util.ClassPathEntryTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.EntryStructTest, name=com.tonicsystems.jarjar.util.EntryStructTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.EntryStructTest, name=com.tonicsystems.jarjar.util.EntryStructTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.EntryStructTest, name=com.tonicsystems.jarjar.util.EntryStructTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.JarProcessorTest, name=com.tonicsystems.jarjar.util.JarProcessorTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.JarProcessorTest, name=com.tonicsystems.jarjar.util.JarProcessorTest]

18:20:56 PIT >> INFO : SLAVE : 18:20:56 PIT >> FINE : Gathering coverage for test Description [testClass=com.tonicsystems.jarjar.util.JarProcessorTest, name=com.tonicsystems.jarjar.util.JarProcessorTest]

[snip]

But when I look at the results, it is showing 0% for all the classes in util package. In contrast, JaCoCo gives 25.5% for the util package with the same tests. For the whole project figures are 15% by PIT, 44% by JaCoCo for line coverage.

Any ideas what I could check?

Br,
-Jarkko

[henry]

On Wednesday, 5 March 2014 17:02:30 UTC, Jarkko Rantavuori wrote:

But when I look at the results, it is showing 0% for all the classes in util package. In contrast, JaCoCo gives 25.5% for the util package with the same tests. For the whole project figures are 15% by PIT, 44% by JaCoCo for line coverage.

Any ideas what I could check?

PIT has a nasty habit of ignoring errors during the coverage stage and reporting 0 coverage, but it sounds like you've checked the output with verbose logging on and not seen anything.

If you can boil the issue down to a minimal example (i.e. 1 class and 1 test) and share the project I'll take a look.

Thanks

Henry

[Jarkko Rantavuori]

Hi Henry,

and thanks for the offer!

Since I really needed this fixed I digged some more into this.

Here is the example project I've been running on Windows, on Eclipse for JaCoCo coverage, command line for PIT testing: https://github.com/eis/pitest-hamcrest-issue-demo

You can see in the repo the two runs, successful one:

https://raw.github.com/eis/pitest-hamcrest-issue-demo/master/target/pitest/201403062208/index.html (showing 44% line coverage)

https://raw.github.com/eis/pitest-hamcrest-issue-demo/master/target/pitest/pitest-out.log (all output on verbose, no problems here)

and broken one:

https://raw.github.com/eis/pitest-hamcrest-issue-demo/master/target/pitest/201403062209/index.html (0% coverage)

https://raw.github.com/eis/pitest-hamcrest-issue-demo/master/target/pitest/pitest-out-nohamcrest.log (again no problems mentioned, except that it doesn't find the tests)

The runs are identical in all other respects, except the successful one has added hamcrest library to the classpath, and broken one hasn't.

Successful one: https://github.com/eis/pitest-hamcrest-issue-demo/blob/master/pit-jarjar.bat

Broken one: https://github.com/eis/pitest-hamcrest-issue-demo/blob/master/pit-jarjar-nohamcrest.bat

Now, in my trimmed down example it is clear on the broken one that something is wrong. However in a big project this is a really nasty issue, since only some of the tests aren't found if hamcrest library is missing, but some are, so you get worse coverage than you should but you still get something, making it a lot harder to notice things are wrong. That's why I didn't see anything special in the verbose logs at first.

Hamcrest missing should've thrown noclassdeffounderror or similar somewhere, but it doesn't. It's swallowed somewhere by PIT it seems.

I'll open an issue also to github about this.

-Jarkko

--
You received this message because you are subscribed to the Google Groups "PIT Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pitusers+u...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[henry]

On Thursday, 6 March 2014 20:25:39 UTC, Jarkko Rantavuori wrote:

The runs are identical in all other respects, except the successful one has added hamcrest library to the classpath, and broken one hasn't.

Yes, the error handling for errors that occur in the child jvms is not good - I look to finally get this fixed under the issue you raised.

Thanks

Henry

[Jarkko Rantavuori]

Hi,

I was finally able to debug this issue.

The problem seems to be this line:

https://github.com/hcoles/pitest/blob/master/pitest/src/main/java/org/pitest/junit/JUnitCustomRunnerTestUnitFinder.java#L62

because if the test has classnotfound error or some other error very early on, this condition is triggered, and test analysis is aborted silently. This will cause low code coderage without any explanations.

When I commented that line out, I get the error message properly:

22:00:59 PIT >> INFO : SLAVE : 22:00:59 PIT >> WARNING : JUnit error for class class com.tonicsystem
s.jarjar.DepFindTest : com.tonicsystems.jarjar.DepFindTest
22:00:59 PIT >> FINE : SLAVE : ERROR Description [testClass=com.tonicsystems.jarjar.DepFindTest, nam
e=com.tonicsystems.jarjar.DepFindTest] -> java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribin
g
22:00:59 PIT >> INFO : SLAVE : java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing
..
22:00:59 PIT >> INFO : Calculated coverage in 0 seconds.
Exception in thread "main" org.pitest.help.PitHelpError: All tests did not pass without mutation whe
n calculating line coverage. Mutation testing requires a green suite.
See http://pitest.org for more details.

The test case is at

https://github.com/eis/pitest-hamcrest-issue-demo

I'll update the details I've found to the issue I opened for this: https://github.com/hcoles/pitest/issues/105

and will prepare a pull request on removing that line so it won't cause silent errors anymore if that is the way to fix it.

However, is that the correct course of action? Is there some reason for that line?

-Jarkko

[Henry Coles]

and will prepare a pull request on removing that line so it won't cause silent errors anymore if that is the way to fix it.

However, is that the correct course of action? Is there some reason for that line?

Thanks for looking into this.

I'm a little surprised that removing that line doesn't cause a test to fail - does the suite still run green?

Henry

[Jarkko Rantavuori]

Hi,

The original problem that I was trying to solve was that PIT was silently failing, without giving any errors, but reporting line coverge 15% in contrast to 44% by JaCoCo. The situation is about missing library in the class path, but the problem PIT has is that it didn't report this in any way.

Commenting that line out makes PIT report the problem, so user is aware of what he should fix in the PIT configuration. If that line is in place, problem is never reported but user gets bad coverage anyway, tests are just skipped.

-Jarkko

[Jarkko Rantavuori]

So to be clear, the tests fail in both cases - but with that line present PIT doesn't let user know that it thinks the tests failed.

With the line commented out PIT reports the failures properly.

However, as pointed out in the issue by KyleRogers (https://github.com/hcoles/pitest/issues/105), there are problems with just removing that ilne - some other behaviour relies on it.

-Jarkko

[Jarkko Rantavuori]

perjantai, 2. toukokuuta 2014 22.32.39 UTC+3 Henry Coles kirjoitti:

Thanks for looking into this.

I'm a little surprised that removing that line doesn't cause a test to fail - does the suite still run green?

Sorry, I think I misunderstood you the first time.

No, it doesn't run green anymore. If I remove that line, these tests start to fail:

 JUnitCustomRunnerTestUnitFinderTest.shouldNotFindTestInNonTestClasses:165 null
  ConfigurationFactoryTest.shouldNotCreateAConfigurationThatFindsTestNGTestsWhenTestNGNotOnClassPath:68 null

Details of fails:

FAILURE! - in org.pitest.junit.JUnitCustomRunnerTestUnitFinderTest
shouldNotFindTestInNonTestClasses(org.pitest.junit.JUnitCustomRunnerTestUnitFinderTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: null
    at org.junit.Assert.fail(Assert.java:92)
    at org.junit.Assert.assertTrue(Assert.java:43)
    at org.junit.Assert.assertTrue(Assert.java:54)
    at org.pitest.junit.JUnitCustomRunnerTestUnitFinderTest.shouldNotFindTestInNonTestClasses(JUnitCustomRunnerTestUnitFinderTest.java:165)

 FAILURE! - in org.pitest.mutationtest.config.ConfigurationFactoryTest
shouldNotCreateAConfigurationThatFindsTestNGTestsWhenTestNGNotOnClassPath(org.pitest.mutationtest.config.ConfigurationFactoryTest)  Time elapsed: 0 sec  <<< FAILURE!
java.lang.AssertionError: null
    at org.junit.Assert.fail(Assert.java:92)
    at org.junit.Assert.assertTrue(Assert.java:43)
    at org.junit.Assert.assertTrue(Assert.java:54)
    at org.pitest.mutationtest.config.ConfigurationFactoryTest.shouldNotCreateAConfigurationThatFindsTestNGTestsWhenTestNGNotOnClassPath(ConfigurationFactoryTest.java:68)

[Jarkko Rantavuori]

I have now a first suggestion for a fix. Fix removes the problematic line and adds a check for test methods instead.

See the discussion here, at the end I've linked to my changes:

https://github.com/hcoles/pitest/issues/105

-Jarkko