Falsely low coverage when test methods are inherited

[Chris Hennick]

I'm not quite sure whether this is a bug in JaCoCo or in Coveralls.io, because JaCoCo doesn't produce reports of its own anywhere I can find.

I recently refactored some test classes to pull out their common functionality into an abstract class, from which they now inherit most of their @Test methods. The inherited methods in turn call overridden methods to instantiate whichever class is under test. But my reported coverage went way down, and went to zero on several classes that I *know* have nonzero coverage!

An example is at https://coveralls.io/builds/13126009/source?filename=src%2Fmain%2Fjava%2Fbetterrandom%2Fprng%2FAesCounterRandom.java. The tests for that class are in these files:

• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/BaseEntropyCountingRandomTest.java
• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/BaseRandomTest.java
• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/AesCounterRandomTest128.java
• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/AesCounterRandomTest256.java
  

Any way to get accurate coverage when the tests are written this way?

[Evgeny Mandrikov]

Hi,

On Tuesday, September 5, 2017 at 10:59:46 AM UTC+2, Chris Hennick wrote:

I'm not quite sure whether this is a bug in JaCoCo or in Coveralls.io, because JaCoCo doesn't produce reports of its own anywhere I can find.

JaCoCo does produce own reports. Even taking your project as example - in file "coverage.sh" you have "mvn jacoco:prepare-agent test jacoco:report" that results in generation of native JaCoCo report in directory "target/site/jacoco".

 

I recently refactored some test classes to pull out their common functionality into an abstract class, from which they now inherit most of their @Test methods. The inherited methods in turn call overridden methods to instantiate whichever class is under test. But my reported coverage went way down, and went to zero on several classes that I *know* have nonzero coverage!

An example is at https://coveralls.io/builds/13126009/source?filename=src%2Fmain%2Fjava%2Fbetterrandom%2Fprng%2FAesCounterRandom.java. The tests for that class are in these files:

• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/BaseEntropyCountingRandomTest.java
• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/BaseRandomTest.java
• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/AesCounterRandomTest128.java
• https://github.com/Pr0methean/BetterRandom/blob/de90df9fef137f022d63db4c079057d67af0364f/src/test/java/betterrandom/prng/AesCounterRandomTest256.java
  

Any way to get accurate coverage when the tests are written this way?

JaCoCo records execution of code and doesn't depend on a way you do this - tests can be even manual.

After execution of "mvn test" in TestNG report there is no execution of "AesCounterRandomTest256", nor "AesCounterRandomTest128". Absence of execution of tests perfectly explains absence of coverage of corresponding code. I'm guessing they are not considered as tests as their name doesn't end on "Test".

Regards,

Evgeny