Wrong coverage shown when testing same class multiple times

[juzer]

Hi everyone,

I have Spring Boot application on which I run coverage check using jacoco-maven-plugin in version 0.7.9. The strange thing I found is that when a class is tested with more then one test class the results are not combined but I get coverage calculated using only one of the test classes.

For example: I have a Service which is covered almost 100% with ServiceTest. And then I have a Controller which depends on Service and a separate ControllerTest which by testing the Controller also touches some bits of the Service class. Now when I run all my tests with JaCoCo I get the coverage of the Service at only about 40% which means only the result of running the ControllerTest were taken into account. And the ServiceTest results were completely ignored although the tests there were executed.

So the question is whether there is something wrong here or this is simply how JaCoCo works?

[Evgeny Mandrikov]

Hi,

On Monday, October 2, 2017 at 2:56:48 PM UTC+2, juzer wrote: 

So the question is whether there is something wrong here or this is simply how JaCoCo works?

Something seems to be wrong here, because coverage after execution of

public class ATest {

  @org.junit.Test

  public void a() {

    new A().a();

  }

}

public class BTest {

  @org.junit.Test

  public void b() {

    new B().b();

  }

}

for

public class A {

  void a() {

  }

}

public class B {

  void b() {

  }

}

is 100% for both A and B classes.

And can't say what's wrong in your case in absence of reproducer/example.

[piotr.n...@gmail.com]

W dniu poniedziałek, 2 października 2017 18:24:06 UTC+2 użytkownik Evgeny Mandrikov napisał:
> And can't say what's wrong in your case in absence of reproducer/example.

Thank you for response Evgeny.

I managed to prepare an example of my case and pinpoint the issue.
https://github.com/juzer/spring-jacoco-demo
It seems to be related to the inheritance hierarchy and abstract classes. I have an AbstractService which is tested by instantiating an anonymous class and it's covered 100%. Then I have a concrete subclass that is also tested (only methods in this class). When I run all the tests however, the coverage of AbstractService is 0. Moving all tests to the ServiceTest results in correct coverage report - 100% on both classes.

Is there anything special about how JaCoCo treats abstract classes?

[Evgeny Mandrikov]

On Wednesday, October 4, 2017 at 5:52:46 PM UTC+2, piotr.n...@gmail.com wrote:

 

Is there anything special about how JaCoCo treats abstract classes?

No. No magic: executed = covered, not executed = not covered. And you can answer such and any following similar questions on your own by trying simple plain java examples.

 

I managed to prepare an example of my case and pinpoint the issue. 
https://github.com/juzer/spring-jacoco-demo 
It seems to be related to the inheritance hierarchy and abstract classes. I have an AbstractService which is tested by instantiating an anonymous class and it's covered 100%. Then I have a concrete subclass that is also tested (only methods in this class). When I run all the tests however, the coverage of AbstractService is 0. Moving all tests to the ServiceTest results in correct coverage report - 100% on both classes. 

I'll have a look later.

[Evgeny Mandrikov]

On Wednesday, October 4, 2017 at 6:33:22 PM UTC+2, Evgeny Mandrikov wrote:

I managed to prepare an example of my case and pinpoint the issue. 
https://github.com/juzer/spring-jacoco-demo 
It seems to be related to the inheritance hierarchy and abstract classes. I have an AbstractService which is tested by instantiating an anonymous class and it's covered 100%. Then I have a concrete subclass that is also tested (only methods in this class). When I run all the tests however, the coverage of AbstractService is 0. Moving all tests to the ServiceTest results in correct coverage report - 100% on both classes. 

I'll have a look later.

Execution of

mvn clean test

produces

Running com.example.demo.ServiceTest

Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.154 sec - in com.example.demo.ServiceTest 

i.e. AbstractServiceTest is not executed!! hence not covered as expected. BTW addition of debug output allows to easily pinpoint if code is really executed.

According to https://stackoverflow.com/questions/25813941/spring-boot-my-unit-tests-are-being-skipped :

"Spring Boot configures the Surefire plugin to run all test classes that have a name ending with Test or Tests but not starting with Abstract."

After renaming into "MyAbstractServiceTest":

Regards,

Evgeny

[piotr.n...@gmail.com]

> i.e. AbstractServiceTest is not executed!! hence not covered as expected. BTW addition of debug output allows to easily pinpoint if code is really executed.
>
>
> According to https://stackoverflow.com/questions/25813941/spring-boot-my-unit-tests-are-being-skipped :
> "Spring Boot configures the Surefire plugin to run all test classes that have a name ending with Test or Tests but not starting with Abstract."

Oh my, your are obviously right! That explains why running only single test succeeds but not when running the whole suite.

That's a nasty "hidden" feature of Spring Boot...

Thanks a lot for your help Evgeny!