Should methods dump(true) or reset() of IAgent class reset/reload classes?

[José Carlos de Campos]

Hi,

I found the methods dump(true) or reset() of IAgent class very useful when we want

to get, for example, periodically coverage. However, I also found that both methods

do not work I was expecting, special for classes with static fields.

Let's suppose we have the following class:

1. public class PublicStaticFields {

2.  public static String className = PublicStaticFields.class.getCanonicalName();

    public static String s;

    static {
5.    s = "SsS";
6.  }
   }

and the following test class which achieves 100% line coverage.

import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;

import org.junit.Test;

public class TestPublicStaticFields {

  @Test
  public void test1() {
    assertEquals(PublicStaticFields.class.getCanonicalName(), PublicStaticFields.className);
    assertEquals("SsS", PublicStaticFields.s);
  }

  @Test
  public void test2() {
    // empty
  }

  @Test
  public void test3() {
    assertEquals(PublicStaticFields.class.getCanonicalName(), PublicStaticFields.className);
    assertEquals("SsS", PublicStaticFields.s);
  }

  @Test
  public void test4() {
    assertNotNull(new PublicStaticFields());
  }
}

Now, suppose we want some periodically coverage. To keep the example simple, let's

assume periodically as per test case. I.e., every time a test case finishes, coverage

is collected and dumped. (something similar to what sonar-jacoco-listeners does)

For this example we should get

• 3 lines covered by test1 (lines 2, 5, and 6)
  
• 0 lines covered by test2
  
• 3 lines covered by test3 (lines 2, 5, and 6)
  
• 4 lines covered by test4 (lines 1, 2, 5, 6)

however we get:

• 3 lines covered by test1 (lines 2, 5, and 6) : As expected
  
• 0 lines covered by test2 : As expected
  
• 0 lines covered by test3 : Lines 2, 5, and 6 should have been covered
  
• 1 line covered by test4 (line 1) : Lines 2, 5, and 6 should have been covered as well
  

I do not think there is any bug in methods dump or reset of class IAgent, I just found it a bit

odd that every time we dump/reset coverage we lost coverage of static fields.

As far I know, the only option to get coverage data of static fields per test case is by loading

all classes under test with a custom classloader. And I don't think JaCoCo implements its

own classloader, so it is not surprising that we lose track of coverage of static fields. Please

correct me if JaCoCo actually implements is own classloader.

Any thoughts on this? Would a JaCoCo's custom classloader be able to address the above

scenario? What would the drawbacks be? As far I understood, classes get instrumented when

they are loaded, assuming they will be reloaded several times, would that mean they would

have to be instrumented several times as well? (I wonder how much would that affect JaCoCo's

runtime).

--

Thanks in advance,

Jose

[Evgeny Mandrikov]

Hi,

IMO term "coverage of static fields" is quite vague. And so not clear what and why you want to measure? From here and given what is explained below any tricks with ClassLoaders sound crazy.

JaCoCo does not record access to fields. It records execution of code that initializes fields. Code to initialize static fields is executed when class is initialized. So behavior that you observe is correct one, because class is initialized only once.

JaCoCo doesn't introduce any new ClassLoaders into application and doesn't do any tricks with existing ones.

If two ClassLoaders load class (not return already loaded class), then at runtime these two classes are different, with all the consequences. Including two initializations of two classes, memory allocation in JVM to represent each, two instrumentations, etc. Not counting that this might be not expected by your application and tests and hence they might work incorrectly - e.g. usage of instance of one class as an argument to the equals method of another class will give you false even if getClass().getName() is the same, classes were loaded from the same class file on disk and have the same code:

  class Foo {

    public boolean equals(Object o) {

      if (o == null || getClass() != o.getClass()) {

        return false;

      }

      ...

    }

  }

As far as JaCoCo concerned: if at runtime you have two different classes that are actually the same code loaded by different class loaders, JaCoCo knows that their code is the same thanks to http://www.jacoco.org/jacoco/trunk/doc/classids.html So recorded coverage will be stored in a single place, even if there will be two instrumentations.

Hope this helps.

Evgeny

[José Carlos de Campos]

Hi Evgeny,

First of all, thanks for your detailed response.

Please find my comments below.

On Tuesday, August 29, 2017 at 9:38:35 PM UTC+1, Evgeny Mandrikov wrote:

Hi,

IMO term "coverage of static fields" is quite vague. And so not clear what and why you want to measure?

In this example I want to measure coverage per test case as if I was running every single

test case on a new JVM. (I'm aware that running each test case on a new JVM would be

insanely expensive, that's the reason I'm exploring other solutions)

The motivation behind is, suppose that test3 from my example actually triggers a bug.

Assuming the current coverage per test case from my example, it won't make any sense

to report failing test case test3 to a developer because it does not cover anything.

From here and given what is explained below any tricks with ClassLoaders sound crazy.

Yes, I do agree with you. Any trick with any classloader would be indeed crazy and most

likely won't work for all cases (as for example the one you kindly described below).

JaCoCo does not record access to fields. It records execution of code that initializes fields. Code to initialize static fields is executed when class is initialized. So behavior that you observe is correct one, because class is initialized only once.

Any suggestion on how can I make coverage per test case 100% accurate if only the first

test case (that makes the class-under-test be loaded and initialised) would record the

execution of code that initialise static fields? I.e., how can I address the fact that coverage

reported for test3 (from my example) is 0, when it should be 3 lines?

JaCoCo doesn't introduce any new ClassLoaders into application and doesn't do any tricks with existing ones.

Good to know, thanks.

 

If two ClassLoaders load class (not return already loaded class), then at runtime these two classes are different, with all the consequences. Including two initializations of two classes, memory allocation in JVM to represent each, two instrumentations, etc. Not counting that this might be not expected by your application and tests and hence they might work incorrectly - e.g. usage of instance of one class as an argument to the equals method of another class will give you false even if getClass().getName() is the same, classes were loaded from the same class file on disk and have the same code:

  class Foo {

    public boolean equals(Object o) {

      if (o == null || getClass() != o.getClass()) {

        return false;

      }

      ...

    }

  }

Thanks for the detailed example.

 

As far as JaCoCo concerned: if at runtime you have two different classes that are actually the same code loaded by different class loaders, JaCoCo knows that their code is the same thanks to http://www.jacoco.org/jacoco/trunk/doc/classids.html So recorded coverage will be stored in a single place, even if there will be two instrumentations.

I'm aware of it.

--

Best,

Jose

[Evgeny Mandrikov]

No code in class is executed by test3 and so 0 is a correct value, not 3. And since no code is executed, then how it can fail? Something is missing in your example/explanation.

What is the target of test3? What it should test / catch? If target is a process of initialization and state after initialization, then you need to isolate this process in this test to guarantee that it happens here (note that if something else used class prior to test1, then even test1 won't observe initialization), so indeed I don't see any other ways than separate JVM or properly coded ClassLoader. If initialization and its test do not have "ClassLoader leak", then JVM and GC will free memory that was used during this process. And as was said before - this won't impact memory consumed by JaCoCo. And I suppose time spent on instrumentation is negligibly small compared to the rest. You can do own benchmarks and compre cases with and without instrumentation/coverage - this is an interesting exercise and we'll be interested to know results. However as a rough approximation - time to instrument rt.jar (20693 classes) of JDK 8u131 on my not so fast machine is about 15 seconds (including reading from disk, unpacking from JAR, instrumentation, packing to JAR and writing back to disk).

[José Carlos de Campos]

On Wednesday, August 30, 2017 at 12:31:22 AM UTC+1, Evgeny Mandrikov wrote:

No code in class is executed by test3 and so 0 is a correct value, not 3.

Coverage of test3 is only 0 because test cases test1 and test3 are executed on the

same JVM. However, when test cases are executed in isolation (i.e., one JVM per

test) both test cases cover 3 lines.

And since no code is executed, then how it can fail? Something is missing in your example/explanation.

Lets change for a moment the following assertion

assertEquals("SsS", PublicStaticFields.s);

of test3 to something like

assertEquals("fooBar", PublicStaticFields.s);

Now test3 fails, right? How would we explain that fails and does not cover anything? 

What is the target of test3? What it should test / catch?

Exactly the same as test1, the process of initialisation.

 

If target is a process of initialization and state after initialization, then you need to isolate this process in this test to guarantee that it happens here (note that if something else used class prior to test1, then even test1 won't observe initialization),

Yes, I totally agree with you. Thing is, there are really bad test suites / cases out there

that are not coded properly. And I'm just trying to find a common solution that could be

accurate even for those shit test suites / cases.

 

so indeed I don't see any other ways than separate JVM

I know maven (by default) forks a new JVM per test class, but I do not know any option

to fork a new JVM per test case. Are you aware of any?

 

or properly coded ClassLoader.

How could that be done? Can you please provide me some pointers?

 If initialization and its test do not have "ClassLoader leak", then JVM and GC will free memory that was used during this process. And as was said before - this won't impact memory consumed by JaCoCo. And I suppose time spent on instrumentation is negligibly small compared to the rest. You can do own benchmarks and compre cases with and without instrumentation/coverage - this is an interesting exercise and we'll be interested to know results. However as a rough approximation - time to instrument rt.jar (20693 classes) of JDK 8u131 on my not so fast machine is about 15 seconds (including reading from disk, unpacking from JAR, instrumentation, packing to JAR and writing back to disk).

20693 classes in just 15 seconds, wow.

--

Best,

Jose

[Evgeny Mandrikov]

On Wednesday, August 30, 2017 at 10:07:02 AM UTC+2, José Carlos de Campos wrote:

On Wednesday, August 30, 2017 at 12:31:22 AM UTC+1, Evgeny Mandrikov wrote:

No code in class is executed by test3 and so 0 is a correct value, not 3.

Coverage of test3 is only 0 because test cases test1 and test3 are executed on the

same JVM. However, when test cases are executed in isolation (i.e., one JVM per

test) both test cases cover 3 lines.

And since no code is executed, then how it can fail? Something is missing in your example/explanation.

Lets change for a moment the following assertion

assertEquals("SsS", PublicStaticFields.s);

of test3 to something like

assertEquals("fooBar", PublicStaticFields.s);

Now test3 fails, right? How would we explain that fails and does not cover anything? 

Code coverage alone will never give you an ultimate answer of why test fails. Code coverage is a way to observe which code was executed. And once again - it is correctly reported that no code was executed.

Speaking of why test fails: If target of this test is process of class initialization, then this test is incorrect as it doesn't test target. Or simply because assertion in test is incorrect. Or because of bug in assertions library, etc.

 

What is the target of test3? What it should test / catch?

Exactly the same as test1, the process of initialisation.

See above.

 

 

If target is a process of initialization and state after initialization, then you need to isolate this process in this test to guarantee that it happens here (note that if something else used class prior to test1, then even test1 won't observe initialization),

Yes, I totally agree with you. Thing is, there are really bad test suites / cases out there

that are not coded properly. And I'm just trying to find a common solution that could be

accurate even for those shit test suites / cases.

 

so indeed I don't see any other ways than separate JVM

I know maven (by default) forks a new JVM per test class, but I do not know any option

to fork a new JVM per test case. Are you aware of any?

No.

 

 

or properly coded ClassLoader.

How could that be done? Can you please provide me some pointers?

There are plenty of materials about ClassLoaders in internet.