Adding method-level coverage

32 views
Skip to first unread message

Antoine Baudoux

unread,
Oct 9, 2021, 6:33:07 AM10/9/21
to JaCoCo and EclEmma Users
Hello, 

Would you welcome a contribution to add method-level coverage?
Method-level coverage (as opposed to line-level, the current default mode for JaCoCo) can be useful to track dead code, i.e. methods that never get called.

It has lower overhead and therefore could be enabled continuously for code running in production, allowing for dead code detection and removal.

If you think that it could be added to the JaCoCo codebase I would be happy to contribute. Please let me know your thoughts.

Kind regards, 

Antoine

Marc Hoffmann

unread,
Oct 19, 2021, 11:11:30 AM10/19/21
to JaCoCo and EclEmma Users
Dear Antoine,

JaCoCo already supports method level coverage.

What exactly is the performance problem that requires a different implementation?

In general we’re very reluctant to such configuration options as they require different implementations for the whole path: Instrumentation, exec files and analysis. And of course causes more questions by users that get these configuration options wrong...

Regards,
-marc



--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/e6da49e8-56a9-4c14-9e9d-fdea4c152349n%40googlegroups.com.

Antoine Baudoux

unread,
Oct 19, 2021, 11:16:56 AM10/19/21
to jac...@googlegroups.com
By "method level coverage" I mean only adding one counter, one instruction at the start of each method to check if the method was called or not. And not adding any branch counter, so that there is less instrumentation overhead.
I did not see a way to do that, if it is already supported I'm sorry about the confusion.

Kind regards, 

Antoine


Marc Hoffmann

unread,
Oct 19, 2021, 11:56:09 AM10/19/21
to jac...@googlegroups.com
Dear Antoine,

adding different counters is actually not supported.

Maintaining different instrumentation/analysis strategies is something we‘re actually not looking for.

Do you have performance meassurements for your use case?

Best regards,
-marc

On 19. Oct 2021, at 17:16, 'Antoine Baudoux' via JaCoCo and EclEmma Users <jac...@googlegroups.com> wrote:



Antoine Baudoux

unread,
Oct 29, 2021, 7:37:07 AM10/29/21
to jac...@googlegroups.com
Do you have performance meassurements for your use case?

Yes, we did some benchmarks with a home-made agent that did function-level instrumentation.
The instrumentation overhead was anywhere from 2x to 5x with instruction-level coverage than function-level coverage.

Marc Hoffmann

unread,
Oct 29, 2021, 8:58:46 AM10/29/21
to JaCoCo and EclEmma Users
Wow, can you share your test setup? Our measurements e.g. running test suites of apache commons-collections show an overhead of about 20%.


Antoine Baudoux

unread,
Nov 1, 2021, 6:32:48 AM11/1/21
to jac...@googlegroups.com
We attached both the jacoco agent and our custom method-level-only agent ("funccover") to some sample codes and averaged their run execution time.

We attached “jacocoagent” and “funccover” to Google Java Formatter to test and compare “funccover”s performance. Results are the average of 200 runs.
Normally formatter’s execution time is around 200 seconds, with “jacocoagent” it runs around 97 seconds (48% overhead) slower and for “funccover” it runs around 30 seconds slower (15% overhead). So our agent adds x3 less overhead for this case.

2) Merge Sort Algorithm
In this test we implemented a program that sorts an array of size 10^7 with a merge sort algorithm. In each binary, we sort the array 50 times per run to be able to ignore the constant overhead of instrumentation since it is only done once per run. There is no I/O bottleneck and most of the time is spent on the CPU. Binary runs around 216 seconds without the instrumentation. Our agent "funccover" adds around 1.97% overhead whilst jacoco adds around 11.4%. Our instrumentation adds 5x less overhead. Results are average of 50x runs.

3) Simple Function Calls - Filtered
We attached “jacocoagent” and “funccover” to a simple program that calls the following function 10^8 times with numbers from 1 to 10^8.

 static int f(int x) {
   if(x == 0) return 0;
   if(x % 2 == 0) return f(x / 2);
   return f(x / 2) + 1;
 }

Results are average of 500x runs.
Here jacoco adds 17% overhead whilst funccover adds less than 7% overhead.

All this makes sense of course, you are updating less often the coverage data if you only add a single probe per method.

Marc Hoffmann

unread,
Nov 1, 2021, 8:13:25 AM11/1/21
to JaCoCo and EclEmma Users
Thanks Antoine, for the detailed measurements! Good to know what the actual overhead in different scenarios is. While I’m a bis surprised by the overhead in the first scenario (for both agents).

Is the code for the “funccover” agent somewhere available? I wonder how much extra code, config options etc this would add to JaCoCo. As said before our resources are very limited and we cannot afford to maintain code for edge cases.

Best regards,
-marc



Antoine Baudoux

unread,
Nov 1, 2021, 8:38:08 AM11/1/21
to jac...@googlegroups.com
The gist of it is simply a stripped down version of Jacoco's MethodProbesAdapter that only inserts a probe at the start of each method : 

public final class MethodProbesAdapter extends MethodVisitor {

  private final MethodProbesVisitor probesVisitor;

  private final IProbeIdGenerator idGenerator;

  /**
   * Create a new adapter instance.
   *
   * @param probesVisitor visitor to delegate to
   * @param idGenerator generator for unique probe ids
   */
  public MethodProbesAdapter(
      final MethodProbesVisitor probesVisitor, final IProbeIdGenerator idGenerator) {
    super(InstrSupport.ASM_API_VERSION, probesVisitor);
    this.probesVisitor = probesVisitor;
    this.idGenerator = idGenerator;
  }

  @Override
  public void visitCode() {
    mv.visitCode();
    probesVisitor.visitProbe(idGenerator.nextId());
  }
}


The rest of the POC is a bunch of forked classes only needed to install this new MethodProbesAdapter into the Jacoco Runtime.

That is the biggest issue we have : It is right now complicated to install a new probe placement strategy into Jacoco. If it was easy we would simply have used our own version of MethodProbesAdapter,no need to change Jacoco at all.

Another use case we have for custom probe placement strategy : We know for sure that certain methods are called in production, and we want to figure out what is happening for the other methods. So we want to insert probes only for the methods for which we are not sure if they are executed or not.

So to summarize, we don't need to add other probe placement strategies to Jacoco, but rather make it easier to develop and install your own.

What do you think?

Marc Hoffmann

unread,
Nov 1, 2021, 8:59:59 AM11/1/21
to jac...@googlegroups.com
Hi Antoine,

I also see this complexity when integrating this alternative instrumentation mode into JaCoCo.

Given the fact that performance wasn’t an issue for our users so far, I would also recommend that you do this in a separate project.

Best regards,
-marc


Reply all
Reply to author
Forward
0 new messages