Getting Coverage in Memory via API

[Yevgeny P]

Hey Team,

Let's say I've started a Java program with Jacoco agent. Is it possible to retrieve the coverage from within the Java program itself via somekind of API or it should be an external program?

I only saw an example doing something similar for a single class without Jacoco agent.

Thanks,

Yevgeny

[Marc Hoffmann]

Hi Yevgeny,

it is possible to talk to the agent through its runtime API: https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/agent/rt/package-summary.html

Regards,

-marc

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/ab4e80bc-1c1e-4189-87dc-640d2ce5858a%40googlegroups.com.

[yev...@fuzzit.dev]

Thanks! I'll try that out.  Also Im trying to start the coverage in a maven plugin, kind like you did here: https://github.com/jacoco/jacoco/blob/master/jacoco-maven-plugin/src/org/jacoco/maven/AbstractAgentMojo.java .

But this doesn't work for also I don't understand how it works in jacoco as the JVM is already started or does it spawn another JVM per test and then it's start the agent like specified in the properties.

On Tuesday, November 5, 2019 at 12:54:45 AM UTC+2, Marc R. Hoffmann wrote:

Hi Yevgeny,

it is possible to talk to the agent through its runtime API: https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/agent/rt/package-summary.html

Regards,

-marc

On 4. Nov 2019, at 21:22, Yevgeny P <yev...@gmail.com> wrote:

Hey Team,

Let's say I've started a Java program with Jacoco agent. Is it possible to retrieve the coverage from within the Java program itself via somekind of API or it should be an external program?

I only saw an example doing something similar for a single class without Jacoco agent.

Thanks,

Yevgeny

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to jac...@googlegroups.com.

[Yevgeny P]

I think I got it, this goal is running before the test goal so it's changes the JVM command-line. I have different problem now - I'm running with MVN_OPTS="-javaagent:<path>" this indeed generates the exec coverage file when the program exits but it I'm not able to use the API as I get:

Caused by: java.lang.IllegalStateException: JaCoCo agent not started. even though it is started as I get the coverage after the program exits.

[Marc Hoffmann]

Interesting. Can you please provide the full stack trace?

To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/290d2d71-da0c-4085-ad5d-81842f2d33fe%40googlegroups.com.

[Yevgeny P]

Hey Marc, Attached is a github repo with complete reproduction steps in the README https://github.com/yevgenypats/jacoco-test

Thanks again.

On Wednesday, November 6, 2019 at 6:40:24 AM UTC+2, Marc R. Hoffmann wrote:

Interesting. Can you please provide the full stack trace?

On 5. Nov 2019, at 11:41, Yevgeny P <yev...@gmail.com> wrote:

I think I got it, this goal is running before the test goal so it's changes the JVM command-line. I have different problem now - I'm running with MVN_OPTS="-javaagent:<path>" this indeed generates the exec coverage file when the program exits but it I'm not able to use the API as I get:

Caused by: java.lang.IllegalStateException: JaCoCo agent not started. even though it is started as I get the coverage after the program exits.

On Tuesday, November 5, 2019 at 8:37:07 AM UTC+2, yev...@fuzzit.dev wrote:

Thanks! I'll try that out.  Also Im trying to start the coverage in a maven plugin, kind like you did here: https://github.com/jacoco/jacoco/blob/master/jacoco-maven-plugin/src/org/jacoco/maven/AbstractAgentMojo.java .

But this doesn't work for also I don't understand how it works in jacoco as the JVM is already started or does it spawn another JVM per test and then it's start the agent like specified in the properties.

On Tuesday, November 5, 2019 at 12:54:45 AM UTC+2, Marc R. Hoffmann wrote:

Hi Yevgeny,

it is possible to talk to the agent through its runtime API: https://www.jacoco.org/jacoco/trunk/doc/api/org/jacoco/agent/rt/package-summary.html

Regards,

-marc

On 4. Nov 2019, at 21:22, Yevgeny P <yev...@gmail.com> wrote:

Hey Team,

Let's say I've started a Java program with Jacoco agent. Is it possible to retrieve the coverage from within the Java program itself via somekind of API or it should be an external program?

I only saw an example doing something similar for a single class without Jacoco agent.

Thanks,

Yevgeny

-- 
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jac...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/ab4e80bc-1c1e-4189-87dc-640d2ce5858a%40googlegroups.com.

-- 
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jac...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/290d2d71-da0c-4085-ad5d-81842f2d33fe%40googlegroups.com.

[Evgeny Mandrikov]

Hi,

On Wednesday, November 6, 2019 at 8:26:43 AM UTC+1, Yevgeny P wrote:

Hey Marc, Attached is a github repo with complete reproduction steps in the README https://github.com/yevgenypats/jacoco-test

AFAIK Maven plugins use child-first classloading strategy ( https://maven.apache.org/guides/mini/guide-maven-classloading.html ),

so the most reasonable explanation - is that you have JaCoCo classes in classpath of plugin and therefore they are used instead of actual classes of agent.

As a proof - https://github.com/yevgenypats/jacoco-test/pull/1

[Yevgeny P]

Thank you, this works!!

[Yevgeny P]

Hey, This indeed works. But the performance is pretty slow in terms of both the call to getExecutionData(0.5s) and parsing the data(0.5s). Is there anything to do with that or I would need to reimplement the instrumentation myself if I would like this to be faster?

[Evgeny Mandrikov]

On Thursday, November 7, 2019 at 8:06:55 AM UTC+1, Yevgeny P wrote: 

the call to getExecutionData(0.5s)

If you use reflective call that was shown by me as is, without usual speedup tricks for reflective calls, then I'm afraid that you're measuring reflection overhead and not at all performance of this method in JaCoCo.

This method boils down mostly to just simple traversal of an entries in a Map.

 

and parsing the data(0.5s).

I'm not sure what you mean by "parsing the data", because there is nothing to parse in this byte array - for further analysis it is used mostly as it is, undergoing only simple bytes into booleans decompression - https://github.com/jacoco/jacoco/blob/v0.8.5/org.jacoco.core/src/org/jacoco/core/data/ExecutionDataReader.java

If you mean actual analysis, then AFAIR on example of JDK rt.jar classes JaCoCo is capable of analysing at speed ~20 Mb of class files per second on even non-warmed JVM, which is thousands of classes and which is IMO pretty good given HDD and OS data access overheads, which scales linearly with amount of classes, and which is much faster than execution of all tests for these classes.

But the performance is pretty slow

Of course if you do both operations in a very tight loop on a big amount of classes with just a tiny bit of something else, then of course compared to this something else, they will look slow. But IMO this is incredibly suspicious use-case and even benchmark to look correct. 

Is there anything to do with that or I would need to reimplement the instrumentation myself if I would like this to be faster?

Instrumentation is not involved in a call of first method and not performed at a time of analysis of its data. 

All in all IMO when speaking about performance, one need to be extremely careful/pedantic and to avoid bold statements about poor performance must make sure that benchmark/measurement/use-case is relevant and makes sense, and must describe not only the use-case, but also the way of measurement and setup/environment.

To add few examples of why this matters: How far your measurement from the time to read same class files from disk? to parse them using ASM library? If not far, then you're actually not measuring performance of JaCoCo code ;) and etc.

Regards,

Evgeny

[Yevgeny P]

Hey Evgeny,

Thanks you for the detailed response, and sorry if I offended JaCoCo performance:) I didn't mean that.

It is totally possible that I'm getting something wrong (I'm far from being a Java expert). I'll share both the use-case and the benchmark code.

I'm porting my jsfuzz library to a new javafuzz (coverage guided fuzzer) library where I want to use JaCoCo as the coverage library.

Essentially for the basic first version of javafuzz I only need the total hit count and I don't need to reset the counters so that should help save some time.

Currently I'm getting 930 execs/s (I had a mistake in my early benchmarks) for both calling getExecutionData and getting the total hitCount.

Maybe it is possible to speed it up both with the reflection tricks + I can add a Pull-Request for JaCoCo to save the total number of hitCounts. Just as reference the "same logic" in jsfuzz and pythonfuzz can get to about 20,000 execs/s (inclduing the overhead of the fuzzer logic itself which is currently not in the benchmark).  

here is a link to the updated repo with the benchmarks.

Much appreciated,

Also Yevgeny:)

[Marc Hoffmann]

Hi Yevgeny,

that sounds like an interesting project! If you publish it let us know, we can add your project to our integration matrix.

I added an experimental API for you that just counts the number of hits (no overhead for serializing the data):

https://github.com/jacoco/jacoco/tree/experimental-hit-count-api

Can you please tell us, whether this improves your performance?

Cheers,

-marc

To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/517ba116-ce05-4b3e-8a73-6d683d1b7be5%40googlegroups.com.

[Yevgeny P]

Hey Marc,

Wow, that was fast! this improves performance by 5-6x. I think this is now feasible for the first version and I can start working on the port. I'll keep you posted once I have the first version ready.

Also, can you push this api to a release version?

Much appreciated!

Yevgeny

On Friday, November 8, 2019 at 5:02:42 PM UTC+2, Marc R. Hoffmann wrote:

Hi Yevgeny,

that sounds like an interesting project! If you publish it let us know, we can add your project to our integration matrix.

I added an experimental API for you that just counts the number of hits (no overhead for serializing the data):

https://github.com/jacoco/jacoco/tree/experimental-hit-count-api

Can you please tell us, whether this improves your performance?

Cheers,

-marc

To unsubscribe from this group and stop receiving emails from it, send an email to jac...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/517ba116-ce05-4b3e-8a73-6d683d1b7be5%40googlegroups.com.

[Marc Hoffmann]

Hallo Yevgeny,

as you can guess from the branch name (“experimental”) I didn’t consider this change for merge into master.

Probes are an implementation detail and the semantic can change whenever we adjust the instrumentation strategy. This actually happens from time to time.

Also due to maintenance reasons we very reluctant when adding new APIs.

May I propose that you continue working on our experimental branch. When you’re feature complete we can check whether the simple probe count is still sufficient (to be honest I don’t really understand what its purpose is) and we can double check where the actual performance bottleneck is located.

Cheers,

-marc

To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/e23e17aa-fdd2-4446-b2b2-3e05f76c0313%40googlegroups.com.

[Yevgeny Pats]

Hey Marc,

Sounds good. I'll keep you posted with the progress.

Thanks!

Yevgeny

To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/D799CE74-8BCD-43D7-B5AC-D18BC1D10803%40mountainminds.com.