No Coverage using @RunWith(Parameterized.class)

511 views
Skip to first unread message

Svante Schubert

unread,
Nov 3, 2021, 4:34:17 PM11/3/21
to JaCoCo and EclEmma Users
Hello coverage experts!

Does anybody have success using JaCoCo and JUnit's @RunWith(Parameterized.class)?

I am working on the ODF file format and I am running a test transformation for every existing ODF test document in our test folder comparing it with the prior reference result.


By using this generic test the coverage of the changes packages should be high, but the coverage of package org/odftoolkit/odfdom/changes is instead zero!

Thanks in advance for any hints!
Svante


Message has been deleted

Marc Hoffmann

unread,
Nov 3, 2021, 5:38:25 PM11/3/21
to JaCoCo and EclEmma Users
Hi Svante,

JaCoCo does not depend on a specific testing frameworks and I don’t think there are technical limitations with @RunWith(Parameterized.class). Maybe it does not work at all in your case and the code is in fact not executed?

If you think there is a problem can you please extract a minimal reproducer to demonstrate the problem?

Thanks,
-marc

--
You received this message because you are subscribed to the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/75f55ae9-9f24-4261-93bc-54c2e1c01367n%40googlegroups.com.

Svante Schubert

unread,
Nov 3, 2021, 6:45:50 PM11/3/21
to jac...@googlegroups.com
Hi Marc,

the tests are running and there are test output files (and debug output from the uncovered files), but you are right the special JUnit class might not be the problem.
Seems, there is only one test providing coverage: https://svanteschubert.github.io/odfdom-coverage-data/ <- I have uploaded the JaCoCo report.
Only one file has coverage, being tested by an integration test:
Now I suspect that I am doing the Maven pom.xml wrong as there are only integration tests and JaCoCo in the reporting, see:

Likely I have overseen something...
Do you have any further hint?

Thanks in advance,
Svante

PS: It is not the small test you asked for, but you may via command line 
  1. clone git repo https://github.com/tdf/odftoolkit/ 
  2. switch to the coverage branch
  3. change to subdirectory odfdom of odftoolkit
  4. call 'mvn install' (using JDK >=9) 
  5. call 'mvn jacoco:report'


You received this message because you are subscribed to a topic in the Google Groups "JaCoCo and EclEmma Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jacoco/dXbkWvvKj5Q/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jacoco+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacoco/3BBA4FEC-403A-46F2-BE9E-2F46E3C356CE%40mountainminds.com.

444687196

unread,
Nov 3, 2021, 6:45:57 PM11/3/21
to Svante Schubert
谢谢你，给我发邮件，我会尽快回复！
这是自动回复……

Marc Hoffmann

unread,
Nov 4, 2021, 3:50:23 AM11/4/21
to JaCoCo and EclEmma Users
Hi Svante,

as you can see on the sessions page you only have execution data for a single library class:


As this is the only (not excluded) IT test executed with maven-failsafe-plugin it looks like that there is simply no coverage for your unit tests executed with the maven-surefire-plugin plugin.

The reason is that you overwrite the argLine which was prepared by JaCoCo before:


Please see JaCoCo documentation how to combine the JaCoCo agent configuration with your own VM arguments correctly:


Probably the line should read:

    <argLine>@{argLine} -Dfile.encoding=${project.build.sourceEncoding}</argLine>

Best refards,
-marc


Svante Schubert

unread,
Nov 5, 2021, 8:44:50 AM11/5/21
to jac...@googlegroups.com
Hi Marc,

Thank you very much that does the trick! :-)

I realized some "noise" on some tests being once executed and once not, but that might be as well a surefire problem:

I will dive into the docu to see if I find more on how the coverage works... Any good pointers?

In addition, does anyone have a pointer to existing open-source tooling (like Jenkins script) that pushes the coverage test results (like above) to some webserver (like GitHub pages)?
My first aim would be to always align the coverage results with a Git commit.

Allow me to paint the whole picture:
As mentioned I am working on the OpenDocument format (I am co-chair of OASIS ODF TC and co-maintainer of the ODF Toolkit).
In the ODF Toolkit and LibreOffice (LO) we have the problem that the tests are taking quite long (in LO even longer than the build). 

I like to evaluate a trick using coverage:
First, I am loading a test document in an ODF application like LO - say only containing "Hello World" - and saving aside all the code coverage during the test. All the source code lines are required to load it (and layout it in LO).
Second, I am loading a second test document - again "Hello World" but this time in bold - and saving aside again the code coverage data for the second test.

I assume that the second document load test, which contains, in addition, the "bold" feature, will have higher test coverage. 
The subtraction of both test coverages should reveal the lines solely required for the "bold" feature. We like to align these feature lines with a very fast feature-smoke test in conjunction with a GIT pre-commit-hook.

In addition, we would be able to "draw" some "feature map" of the source code for newcomers. 

Is anyone aware of existing research in this area? I have only skimmed the research papers your project has listed: https://www.jacoco.org/research/index.html
Our work will be all open-source. We were able to receive a small grant from the German Ministry of Research.
We are Thorsten Behrens - taking over the C++ part, his son Linus Behrens (Python) and myself for Java.

Last but not least, imagine we would be "the kings" of JCov, Jacoco, Cobertura... all Java coverage tools, wouldn't we think about reusing modules? Like front/backend as Clang is using?

Have a great weekend & greetings from Berlin!
Svante

PS: One design aspect: The most revolutionary thing I observed in the past years in IDEs is the tendency to move from all-source test/compile to incremental testing/compilation.
My favourite example is the Visual Studio Code (VSCode) extension (nice idea to split the language-dependent logic into a server part (language server - quick intro) and a stupid-only-layouting client part of the VSCode extension), where the official VSCode Rust extension switched to the incremental approach. 


Marc Hoffmann

unread,
Nov 5, 2021, 12:16:00 PM11/5/21
to jac...@googlegroups.com
Hi Svante,

regarding the second part of your email: This is referred to as “incremental code coverage”, there was research on this based on JaCoCo/EclEmma (see https://www.cs.ubc.ca/tr/2008/tr-2008-14).

I think you’re scenario is supported using the JaCoCo API. Here are the the steps:

1) Collect execution data for loading plain “Hello World” (exec1)
2) Collect execution data for loading bold “Hello World” (exec2)
4) Generate Coverage report with exec data from step 3)

This report should only show code which was executed in addition to render the bold text.

Cheers,
-marc


Svante Schubert

unread,
Nov 15, 2021, 7:39:35 AM11/15/21
to jac...@googlegroups.com
Hi Marc

Am Fr., 5. Nov. 2021 um 17:16 Uhr schrieb Marc Hoffmann <hoff...@mountainminds.com>:
Hi Svante,

regarding the second part of your email: This is referred to as “incremental code coverage”, there was research on this based on JaCoCo/EclEmma (see https://www.cs.ubc.ca/tr/2008/tr-2008-14).

That was the one paper I had already read, but now I took the opportunity to contact Kaitlin and Gail.
Kaitlin was instantly answering and although their source code is available, she strongly suggested building upon your JaCoco code.

In addition, she wrote:
"For what it's worth, I now make a distinction between differential code coverage (run two sessions, diff them) and incremental code coverage (one session, where you get to a point in the program, turn on code coverage, run something, and then turn code coverage collection off).  I found differential code coverage to be much more useful than incremental code coverage.

The first issue is garbage collection: you are likely to get garbage collection at some point in any quantum of code running, which incremental code coverage will capture.  Differential code coverage will likely have garbage collection in both, so the user won't see it.

I can't remember the second issue well enough to really describe it down concretely, but my memory is that there was more "initialization" code which got surfaced in incremental code coverage than in differential code coverage.  I think it was things like initializing draw routines for the UI -- something which happens all the time and is not interesting -- which differential code coverage hid for the user that incremental code coverage still showed."
 
I think you’re scenario is supported using the JaCoCo API. Here are the the steps:

1) Collect execution data for loading plain “Hello World” (exec1)
2) Collect execution data for loading bold “Hello World” (exec2)
4) Generate Coverage report with exec data from step 3)

I will see if I can bundle the above functionality from your given examples 
I assume you suggest doing all steps from within the JaCoCo API, right?
 
This report should only show code which was executed in addition to render the bold text.

Great! Many thanks, Marc!
Svante
 

Marc Hoffmann

unread,
Nov 15, 2021, 7:57:47 AM11/15/21
to JaCoCo and EclEmma Users
Hi Svante,

nice that Kaitlin responded to you!

Not sure about Garbage collection though: This should not have an impact on Code coverage unless you implemented finalize() methods (which are deprecated now) or ReferenceQueues.

> I assume you suggest doing all steps from within the JaCoCo API, right?

Correct, we don’t provide a tool integration with diff support.

Cheers,
-marc


Svante Schubert

unread,
Dec 2, 2021, 9:30:58 AM12/2/21
to jac...@googlegroups.com
Hello Marc,

I must admit, I have some difficulties bringing the 4 steps you mentioned below in synch with the JaCoCo Code and my upcoming user scenarios. ;-)

One user scenario in mind is to establish a pre-commit hook that requests from committers of new features to add also some feature regression test, initially by testing the coverage-delta of a changed class.
Or the same scenario just slightly different (but less controversial) - the usability feature to show automatically the statistic of changed coverage along with every received pull request (PR).

So, Initially, your 4 step solution recipe seemed quite easy as you pointed me to the exact method ExecutionData.merge(..) and I thought I just have to go "backwards" (compiling the project in my IDE and to a "used-by IDE query").
But I have not found a way to load existing persistent coverage data (XML/binary). This seemed helpful, as we try to bridge the usage of several coverage tools and we realized that other none-Java coverage frameworks like "Coverage.py for Python" and "GCOV for C++" are using the Cobertura XML as a common denominator. At least we found already a transformation from Jacoco to Cobertura XML (the other way around might exist somewhere and not that difficult). Our attempt is to reuse some coverage tooling among various code coverage tools.

In addition, if a Java Source (and its class) was changed, how can we find the related ExecutionData of the changed class, which are based on the class hash?
I believe your design likely comes from the class-only-exist scenario, while I would rather take the sources and even more some version control system as a prerequisite for my subtype of scenarios. 
The coverage identification could be done by file name (in Java package & class name), while in addition Git information (as the commit and/or file hash) could be added.
Especially helpful would be to receive to identify by git information any class renaming/movements.
(I personally would initially focus on Git, but most version control systems could be easily abstracted in an API.)

Does this make sense to you, Marc? What would you suggest how to proceed?

I really would like to invite you for some tea/beer and discuss this over aside from a conference! But damn Corona, but if you like to share a remote tea and just discuss/brainstorm a few things, just ping me directly.
This might remove some basic errors from my perspective as I am a newcomer in this field - even the wording Agent & probe was difficult to understand until I found out it references to JXM. Later I found probe very well (but a bit hidden) explained in "Why can't JaCoCo simply use the class name to identify classes?" at  https://www.jacoco.org/jacoco/trunk/doc/classids.html, but regarding docu I am in a glass house and not throwing any stone ;-)

Looking forward to hearing from you, Marc!

Hope you have a great week!
Svante

Ullrich Hafner

unread,
Dec 2, 2021, 11:13:58 AM12/2/21
to jac...@googlegroups.com
Is anyone aware of existing research in this area? I have only skimmed the research papers your project has listed: https://www.jacoco.org/research/index.html

I’m currently improving with one of my master's students the JaCoCo code coverage reports in Jenkins. We are focussing on a different goal but maybe our approach is helpful as well: we are trying to compute a delta report of two coverage analyses, on the one hand the coverage of a pull request and on the other hand the coverage of the target branch build. The goal is to give developers feedback about the quality of the pull request. Did the coverage increase or decrease? How is the coverage of the actual changes? 

A sketch of our planned approach (we just startet the thesis):
1) read the coverage of the pull request build into a model (JaCoCo, Cobertura, etc. - the format actually does not matter) and map it to the source files
2) read the coverage of the target branch build into a model (JaCoCo, Cobertura, etc. - the format actually does not matter) and map it to the source files
3) read the changed lines and files from Git 
4) combine the results of 1-3

Our model is based on the source code, so we do not use class files anymore. Just package, file, class, method.

Svante Schubert

unread,
Dec 5, 2021, 9:55:36 AM12/5/21
to JaCoCo and EclEmma Users
Dear Ulrich,

Thank you for sharing the interesting details of your project on PR Coverage Diff. 
Your project shares a few ideas with what we are doing currently. 
Aside from me, there is Thorsten and Linus Behrens working on the PrototypeFund project Cover-Rest we are sponsored till March.
My part of Cover-Rest is the Java part and I would like to add the new functionality to the ODFDOM project I work on, while Linus and Thorsten are focusing on LibreOffice (C++ & Python).
As we all work with/on open-source perhaps we should be able to find some synergies and collaborate on our mutual efforts.
Would in any case be awesome to have our ideas challenged & discussed - even if outright collaboration is not yet there - always good to challenge his own view by some tough & honest reviews!

From a bird's eye view, it seems to us that all involved projects require syntax: Not only to interchange data in an easy interoperable way among modules (& even across different coverage tools) but also to be able to archive interoperable data!
In regard to the interchange/archive format decision for coverage semantic, We don't feel we've got the full picture yet, but if we would have to make a design decision today, we would likely choose the Cobertura XML as the common denominator between various coverage tools. Yes, Cobertura is likely a dead project (over a year no commit with 200 issues), but there is this python transformation cover2cover from JaCoCo XML to Cobertura XML (last update 5 years ago) but no transformation the other way around and the leading Python and C++ coverage tools (GCOV/LCOV of C++ & coverage.py Python) are compatible to Cobertura XML (no idea why likely it was added when JaCoCo did not exist and Cobertura was striving). But in fact, the syntax does not really matter as long there is no significant semantic super/subset (we're not aware of any), which we have not found out, yet. We have not stumbled across any grammar and detailed descriptions of either format to compare them - did anyone on this list? Or any references to the current state of coverage tooling and its history?

If you take a look at the above-mentioned cover2cover transformation they explained (5 years) ago some superior GUI features of the Jenkins Cobertura plugin over the Jenkins JaCoCo plugin:
  1. Jenkins JaCoCo Plugin: https://wiki.jenkins-ci.org/display/JENKINS/JaCoCo+Plugin
  2. Jenkins Cobertura Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin
^^Is this the plugin you are working on, Ulrich? Do you agree with his observations and/or are they still up-to-date?
Again, I really do not understand why both plugins don't work together, like using cover2cover to abstract from the Syntax and share the same GUI, likely it might be this "Life of Brian" of "People's Front of Judea" and "Judean People's Front" thing or in any case it is still possible - as both are using the same open-source MIT license!

Last but not least, Thorsten and I have discussed how to wrap Git's complexity by an interoperable syntax. Naturally, the long existent syntax for text changes, the patch as DIFF format and the syntax for "diff among patches" the interdiff format might be some good start. What do you and the others think?

Ulrich, if you (or anyone interested in the topic) like to connect just drop me a line directly (e.g. via LinkedIn)!

Have a nice Sunday!
Svante

Ullrich Hafner

unread,
Dec 7, 2021, 7:53:27 AM12/7/21
to jac...@googlegroups.com
Am 05.12.2021 um 15:55 schrieb Svante Schubert <svante....@gmail.com>:

Dear Ulrich,

Thank you for sharing the interesting details of your project on PR Coverage Diff. 
Your project shares a few ideas with what we are doing currently. 
Aside from me, there is Thorsten and Linus Behrens working on the PrototypeFund project Cover-Rest we are sponsored till March.
My part of Cover-Rest is the Java part and I would like to add the new functionality to the ODFDOM project I work on, while Linus and Thorsten are focusing on LibreOffice (C++ & Python).
As we all work with/on open-source perhaps we should be able to find some synergies and collaborate on our mutual efforts.
Would in any case be awesome to have our ideas challenged & discussed - even if outright collaboration is not yet there - always good to challenge his own view by some tough & honest reviews!

From a bird's eye view, it seems to us that all involved projects require syntax: Not only to interchange data in an easy interoperable way among modules (& even across different coverage tools) but also to be able to archive interoperable data!
In regard to the interchange/archive format decision for coverage semantic, We don't feel we've got the full picture yet, but if we would have to make a design decision today, we would likely choose the Cobertura XML as the common denominator between various coverage tools. Yes, Cobertura is likely a dead project (over a year no commit with 200 issues), but there is this python transformation cover2cover from JaCoCo XML to Cobertura XML (last update 5 years ago) but no transformation the other way around and the leading Python and C++ coverage tools (GCOV/LCOV of C++ & coverage.py Python) are compatible to Cobertura XML (no idea why likely it was added when JaCoCo did not exist and Cobertura was striving). But in fact, the syntax does not really matter as long there is no significant semantic super/subset (we're not aware of any), which we have not found out, yet. We have not stumbled across any grammar and detailed descriptions of either format to compare them - did anyone on this list? Or any references to the current state of coverage tooling and its history?

If you take a look at the above-mentioned cover2cover transformation they explained (5 years) ago some superior GUI features of the Jenkins Cobertura plugin over the Jenkins JaCoCo plugin:
  1. Jenkins JaCoCo Plugin: https://wiki.jenkins-ci.org/display/JENKINS/JaCoCo+Plugin
  2. Jenkins Cobertura Plugin: https://wiki.jenkins-ci.org/display/JENKINS/Cobertura+Plugin
^^Is this the plugin you are working on, Ulrich? Do you agree with his observations and/or are they still up-to-date?

Well, actually both Jenkins plugins are „deprecated“ in my view: we are currently moving all of our efforts into the 
One plugin to rule them all ;-) It is intended to be used as a general code coverage UI that shows the results of every existing code coverage tool. 
We have converters from Cobertura, JaCoCo, Istanbul, lcov, … 
Here is the plugin in action (on its own code base):

The idea is taken from my warnings plugin which does the same thing for static analysis: provide *one* Java object model to represent warnings of *all* static analysis tools. The code-coverage-api-plugin also is intended to provide *one* Java object model for *all* coverage tools. In order to achieve that it is requited to talk about an object model in Java. The actual XML or JSON representation is irrelevant in my view, so if you are using Cobertura XML or JaCoCo should be transparent. 


Again, I really do not understand why both plugins don't work together, like using cover2cover to abstract from the Syntax and share the same GUI, likely it might be this "Life of Brian" of "People's Front of Judea" and "Judean People's Front" thing or in any case it is still possible - as both are using the same open-source MIT license!

Last but not least, Thorsten and I have discussed how to wrap Git's complexity by an interoperable syntax. Naturally, the long existent syntax for text changes, the patch as DIFF format and the syntax for "diff among patches" the interdiff format might be some good start. What do you and the others think?

Ulrich, if you (or anyone interested in the topic) like to connect just drop me a line directly (e.g. via LinkedIn)!


I will do so!


Reply all
Reply to author
Forward
0 new messages