Coverage for OpenJDK classes, multiple JUnit tests
176 views
Skip to first unread message
Marko Dimjasevic
Jan 3, 2013, 3:33:32 PM1/3/13
to JaCoCo and EclEmma Users
Hi all,
I'm trying to get the code coverage for a few OpenJDK 6 classes by
running Randoop first (which generates hundreds of JUnit tests), and
then get the code coverage of the OpenJDK classes under these tests. I
have an Ant tasks file for all of this. In particular, now I'm trying
to get the code coverage of Randoop-generated JUnit tests for the
java.util.BitSet class. There is the "entrance point" class
RandoopTest, which then depends on RandoopTest?.java (? is 0, 1, 2,
etc.) tests. As far as I can see, I have to have the main method as an
entrance point, so I created a simple
bitsetclassextractor.Bitsetclassextractor class that creates a new
BitSet object and performs a few simple operations on it. Here's part
of my build.xml file:
<target name="test" depends="source-compile,tests-compile"
description="Runs JUnit test units with Jacoco in the background to
determine the code coverage">
<jacoco:coverage destfile="${result.exec.file}">
<junit fork="true" forkmode="once">
<test name="RandoopTest"/>
<classpath>
<pathelement location="/usr/share/java/junit4.jar"/
>
<pathelement location="${result.classes.dir}"/>
</classpath>
</junit>
</jacoco:coverage>
<jacoco:coverage destfile="${result.exec.file}">
<java
classname="bitsetclassextractor.Bitsetclassextractor" fork="true">
<classpath>
<pathelement location="/usr/share/java/junit4.jar"/
>
<pathelement location="${result.classes.dir}"/>
</classpath>
</java>
</jacoco:coverage>
</target>
<target name="report" depends="test">
<jacoco:report>
<executiondata>
<file file="${result.exec.file}" />
</executiondata>
<structure name="JaCoCo report for Randoop-generated tests
code coverage">
<classfiles>
<fileset dir="${result.classes.dir}">
</fileset>
</classfiles>
<sourcefiles encoding="UTF-8">
<fileset dir="${src.dir}" />
<fileset dir="${src.tests.dir}" />
</sourcefiles>
</structure>
<html destdir="${result.report.dir}" />
<csv destfile="${result.report.dir}/report.csv" />
<xml destfile="${result.report.dir}/report.xml" />
</jacoco:report>
</target>
In the html report I can see that the default package (where Randoop
test classes reside) has a coverage of ~90%, the java.util package
(where the BitSet class is) has 0%, and the bitsetclassextractor
package has a coverage of 100%.
How do I get a coverage of more than 0% for the java.util.BitSet
class? Does it have something to do with the way OpenJDK classes
should be compiled? This is how I compile the java.util.BitSet class:
<target name="source-compile" description="Compiles the system
under test">
<mkdir dir="${result.classes.dir}" />
<javac srcdir="${src.dir}" destdir="${result.classes.dir}"
debug="true" includeantruntime="false" classpath="/usr/share/java/
junit4.jar"/>
</target>
Regards,
Marko Dimjašević
I'm trying to get the code coverage for a few OpenJDK 6 classes by
running Randoop first (which generates hundreds of JUnit tests), and
then get the code coverage of the OpenJDK classes under these tests. I
have an Ant tasks file for all of this. In particular, now I'm trying
to get the code coverage of Randoop-generated JUnit tests for the
java.util.BitSet class. There is the "entrance point" class
RandoopTest, which then depends on RandoopTest?.java (? is 0, 1, 2,
etc.) tests. As far as I can see, I have to have the main method as an
entrance point, so I created a simple
bitsetclassextractor.Bitsetclassextractor class that creates a new
BitSet object and performs a few simple operations on it. Here's part
of my build.xml file:
<target name="test" depends="source-compile,tests-compile"
description="Runs JUnit test units with Jacoco in the background to
determine the code coverage">
<jacoco:coverage destfile="${result.exec.file}">
<junit fork="true" forkmode="once">
<test name="RandoopTest"/>
<classpath>
<pathelement location="/usr/share/java/junit4.jar"/
>
<pathelement location="${result.classes.dir}"/>
</classpath>
</junit>
</jacoco:coverage>
<jacoco:coverage destfile="${result.exec.file}">
<java
classname="bitsetclassextractor.Bitsetclassextractor" fork="true">
<classpath>
<pathelement location="/usr/share/java/junit4.jar"/
>
<pathelement location="${result.classes.dir}"/>
</classpath>
</java>
</jacoco:coverage>
</target>
<target name="report" depends="test">
<jacoco:report>
<executiondata>
<file file="${result.exec.file}" />
</executiondata>
<structure name="JaCoCo report for Randoop-generated tests
code coverage">
<classfiles>
<fileset dir="${result.classes.dir}">
</fileset>
</classfiles>
<sourcefiles encoding="UTF-8">
<fileset dir="${src.dir}" />
<fileset dir="${src.tests.dir}" />
</sourcefiles>
</structure>
<html destdir="${result.report.dir}" />
<csv destfile="${result.report.dir}/report.csv" />
<xml destfile="${result.report.dir}/report.xml" />
</jacoco:report>
</target>
In the html report I can see that the default package (where Randoop
test classes reside) has a coverage of ~90%, the java.util package
(where the BitSet class is) has 0%, and the bitsetclassextractor
package has a coverage of 100%.
How do I get a coverage of more than 0% for the java.util.BitSet
class? Does it have something to do with the way OpenJDK classes
should be compiled? This is how I compile the java.util.BitSet class:
<target name="source-compile" description="Compiles the system
under test">
<mkdir dir="${result.classes.dir}" />
<javac srcdir="${src.dir}" destdir="${result.classes.dir}"
debug="true" includeantruntime="false" classpath="/usr/share/java/
junit4.jar"/>
</target>
Regards,
Marko Dimjašević
Marc R. Hoffmann
Jan 4, 2013, 5:19:31 AM1/4/13
to jac...@googlegroups.com
Hi,
in the HTML report at the top right corner there is a link "Sessions".
If you open this page you will see a list of all classes where execution
data has been collected for.
I don't know you exact setup, but in general classes from the
bootclasspath (like java.util) are always excluded by the JaCoCo agent.
Best regards,
-marc
in the HTML report at the top right corner there is a link "Sessions".
If you open this page you will see a list of all classes where execution
data has been collected for.
I don't know you exact setup, but in general classes from the
bootclasspath (like java.util) are always excluded by the JaCoCo agent.
Best regards,
-marc
Marko Dimjašević
Jan 7, 2013, 11:47:46 AM1/7/13
to jac...@googlegroups.com
"Marc R. Hoffmann" <hoff...@mountainminds.com> writes:
>
> I don't know you exact setup, but in general classes from the
> bootclasspath (like java.util) are always excluded by the JaCoCo
> agent.
Code coverage for OpenJDK classes (including those at java.util) is what
>
> I don't know you exact setup, but in general classes from the
> bootclasspath (like java.util) are always excluded by the JaCoCo
> agent.
I need. Is it possible to get coverage with Jacoco for those?
Cheers,
Marko Dimjašević
Marc Hoffmann
Jan 7, 2013, 12:24:40 PM1/7/13
to jac...@googlegroups.com
Hi,
we do have a pending feature request for this:
https://github.com/jacoco/jacoco/issues/49
But even if we implement this the feature will come with some
limitations: In particular JDK classes which are used by the Agent
itself can probably not be tracked.
Alternatively you might try the new offline instrumentation feature and
pre-process JDK classes:
http://www.eclemma.org/jacoco/trunk/doc/offline.html
Best regards,
-marc
we do have a pending feature request for this:
https://github.com/jacoco/jacoco/issues/49
But even if we implement this the feature will come with some
limitations: In particular JDK classes which are used by the Agent
itself can probably not be tracked.
Alternatively you might try the new offline instrumentation feature and
pre-process JDK classes:
http://www.eclemma.org/jacoco/trunk/doc/offline.html
Best regards,
-marc
Marko Dimjašević
Jan 8, 2013, 7:05:05 PM1/8/13
to jac...@googlegroups.com
Marc Hoffmann <hoff...@mountainminds.com> writes:
>
> Alternatively you might try the new offline instrumentation feature
> and pre-process JDK classes:
> http://www.eclemma.org/jacoco/trunk/doc/offline.html
I tried with Jacoco 0.6.1, but it doesn't have the instrument task. So I
>
> Alternatively you might try the new offline instrumentation feature
> and pre-process JDK classes:
> http://www.eclemma.org/jacoco/trunk/doc/offline.html
downloaded the code from the git repository, compiled org.jacoco.ant,
and used it to get the code coverage for the java.util.BitSet class, but
still it is not covered (checked it in the Sessions section too).
I've attached my build file. Can you see what's wrong with it?
Cheers,
Marko Dimjašević
Marc Hoffmann
Jan 9, 2013, 3:41:58 AM1/9/13
to jac...@googlegroups.com
I don't think it is possible to overwrite system classes with the
application class path. Probably you need to prepend the boot classpath
with the instrumented version of the classes.
Also I do not understand how this can work:
<java classname="java.util.BitSet" fork="true"> ...</java>
Does your version of BitSet declare a main method?
If I find some time these day I try to hack an example how to put JRE
classes under code coverage.
Cheers,
-marc
application class path. Probably you need to prepend the boot classpath
with the instrumented version of the classes.
Also I do not understand how this can work:
<java classname="java.util.BitSet" fork="true"> ...</java>
Does your version of BitSet declare a main method?
If I find some time these day I try to hack an example how to put JRE
classes under code coverage.
Cheers,
-marc
Marko Dimjašević
Jan 9, 2013, 12:34:24 PM1/9/13
to jac...@googlegroups.com
Marc Hoffmann <hoff...@mountainminds.com> writes:
> I don't think it is possible to overwrite system classes with the
> application class path. Probably you need to prepend the boot
> classpath with the instrumented version of the classes.
I'm not sure what this means. Can you provide an example for this?
> I don't think it is possible to overwrite system classes with the
> application class path. Probably you need to prepend the boot
> classpath with the instrumented version of the classes.
> Also I do not understand how this can work:
>
> <java classname="java.util.BitSet" fork="true"> ...</java>
>
> Does your version of BitSet declare a main method?
but I do have some simple class with a static main method that creates
an object of java.util.BitSet class. So, in the jacoco:coverage section,
I should provide my own custom class with a main method instead of
java.util.BitSet, right?
> If I find some time these day I try to hack an example how to put JRE
> classes under code coverage.
Thank you for your help this far!
Cheers,
Marko
Marc R. Hoffmann
Jan 14, 2013, 3:40:43 PM1/14/13
to jac...@googlegroups.com
Hi Marko,
finally I found some time to hack a prototype for code coverage on JRE
classes, here is the ant build file: https://gist.github.com/4533161
Note that
1) the example uses offline instrumentation, so please use the latest
0.6.2 snapshot build
2) the referenced Main class is a placeholder for the actual test in
this example
3) While it works on java.util.BitSet instrumenting many other
classes lead to errors at JVM startup. The problem is that JaCoCo itself
uses JRE classes. If classes used by instrumented classes itself are
instrumented, you typically end-up with an StackOverflowError.
Best regards,
-marc
finally I found some time to hack a prototype for code coverage on JRE
classes, here is the ant build file: https://gist.github.com/4533161
Note that
1) the example uses offline instrumentation, so please use the latest
0.6.2 snapshot build
2) the referenced Main class is a placeholder for the actual test in
this example
3) While it works on java.util.BitSet instrumenting many other
classes lead to errors at JVM startup. The problem is that JaCoCo itself
uses JRE classes. If classes used by instrumented classes itself are
instrumented, you typically end-up with an StackOverflowError.
Best regards,
-marc
Marko Dimjašević
Jan 15, 2013, 5:58:01 PM1/15/13
to jac...@googlegroups.com
Hi Marc,
for me.
> 1) the example uses offline instrumentation, so please use the
> latest 0.6.2 snapshot build
What exactly do you mean by latest 0.6.2 snapshot build? This is how I
did it:
1. git pull
2. mvn clean install
3. copy org.jacoco.ant/target/org.jacoco.ant-0.6.2-SNAPSHOT-nodeps.jar
to my project and name it jacocoant.jar
4. copy org.jacoco.agent/target/classes/jacocoagent.jar to my project
Then I run Ant with your build file where I changed only this step:
> 2) the referenced Main class is a placeholder for the actual test in
> this example
to my own placeholder class.
When I run it, I get the following error:
test:
[java] Error occurred during initialization of VM
[java] java.lang.ExceptionInInitializerError
[java] at java.util.BitSet.$jacocoInit(BitSet.java)
[java] at java.util.BitSet.<clinit>(BitSet.java)
[java] at sun.net.www.ParseUtil.<clinit>(ParseUtil.java:52)
[java] at sun.misc.Launcher.getFileURL(Launcher.java:442)
[java] at sun.misc.Launcher$ExtClassLoader.getExtURLs(Launcher.java:190)
[java] at sun.misc.Launcher$ExtClassLoader.<init>(Launcher.java:161)
[java] at sun.misc.Launcher$ExtClassLoader$1.run(Launcher.java:145)
[java] at java.security.AccessController.doPrivileged(Native Method)
[java] at sun.misc.Launcher$ExtClassLoader.getExtClassLoader(Launcher.java:138)
[java] at sun.misc.Launcher.<init>(Launcher.java:71)
[java] at sun.misc.Launcher.<clinit>(Launcher.java:59)
[java] at java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1304)
[java] at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1286)
[java] Caused by: java.lang.NullPointerException
[java] at sun.net.www.ParseUtil.encodePath(ParseUtil.java:121)
[java] at sun.net.www.ParseUtil.encodePath(ParseUtil.java:97)
[java] at sun.net.www.ParseUtil.fileToEncodedURL(ParseUtil.java:265)
[java] at sun.misc.Launcher.getFileURL(Launcher.java:442)
[java] at sun.misc.Launcher.pathToURLs(Launcher.java:381)
[java] at sun.misc.Launcher.access$300(Launcher.java:57)
[java] at sun.misc.Launcher$1.run(Launcher.java:368)
[java] at java.security.AccessController.doPrivileged(Native Method)
[java] at sun.misc.Launcher.getBootstrapClassPath(Launcher.java:351)
[java] at java.lang.ClassLoader.getBootstrapClassPath(ClassLoader.java:1137)
[java] at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1112)
[java] at java.lang.ClassLoader.getSystemResource(ClassLoader.java:1073)
[java] at java.lang.ClassLoader.getSystemResourceAsStream(ClassLoader.java:1181)
[java] at java.lang.Class.getResourceAsStream(Class.java:2045)
[java] at org.jacoco.agent.rt.internal_5f5e791.ConfigLoader.load(ConfigLoader.java:33)
[java] at org.jacoco.agent.rt.internal_5f5e791.Offline.<clinit>(Offline.java:29)
[java] at java.util.BitSet.$jacocoInit(BitSet.java)
[java] at java.util.BitSet.<clinit>(BitSet.java)
[java] at sun.net.www.ParseUtil.<clinit>(ParseUtil.java:52)
[java] at sun.misc.Launcher.getFileURL(Launcher.java:442)
[java] at sun.misc.Launcher$ExtClassLoader.getExtURLs(Launcher.java:190)
[java] at sun.misc.Launcher$ExtClassLoader.<init>(Launcher.java:161)
[java] at sun.misc.Launcher$ExtClassLoader$1.run(Launcher.java:145)
[java] at java.security.AccessController.doPrivileged(Native Method)
[java] at sun.misc.Launcher$ExtClassLoader.getExtClassLoader(Launcher.java:138)
[java] at sun.misc.Launcher.<init>(Launcher.java:71)
[java] at sun.misc.Launcher.<clinit>(Launcher.java:59)
[java] at java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1304)
[java] at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1286)
[java]
BUILD FAILED
Any idea what went wrong?
Cheers,
Marko
> Hi Marko,
>
> finally I found some time to hack a prototype for code coverage on JRE
> classes, here is the ant build file: https://gist.github.com/4533161
Most probably I'm doing something wrong, but this example isn't working
>
> finally I found some time to hack a prototype for code coverage on JRE
> classes, here is the ant build file: https://gist.github.com/4533161
for me.
> 1) the example uses offline instrumentation, so please use the
> latest 0.6.2 snapshot build
did it:
1. git pull
2. mvn clean install
3. copy org.jacoco.ant/target/org.jacoco.ant-0.6.2-SNAPSHOT-nodeps.jar
to my project and name it jacocoant.jar
4. copy org.jacoco.agent/target/classes/jacocoagent.jar to my project
Then I run Ant with your build file where I changed only this step:
> 2) the referenced Main class is a placeholder for the actual test in
> this example
When I run it, I get the following error:
test:
[java] Error occurred during initialization of VM
[java] java.lang.ExceptionInInitializerError
[java] at java.util.BitSet.$jacocoInit(BitSet.java)
[java] at java.util.BitSet.<clinit>(BitSet.java)
[java] at sun.net.www.ParseUtil.<clinit>(ParseUtil.java:52)
[java] at sun.misc.Launcher.getFileURL(Launcher.java:442)
[java] at sun.misc.Launcher$ExtClassLoader.getExtURLs(Launcher.java:190)
[java] at sun.misc.Launcher$ExtClassLoader.<init>(Launcher.java:161)
[java] at sun.misc.Launcher$ExtClassLoader$1.run(Launcher.java:145)
[java] at java.security.AccessController.doPrivileged(Native Method)
[java] at sun.misc.Launcher$ExtClassLoader.getExtClassLoader(Launcher.java:138)
[java] at sun.misc.Launcher.<init>(Launcher.java:71)
[java] at sun.misc.Launcher.<clinit>(Launcher.java:59)
[java] at java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1304)
[java] at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1286)
[java] Caused by: java.lang.NullPointerException
[java] at sun.net.www.ParseUtil.encodePath(ParseUtil.java:121)
[java] at sun.net.www.ParseUtil.encodePath(ParseUtil.java:97)
[java] at sun.net.www.ParseUtil.fileToEncodedURL(ParseUtil.java:265)
[java] at sun.misc.Launcher.getFileURL(Launcher.java:442)
[java] at sun.misc.Launcher.pathToURLs(Launcher.java:381)
[java] at sun.misc.Launcher.access$300(Launcher.java:57)
[java] at sun.misc.Launcher$1.run(Launcher.java:368)
[java] at java.security.AccessController.doPrivileged(Native Method)
[java] at sun.misc.Launcher.getBootstrapClassPath(Launcher.java:351)
[java] at java.lang.ClassLoader.getBootstrapClassPath(ClassLoader.java:1137)
[java] at java.lang.ClassLoader.getBootstrapResource(ClassLoader.java:1112)
[java] at java.lang.ClassLoader.getSystemResource(ClassLoader.java:1073)
[java] at java.lang.ClassLoader.getSystemResourceAsStream(ClassLoader.java:1181)
[java] at java.lang.Class.getResourceAsStream(Class.java:2045)
[java] at org.jacoco.agent.rt.internal_5f5e791.ConfigLoader.load(ConfigLoader.java:33)
[java] at org.jacoco.agent.rt.internal_5f5e791.Offline.<clinit>(Offline.java:29)
[java] at java.util.BitSet.$jacocoInit(BitSet.java)
[java] at java.util.BitSet.<clinit>(BitSet.java)
[java] at sun.net.www.ParseUtil.<clinit>(ParseUtil.java:52)
[java] at sun.misc.Launcher.getFileURL(Launcher.java:442)
[java] at sun.misc.Launcher$ExtClassLoader.getExtURLs(Launcher.java:190)
[java] at sun.misc.Launcher$ExtClassLoader.<init>(Launcher.java:161)
[java] at sun.misc.Launcher$ExtClassLoader$1.run(Launcher.java:145)
[java] at java.security.AccessController.doPrivileged(Native Method)
[java] at sun.misc.Launcher$ExtClassLoader.getExtClassLoader(Launcher.java:138)
[java] at sun.misc.Launcher.<init>(Launcher.java:71)
[java] at sun.misc.Launcher.<clinit>(Launcher.java:59)
[java] at java.lang.ClassLoader.initSystemClassLoader(ClassLoader.java:1304)
[java] at java.lang.ClassLoader.getSystemClassLoader(ClassLoader.java:1286)
[java]
BUILD FAILED
Any idea what went wrong?
Cheers,
Marko
Marc R. Hoffmann
Jan 16, 2013, 1:32:11 AM1/16/13
to jac...@googlegroups.com
Hi Marko,
as I said: This will not work for classes which are used by the JVM
itself for bootstrapping. It looks like that your JRE Version needs
BitSet to parse some URLs. So BitSet cannot be instrumented.
What if you move the classes under test to a different package (e.g.
test.java.util.BitSet) and test them as regular application classes?
Cheers,
-marc
--
Marc Hoffmann
hoff...@mountainminds.com
_______________________________________________
Mountainminds GmbH & Co. KG
Nussbaumstr. 4 * 80336 Muenchen * Germany
Phone/Fax +49-700-68664637 * 0700-MTNMINDS
Registergericht Muenchen * HRA 80201
Mountainminds Verwaltungs GmbH
Registergericht Muenchen * HRB 143183
Geschaeftsfuehrer Marc Hoffmann
as I said: This will not work for classes which are used by the JVM
itself for bootstrapping. It looks like that your JRE Version needs
BitSet to parse some URLs. So BitSet cannot be instrumented.
What if you move the classes under test to a different package (e.g.
test.java.util.BitSet) and test them as regular application classes?
Cheers,
-marc
Marc Hoffmann
hoff...@mountainminds.com
_______________________________________________
Mountainminds GmbH & Co. KG
Nussbaumstr. 4 * 80336 Muenchen * Germany
Phone/Fax +49-700-68664637 * 0700-MTNMINDS
Registergericht Muenchen * HRA 80201
Mountainminds Verwaltungs GmbH
Registergericht Muenchen * HRB 143183
Geschaeftsfuehrer Marc Hoffmann
Marko Dimjašević
Jan 16, 2013, 11:32:56 AM1/16/13
to jac...@googlegroups.com
"Marc R. Hoffmann" <hoff...@mountainminds.com> writes:
>
> What if you move the classes under test to a different package
> (e.g. test.java.util.BitSet) and test them as regular application
> classes?
Actually, that's exactly what I did for a set of JDK classes after this
>
> What if you move the classes under test to a different package
> (e.g. test.java.util.BitSet) and test them as regular application
> classes?
offline instrumentation approach didn't work and it works! In that way
you don't need offline instrumentation at all.
I appreciate your help!
Cheers,
Marko
Reply all
Reply to author
Forward
This conversation is locked
You cannot reply and perform actions on locked conversations.
0 new messages