Should I parse the report generated by JaCoCo

[Ognjen Bubalo]

Hello!

I was wondering, if JaCoCo generates the reports for us, maybe it is a good idea to show exactly the same on Jenkins, and not to use the API to do everything from scratch.

Of course to show charts I would need the JaCoCo API.

What do you think?

Br, Ogi

[Mickael Istria]

Hi,

I'm fine with JaCoCo HTML reports. If you can easily embed them into Jenkins coverage view, then please do.

IMHO, there are only 2 kinds of reports: Jacoco.exec, and other XML and HTML reports.
Using directly jacoco.exec would be much more convenient for end-users, since getting it is just about setting the javaageent JVM property. That's for example the only file Sonar requires (along with Sources). But it's probably much more work to turn it into nice reports.
Other reports are those generated by Jacoco tasks or Maven plugin. You currently require XML reports, so all your users are able to generate XML reports: they will easily be able to generate HTML reports. So requiring HTML is no additional effort to requiring XML for users. Both HTML and XML reports require additional build steps which have limitations (http://sourceforge.net/tracker/?func=detail&atid=883354&aid=3474708&group_id=177969 and http://sourceforge.net/tracker/?func=detail&atid=883354&aid=3535229&group_id=177969 ).

So I think it's either you use only jacoco.exec, either you can re-use any report (XML, HTML...) generated by JaCoCo. The second approach is probably the best since we are in the early life of this plugin.

--
Mickael Istria
Eclipse developer at JBoss, by Red Hat
My blog - My Tweets

[Marc R. Hoffmann]

Hi Ogi,

actually this was my original proposal: Simply use the xml report
generated by the JaCoCo reporting task/goal. Otherwise you're going to
duplicate all the effort. Also the configuration of the plug-in will
probably become quite tricky because you have to ouline the
module/class/source structure.

I know there are limitations for multi-module Maven builds. But I would
suggest to put the effort in an improved version of JaCoCo's reporting goal.

Cheers,
-marc

[Ognjen Bubalo]

I think I did not explain properly what I want.

With Jonathan's help we integrated the ReportMojo into the plugin. I generated the exact report using only the exec file, the class and source files, so now I have HTML files. And I was thinking of parsing them (HTML source) and put them into Jenkink's structure, so it would look exacty as a JaCoCo report looks. Users do not need to get use to 2 views, and our work would be simpler too.

We realized that throwing away the XML parsing would be better right now.
Of course if we want a fast release we can use the existing code and generated HTML files with sourcefile highlighting to fix the 404 issue. That would work, but if we have already all the reports we need I think using it would be reasonable.

Br,
ogi

[Jonathan Fuerth]

I just want to jump in and clarify our change in direction.

A few days ago, Ogi and I sat down and looked at the report mojo in the maven plugin. It's just a single class that gathers some configuration data (most importantly, the location of the jacoco.exec file, the java classes, and the java sources) from the pom.xml and then uses the JaCoCo reporting library to generate the XML and HTML reports.

Then we adapted this class to work within Jenkins and do the same thing: gather configuration data (this time from the job config rather than the pom) and then use the JaCoCo reporting library to generate the XML and HTML reports. After sorting out the asm version conflict, this approach now works!

I think this approach to the integration has several advantages over the XML based integration:

- less burden on users (no need to configure the build to generate the reports) — this was Mickael's point
- produces output identical to the report mojo/ant task
- gives us access to the JaCoCo library for merging coverage data and gathering interesting metrics to plot
- requires much less code in the Jenkins plugin vs. parsing the xml

In all, I think my original adaptation of the emma Jenkins plugin was a useful stopgap, but switching to the .exec file will let us trivially solve all the major issues reported so far while also significantly reducing the amount of code in the Jenkins plugin.

-Jonathan

[Jonathan Fuerth]

Well said, Ogi. Looks like we were writing the same email at the same time. :)

-Jonathan

[Ognjen Bubalo]

Yes, this was good :)

[Mickael Istria]

On 07/13/2012 02:15 PM, Jonathan Fuerth wrote:

Then we adapted this class to work within Jenkins and do the same thing: gather configuration data (this time from the job config rather than the pom) and then use the JaCoCo reporting library to generate the XML and HTML reports. After sorting out the asm version conflict, this approach now works!

That's awesome! Can't wait to give it a try!

I think this approach to the integration has several advantages over the XML based integration:

- less burden on users (no need to configure the build to generate the reports) � this was Mickael's point

- produces output identical to the report mojo/ant task
- gives us access to the JaCoCo library for merging coverage data and gathering interesting metrics to plot
- requires much less code in the Jenkins plugin vs. parsing the xml

Agreed on these advantages. Especially #1 ;)

Switching to the .exec file will let us trivially solve all the major issues reported so far while also significantly reducing the amount of code in the Jenkins plugin.

Good continuation with that. That's IMHO the good direction, at least for our use-cases.
I understand Marc's concerns about duplication of efforts between reports tasks/mojos and Jenkins plugin, but I think it may be more sustainable to rely on Jacoco APIs than on output of report tools, and since the added value of tools over API is limited (the ReportMojo is pretty easy and is simple usage of API) it does not provide a lot of help to take as input XML files instead of .exec files, the step to go from .exec to reports is easy to implement into Jenkins plugin and provides a great usability improvement.

So congrats for this effort, please tell me whenever I can give a try!

Cheers

[Mickael Istria]

On 07/13/2012 02:15 PM, Ognjen Bubalo wrote:

With Jonathan's help we integrated the ReportMojo into the plugin. I generated the exact report using only the exec file, the class and source files, so now I have HTML files. And I was thinking of parsing them (HTML source) and put them into Jenkink's structure, so it would look exacty as a JaCoCo report looks. Users do not need to get use to 2 views, and our work would be simpler too.

We realized that throwing away the XML parsing would be better right now.
Of course if we want a fast release we can use the existing code and generated HTML files with sourcefile highlighting to fix the 404 issue. That would work, but if we have already all the reports we need I think using it would be reasonable.

Did you make progress on this topic ? Do you have a planned release date for this ability to take directly jacoco.exec as input ?
I'm eager to finally get Jacoco reports for JBoss Tools jobs on Jenkins. Is there anything I can do to help you get it working (I can code) ?

[Ognjen Bubalo]

Hi Mickael,

I am currently implementing the sourcefile highlighting. It should be finished today.

Currently I am stuck with the Digester. It parses the jacoco.xml. To have the sourcefile highlighting I need to call one method during the parsing with a specific parameter, but for some reason it does not work for me.

Digester creates a stack, creates objects and puts them in the stack while reading the XML. When it reads a specific "xml path" Digester can call a method, or and object using the given class.

It should be a small problem, so soon we will have the ability to check the highlighted sourcefiles. After that I think I need a week or two to change everything from xml parsing to exec file usage.
To do this I think I will use the generated HTML reports. After every build I will store it in the build's folder. I think I would not use the JaCoCo API because I would need to store the exec files, sourcefiles, classfiles. So to show the reports I'll try to copy the needed HTML table parts and show it in Jenkins, it would look like exactly like in JaCoCo. BUT I think this whole concept is still under questionmark. Because to have a working plugin I only need one or two information from the HTML files or exec  (for e.g. which is the biggest coverage number in one group, this is needed to show a proper green/red bars and charts as Marc proposed). So I think we will soon have to discuss again what is a better way.

After this is done I will implement the plugin to work also under multiple modules.

Br, ogi

[Jonathan Fuerth]

Hi Ogi,

On Wed, Jul 25, 2012 at 6:10 AM, Ognjen Bubalo <ognjen...@gmail.com> wrote:
> I am currently implementing the sourcefile highlighting. It should be
> finished today.
>
> Currently I am stuck with the Digester. It parses the jacoco.xml. To have
> the sourcefile highlighting I need to call one method during the parsing
> with a specific parameter, but for some reason it does not work for me.
>
> Digester creates a stack, creates objects and puts them in the stack while
> reading the XML. When it reads a specific "xml path" Digester can call a
> method, or and object using the given class.

I've had a lot of experience with Apache Digester. I can help you with
this if you think it would help. But I have some questions about your
direction here...

> It should be a small problem, so soon we will have the ability to check the
> highlighted sourcefiles. After that I think I need a week or two to change
> everything from xml parsing to exec file usage.

Does this mean you've decided to implement the source highlighting
feature based on the JaCoCo XML report?

> To do this I think I will use the generated HTML reports. After every build
> I will store it in the build's folder. I think I would not use the JaCoCo
> API because I would need to store the exec files, sourcefiles, classfiles.

Sure, I think this is the best approach. Just generate all the HTML
snippets you need, and save them with the build.

Even if you want to use the jacoco.xml file for generating
line-by-line coverage reports (which I don't think is optimal), you
will still need the source code.

> So to show the reports I'll try to copy the needed HTML table parts and show
> it in Jenkins, it would look like exactly like in JaCoCo. BUT I think this
> whole concept is still under questionmark. Because to have a working plugin
> I only need one or two information from the HTML files or exec (for e.g.
> which is the biggest coverage number in one group, this is needed to show a
> proper green/red bars and charts as Marc proposed). So I think we will soon
> have to discuss again what is a better way.

I think the information for the green/red bars will be part of the
HTML you will save with the build. And the other counters that you
want to build plots of can be queried from the API at build time and
saved with the build. You shouldn't need to use the JaCoCo API when
rendering pages; you will just load jacoco.exec during the post-build
phase, save everything you might ever need (HTML reports, the values
of various counters) using the Jenkins API, and then refer to these
saved values when rendering pages in Jelly/Stapler.

Does this make sense?

> After this is done I will implement the plugin to work also under multiple
> modules.

Yes, that's the next trick. One step at a time. :)

-Jonathan

[Ognjen Bubalo]

Hi Jonathan,

I am one day late :(.

Does this mean you've decided to implement the source highlighting

feature based on the JaCoCo XML report?

 

No, I wanted this as a temporary solution, to have more experience and to meet the sourcehighlighting requirement.

Even if you want to use the jacoco.xml file for generating
line-by-line coverage reports (which I don't think is optimal), you
will still need the source code.

I need it only once at the build. JaCoCo generates the sourcehighlighting of the actual build, the plugin copies the whole generated HTML structure to the buildfolder (where the previous builds are stored), so we do not need to regenerate it. Hmm, I copied somewhere else the sourcecode and the sourcehighlighting is still working in the generated HTMLs, so maybe I do not need to store the sourcecode.

What do you think?

Br, ogi

[Matthias Erche]

Hi Ogi,

Am Donnerstag, 26. Juli 2012 08:49:48 UTC+2 schrieb Ognjen Bubalo:

I need it only once at the build. JaCoCo generates the sourcehighlighting of the actual build, the plugin copies the whole generated HTML structure to the buildfolder (where the previous builds are stored), so we do not need to regenerate it. Hmm, I copied somewhere else the sourcecode and the sourcehighlighting is still working in the generated HTMLs, so maybe I do not need to store the sourcecode.

The sources are not needed to view the HTML reports. For every Java class a HTML page is created.

- Matthias 

[Ognjen Bubalo]

Hi Matthias,

Thank you!

ogi

[Ognjen Bubalo]

Ok, I want to pass one parameter to the MethodReport class while parsing the XML with Digester in the CoverageReport class. This is the buildpath. I need it because I want to set a link in the MethodReport class, so my jelly (breakdownMethodsTable.jelly) can connect the highlighted sourcefiles using the MethodReport's sourceFilePath parameter.

The problem is, that I have to set this paramether adding rules before the parsing starts (and the Objects are created, so I cannot reach them). 

 

Another way would be just call the methodReport.getParent().getParent().getBuild.getURL().. or just in the PackageReport to call the getParent.getURL and then to pass to its childs (ClassReport->MethodReport). I tried it and got exception always (for calling getParent()). So I tried something like:

digester.addCallMethod("report/package", "setParent", 1);

but something is not ok.

Jonathan can you please help me with this? I committed this experiments into a branch.

Thank you,

Ogi

[Ognjen Bubalo]

Hi,

I would like to ask if someone has experience with HTML parsing and could recommend me a good parser.

Marc, I would like to ask, is there a simple solution to get informations like "which is the biggest coverage in one group" (I need it for drawing the green/red bar) from the EXEC file? If not I would get it from the HTML files.

Thanks!

Br, Ogi

[Marc R. Hoffmann]

> Marc, I would like to ask, is there a simple solution to get
> informations like "which is the biggest coverage in one group" (I
> need it for drawing the green/red bar) from the EXEC file?

I'm afraid not. The exec files are worthless without the corresponding
class files and their corresponding group structure.

You can open exec files with Eclipse when EclEmma is installed.
Basically exec files contain the executed probes per loaded class.

-marc

[Mickael Istria]

On 07/26/2012 06:30 PM, Ognjen Bubalo wrote:

I would like to ask if someone has experience with HTML parsing and could recommend me a good parser.

An ex-colleague one day told me with a lof of enthusiasm about jericho, but I've never tried it by myself.

[Marc Hoffmann]

Hi all,

I just want to make clear that I do not consider the HTML files
produced by JaCoCo as API. The format might change any time in future.

What is the exact information we want to get out of the HTML pages? We
can easily add additional XML files (with the JaCoCo Schema, which is
API) to the HTML report, for example a summary for every group.

Cheers,
-marc

[Ognjen Bubalo]

Okok. I was thinking of getting this information when we have all the resources and then save it (for e.g. in a simple xml file), but we have it already in the HTML files. I was thinking of parsing the HTML files but yes, it can change so we need another way.

The idea behind the HTML parsing was that the report could look exactly like in JaCoCo, but if the existing table is ok (in the plugin), then I need only informations like "which is the biggest coverage in one group", some summary etc. and it will work. This information is not in the XML, but the HTML report contains it. If we could dump this information somewhere it would be great.

What do you think?

Br, Ogi

[Marc Hoffmann]

Hi Ogi,

all the information in the HTML report is also available from the XML
report (except the source code of course).

Some figures in the HTML report are derived (like percentage) but this
should be simple math.

Cheers,
-marc

[Jonathan Fuerth]

I'm afraid we're going off the rails a bit here (anyone, please feel
free to disagree!). Relying on jacoco.exec, jacoco.xml, the jacoco
library and some workalike classes in the jenkins plugin all at the
same time; parsing HTML that came from library calls on data we
already have free access to.

My feeling is that the Jenkins plugin should work like this:

At post-build time:

1. Find the jacoco.exec file(s) either by searching the workspace or
by looking at one or more user-specified locations
2. Find the .java and .class files also by searching or by looking in
user-specified locations
3. Use the JaCoCo library to generate the following (all of which will
be saved with the Jenkins build so it can be retrieved later):
a. all line-by-line coverage reports in HTML format
b. all module, package, and class reports (these are the pages with
the red/green bars that link to each other, and eventually the
line-by-line reports)
c. a handful of statistics that are interesting to plot on a
timeline (like total covered, partly covered, and missed lines;
complexity; etc.)

When rendering a page (after the build has completed):

1. Retrieve the data that was stored with the build in step 3 above
(part a, b, or c as needed by the page that's being rendered)
2. Pass the retrieved data to stapler


Things I don't think it should do, given my present understanding of everything:

- use the jacoco.xml file (we can just generate everything we need at
build time)
- parse HTML output (we can/should modify what is generated by the
jacoco reporting library so we only get the parts we need)


We had discussed lots of alternatives on this list before, and I
believe this is the most sensible approach. Parts 1, 2, and 3 of the
post-build steps I've outlined above are already prototyped and
working. AFAIK, the only remaining work is to implement
configurability and to save the generated data so it can be retrieved
later.

What does everyone else think?

-Jonathan

[Ognjen Bubalo]

Hi, 

This is ok for me. How should we retreive and save the data? In which form? I understand that we can use the API to retrieve the data we need, but we need a good way to save this data. Wouldn't be good to generate the XML report with the API + the highlighted HTML sourcefiles? So our data would be saved in an acceptable format, also we wouldn't have to change the existing plugin. Or is there another, better way to simply save the data we need? 

Currently the plugin saves the highlighted sourcefiles after every build, I only need to connect it with the MethodReport by passing a parameter, but this is some Digester magic :)

What do you think?

Br, Ogi

[Jonathan Fuerth]

On Jul 27, 2012 6:22 PM, "Ognjen Bubalo" <ognjen...@gmail.com> wrote:
> This is ok for me. How should we retreive and save the data? In which form? I understand that we can use the API to retrieve the data we need, but we need a good way to save this data. Wouldn't be good to generate the XML report with the API + the highlighted HTML sourcefiles? So our data would be saved in an acceptable format, also we wouldn't have to change the existing plugin. Or is there another, better way to simply save the data we need? 

After both of us taking a few hours to investigate today, the answer seems to be that we can save as much data as we want, and in whichever files we want, using instances of the Jenkins FilePath class. The root path for storing and retrieving files under a particular build is build.getRootDir().

At post-build time, the Emma plugin uses this technique to make copies of all the Emma XML files. So we have a working example of how to use this technique.

>
> Currently the plugin saves the highlighted sourcefiles after every build, I only need to connect it with the MethodReport by passing a parameter, but this is some Digester magic :)

As I mentioned, I do have lots of experience with Digester. But I think our goal of eliminating all this duplication and relying as much as possible on what JaCoCo can provide is a good one. Now that we know how to store arbitrary files associated with a build, and then retrieve them later, I think we're in good shape.

-Jonathan

[Velganesh Subramanian]

Hello all, I started using JaCoCo Jenkins plugin from last month. Our application is an Eclipse RCP and we obviously have a multi-module Maven project. I couldn't find a suitable example of using JaCoCo Maven plugin to create xml reports for multi-module Maven project. We use Jenkins and we have an ant file to convert multiple jacoco.exec files into one xml. We use your plugin (JaCoCo Coverage Reports as a post-build action) to get the trend. We are not able to get the table with complete details except for this line.

Overall Coverage Summary

all classes

86.1% 454/527

70.8% 5145/7264

64.0% 34335/53621

55.5% 7987/14394

66.2% 155673/235332

53.5% 6975/13049

I love the way you are progressing now - scan for exec, java and class files and link automatically for each build. I just subscribed to this group and will wait for your results.

[Mickael Istria]

On 08/07/2012 07:49 AM, Velganesh Subramanian wrote:

Hello all, I started using JaCoCo Jenkins plugin from last month. Our application is an Eclipse RCP and we obviously have a multi-module Maven project. I couldn't find a suitable example of using JaCoCo Maven plugin to create xml reports for multi-module Maven project.

If you use the jacoco-maven-plugin, you can have a look at this feature request and patch, that work fine for me locally: https://sourceforge.net/tracker/?func=detail&aid=3474708&group_id=177969&atid=883354
It requires your to configure the prepare-agent mojo to put a single Exec file for all your tests at the root of your project, using the outputPath and append options (you can already do that).
And this patch allows the report Mojo to look at all children of your parent pom in order to generate XML reports. Then Jacoco Jenkins plugin can consume this XML reports, which contains all necessary stuff.
Feel free to add comment to this ticket to encourage Jacoco Maven plugin developers to look at this patch ASAP.

I love the way you are progressing now - scan for exec, java and class files and link automatically for each build.

Yup, that's great!

[Ognjen Bubalo]

Hi Everyone,

The plugin now handles modules too. I just need minor fixes to do.

Br, ogi

[Mickael Istria]

Cool,
Please annonce on the mailing-list when we can give a try. I'm currently stuck as described in https://issues.jenkins-ci.org/browse/JENKINS-14274?focusedCommentId=166557&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-166557

Cheers,