Java Test Coverage

[Rob Figueiredo]

Hi all,

I was hoping to get some kind of example or high-level guidance of how coverage should be used, in particular report generation. Previously, we used a jacoco ant task to produce reports in a known directory where they can get picked up by TeamCity. We've had a lot of trouble figuring out how to get this to work with bazel.

The command-line reference talks about a --coverage_report_generator option; after spending too much time trying to get that to work (and then resorting to searching through the bazel code), I concluded that it did literally nothing. If I'm missing something about this or there's some other way to integrate a report tool that would be nice to know.

More philosophically, I would be interested to know if there were any plans to make test output more extensible. For writing test outputs beyond logs, I found that you can write to $TEST_UNDECLARED_OUTPUTS_DIR/outputs.zip ; this seems hacky and is undocumented, surely there should be a better way to generate a test output file. This is what keeps us from just using jacoco directly -- we can set it up as an agent, but have nowhere to put the file it generates.

Two notes:

(1) it seems like a missed opportunity to not have some way to make a rule depend on test output as an input (this seems like it would be the natural way to implement a coverage report generator); perhaps that was considered out of scope for bazel.

(2) once we get bazel coverage to work, I think we would need to specify different values of --instrumentation_filter for each test suite, perhaps by creating a wrapper script for each. This is necessary to filter the coverage report to the relevant code (e.g. the package being tested) -- why would we care how much of a util library is covered by a non-util test? We have a monorepo in the google3 model, and this challenge seems like the common case.. has it been considered to move specification of --instrumentation_filter to the rule level, which could handle this pretty naturally?

Thank you very much for your help!

Rob

[Irina Iancu]

Hi Rob,

First, I want to appologize because coverage in bazel is not 100% open sourced and it lacks some features. Let me see how I can help.

I was hoping to get some kind of example or high-level guidance of how coverage should be used, in particular report generation.

 

bazel coverage creates a .dat file containing coverage information. Its location is printed at the end of the bazel coverage run. You can use that file as input to genthml to get the coverage results as html.

 The command-line reference talks about a --coverage_report_generator option; after spending too much time trying to get that to work (and then resorting to searching through the bazel code), I concluded that it did literally nothing

 

Unfortunately you are right, it does nothing in bazel. The coverage report action has not been open sourced yet. If we open source the action, writing a report generator should be straight forward, as it would have to follow a protocol. (this also addresses note (1) )

More philosophically, I would be interested to know if there were any plans to make test output more extensible. For writing test outputs beyond logs, I found that you can write to $TEST_UNDECLARED_OUTPUTS_DIR/outputs.zip ; this seems hacky and is undocumented, surely there should be a better way to generate a test output file. This is what keeps us from just using jacoco directly -- we can set it up as an agent, but have nowhere to put the file it generates.

What do you mean by making the test output more extensible? I was not even aware of $TEST_UNDECLARED_OUTPUTS_DIR/outputs.zip so let's not go in that direction because as you said, it seems hacky. But bazel coverage already uses jacoco and outputs the generated file. Maybe I am missing something here? Can you elaborate more?

(2) once we get bazel coverage to work, I think we would need to specify different values of --instrumentation_filter for each test suite, perhaps by creating a wrapper script for each. This is necessary to filter the coverage report to the relevant code (e.g. the package being tested) -- why would we care how much of a util library is covered by a non-util test? We have a monorepo in the google3 model, and this challenge seems like the common case.. has it been considered to move specification of --instrumentation_filter to the rule level, which could handle this pretty naturally?

This wasn't considered as far as I know. From what I've seen already, people run coverage on multiple test rules at once and include all the needed filters in --instrumentation_filter, that way they get code coverage for everything. How do you plan to run coverage? Are you more concerned with running it locally or within a CI?

[ssuc...@pepperdata.com]

On Friday, November 3, 2017 at 7:09:10 PM UTC-4, Irina Iancu wrote:
...

> bazel coverage creates a .dat file containing coverage information. Its location is printed at the end of the bazel coverage run. You can use that file as input to genthml to get the coverage results as html.

I tried using this file as input to genhtml, but I was stymied because my Java source files are not all in the same directory. I've got some that are in src/main/java, some that are in src/test/java, and one that is autogenerated as part of a Bazel rule. (So it only exists in bazel-genfiles).

I can imagine writing a Bazel rule to collect all these source files into a new directory and run genhtml in there. It's a little cumbersome, but could work. It'd be ideal if I could create a report as part of some build rule. The only part that I don't know how to do there is to get the coverage.dat - I only know how to do that by running 'bazel coverage' - is there some implicit rule that I can use that causes the same effect of generating the coverage.dat file?

Otherwise, my best idea is some automation outside of Bazel that runs 'bazel coverage', puts the file in some place that another rule can use it as a source file in a 'bazel build' rule. Is this the best I can do, or is there something better?

Thanks, Sean

[peter.v...@gmail.com]

On Monday, 20 November 2017 02:58:15 UTC, ssuc...@pepperdata.com wrote:
> On Friday, November 3, 2017 at 7:09:10 PM UTC-4, Irina Iancu wrote:
> ...

> I can imagine writing a Bazel rule to collect all these source files into a new directory and run genhtml in there. It's a little cumbersome, but could work. It'd be ideal if I could create a report as part of some build rule. The only part that I don't know how to do there is to get the coverage.dat - I only know how to do that by running 'bazel coverage' - is there some implicit rule that I can use that causes the same effect of generating the coverage.dat file?
>
> Otherwise, my best idea is some automation outside of Bazel that runs 'bazel coverage', puts the file in some place that another rule can use it as a source file in a 'bazel build' rule. Is this the best I can do, or is there something better?

Being able to refer to both test and coverage artifacts in the same way as other outputs would be useful for general quality analysis. For example I have a (fairly crude) SonarQube executable rule which pulls together various build sources and artifacts and runs the SQ Scanner against them. But it's missing the test and coverage metrics as there's no way to feed the data into the rule.

It would really fill in the gaps if we could include in the deps/runfiles of a rule the (possibly transformed/aggregated) test and coverage reports.

But right now I'm in the same boat as you - a separate extra-bazel job which has to be manually aware of all these artifact locations.

Thinking about it again now, I guess the authoritative way is to use the build events stream/artifacts[1] to find the info, rather than manually configure artifact locations...

[1]: https://docs.bazel.build/versions/master/build-event-protocol.html