Measuring test coverage for plugins broken?

110 views
Skip to first unread message

Edwin Kempin

unread,
Dec 4, 2020, 8:23:09 AM12/4/20
to Repo and Gerrit Discussion
Hi,

Does anyone know how to measure the test coverage of plugin code?

In the past it was possible to get the test coverage for plugin code by
* configuring the plugin as submodule and having its sources in the tree
* running the tools/coverage.sh script

I’m sure that this worked because I used this to inspect the test coverage of the checks plugin back then when we wrote this plugin.

Now I’m trying to do the same for the code-owners plugin.

To make the tools/coverage.sh script run for me I need to add the following 2 lines at the beginning (this I needed to do already back then, just mentioning it here if case anyone else needs this):
shopt -s expand_aliases
source ~/.bash_profile

Running tools/coverage.sh runs all tests (including the code-owners test) and produces a report. The report shows test coverage for the test classes of the code-owners plugin (though with weird 'tmp' paths, see [1]) but test coverage for the code-owners backend classes is missing.

Does anyone see what's wrong?

Thanks,
Edwin

[1] https://imgur.com/vpmInqA

Edwin Kempin

unread,
Dec 4, 2020, 8:32:07 AM12/4/20
to Repo and Gerrit Discussion
While looking at the script, I noticed some issues which I fixed in [1] but it didn't help with the actual issue.

David Ostrovsky

unread,
Dec 5, 2020, 3:21:50 PM12/5/20
to Repo and Gerrit Discussion
I tried to reproduce the problem and created these reproducers: [1],[2]
and filed this issue upstream: [3].

Now that we understand the problem, the solution could be also implemented
in code-owners plugin. I refactored the build to split the source rules to match
the java packages: [4]. Now test coverage works as expected: [5].

BTW, you shouldn't actually need custom `tools/coverage.sh` script.
Bazel is shipping lcov merge utility. Just call the coverage command
like this:

  $ bazel coverage --test_output=all \
    plugins/code-owners/... \
    --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator \
    --combined_report=lcov

This should produce merged coverage.dat file:

[...]
INFO: LCOV coverage report is located at /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/execroot/gerrit/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat

$ ls -all bazel-out/_coverage/_coverage_report.dat
-r-xr-xr-x 1 davido users 300658 Dec  5 19:59 bazel-out/_coverage/_coverage_report.dat

Edwin Kempin

unread,
Dec 9, 2020, 8:53:58 AM12/9/20
to David Ostrovsky, Repo and Gerrit Discussion
Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!
Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?
These changes likely break our internal build and it's safer for me to do this step by step.

Thanks, Edwin


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a4f87614-cb76-4183-9819-6ce979a8db5bn%40googlegroups.com.

David Ostrovsky

unread,
Dec 9, 2020, 9:21:14 AM12/9/20
to Repo and Gerrit Discussion
Edwin Kempin schrieb am Mittwoch, 9. Dezember 2020 um 14:53:58 UTC+1:
Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!

You are wellcome!
 
Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?

Sure, feel free to split that change. Thanks.

Edwin Kempin

unread,
Jan 29, 2021, 10:13:39 AMJan 29
to Repo and Gerrit Discussion
On Wednesday, December 9, 2020 at 3:21:14 PM UTC+1 David Ostrovsky wrote:
Edwin Kempin schrieb am Mittwoch, 9. Dezember 2020 um 14:53:58 UTC+1:
Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!

You are wellcome!
 
Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?

Sure, feel free to split that change. Thanks.


Since then the test coverage report mostly works [1] by running:

bazel coverage --test_output=all     plugins/code-owners/...     --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator     --combined_report=lcov && genhtml -o . --ignore-errors source bazel-out/_coverage/_coverage_report.dat

But I noticed that some packages are missing, e.g. "com.google.gerrit.plugins.codeowners.validation". 
David, or anyone else, do you have an idea why some package do not show up?

David Ostrovsky

unread,
Jan 30, 2021, 3:14:13 PMJan 30
to Repo and Gerrit Discussion
Edwin Kempin schrieb am Freitag, 29. Januar 2021 um 16:13:39 UTC+1:
On Wednesday, December 9, 2020 at 3:21:14 PM UTC+1 David Ostrovsky wrote:
Edwin Kempin schrieb am Mittwoch, 9. Dezember 2020 um 14:53:58 UTC+1:
Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!

You are wellcome!
 
Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?

Sure, feel free to split that change. Thanks.


Thanks.
 
Since then the test coverage report mostly works [1] by running:

bazel coverage --test_output=all     plugins/code-owners/...     --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator     --combined_report=lcov && genhtml -o . --ignore-errors source bazel-out/_coverage/_coverage_report.dat


Great.
 
But I noticed that some packages are missing, e.g. "com.google.gerrit.plugins.codeowners.validation". 
David, or anyone else, do you have an idea why some package do not show up?


If you run the above command, you can see this log output:

INFO: Using default value for --instrumentation_filter: 
"^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/api[/:],
^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/batch[/:],
^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/restapi[/:],
^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/testsuite[/:],
^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/backend[/:],
^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/restapi[/:],
^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/util[/:]"

The instrumentation filter is inferred from bazel test packages. As can be seen from
the `tree -d` result: [1], the source and test packages differ in code-owners plugin. To
help bazel, you need to override the instrumentation filter. This command should do
what you want, see this coverage result: [2]

  $ bazel coverage --test_output=all plugins/code-owners/... --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator --combined_report=lcov --instrumentation_filter="^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/testsuite[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/api[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/api/impl[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/backend[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/common[/:],,^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/metrics[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/restapi[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/testing[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/util[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/validation[/:]"

Obviously, you wouldn't need it, if the source and test packages would be identical.


Edwin Kempin

unread,
Feb 1, 2021, 3:19:39 AMFeb 1
to David Ostrovsky, Repo and Gerrit Discussion
 Awesome! David, thank you very much! This works :)

Obviously, you wouldn't need it, if the source and test packages would be identical.


--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages