Measuring test coverage for plugins broken?

[Edwin Kempin]

Hi,

Does anyone know how to measure the test coverage of plugin code?

In the past it was possible to get the test coverage for plugin code by
* configuring the plugin as submodule and having its sources in the tree
* running the tools/coverage.sh script

I’m sure that this worked because I used this to inspect the test coverage of the checks plugin back then when we wrote this plugin.

Now I’m trying to do the same for the code-owners plugin.

To make the tools/coverage.sh script run for me I need to add the following 2 lines at the beginning (this I needed to do already back then, just mentioning it here if case anyone else needs this):
shopt -s expand_aliases
source ~/.bash_profile

Running tools/coverage.sh runs all tests (including the code-owners test) and produces a report. The report shows test coverage for the test classes of the code-owners plugin (though with weird 'tmp' paths, see [1]) but test coverage for the code-owners backend classes is missing.

Does anyone see what's wrong?

Thanks,
Edwin

[1] https://imgur.com/vpmInqA

[Edwin Kempin]

While looking at the script, I noticed some issues which I fixed in [1] but it didn't help with the actual issue.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/290799

[David Ostrovsky]

I tried to reproduce the problem and created these reproducers: [1],[2]

and filed this issue upstream: [3].

Now that we understand the problem, the solution could be also implemented

in code-owners plugin. I refactored the build to split the source rules to match

the java packages: [4]. Now test coverage works as expected: [5].

BTW, you shouldn't actually need custom `tools/coverage.sh` script.

Bazel is shipping lcov merge utility. Just call the coverage command

like this:

  $ bazel coverage --test_output=all \

    plugins/code-owners/... \

    --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator \

    --combined_report=lcov

This should produce merged coverage.dat file:

[...]

INFO: LCOV coverage report is located at /home/davido/.cache/bazel/_bazel_davido/5c01f4f713b675540b8b424c5c647f63/execroot/gerrit/bazel-out/_coverage/_coverage_report.dat

 and execpath is bazel-out/_coverage/_coverage_report.dat

$ ls -all bazel-out/_coverage/_coverage_report.dat

-r-xr-xr-x 1 davido users 300658 Dec  5 19:59 bazel-out/_coverage/_coverage_report.dat

[1] https://github.com/davido/bazel_coverage_src_and_test_same_package

[2] https://github.com/davido/bazel_coverage_src_and_test_different_package

[3] https://github.com/bazelbuild/bazel/issues/12636

[4] https://gerrit-review.googlesource.com/c/plugins/code-owners/+/290908

[5] https://imgur.com/a/WkwBRYN

[Edwin Kempin]

Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!

Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?
These changes likely break our internal build and it's safer for me to do this step by step.

Thanks, Edwin

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/a4f87614-cb76-4183-9819-6ce979a8db5bn%40googlegroups.com.

[David Ostrovsky]

Edwin Kempin schrieb am Mittwoch, 9. Dezember 2020 um 14:53:58 UTC+1:

Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!

You are wellcome!

 

Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?

Sure, feel free to split that change. Thanks.

[Edwin Kempin]

On Wednesday, December 9, 2020 at 3:21:14 PM UTC+1 David Ostrovsky wrote:

Edwin Kempin schrieb am Mittwoch, 9. Dezember 2020 um 14:53:58 UTC+1:

Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!

You are wellcome!

 

Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?

Sure, feel free to split that change. Thanks.

This was done by https://gerrit-review.googlesource.com/q/hashtag:split-monolithic-plugin-target

Since then the test coverage report mostly works [1] by running:

bazel coverage --test_output=all     plugins/code-owners/...     --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator     --combined_report=lcov && genhtml -o . --ignore-errors source bazel-out/_coverage/_coverage_report.dat

But I noticed that some packages are missing, e.g. "com.google.gerrit.plugins.codeowners.validation". 

David, or anyone else, do you have an idea why some package do not show up?

[1] https://i.imgur.com/9NsJ2Hp_d.webp?maxwidth=1520&fidelity=grand

[David Ostrovsky]

Edwin Kempin schrieb am Freitag, 29. Januar 2021 um 16:13:39 UTC+1:

On Wednesday, December 9, 2020 at 3:21:14 PM UTC+1 David Ostrovsky wrote:

Edwin Kempin schrieb am Mittwoch, 9. Dezember 2020 um 14:53:58 UTC+1:

Hi David,

Sorry for replying late, but I was OOO for a few days.

Thank you so much David, you are the best!
With your change, generating the test coverage works for me too!

You are wellcome!

 

Do you mind if I split your change into smaller chunks so that I can apply it piece by piece?

Sure, feel free to split that change. Thanks.

This was done by https://gerrit-review.googlesource.com/q/hashtag:split-monolithic-plugin-target

Thanks.

 

Since then the test coverage report mostly works [1] by running:

bazel coverage --test_output=all     plugins/code-owners/...     --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator     --combined_report=lcov && genhtml -o . --ignore-errors source bazel-out/_coverage/_coverage_report.dat

Great.

 

But I noticed that some packages are missing, e.g. "com.google.gerrit.plugins.codeowners.validation". 

David, or anyone else, do you have an idea why some package do not show up?

If you run the above command, you can see this log output:

INFO: Using default value for --instrumentation_filter: 

"^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/api[/:],

^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/batch[/:],

^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/restapi[/:],

^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/testsuite[/:],

^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/backend[/:],

^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/restapi[/:],

^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/util[/:]"

The instrumentation filter is inferred from bazel test packages. As can be seen from

the `tree -d` result: [1], the source and test packages differ in code-owners plugin. To

help bazel, you need to override the instrumentation filter. This command should do

what you want, see this coverage result: [2]

  $ bazel coverage --test_output=all plugins/code-owners/... --coverage_report_generator=@bazel_tools//tools/test:coverage_report_generator --combined_report=lcov --instrumentation_filter="^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/acceptance/testsuite[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/api[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/api/impl[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/backend[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/common[/:],,^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/metrics[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/restapi[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/testing[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/util[/:],^//plugins/code-owners/java/com/google/gerrit/plugins/codeowners/validation[/:]"

Obviously, you wouldn't need it, if the source and test packages would be identical.

[1] https://imgur.com/a/uNuaXmb

[2] https://imgur.com/a/koL8r6L

[Edwin Kempin]

 Awesome! David, thank you very much! This works :)

Obviously, you wouldn't need it, if the source and test packages would be identical.

[1] https://imgur.com/a/uNuaXmb

[2] https://imgur.com/a/koL8r6L

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/bfc5414d-d5d0-4558-965e-a0b1f4e98bd7n%40googlegroups.com.