Bazel Coverage Non-Test Targets

[mvo...@argo.ai]

I am currently using `bazel coverage //...` to generate a whole-project coverage report, but it appears that only test targets are analyzed. I understand that there will be zero executed lines for non-test targets, but I would like the coverage report to include the executable lines from all targets when generating the report.

This was accomplished before by ensuring that all targets were compiled with the coverage tooling so that *.gcno files were created for each source file even if it was not exercised for a test, that way the total of all theoretical executable lines would be calculated, rather than only those directly referenced from test targets.

Is there a way to expand the test report from bazel to contain the line-count metrics for non-executed code?

Thanks,

[mvo...@argo.ai]

BTW, we are talking bazel 0.23, and hoping to get the data out into the generated coverage.dat LCOV files.

[Marcel Hlopko]

+Irina Iancu 

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/e68f279e-3797-4272-8d53-f39a924207f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[Kevin Gessner]

On Wed, Jun 5, 2019 at 10:05 PM <mvo...@argo.ai> wrote:

On Wednesday, June 5, 2019 at 3:41:51 PM UTC-4, mvo...@argo.ai wrote:
> I am currently using `bazel coverage //...` to generate a whole-project coverage report, but it appears that only test targets are analyzed.  I understand that there will be zero executed lines for non-test targets, but I would like the coverage report to include the executable lines from all targets when generating the report.
>
> This was accomplished before by ensuring that all targets were compiled with the coverage tooling so that *.gcno files were created for each source file even if it was not exercised for a test, that way the total of all theoretical executable lines would be calculated, rather than only those directly referenced from test targets.
>
> Is there a way to expand the test report from bazel to contain the line-count metrics for non-executed code?
>

 

We hit this same problem, and worked around it by generating an empty test for every java_library package, forcing it to be picked up during coverage.  It's done by shadowing java_library with a macro (which we already did for default javacopts, see [1]) that creates both a native.java_library and a native.java_test for coverage.  The code we use is here if you'd like to use it: https://gist.github.com/kevingessner/8e4c9856a5f630eb12ea4b888fb85a6c

It's not ideal, as it creates a lot of extra targets, but filtering those tests by tag helps to keep the noise to coverage runs only.  Hope this helps!

-- Kevin

1: https://github.com/bazelbuild/bazel/issues/3427

> Thanks,

BTW, we are talking bazel 0.23, and hoping to get the data out into the generated coverage.dat LCOV files.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/e68f279e-3797-4272-8d53-f39a924207f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFuL9G%3DPY2yw1vLDinXOCp3Ddn3Bn1U8C1gM180t%3DO2y4jKXPg%40mail.gmail.com.

[Irina Iancu]

Running `bazel coverage //...` should trigger computing coverage for all the files from the transitive closure starting from all executable targets (test and binary). Do you also need to gather data about sources of libraries that don't get on the transitive path of any executable? If yes, can you explain in more details the reasons why?

You can also add "--coverage_report_generator=@bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main --combined_report=lcov" to your coverage command to get one combined report for everything in the project.

Also note that baseline coverage doesn't work for C++ [1], so this may also be a cause of why you're missing files with zero coverage from your reports.

[1] https://github.com/bazelbuild/bazel/issues/5716

On Wednesday, June 12, 2019 at 4:29:15 PM UTC+2, Kevin Gessner wrote:

On Wed, Jun 5, 2019 at 10:05 PM <mvo...@argo.ai> wrote:

On Wednesday, June 5, 2019 at 3:41:51 PM UTC-4, mvo...@argo.ai wrote:
> I am currently using `bazel coverage //...` to generate a whole-project coverage report, but it appears that only test targets are analyzed.  I understand that there will be zero executed lines for non-test targets, but I would like the coverage report to include the executable lines from all targets when generating the report.
>
> This was accomplished before by ensuring that all targets were compiled with the coverage tooling so that *.gcno files were created for each source file even if it was not exercised for a test, that way the total of all theoretical executable lines would be calculated, rather than only those directly referenced from test targets.
>
> Is there a way to expand the test report from bazel to contain the line-count metrics for non-executed code?
>

 

We hit this same problem, and worked around it by generating an empty test for every java_library package, forcing it to be picked up during coverage.  It's done by shadowing java_library with a macro (which we already did for default javacopts, see [1]) that creates both a native.java_library and a native.java_test for coverage.  The code we use is here if you'd like to use it: https://gist.github.com/kevingessner/8e4c9856a5f630eb12ea4b888fb85a6c

It's not ideal, as it creates a lot of extra targets, but filtering those tests by tag helps to keep the noise to coverage runs only.  Hope this helps!

-- Kevin

1: https://github.com/bazelbuild/bazel/issues/3427

> Thanks,

BTW, we are talking bazel 0.23, and hoping to get the data out into the generated coverage.dat LCOV files.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/e68f279e-3797-4272-8d53-f39a924207f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.

[mvo...@argo.ai]

We are working on gathering data about sources for libraries that do not have a path to an executable in order to capture coverage statistics for all written code. We are doing work on a modular system which dynamically loads library code at runtime.

Thanks for the tip on the combined coverage, I had some trouble getting the right command in the past, but I'll give that a shot.

What I am doing now to get baseline coverage is to run "lcov --quiet --no-external --capture --initial --directory bazel-bin --base-directory . --output-file bazel-testlogs/initial_coverage.dat" as bazel does leave *.gcno files in bazel-bin which can be analyzed by lcov. I later combine this initial_coverage.dat with the rest of the lcov output.