Python test coverage support

[Hyrum Wright]

I'm interesting in having Bazel support code coverage for Python tests.

From https://github.com/bazelbuild/bazel/issues/1118, it seems like the basic infrastructure is there.

https://docs.google.com/document/d/1WRQTXQBV-m1_HDkAPvRSfGIrDW4pkI7ATPWtNdC8k9E/edit describes the ideal world of how coverage should work.  The doc looks to be a bit dated, and since Java coverage is already implemented it would seem that the basic infrastructure exists for running coverage.

Python already has tooling for running coverage outside of Bazel: https://coverage.readthedocs.io/en/coverage-4.4.1/  It would be nice to be able to integrate that tooling into Bazel for use on Python tests.

Any thoughts, updates, or pointers here?  Is supporting python coverage in Bazel a fool's errand, or actually feasible?

-Hyrum

[Hyrum Wright]

Ping.

I'm happy to contribute effort here, but it would probably be useful to have some pointers.

[Matthieu Poncin]

I would be very interested in this too

[Hyrum Wright]

Any pointers here?  Is this even the right place to ask?

I'm happy to do much of the legwork to make python coverage support happen, but it will go much quicker if somebody with more familiarity than myself can give me a brief overview of what needs to be done.

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/43a48c26-ce65-4cc0-9e87-fd99c16b726f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marcel Hlopko]

+Ulf Adams (but he's currently out sick, get well soon Ulf!) for the high level vision.

+Yilei Yang for python insight

+Dmitry Lomov for vision :)

Last thing I heard was that we don't want to invest much into language specific coverage support. Ideally we would have a cross language coverage framework, but afaik nobody is working on this currently. 

On Tue, Nov 14, 2017 at 7:57 AM Hyrum Wright <hwr...@duolingo.com> wrote:

Any pointers here?  Is this even the right place to ask?

I'm happy to do much of the legwork to make python coverage support happen, but it will go much quicker if somebody with more familiarity than myself can give me a brief overview of what needs to be done.

-Hyrum

On Tue, Nov 7, 2017 at 1:20 AM, Matthieu Poncin <matt...@yousician.com> wrote:

I would be very interested in this too

On Monday, October 30, 2017 at 4:42:58 PM UTC, Hyrum Wright wrote:

I'm interesting in having Bazel support code coverage for Python tests.

From https://github.com/bazelbuild/bazel/issues/1118, it seems like the basic infrastructure is there.

https://docs.google.com/document/d/1WRQTXQBV-m1_HDkAPvRSfGIrDW4pkI7ATPWtNdC8k9E/edit describes the ideal world of how coverage should work.  The doc looks to be a bit dated, and since Java coverage is already implemented it would seem that the basic infrastructure exists for running coverage.

Python already has tooling for running coverage outside of Bazel: https://coverage.readthedocs.io/en/coverage-4.4.1/  It would be nice to be able to integrate that tooling into Bazel for use on Python tests.

Any thoughts, updates, or pointers here?  Is supporting python coverage in Bazel a fool's errand, or actually feasible?

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/43a48c26-ce65-4cc0-9e87-fd99c16b726f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CALmYJEVkJUCQm_TPf5YGkcfDRVgxbxPYRg_0MbSF-P-ZLmg6gg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

-- 

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[Hyrum Wright]

Thanks for the response.

Do you know where any of this is documented, and what the potential timelines are?  Coverage is an important test metric in a lot of use cases, particularly for languages like Python which lack compile-time checks.  We'd really like to use Bazel for python projects, and having coverage support is a big part of that.  I'm willing to contribute toward making that happen.

On Tue, Nov 14, 2017 at 6:08 PM, Marcel Hlopko <hlo...@google.com> wrote:

+Ulf Adams (but he's currently out sick, get well soon Ulf!) for the high level vision.

+Yilei Yang for python insight

+Dmitry Lomov for vision :)

Last thing I heard was that we don't want to invest much into language specific coverage support. Ideally we would have a cross language coverage framework, but afaik nobody is working on this currently. 

On Tue, Nov 14, 2017 at 7:57 AM Hyrum Wright <hwr...@duolingo.com> wrote:

Any pointers here?  Is this even the right place to ask?

I'm happy to do much of the legwork to make python coverage support happen, but it will go much quicker if somebody with more familiarity than myself can give me a brief overview of what needs to be done.

-Hyrum

On Tue, Nov 7, 2017 at 1:20 AM, Matthieu Poncin <matt...@yousician.com> wrote:

I would be very interested in this too

On Monday, October 30, 2017 at 4:42:58 PM UTC, Hyrum Wright wrote:

I'm interesting in having Bazel support code coverage for Python tests.

From https://github.com/bazelbuild/bazel/issues/1118, it seems like the basic infrastructure is there.

https://docs.google.com/document/d/1WRQTXQBV-m1_HDkAPvRSfGIrDW4pkI7ATPWtNdC8k9E/edit describes the ideal world of how coverage should work.  The doc looks to be a bit dated, and since Java coverage is already implemented it would seem that the basic infrastructure exists for running coverage.

Python already has tooling for running coverage outside of Bazel: https://coverage.readthedocs.io/en/coverage-4.4.1/  It would be nice to be able to integrate that tooling into Bazel for use on Python tests.

Any thoughts, updates, or pointers here?  Is supporting python coverage in Bazel a fool's errand, or actually feasible?

-Hyrum

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/43a48c26-ce65-4cc0-9e87-fd99c16b726f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CALmYJEVkJUCQm_TPf5YGkcfDRVgxbxPYRg_0MbSF-P-ZLmg6gg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Ulf Adams]

Sure, give it a try. It shouldn't be too hard to make it work.

Ideally, all languages output a single unified format (into separate per-process files), and we provide a tool to merge the files and which Bazel runs as part of the test action, and then there's another tool to post-process (e.g., to generate an HTML report or whatnot). Tool for merging exists, but it currently only handles whatever we output from the Java coverage runner.

https://github.com/bazelbuild/bazel/blob/8328dc9e3b0d473756ec94508744afefe4d8eaf2/tools/test/LcovMerger/java/com/google/devtools/lcovmerger/LcovMerger.java

[Ulf Adams]

Oh yeah, the tool is just a stub right now. Duh!

[Hyrum Wright]

Thanks, Ulf.

Where in the tree are the appropriate hooks to intercept the `bazel coverage` command and run the tests with a different running (e.g., a vendored copy of coverage.py)?

[Hyrum Wright]

After doing a bunch of spelunking, I've managed to track this down.  I'm recording my findings here, mostly so I don't forget them over the next few days, but also in the optimistic hope that somebody on the core team can read and comment on the validity of the approach.

Background

Python coverage support is provided by coverage.py, a modified test runner which instruments the code as it is running and then provides output in a proprietary format.  This format can be read through an API or transformed into xml or other formats.  Multiple independent test runs can be combined into a single report through the coverage tool.

Approach

To use coverage, we inject it as part of the test running run running with bazel coverage.  An initial attempt at doing so is here:

https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage

Remaining issues:

1. Using the vendored coverage module included with Bazel, rather than depending on the system coverage module.
2. Finding where the coverage output is being placed.
3. Merging multiple coverage reports into a single one for a given Bazel invocation.

I'd appreciate if somebody with familiarity with Bazel could look over the above branch, and offer pointers on the issues above.

Thanks,

-Hyrum

[Ulf Adams]

Sorry I was out sick for the last week.

What you describe sounds about right.

Ideally, we'd make it output the coverage in lcov format, and change the LcovMerger tool to actually merge multiple lcov format files.

(I don't have any particular preference for lcov versus some other format, but that's what Google uses internally across most of the languages we write code for, and I didn't find anything that was clearly technically superior or more widely used than lcov. Although I hear that Haskell does per-character coverage, rather than per-line, and I'm not sure lcov supports that.)

[Hyrum Wright]

For my current purposes, I'm more interested in just getting the coverage tooling running.  After it is generating output, we can worry about proper formatting and merging.

To get the tooling running, it would be helpful to know the answers to these questions:

•  Where is the location of the vendored copy of coverage, so that it can be added to the python path here: https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage#diff-34d8cb4d69c74a59002d79684518a778R151  (Note that this isn't for the test itself, but inside of the python stub.)
  
  
•  Where are the resulting files written?  In which part of the the bazel symlink hierarchy should I look for the output written by the test runner?  Is that location considered temporary?
  

Thanks,

-Hyrum

[Ulf Adams]

On Mon, Nov 27, 2017 at 2:15 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

For my current purposes, I'm more interested in just getting the coverage tooling running.  After it is generating output, we can worry about proper formatting and merging.

To get the tooling running, it would be helpful to know the answers to these questions:

•  Where is the location of the vendored copy of coverage, so that it can be added to the python path here: https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage#diff-34d8cb4d69c74a59002d79684518a778R151  (Note that this isn't for the test itself, but inside of the python stub.)
  

I don't see a vendored copy of coverage.py - why do you think we are vendoring one?

 

• 
  
•  Where are the resulting files written?  In which part of the the bazel symlink hierarchy should I look for the output written by the test runner?  Is that location considered temporary?
  

The result files need to be written to a temporary directory COVERAGE_DIR. The test runner should take steps to ensure unique file names. If you run locally without sandboxing, the coverage directory won't be deleted, so you should be able to see the results after a test run.

[Hyrum Wright]

On Tue, Nov 28, 2017 at 3:43 AM, Ulf Adams <ulf...@google.com> wrote:

On Mon, Nov 27, 2017 at 2:15 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

For my current purposes, I'm more interested in just getting the coverage tooling running.  After it is generating output, we can worry about proper formatting and merging.

To get the tooling running, it would be helpful to know the answers to these questions:

•  Where is the location of the vendored copy of coverage, so that it can be added to the python path here: https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage#diff-34d8cb4d69c74a59002d79684518a778R151  (Note that this isn't for the test itself, but inside of the python stub.)
  

I don't see a vendored copy of coverage.py - why do you think we are vendoring one?

I've added the vendored copy on my branch:

https://github.com/hwright/bazel/tree/py-coverage

With that branch, I can explicitly add "@bazel_tools//third_party/py/coverage" to the dependencies of a py_test, and the test stub will find the coverage module.  I'd like for the test stub to be able to find that module without having to explicitly include it in the test dependencies, since it is only required for running coverage, and it is always required when running coverage.

 

 

• 
  
•  Where are the resulting files written?  In which part of the the bazel symlink hierarchy should I look for the output written by the test runner?  Is that location considered temporary?
  

The result files need to be written to a temporary directory COVERAGE_DIR. The test runner should take steps to ensure unique file names. If you run locally without sandboxing, the coverage directory won't be deleted, so you should be able to see the results after a test run.

Can you point me at the documentation for COVERAGE_DIR?  Is this something which bazel creates when running in coverage mode?  I assume it's an environment variable; in which environments is it set?

[Ulf Adams]

On Tue, Nov 28, 2017 at 2:57 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Tue, Nov 28, 2017 at 3:43 AM, Ulf Adams <ulf...@google.com> wrote:

On Mon, Nov 27, 2017 at 2:15 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

For my current purposes, I'm more interested in just getting the coverage tooling running.  After it is generating output, we can worry about proper formatting and merging.

To get the tooling running, it would be helpful to know the answers to these questions:

•  Where is the location of the vendored copy of coverage, so that it can be added to the python path here: https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage#diff-34d8cb4d69c74a59002d79684518a778R151  (Note that this isn't for the test itself, but inside of the python stub.)
  

I don't see a vendored copy of coverage.py - why do you think we are vendoring one?

I've added the vendored copy on my branch:

https://github.com/hwright/bazel/tree/py-coverage

With that branch, I can explicitly add "@bazel_tools//third_party/py/coverage" to the dependencies of a py_test, and the test stub will find the coverage module.  I'd like for the test stub to be able to find that module without having to explicitly include it in the test dependencies, since it is only required for running coverage, and it is always required when running coverage.

Ah, I see. You'd have to change the definition of py_test in Bazel to get the dependency.

 

 

 

• 
  
•  Where are the resulting files written?  In which part of the the bazel symlink hierarchy should I look for the output written by the test runner?  Is that location considered temporary?
  

The result files need to be written to a temporary directory COVERAGE_DIR. The test runner should take steps to ensure unique file names. If you run locally without sandboxing, the coverage directory won't be deleted, so you should be able to see the results after a test run.

Can you point me at the documentation for COVERAGE_DIR?  Is this something which bazel creates when running in coverage mode?  I assume it's an environment variable; in which environments is it set?

It's an environment variable giving an absolute path to a directory. It's set for all tests when running under blaze coverage.

It should be documented here:

https://docs.bazel.build/versions/master/test-encyclopedia.html

It's actually documented here:

https://docs.google.com/document/d/1WRQTXQBV-m1_HDkAPvRSfGIrDW4pkI7ATPWtNdC8k9E/edit

[Hyrum Wright]

On Tue, Nov 28, 2017 at 9:11 AM, Ulf Adams <ulf...@google.com> wrote:

On Tue, Nov 28, 2017 at 2:57 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Tue, Nov 28, 2017 at 3:43 AM, Ulf Adams <ulf...@google.com> wrote:

On Mon, Nov 27, 2017 at 2:15 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

For my current purposes, I'm more interested in just getting the coverage tooling running.  After it is generating output, we can worry about proper formatting and merging.

To get the tooling running, it would be helpful to know the answers to these questions:

•  Where is the location of the vendored copy of coverage, so that it can be added to the python path here: https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage#diff-34d8cb4d69c74a59002d79684518a778R151  (Note that this isn't for the test itself, but inside of the python stub.)
  

I don't see a vendored copy of coverage.py - why do you think we are vendoring one?

I've added the vendored copy on my branch:

https://github.com/hwright/bazel/tree/py-coverage

With that branch, I can explicitly add "@bazel_tools//third_party/py/coverage" to the dependencies of a py_test, and the test stub will find the coverage module.  I'd like for the test stub to be able to find that module without having to explicitly include it in the test dependencies, since it is only required for running coverage, and it is always required when running coverage.

Ah, I see. You'd have to change the definition of py_test in Bazel to get the dependency.

I spent some time last week digging around in the python rules and trying to figure out where the extra dependency would get injected.  I came up empty, but had some promising leads.  Specifically, it looks like there are multiple implementations of the py_test rule in Bazel:

https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/rules/python/PyTest.java

https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPyTestRule.java

Which one is considered canonical?

[Ulf Adams]

On Tue, Nov 28, 2017 at 3:55 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Tue, Nov 28, 2017 at 9:11 AM, Ulf Adams <ulf...@google.com> wrote:

On Tue, Nov 28, 2017 at 2:57 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Tue, Nov 28, 2017 at 3:43 AM, Ulf Adams <ulf...@google.com> wrote:

On Mon, Nov 27, 2017 at 2:15 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

For my current purposes, I'm more interested in just getting the coverage tooling running.  After it is generating output, we can worry about proper formatting and merging.

To get the tooling running, it would be helpful to know the answers to these questions:

•  Where is the location of the vendored copy of coverage, so that it can be added to the python path here: https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage#diff-34d8cb4d69c74a59002d79684518a778R151  (Note that this isn't for the test itself, but inside of the python stub.)
  

I don't see a vendored copy of coverage.py - why do you think we are vendoring one?

I've added the vendored copy on my branch:

https://github.com/hwright/bazel/tree/py-coverage

With that branch, I can explicitly add "@bazel_tools//third_party/py/coverage" to the dependencies of a py_test, and the test stub will find the coverage module.  I'd like for the test stub to be able to find that module without having to explicitly include it in the test dependencies, since it is only required for running coverage, and it is always required when running coverage.

Ah, I see. You'd have to change the definition of py_test in Bazel to get the dependency.

I spent some time last week digging around in the python rules and trying to figure out where the extra dependency would get injected.  I came up empty, but had some promising leads.  Specifically, it looks like there are multiple implementations of the py_test rule in Bazel:

https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/rules/python/PyTest.java

https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPyTestRule.java

BazelPyTestRule declares the py_test() Skylark function and parameters. PyTest is the implementation. BazelPyTest is just a thin wrapper around PyTest that injects the BazelPythonSemantics.

Unfortunately, I think we didn't add a mechanism (yet?) that allows to declare a conditional dependency if coverage is enabled. You'll have to unconditionally make all py_test rules (or even all py_* rules) statically depend on the coverage target. In Java-land, all Java targets depend on a Java coverage target - however, we make sure to only use that if coverage is enabled, so the cost of that is one additional loaded package.

[Hyrum Wright]

Update:

I've now got a Bazel which can run python coverage.  Anybody that's interested can see the relevant patch here:

https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage

The method for making the vendored coverage module available to the test stub seems like quite the hack: plumbing a path from the install path back through the BazelPythonConfiguration.  If there's a better way to do this (or one that would make merging this branch more palatable) please let me know.

The next step seems to be to update tools/test/collect_coverage.sh to combine the output generated by bazel coverage ....  In looking at collect_coverage.sh, I can't really tell if it's trying to be Java-specific or C++ specific or something else.  It looks to have a dependency on the system-installed /usr/bin/lcov binary, which doesn't seem ideal from a hermeticity perspective.

I plan to use coverage.py's own faculties for merging its output.  Is collect_coverage.sh the right place to do so?  If so, how can I get the path to the vendored coverage installation in the Bazel install base from within that script?  Is there a way to differentiate the different coverage tools producing output that script consumes?

Thanks again for the pointers,

-Hyrum

[Ulf Adams]

On Tue, Nov 28, 2017 at 9:30 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

Update:

I've now got a Bazel which can run python coverage.  Anybody that's interested can see the relevant patch here:

https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage

The method for making the vendored coverage module available to the test stub seems like quite the hack: plumbing a path from the install path back through the BazelPythonConfiguration.  If there's a better way to do this (or one that would make merging this branch more palatable) please let me know.

It needs to be a dependency on a rule or target, for hermeticity.

 

The next step seems to be to update tools/test/collect_coverage.sh to combine the output generated by bazel coverage ....  In looking at collect_coverage.sh, I can't really tell if it's trying to be Java-specific or C++ specific or something else.  It looks to have a dependency on the system-installed /usr/bin/lcov binary, which doesn't seem ideal from a hermeticity perspective.

It's trying to be language-agnostic. The idea is that all test runners generate the same format into a common directory, and then we can use a simple tool to merge all the results. Having language-specific sections in collect_coverage.sh is a problem - if we can't do it in a language-agnostic way then we need to find a way to inject language-specific parts into the processing without having them hard-coded in the script. (Although I could be convinced to defer such work.)

Note that Bazel only declares a single coverage output file per test at this time, so if we want that to work for multi-lingual integration tests (which would be nice), they all have to be in the same format. There have been calls for allowing multiple output files / formats, but it's going to be a strictly worse user experience, as we then can't really post-process the results, e.g., to generate an html report. As a temporary workaround, I hear that some people are generating coverage data into the undeclared outputs directory.

 

I plan to use coverage.py's own faculties for merging its output.  Is collect_coverage.sh the right place to do so?  If so, how can I get the path to the vendored coverage installation in the Bazel install base from within that script?  Is there a way to differentiate the different coverage tools producing output that script consumes?

As I said, we don't currently have a way of injecting language-specific information into the script. The way we do it at Google is that we have hard-coded support for specific languages, i.e., we haven't solved this problem yet. I'd prefer not to repeat that, but I'm also sympathetic to a desire to have it work without rewriting the entire infrastructure.

The InstrumentedFilesProvider currently has a way of injecting environment variables from the rule implementations to the test. You might be able to make that do what you need, but it's probably insufficient to handle the general case.

[Hyrum Wright]

On Wed, Nov 29, 2017 at 3:03 AM, Ulf Adams <ulf...@google.com> wrote:

On Tue, Nov 28, 2017 at 9:30 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

Update:

I've now got a Bazel which can run python coverage.  Anybody that's interested can see the relevant patch here:

https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage

The method for making the vendored coverage module available to the test stub seems like quite the hack: plumbing a path from the install path back through the BazelPythonConfiguration.  If there's a better way to do this (or one that would make merging this branch more palatable) please let me know.

It needs to be a dependency on a rule or target, for hermeticity.

Right.

My initial attempts to add "@bazel_tools//third_party/py/coverage" to the dependencies from within the PyTest implementation were a bit muddled.  My naive approach was to clone the existing RuleContext and replace it with one which has that target as an additional dependency here: https://github.com/bazelbuild/bazel/blob/91fb38e92ace6cf14ce5da6527d71320b4e3f3d2/src/main/java/com/google/devtools/build/lib/rules/python/PyTest.java#L40

However, RuleContext has both a private constructor and a private Builder, so it would appear that swapping it out for another would be impossible.  Do you have pointers on the mechanics for adding additional dependencies from within a rule implementation?

 

 

The next step seems to be to update tools/test/collect_coverage.sh to combine the output generated by bazel coverage ....  In looking at collect_coverage.sh, I can't really tell if it's trying to be Java-specific or C++ specific or something else.  It looks to have a dependency on the system-installed /usr/bin/lcov binary, which doesn't seem ideal from a hermeticity perspective.

It's trying to be language-agnostic. The idea is that all test runners generate the same format into a common directory, and then we can use a simple tool to merge all the results. Having language-specific sections in collect_coverage.sh is a problem - if we can't do it in a language-agnostic way then we need to find a way to inject language-specific parts into the processing without having them hard-coded in the script. (Although I could be convinced to defer such work.)

I suspect that doing this in a language-agnostic fashion will require a language-agnostic format.  I don't know that the lcov format is such (though to be honest, I haven't been able to find a spec of the lcov format, so I'm mostly guessing there).  Deciding on and implementing said language-agnostic format is far beyond the scope of this work.

 

Note that Bazel only declares a single coverage output file per test at this time, so if we want that to work for multi-lingual integration tests (which would be nice), they all have to be in the same format. There have been calls for allowing multiple output files / formats, but it's going to be a strictly worse user experience, as we then can't really post-process the results, e.g., to generate an html report. As a temporary workaround, I hear that some people are generating coverage data into the undeclared outputs directory.

 

I plan to use coverage.py's own faculties for merging its output.  Is collect_coverage.sh the right place to do so?  If so, how can I get the path to the vendored coverage installation in the Bazel install base from within that script?  Is there a way to differentiate the different coverage tools producing output that script consumes?

As I said, we don't currently have a way of injecting language-specific information into the script. The way we do it at Google is that we have hard-coded support for specific languages, i.e., we haven't solved this problem yet. I'd prefer not to repeat that, but I'm also sympathetic to a desire to have it work without rewriting the entire infrastructure.

The InstrumentedFilesProvider currently has a way of injecting environment variables from the rule implementations to the test. You might be able to make that do what you need, but it's probably insufficient to handle the general case.

I'll take a look at these, and will probably be back with more questions.

I'd love to have a general system in place, but that appears a much bigger undertaking and I'm prepared for right now.

[Ulf Adams]

On Wed, Nov 29, 2017 at 2:55 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Wed, Nov 29, 2017 at 3:03 AM, Ulf Adams <ulf...@google.com> wrote:

On Tue, Nov 28, 2017 at 9:30 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

Update:

I've now got a Bazel which can run python coverage.  Anybody that's interested can see the relevant patch here:

https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage

The method for making the vendored coverage module available to the test stub seems like quite the hack: plumbing a path from the install path back through the BazelPythonConfiguration.  If there's a better way to do this (or one that would make merging this branch more palatable) please let me know.

It needs to be a dependency on a rule or target, for hermeticity.

Right.

My initial attempts to add "@bazel_tools//third_party/py/coverage" to the dependencies from within the PyTest implementation were a bit muddled.  My naive approach was to clone the existing RuleContext and replace it with one which has that target as an additional dependency here: https://github.com/bazelbuild/bazel/blob/91fb38e92ace6cf14ce5da6527d71320b4e3f3d2/src/main/java/com/google/devtools/build/lib/rules/python/PyTest.java#L40

The dependency needs to be statically declared in PyTestRule. When the rule is being evaluated it is too late - Bazel loads and analyzes all dependencies before evaluating the rule, and there's no mechanism to retry or to analyze on demand.

The main reason for that is performance - we analyze as much in parallel as possible using a thread pool. We can't just block the current thread - we'd risk running out of threads in the thread pool, and using the current thread to analyze dependencies serializes all the analysis. What we do elsewhere is to return a sentinel (or throw an exception) to indicate that the thread would block on X, and return from the current thing so we can reuse the current thread, and retry once X is done. However, this is not exposed to rule analysis, and there are significant performance implications of doing so as well as increased coding complexity.

[Hyrum Wright]

On Thu, Nov 30, 2017 at 3:42 AM, Ulf Adams <ulf...@google.com> wrote:

On Wed, Nov 29, 2017 at 2:55 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Wed, Nov 29, 2017 at 3:03 AM, Ulf Adams <ulf...@google.com> wrote:

On Tue, Nov 28, 2017 at 9:30 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

Update:

I've now got a Bazel which can run python coverage.  Anybody that's interested can see the relevant patch here:

https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage

The method for making the vendored coverage module available to the test stub seems like quite the hack: plumbing a path from the install path back through the BazelPythonConfiguration.  If there's a better way to do this (or one that would make merging this branch more palatable) please let me know.

It needs to be a dependency on a rule or target, for hermeticity.

Right.

My initial attempts to add "@bazel_tools//third_party/py/coverage" to the dependencies from within the PyTest implementation were a bit muddled.  My naive approach was to clone the existing RuleContext and replace it with one which has that target as an additional dependency here: https://github.com/bazelbuild/bazel/blob/91fb38e92ace6cf14ce5da6527d71320b4e3f3d2/src/main/java/com/google/devtools/build/lib/rules/python/PyTest.java#L40

The dependency needs to be statically declared in PyTestRule. When the rule is being evaluated it is too late - Bazel loads and analyzes all dependencies before evaluating the rule, and there's no mechanism to retry or to analyze on demand.

There is no class named PyTestRule.  I'm assuming you meant BazelPyTestRule: https://github.com/bazelbuild/bazel/blob/e222ba362c4c8c0cc0c76b308ee899e75098746a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPyTestRule.java

That seems like a fine place to inject the new dependency, but again the mechanics of doing so are a bit daunting to somebody unfamiliar with the Bazel codebase (and unable to find documentation about it).  For example, it looks like we need to override the existing "deps" attribute as part of the RuleContext.Builder, but we don't have a way to get the existing value before overwriting it.  We need that value so we can append to the list, rather than just substituting it.  I can't find any other instance in the Bazel codebase where an attribute with a label list is extended, so I'm not even sure that facility exists.

Again, any pointers would here would be useful.

[Hyrum Wright]

On Fri, Dec 1, 2017 at 11:54 AM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Thu, Nov 30, 2017 at 3:42 AM, Ulf Adams <ulf...@google.com> wrote:

On Wed, Nov 29, 2017 at 2:55 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

On Wed, Nov 29, 2017 at 3:03 AM, Ulf Adams <ulf...@google.com> wrote:

On Tue, Nov 28, 2017 at 9:30 PM, Hyrum Wright <hwr...@duolingo.com> wrote:

Update:

I've now got a Bazel which can run python coverage.  Anybody that's interested can see the relevant patch here:

https://github.com/bazelbuild/bazel/compare/master...hwright:py-coverage

The method for making the vendored coverage module available to the test stub seems like quite the hack: plumbing a path from the install path back through the BazelPythonConfiguration.  If there's a better way to do this (or one that would make merging this branch more palatable) please let me know.

It needs to be a dependency on a rule or target, for hermeticity.

Right.

My initial attempts to add "@bazel_tools//third_party/py/coverage" to the dependencies from within the PyTest implementation were a bit muddled.  My naive approach was to clone the existing RuleContext and replace it with one which has that target as an additional dependency here: https://github.com/bazelbuild/bazel/blob/91fb38e92ace6cf14ce5da6527d71320b4e3f3d2/src/main/java/com/google/devtools/build/lib/rules/python/PyTest.java#L40

The dependency needs to be statically declared in PyTestRule. When the rule is being evaluated it is too late - Bazel loads and analyzes all dependencies before evaluating the rule, and there's no mechanism to retry or to analyze on demand.

There is no class named PyTestRule.  I'm assuming you meant BazelPyTestRule: https://github.com/bazelbuild/bazel/blob/e222ba362c4c8c0cc0c76b308ee899e75098746a/src/main/java/com/google/devtools/build/lib/bazel/rules/python/BazelPyTestRule.java

That seems like a fine place to inject the new dependency, but again the mechanics of doing so are a bit daunting to somebody unfamiliar with the Bazel codebase (and unable to find documentation about it).  For example, it looks like we need to override the existing "deps" attribute as part of the RuleContext.Builder, but we don't have a way to get the existing value before overwriting it.  We need that value so we can append to the list, rather than just substituting it.  I can't find any other instance in the Bazel codebase where an attribute with a label list is extended, so I'm not even sure that facility exists.

Again, any pointers would here would be useful.

Any examples of how to dynamically insert items into the "deps" list of a build rule would also be useful.

Or even "we don't have that ability, this is going to be a giant yak shave", so I don't spend a ton of time chasing down something which doesn't exist.

-Hyrum

[Ulf Adams]

We usually add another attribute (say "$pycov") that points at the right target, and then merge that with what we get back from deps if coverage is enabled.

[Hyrum Wright]

Thank you for that pointer!  With this information, I've gotten the implicit dependency on the coverage module to work in https://github.com/hwright/bazel/commit/67af226eb963445d970385ea7567eb9c728426f9

As always, I'm open to suggestions on how that could be improved, but for right now, it works.

Earlier in this discussion, you mentioned that each test should output it's coverage information in a unique location.  I'm currently using COVERAGE_OUTPUT_FILE for that purpose, since it is unique for each python test.  As I work out how to combine those coverage results into a single output file, do you have any suggestions on where that combined output file should go?

-Hyrum

[Ulf Adams]

I was out for a long weekend and am now catching up on email.

The COVERAGE_OUTPUT_FILE is unique per-test, correct. However, if you have an integration test with multiple components (maybe multiple languages, maybe multiple binaries in the same language), then they can't all write to this file. Instead, they should write to a unique id under COVERAGE_DIR, and then we need merge all the files as a post-process at the end of the test execution. There's code in test-setup.sh to do that, except it can't handle multiple files right now.

 

-Hyrum

[joshuas...@verbsurgical.com]

On Monday, October 30, 2017 at 11:42:58 AM UTC-5, Hyrum Wright wrote:
> I'm interesting in having Bazel support code coverage for Python tests.
>
>
> From https://github.com/bazelbuild/bazel/issues/1118, it seems like the basic infrastructure is there.
>
>
> https://docs.google.com/document/d/1WRQTXQBV-m1_HDkAPvRSfGIrDW4pkI7ATPWtNdC8k9E/edit describes the ideal world of how coverage should work.  The doc looks to be a bit dated, and since Java coverage is already implemented it would seem that the basic infrastructure exists for running coverage.
>
>
>
> Python already has tooling for running coverage outside of Bazel: https://coverage.readthedocs.io/en/coverage-4.4.1/  It would be nice to be able to integrate that tooling into Bazel for use on Python tests.
>
>
> Any thoughts, updates, or pointers here?  Is supporting python coverage in Bazel a fool's errand, or actually feasible?
>
>
> -Hyrum

Has this functionality been implemented yet or any progress been made on addressing this issue since 2017? My team would be very interested in this functionality as we currently run coverage on every bit of our code base except for python tests because we haven't found a way to do so in Bazel. Please let me know. Thanks

-Josh

[Ulf Adams]

I filed https://github.com/bazelbuild/bazel/issues/10660 just now, which documents a way to collect coverage for python.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/82532b8f-2696-4525-838b-a670bec072f0%40googlegroups.com.