Bazel and pytest

[Hyrum Wright]

I'm trying to run tests of external packages which use pytest.  It appears they need to use pytest as a test runner, but Bazel just executes the test binary directly.  Is there a way to tell Bazel to use pytest as a test runner, rather than direct execution?

Relatedly, is there a simple way to use pytest as the test runner for first-party tests?

-Hyrum

[rodr...@google.com]

On Wednesday, November 8, 2017 at 8:23:39 PM UTC+1, Hyrum Wright wrote:
> I'm trying to run tests of external packages which use pytest.  It appears they need to use pytest as a test runner, but Bazel just executes the test binary directly.  Is there a way to tell Bazel to use pytest as a test runner, rather than direct execution?

I'm not sure if this is the right way, but you can run pytest tests by including a main file that runs them:

cat `which pytest` > test.py

py_test(
name = "test",
srcs = [
"test.py",
"test_lib.py",
],
deps = [
":lib",
requirement("pytest"),
],
)

where pytest is pulled in through rules_python's pip_import, :lib is the module to test, and test_lib.py contains the tests.

To make it easier, you could create a macro that uses a genrule to create the main file, and then uses a py_test() rule to run it.

[Doug Greiman]

I append the following to source files where I use pytest:

----- foo_test.py -----

if __name__ == '__main__':

    pytest.main([__file__])

---------------------------

This plus `chmod +x` lets one directly execute the file, or turn it into a Bazel py_test() as described by rodrigoq

[Doug Greiman]

pytest's philosophy of implicitly finding tests by scouring the file system is pretty much directly in opposition to Bazel's philosophy of correctness through explicitness, so any deeper integration would likely be brittle, or unsatisfying, or both.

It's probably possible though.  You would write a plugin to pytest that finds tests and generates a .bzl file containing generated py_test() targets.  You would either check that .bzl file into your source tree, and remember to rerun the test-finding script every so often, or leave the .bzl file as a generated artifact and have a workspace rule that runs it before every build.

[vector...@outlook.com]

That's pretty good, but it won't tell bazel if the tests pass or fail. You need to add sys.exit to pass the exit code back to bazel. That being said, I'm having an issue with pytest and bazel getting confused by the module names.

This is my pytest log.

exec ${PAGER:-/usr/bin/less} "$0" || exit 1
-----------------------------------------------------------------------------
============================= test session starts ==============================
platform linux -- Python 3.5.2, pytest-3.5.0, py-1.5.3, pluggy-0.6.0
rootdir: /home/vagrant/.cache/bazel/_bazel_vagrant/14760d7c33ea3e90d48de59613ccfeab/bazel-sandbox/7080476036616459591/execroot/vibes/bazel-out/k8-py3-fastbuild/bin/graphql/test_schema.runfiles/vibes, inifile:
collected 0 items / 1 errors

==================================== ERRORS ====================================
___________________ ERROR collecting graphql/test_schema.py ____________________
ImportError while importing test module '/home/vagrant/.cache/bazel/_bazel_vagrant/14760d7c33ea3e90d48de59613ccfeab/bazel-sandbox/7080476036616459591/execroot/vibes/bazel-out/k8-py3-fastbuild/bin/graphql/test_schema.runfiles/vibes/graphql/test_schema.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
ImportError: No module named 'graphql.test_schema'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.46 seconds ============================

[Clint Harrison]

This is probably an issue because of the `__init__.py` created by Bazel in your test directory. Try adding `legacy_create_init = False` on your `py_test` rule.