How to test properties of Bazel rules?

[Markus Teufelberger]

Hi all,

I was wondering if there are some already existing methods to test some guarantees that are explicitly or implicitly encoded in Bazel rules.

An example for an implicit guarantee would be the determinism of an action:
"If a rule with the exact same inputs and under the same conditions is ran multiple times, the output will be bitwise identical."
(This is currently assumed to be true by Bazel, otherwise caching would not work)

An example for an explicit guarantee would be some potentially custom properties of rules, e.g. https://bazel.build/versions/master/docs/be/python.html#py_library.srcs_version:
"If the srcs_version string is set, the described code will work with the described versions of Python."

Both examples could be testable (e.g. build everything twice and hash the outputs; run a linter or something similar to check if the Python version is supported) but currently don't seem to actually be tested by default. Test rules seem more concerned with the inputs and outputs of rules, not rules themselves.

Are there some "meta_test" rules (rules that test that rules and their inputs behave as they claim to do) hidden away somewhere or not yet released, or is this out of scope and developers are the only ones responsible to make sure that their code is deterministic to begin with or to check that the srcs_version flag is set correctly when writing their BUILD files? I think it would be very helpful in some cases to be able to find non-determinism in some compile steps or to be able to check at the time of writing a rule if it actually holds for various toolchains.

Looking forward to your thoughts,
Markus

[ittai zeidman]

Hi Markus,
I invite you to take a look at rules_scala and specifically at test_run.sh which does many of the things you say.
We're not really happy with it because it's currently not being run by Bazel but it at least gives us high confidence when making changes.
Also,
There are tests in the Bazel codebase which create a scratch workspace with code, run bazel and make assertions about correctness. AFK but search for java sandwich for example.