Custom TestRunner implementation

[James Bartlett]

Hi all,
I was wondering if there's a way to implement our own custom test runner in starlark.

Our use case is to run all of our tests (py_tests, cc_tests, sh_tests, etc) in a custom test runner that setups a QEMU environment. I'm aware of the `--run_under` flag. However, I'm hoping there's something more sophisticated than that, so that I can use different test runners for different sets of rules, configurations, etc.

[James Bartlett]

I suppose another way to ask this question is: Is there a way to write custom execution strategies?

[Alex Eagle]

I've never heard of someone adding a custom execution strategy, and I think that's because it's not typically done that way. Bazel treats tests as opaque binaries. I think you can model this differently, rather than something outside of the spawn strategy that causes the test runner process to be in a QEMU environment, you can have Bazel spawn the test runner in the normal way, and then the test runner itself is responsible for creating the qemu environment and handing off to a subprocess that runs inside the emulator. If you want different environments for a single rule, you could use a macro so you get a pair of `foo_native_test` and `foo_emulated_test`, the latter using `args` so that the test runner process knows which way to execute the tests.

Does that help?

-Alex

[Carlo Contavalli]

In our environment we run tests in qemu. But the same test, often, we want to run in different qemu environments (eg: different kernel, different OS, different CPU, ...).

And at times we need multiple qemus to run for a test (eg, moral equivalent of one qemu running a service, another qemu trying to exercise it).

The approach we used is to create a set of rules that allow to define a qemu environment, and a set of targets (or preparation steps) to run in that environment.

The qemu rules can then be used as `bazel test`, or `bazel run`. Collect logs and outputs in a format that's bazel compatible, so BES endpoints, pass/fail, all work correctly.

The rules are open source, defined in the directory here:

https://github.com/enfabrica/enkit/blob/master/bazel/linux/qemu.bzl

Unfortunately we don't have much documentation (yet) other than what is in the .bzl files, and using the rules requires some buy in in terms of having a kernel/rootfs available in the WORKSPACE.

Ramping up on our rules may be a non trivial cost as is, but they are fairly complete and generic. May serve as inspiration to create your own (or help grow ours, PRs welcome).

Here's an example of what a target looks like:

```

py_binary(

    name = "hello-world"

    ...

)

vm_bundle(
    name = "hello-world-testing",

    run = ":hello-world",
    run_cmds = [
        "export ENVIRONMENT=testing",
    ],

    # can have pre-run and post-run steps or checks.
  )

qemu_test(
    name = "hello-world-x86-64",
    kernel_image = "@minimal-latest-kernel//:image",
    run = [  # one or more binary targets or vm_bundle targets
        ":hello-world-testing",
    ],
    setup = COMMON_TEST_SETUP + [ # arbitrary executable or bundle steps to run before the target.
        "//vm/setup:bring-up-db",
        "//vm/setup:create-users",
    ],

    # can define tear down steps, and things to check after the VM runs.
)

```

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/9ac02fca-7c6f-41f4-9156-37871326e04cn%40googlegroups.com.

[Nick Kirkby]

Hi James. I've not yet defined custom rules but I am making use of qemu to run tests under bazel with the `--run_under` flag. I am using platforms and the `--test_tag_filters` option to select tests based on environment. See examples below.

https://github.com/driftregion/iso14229/blob/0.7.0/test_all.sh

https://github.com/driftregion/iso14229/blob/0.7.0/.bazelrc

Does this help?