Generating test data at test time

[Giovanni Funchal]

Hi,

I currently generate test data, which is used only by my tests, with a simple genrule that runs a command. However the genrule runs at build time despite the fact it has testonly=True, and that causes some problems for me, because the test data is only really needed when actually running tests.

Is it possible to have a custom rule which runs at test time to generate the test data? I had a look at actions.run_shell wrapped in a rule with test = True, but it does not seem to quite do what I want, in particular I can't figure out how to declare "outputs" of a test rule. A minimal example would be appreciated.

Thanks,

Gio

[Alex Humesky]

The testonly attribute only prevents non-test targets from depending on that target (e.g., so that you don't accidentally get test data in your release binary). It doesn't change how or when the target gets built.

It's not totally clear to me what the problem is though. It seems pretty natural to generate test data in a genrule, or some other rule separate from the test target, so that the test data could be shared and cached among different test targets. Do you mean that when you do a "bazel build", you don't want the test data to be built, because that data is only needed when you do a "bazel test" (i.e., when the test binary is actually run)?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/5b737497-2c87-4fd6-a7a7-2a05709add57n%40googlegroups.com.

[Giovanni Funchal]

> Do you mean that when you do a "bazel build", you don't want the test data to be built, because that data is only needed when you do a "bazel test" (i.e., when the test binary is actually run)?

Yes, exactly. I want to run a custom command just before the test binary is run. I wonder if perhaps a way to do that could be to wrap the test rule in another rule.

Thanks,

Gio

[Alex Humesky]

Gotcha, so there isn't really a straightforward way to do this, because "bazel test" is "bazel build" + "run this binary that is a test" in one command. Your build files tell bazel that this binary needs that data, so bazel will build that data. Building the data only for the test command also introduces other surprising behavior, for example if there's some build error in data generation, then a "bazel build" of the test target would succeed, but a "bazel test" of the same target would fail to build.

All that said, I think you can get this behavior by using a select to select the data dependency only when the command is a test command, and differentiating the build and test commands by adding a --define to your workspace's .bazelrc file:

.bazelrc:

test --define=is_test_command=true

BUILD:

config_setting(
  name = "is_test_command",
  values = {"define": "is_test_command=true"},
)

sh_test(
  name = "test",
  srcs = ["test.sh"],
  data = select({
    ":is_test_command": ["data"],
    "//conditions:default": [],
  }),
)

genrule(
  name = "gen_data",
  outs = ["data"],
  cmd = "echo generating data ; echo generated data > $@",
  testonly = True,
  tags = ["manual"], # exclude from target patterns
)

test.sh:

echo test data is:
cat data

$ bazel clean
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel build test
INFO: Analyzed target //:test (46 packages loaded, 499 targets configured).
INFO: Found 1 target...
Target //:test up-to-date:
  bazel-bin/test
INFO: Elapsed time: 0.516s, Critical Path: 0.00s
INFO: 4 processes: 4 internal.
INFO: Build completed successfully, 4 total actions

$ ls bazel-bin
test  test.runfiles  test.runfiles_manifest

$ bazel clean
INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

$ bazel test test --test_output=streamed
WARNING: Streamed test output requested. All tests will be run locally, without sharding, one at a time
INFO: Analyzed target //:test (47 packages loaded, 502 targets configured).
INFO: Found 1 test target...
INFO: From Executing genrule //:gen_data:
generating data
test data is:
generated data
Target //:test up-to-date:
  bazel-bin/test
INFO: Elapsed time: 0.624s, Critical Path: 0.15s
INFO: 6 processes: 3 internal, 3 linux-sandbox.
INFO: Build completed successfully, 6 total actions
//:test                                                                  PASSED in 0.0s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 6 total actions

$ ls bazel-bin
data  test  test.runfiles  test.runfiles_manifest

$ bazel run test
INFO: Build option --define has changed, discarding analysis cache.
INFO: Analyzed target //:test (0 packages loaded, 499 targets configured).
INFO: Found 1 target...
Target //:test up-to-date:
  bazel-bin/test
INFO: Elapsed time: 0.217s, Critical Path: 0.02s
INFO: 3 processes: 3 internal.
INFO: Build completed successfully, 3 total actions
INFO: Build completed successfully, 3 total actions
exec ${PAGER:-/usr/bin/less} "$0" || exit 1
Executing tests from //:test
-----------------------------------------------------------------------------
test data is:
cat: data: No such file or directory

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/199db1f0-b31a-41c9-9ac1-a20f2194b05an%40googlegroups.com.

[Giovanni Funchal]

Thank you! This worked great for my use case.

Thanks,

Gio