Action environments: action_env, use_default_shell_env, and incompatible_strict_action_env

[Geoffrey Martin-Noble]

Hey Bazel folks,

I've been fielding a user patch to add use_default_shell_env to a custom Starlark rule using ctx.actions.run (https://reviews.llvm.org/D109873). In trying to evaluate the merits of that and whether it's the right fix for the user issue, I found a real dearth of documentation on what action environments look like. I ended up with a cobbled-together understanding based on various Bazel GitHub issues. I think it would be helpful to provide such documentation, which would discuss what environment actions run in, how use_default_shell_env affects that (what is the environment if *not* the default one??), how incompatible_strict_action_env affects that and any known issues with all this (which it appears there are quite a few of). Perhaps there is such documentation already, in which case I think it should probably be linked from https://docs.bazel.build/versions/main/skylark/lib/actions.html#run.use_default_shell_env, https://docs.bazel.build/versions/main/command-line-reference.html#flag--action_env, and https://docs.bazel.build/versions/main/command-line-reference.html#flag--incompatible_strict_action_env. I would also expect it to turn up from searches like "bazel action environment".

Thanks,

Geoffrey

[Keith Smiley]

+1 for docs on this. Note that it also plays strangely with Darwin builds since DEVELOPER_DIR and friends aren't correctly propagated when use_default_shell_env is set. https://github.com/bazelbuild/bazel/issues/12049

--
Keith Smiley

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAKeN2ed3CE6SZFR7N97fyL7yMc6qNeHb7OEPaQj0JJFjtqfjZQ%40mail.gmail.com.

[Marcel Kirsche]

Any update on this? I'm also really confused how those settings play together. Could you summarize your findings and current understanding of it Geoffrey?

--

Marcel

[Fabian Meumertzheim]

https://github.com/bazelbuild/bazel/issues/12049 is fixed in Bazel 7 and the fix is in the process of being backported to 6.4.0, where it will be available with the flag `--incompatible_merge_fixed_and_default_shell_env`.

Based on my understanding of "--action_env" and friends, I would suggest the following:

* Generally only use `--action_env` to pass in relatively generic environment variables that are required to run binaries on the executing machine, e.g. PATH, LD_LIBRARY_PATH. Since this applies to many rules, more widespread use of `--action_env` can result in an increase in cache misses.

* When writing a custom rule, supply `use_default_shell_env = True` to `ctx.actions.{run,run_shell}` if there is a chance that binaries run by your action may need these variables. For example, if your ruleset relies on essentially statically built and hermetically fetched SDKs, don't set this to `True` as the action shouldn't care about `PATH`. If your rule runs more general tools configurable by end users, you should probably set this to `True`. Keep in mind that in Bazel versions before the fix `use_default_shell_env = True` means that you can't also use `env`.

* `--incompatible_strict_action_env` is essentially equivalent to some platform-specific `--action_env` definitions for `PATH` and similar variables. No magic here, just default values that work somewhat well but don't in special situations. They are better than nothing (because `PATH` is a fixed value which greatly helps caching), but you won't lose much if you set the variables yourself.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/1eea8bc5-1d6d-4743-b26f-1f33fb42da97n%40googlegroups.com.

[Marcel Kirsche]

Thanks for the clarification!

Based on my observations, if I don't supply `use_default_shell_env`, I cannot alter the env with `--action_env`. However, it seems like the whole environment from the host is accessible! If I do export FOO=BAR, it's visible to the action.

This seems to be the case at least for bazel test & run, don't know about intermediate build actions. Can you confirm that, or did I do something wrong?

[Marcel Kirsche]

Also, I fear that this environment from the host is not captured inside the action key, if not explicitly specified. Can someone also clarify which part of the env under which circumstances is registered as part of the action key for the purpose of caching?

[Fabian Meumertzheim]

If you `bazel run`, the entire host environment will be visible to the process. This is how you run non-hermetic actions with Bazel, so that's fine.

If you `bazel test`, only the `--action_env` and `--test_env` should be seen, plus some other variables (such as PATH). All of them should be part of the cache key.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6e3c4cc0-cb83-44e0-bba5-5915716dc0e2n%40googlegroups.com.

[Marcel Kirsche]

That's interesting. I did some tests and it seems like there is a difference between run, test and build actions. These are my observations:

For Bazel run (terminating run actions):

• The whole host environment is accessible from the binary that shall be ran

• –action_env is always taken into account

For Bazel test (terminating test actions):

• PATH is always taken from the host and accessible by actions

• –action_env is always taken into account

• USER is set based on the current user running Bazel (simply changing the env variable on the host [e.g. export USER=FOO] without switching the user does not change this env var in Bazel.  However, it affects the path to the local cache /home/<REAL_USER>/.cache/bazel/_bazel_<USER_ENV>/……). It can be changed with –action_env though.

For bazel build (all build actions):

• By default, NO environment variables are available to the action, this cannot be changed with –action_env

• When setting use_default_shell_env  = True on the action

  • PATH will automatically become exposed to the action

  • Env set by –action_env / –host_action_env will be exposed to the action

What's a bit strange is that, for testing, a change (e.g export PATH=$PATH:/foo/bar) in the hosts PATH environment variable can be observed within the testing action, however it does not trigger a change in the action key.

[Marcel Kirsche]

In addition to that, running bazel test under a different user exposes different values for the USER variable to test actions. However the action key seems to be stable between different users.