Calling scripts from bazelrc

[Ulrik Falklof]

Hi,

What would you think about adding support for calling scripts from bazelrc files?

E.g. by complementing:

import foo.bazelrc
try-import foo.bazelrc

with support also for:

execute foo.sh
try-execute foo.sh

The scripts could do anything. And any text it is printing on stdout could be imported and evaluated as bazelrc statements.

Example of scripts that could be referenced via execute or try-execute:

• Place output based on what local file systems are mounted:
  

if [ -d "/local" ]; then
  echo startup --output_user_root=/local/${USER}/.cache/bazel/
fi

• Setup --disk_cache with workaround for missing native max size support:
  

DISK_CACHE_LOCATION=/tmp/bazel-disk-cache
MAX_SIZE_GB=20
clear_cache_if_too_large.sh ${DISK_CACHE_LOCATION} ${MAX_SIZE_GB}
echo build --disk_cache=${DISK_CACHE_LOCATION}

• Start bazel-remote cache proxy on demand:

LOCAL_BAZEL_REMOTE_PORT=8080
IDLE_TIMEOUT_SECONDS=3600
# https://github.com/buchgr/bazel-remote can be started
# on-demand and automatically terminate after idle timeout.
start_local_bazel_remote_cache_proxy_if_not_already_running.sh ${LOCAL_BASEL_REMOTE_PORT} ${IDLE_TIMEOUT_SECONDS}
echo build --remote_cache=http://localhost:${LOCAL_BASEL_REMOTE_PORT}

• Enable --config=CI for jenkins, even when not having control of the actual bazel invocations:
  

if [ "$USER" == "jenkins" ]; then
  echo build --config=CI
fi

 

• Add USER and HOST headers/keywords:

echo build --bes_keywords=USER=$USER
echo build --bes_keywords=HOST=$HOST
echo build --remote_header=USER=$USER
echo build --remote_header=HOST=$HOST
echo build --remote_exec_header=USER=$USER
echo build --remote_exec_header=HOST=$HOST

 

• Enable remote execution depending on policy for the local host:

if should_host_use_remote_exeution.sh $HOST; then
  echo build --remote_execution=grpc://machine.com:1234
  echo build --remote_instance_name=foo
  echo build --remote_execution_priority=8
fi

 

• Secondary cache if the primary is down:
  

PRIMARY_REMOTE_CACHE=http://10.0.0.5:8080
SECONDARY_REMOTE_CACHE=http://10.0.0.6:8080
if is_answering_http_request.sh $PRIMARY_REMOTE_CACHE; then
  echo build --remote_cache=${PRIMARY_REMOTE_CACHE}
else
  echo build --remote_cache=${SECONDARY_REMOTE_CACHE}
fi
# An alternative could be to probe both in parallel

# and pick the one that answers first.

 

Motivation, why executing scripts from bazelrc files, instead of as wrapper scripts around bazel?

• Allows fine granular snippets maintained by different people, instead of monolith wrapper scripts.
• Precedence the same way as import statements, allows natural and flexible precedence between options from different sources. E.g. command line parameters can override.
• Works well in combination with --announce_rc.

What do you think?

Regards,

Ulrik Falklöf

[mich...@google.com]

Sounds like an interesting idea if there's enough demand and we can sort out all of the corner cases and confirm there's no viable alternative. Some of these seem like they might be ok features to bake into bazel, or instead add some sort of plugin framework to support?

Off the top of my head, things we need to consider for the script support would be...

• Making this opt-in only - running arbitrary un-sandboxed code will not fly in many environments
• How to prevent non-determinism and slowness from ruining the user experience
• ie, options changing in a way that cause cache flushes or server restarts

[Austin Schuh]

You can make this work today if you want. /usr/bin/bazel is a shell
script which looks for //tools/bazel and calls it if it exists. You
could call the shell commands there, and pass those on to
/usr/bin/bazel-real, or even better, pass them to a version of bazel
tied to the repo. This is essentially what bazelisk is trying to
solve. I would suspect that the solution would fit better in a tool
like bazelisk than in bazel core.

Austin

> --
> You received this message because you are subscribed to the Google Groups "bazel-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/df90c1aa-ad2b-4e41-b941-e431ed009f14%40googlegroups.com.

[Tony Aiuto]

+1 to Austin's idea.  That puts all the non-determinism made possible

by local script execution in the tool which calls bazel rather than bazel itself.

It also makes the functionality much easier to test, because we can do it outside

of the bazel command and rc parsing code.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CABsbf%3DF-xdMiQdXV7tyQnDsn4s6O9k8buH97dXrku5hQ7k%3DFaQ%40mail.gmail.com.

[Ulrik Falklöf]

Thanks for feedback Michajlo!

Regarding non-determinism:

• I’m thinking that If a script would change flags or input files, in a way that would affect the final artifacts, that would be detected by the bazel server process and sandbox. Just like such manual changes are detected. Perhaps I’m missing something. What do you have in mind Michajlo?

Regarding slowness:

• Yes, the scripts could occasionally generate flags that causes server restart. The person writing a script would have to consider that. For most use cases I can think of, such flags would change seldom, and would then not be a problem in practice.

/Ulrik

[Ulrik Falklöf]

Hi Tony and Austin,

The idea is not only about executing scripts, but also generating bazel options from those scripts in powerful and flexible way.

Example of .sh scripts generating bazelrc textual fragments:

.bazelrc contains:

import aaa.bazelrc
execute bbb.sh
import some/other/component/ccc.bazelrc

some/other/component/ccc.bazelrc contains:

execute ddd.sh
import eee.bazelrc

Benefits:

• Different organizations could maintain bbb.sh and ddd.sh separately in different components, and hook in additional ones at will via existing bazelrc file structure.
  
• --announce_rc could tell exactly from which script, or bazelrc file, flags are coming.
• Flexible precedence based on in which order user places execute/import statements.
• E.g. options generated from bbb.sh would have lower precedence than options from ccc.bazelrc, but higher than from aaa.bazelrc
• Scripts could generate bazelrc options that takes effect only for certain --config=

With your suggestion, how would options generated from scripts outside bazel, be passed to bazel? I assume that simply passing them as command line options (e.g. to bazel-real) would lose the benefits above, or do you see a way to still fulfill them?

Thanks,

Ulrik

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CACpMnusq1M%2BwKUEGOHbBvdLw-y9nq%3D7Zv2BYhTk6h4p7wsN-jg%40mail.gmail.com.

[Austin Schuh]

Sure, but those are really the same problem statement. Your example
is equivalent to, or could be made equivalent to:

bazelisk or tools/bazel script:
bbb.sh > bbb.bazelrc
ddd.sh > ddd.bazelrc
exec bazel-real

.bazelrc:
import aaa.bazelrc
import bbb.bazelrc
import some/other/component/cc.bazelrc

import some/other/component/cc.bazelrc:
import ddd.bazelrc
import eee.bazelrc

There's some automation that could be added around this to handle the
discovery pretty easily. Maybe ".bazelisk.sh" files in the repo get
automatically executed? Something else? A folder which has the list
of scripts?

It wouldn't be hard to also have bbb.sh and ddd.sh return a list of
arguments that bazelisk adds to the command line.

The trick with a tool is always to find the core set of features which
is flexible enough to do work, but simple.

Austin

[Tony Aiuto]

I'm with Austin on this. You could do all the things you mentioned without needing Bazel to run the scripts.

[Michajlo Matijkiw]

On Mon, Apr 13, 2020 at 1:08 AM Ulrik Falklöf <ulrik....@gmail.com> wrote:

Thanks for feedback Michajlo!

Regarding non-determinism:

• I’m thinking that If a script would change flags or input files, in a way that would affect the final artifacts, that would be detected by the bazel server process and sandbox. Just like such manual changes are detected. Perhaps I’m missing something. What do you have in mind Michajlo?

I meant that the non-determinism could cause slowness (noted below) and otherwise confusing or unexpected results that could be difficult to attribute to the scripts.

[Ulrik Falklöf]

If generating bazelrc files early on (e.g. via bazelisk) and hard coding the explicit paths to import in other bazelrc files, then how would you prevent race conditions when concurrent builds generates the same bazelrc file with potentially different content?

/Ulrik