Changes to how Bazel accesses the internal tools under //tools

[Lukács T. Berki]

Hello there,

After c74ee3784d3fd3a722f92c39805e57b4353a4509, Bazel will use tools embedded in the Bazel binary itself and not under the //tools directory. This is nice because no one has to worry about providing a //tools directory to Bazel anymore because it's embedded in the self-extracting Bazel binary. 

The slight downside is that extracting the Bazel binary takes significantly longer, because there is more stuff in it (we have been paying that price since about two weeks ago). This hits developers the hardest, because everyone else changes their Bazel version much less frequently, and this only happens once per Bazel version. If this proves to be a problem, we can create a "Bazel development" binary that does not contain these embedded tools.

Skylark sources (i.e. things under tools/build_defs and tools/build_rules) are still fetched from the main repository. John is working on fetching them from the Bazel binary, too.

If, for some reason (for example, you are tweaking one of these tools) you want Bazel to use a different version than the one embedded in it, you can add

local_repository(name = "bazel_tools", path=<path to the Bazel source tree>)

to your WORKSPACE file.

If you find mysterious problems with accessing tools in Bazel, write me (and bazel-discuss!) -- this is quite an intrusive change and I can imagine that there are cases I haven't considered.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Maximillianstr. 11-15 | 80539 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[Erik Kuefler]

Will this have any effect on people who are still building Bazel from source rather than using a binary release?

[Lukács T. Berki]

Building Bazel from source will also get you a Bazel version that uses the tools embedded into itself, but it should Just Work (TM).

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/3abf2880-ee5f-4069-a385-7f3a08d98247%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Erik Kuefler]

Cool. Just thought of one other thing - we rely on workspace configurations checked into tools/bazel.rc in our local repo. Is this going to cause a problem there or will Bazel still respect a local tools directory?

[Brian Silverman]

Is there any way to use a non-embedded version of only a few of the tools? For example, I have a customized //tools/test:test-setup.sh I want to keep, but it would be nice to not have copies of all the other files in //tools too.

Currently, it's easy to keep using a custom CROSSTOOL with --crosstool_top=//tools/cpp:toolchain. Would adding similar flags for some of the other major components in //tools make sense? Or maybe some kind of "local_overlay_repository" which can use WORKSPACE-relative paths?

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAOu%2B0LUbCi9OZy4SoypF%3D7biwMJvWW5MywXiO0cyE_2X5BLasQ%40mail.gmail.com.

[Lukács T. Berki]

On Tue, Oct 13, 2015 at 11:10 PM, Brian Silverman <bsilve...@gmail.com> wrote:

Is there any way to use a non-embedded version of only a few of the tools? For example, I have a customized //tools/test:test-setup.sh I want to keep, but it would be nice to not have copies of all the other files in //tools too. 

No, unfortunately, it's currently an all-or-nothing affair. Ulf brought up the idea of having different repositories for different languages (i.e. direct subdirectories if //tools), but I'd first like to sort all the issues out with the migration to a separate tools repository. It was a fiddly migration and I'm not very confident yet that I know of all the issues.

Currently, it's easy to keep using a custom CROSSTOOL with --crosstool_top=//tools/cpp:toolchain. Would adding similar flags for some of the other major components in //tools make sense? Or maybe some kind of "local_overlay_repository" which can use WORKSPACE-relative paths?

By local_overlay_repository you mean some rule that would replace only part of another repository (in this case, bazel_tools)? TBH, that sounds a bit hackish  to me. Separate repositories for separate languages is the way to go.

Adding more *_top flags... well, that's certainly a solution and we'll probably end up being there, but I'd rather put it off as long as possible, because having this many command line options is not fun.

 

[Alex Humesky]

On Tue, Oct 13, 2015 at 4:34 PM Erik Kuefler <ekue...@gmail.com> wrote:

Cool. Just thought of one other thing - we rely on workspace configurations checked into tools/bazel.rc in our local repo. Is this going to cause a problem there or will Bazel still respect a local tools directory?

Looks like this is just a path, so it should still work:

https://github.com/bazelbuild/bazel/blob/15e9a506118b4067655e5e7f72ff852be5b797ca/src/main/cpp/option_processor.cc#L157

 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CANK5_0pzMGJ%3Dt_p_o2GLUL_r%2Bep-YdqRjRREqw%3DtWbhFP%2B0%3DYQ%40mail.gmail.com.

[Brian Silverman]

I went to actually try adding local_repository(name='bazel_tools', path='.') to my WORKSPACE and ran into a problem: it needs an absolute path. I tried changing that, and then ran into the problem of needing a lot more packages copied into both the tools directory in my repository and other places (like third_party) because Bazel no longer looks for those in the package_path. It seems like there isn't a good way any more to use a tools directory checked in to the same repository as the code. Am I missing something?

Adding just one would fix the problem for the only one I actually have a use case for changing: --test_runtime (defaults to Constants.TOOLS_REPOSITORY + "//tools/test:runtime"). If I write that patch, would you be willing to merge it?

 

[Lukács T. Berki]

On Sat, Oct 17, 2015 at 9:04 PM, Brian Silverman <bsilve...@gmail.com> wrote:

I went to actually try adding local_repository(name='bazel_tools', path='.') to my WORKSPACE and ran into a problem: it needs an absolute path. I tried changing that, and then ran into the problem of needing a lot more packages copied into both the tools directory in my repository and other places (like third_party) because Bazel no longer looks for those in the package_path. It seems like there isn't a good way any more to use a tools directory checked in to the same repository as the code. Am I missing something?

No, not really. We are planning to separate the single tools repository into multiple per-language ones, but I am still dealing with the fallout of moving things to that single repository, so it will take a while.

 

On Wed, Oct 14, 2015 at 4:30 AM, Lukács T. Berki <lbe...@google.com> wrote:

On Tue, Oct 13, 2015 at 11:10 PM, Brian Silverman <bsilve...@gmail.com> wrote:

Is there any way to use a non-embedded version of only a few of the tools? For example, I have a customized //tools/test:test-setup.sh I want to keep, but it would be nice to not have copies of all the other files in //tools too. 

No, unfortunately, it's currently an all-or-nothing affair. Ulf brought up the idea of having different repositories for different languages (i.e. direct subdirectories if //tools), but I'd first like to sort all the issues out with the migration to a separate tools repository. It was a fiddly migration and I'm not very confident yet that I know of all the issues.

Currently, it's easy to keep using a custom CROSSTOOL with --crosstool_top=//tools/cpp:toolchain. Would adding similar flags for some of the other major components in //tools make sense? Or maybe some kind of "local_overlay_repository" which can use WORKSPACE-relative paths?

By local_overlay_repository you mean some rule that would replace only part of another repository (in this case, bazel_tools)? TBH, that sounds a bit hackish  to me. Separate repositories for separate languages is the way to go.

Adding more *_top flags... well, that's certainly a solution and we'll probably end up being there, but I'd rather put it off as long as possible, because having this many command line options is not fun.

Adding just one would fix the problem for the only one I actually have a use case for changing: --test_runtime (defaults to Constants.TOOLS_REPOSITORY + "//tools/test:runtime"). If I write that patch, would you be willing to merge it?

Sigh -- this is Yet Another --*_top command line option that I don't really like, but I do acknowledge the necessity of, so I would grudgingly merge the change. I'd like a second opinion. Ulf?

[Ross Light]

Brian: there is a bug from a while back (https://github.com/bazelbuild/bazel/issues/269) about supporting relative paths in local_repository.  I'm not sure how far Kristina got in addressing it.

[Brian Silverman]

OK. I sent out https://bazel-review.googlesource.com/2150 with you guys as reviewers. Thanks for being so receptive to changes on stuff like this. :)

[Lukács T. Berki]

Out of curiosity, what do you have in your custom test-setup.sh? Of all the ancillary tools Bazel uses, I thought this was the one that requires the least amount of tweaking.

[Brian Silverman]

The main things it does are set up a separate network namespace and support running the test on another machine over SSH. I need the separate network namespace because I have third-party tests which otherwise become flaky because they all bind to the same ports.

I also have it re-mount the source tree and /tmp read-only to catch tests which try to write there (had issues in the past with third-party tests being flaky when run in parallel because they did that...). It also sets GTEST_SHARD_INDEX and GTEST_TOTAL_SHARDS based on the Bazel environment variables so sharding works with googletest.

[Han-Wen Nienhuys]

On Tue, Oct 20, 2015 at 7:48 PM, Brian Silverman <bsilve...@gmail.com> wrote:
> The main things it does are set up a separate network namespace and support
> running the test on another machine over SSH.

Is this for use with --run_under, or is this a separate mechanism?

> I need the separate network
> namespace because I have third-party tests which otherwise become flaky
> because they all bind to the same ports.

We had a contribution which sets up network NS's for sandboxed
execution. Is this a request to have this wired up with some option or
testtag?

> I also have it re-mount the source tree and /tmp read-only to catch tests
> which try to write there (had issues in the past with third-party tests
> being flaky when run in parallel because they did that...). It also sets

isn't this fixed with sandboxing?

> GTEST_SHARD_INDEX and GTEST_TOTAL_SHARDS based on the Bazel environment
> variables so sharding works with googletest.

We should fix Bazel so gtest sharding works out of the box.

--
Han-Wen Nienhuys
Google Munich
han...@google.com

[Brian Silverman]

Actually, it looks like maybe it's reasonable to move away from my custom test-setup.sh because there are better places for all of the functionality.

On Tue, Oct 20, 2015 at 4:54 PM, Han-Wen Nienhuys <han...@google.com> wrote:

On Tue, Oct 20, 2015 at 7:48 PM, Brian Silverman <bsilve...@gmail.com> wrote:
> The main things it does are set up a separate network namespace and support
> running the test on another machine over SSH.

Is this for use with --run_under, or is this a separate mechanism?

What I have is separate. However, now that you mention it, I'm not sure why it doesn't use --run_under instead...

> I need the separate network
> namespace because I have third-party tests which otherwise become flaky
> because they all bind to the same ports.

We had a contribution which sets up network NS's for sandboxed
execution. Is this a request to have this wired up with some option or
testtag?

Having that (including a separate IPC namespace too) would obviate the need for that part of it. Did we ever decide which way that should default?

> I also have it re-mount the source tree and /tmp read-only to catch tests
> which try to write there (had issues in the past with third-party tests
> being flaky when run in parallel because they did that...). It also sets

isn't this fixed with sandboxing?

On second thought, I think sandboxing does cover everything for this. I have it mounting various folders read-only to catch tests trying to write there, but relying on sandboxing to handle that also works. Is it reasonable to assume that tests will always run in the sandbox for the foreseeable future?

> GTEST_SHARD_INDEX and GTEST_TOTAL_SHARDS based on the Bazel environment
> variables so sharding works with googletest.

We should fix Bazel so gtest sharding works out of the box.

I'd be happy to submit a change to do that. Is setting the environment variables from test-setup.sh or Java preferred?

[Han-Wen Nienhuys]

On Wed, Oct 21, 2015 at 1:36 AM, Brian Silverman <bsilve...@gmail.com> wrote:
> Actually, it looks like maybe it's reasonable to move away from my custom
> test-setup.sh because there are better places for all of the functionality.

great!

>> > I need the separate network
>> > namespace because I have third-party tests which otherwise become flaky
>> > because they all bind to the same ports.
>>
>> We had a contribution which sets up network NS's for sandboxed
>> execution. Is this a request to have this wired up with some option or
>> testtag?
>
>
> Having that (including a separate IPC namespace too) would obviate the need
> for that part of it. Did we ever decide which way that should default?

I think the default should be separate namespace. Inside google, using
the network is a perennial source of flakiness.

>> > I also have it re-mount the source tree and /tmp read-only to catch
>> > tests
>> > which try to write there (had issues in the past with third-party tests
>> > being flaky when run in parallel because they did that...). It also sets
>>
>> isn't this fixed with sandboxing?
>
>
> On second thought, I think sandboxing does cover everything for this. I have
> it mounting various folders read-only to catch tests trying to write there,
> but relying on sandboxing to handle that also works. Is it reasonable to
> assume that tests will always run in the sandbox for the foreseeable future?

Yes, I think so.

>> > GTEST_SHARD_INDEX and GTEST_TOTAL_SHARDS based on the Bazel environment
>> > variables so sharding works with googletest.
>>
>> We should fix Bazel so gtest sharding works out of the box.
>
>
> I'd be happy to submit a change to do that. Is setting the environment
> variables from test-setup.sh or Java preferred?

the shell script. Let's keep the java as simple as possible.

[Lukács T. Berki]

The change looks good now, but if you don't actually need it, let's not merge it, because the less knobs we have, the better.

--run_under should work.

+1. IIRC Bazel already sets the TEST_TOTAL_SHARDS, TEST_SHARD_INDEX and other environment variables that are needed for sharding, except that they are not prefixed with "G".