Bazel flag for using as-specified RBE output path semantics?

[Lukács T. Berki]

Hey Chi,

As a followup to https://github.com/bazelbuild/bazel/issues/13188, it looks like that there are numerous breakages with Bazel that result from the fact that Bazel specifies the paths of action outputs relative to the root directory of the action (as RBE behaves) and not relative to the working directory (as specified).

Presumably, all the other remote execution engines work as specified. RBE will, eventually, and then we'll have to change Bazel anyway, but in the meantime, we could unblock users of other execution engines by implementing the --incompatible_remote_output_paths_relatitive_to_working_directory command line option now.

WDYT? Do you have capacity for that these days?

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[Chi Wang]

Hi Lukács,

I can work on this now but not sure whether we should introduce the flag --incompatible_remote_output_paths_relatitive_to_working_directory.

From the perspective of remote execution server, Bazel had been following the API spec before the change https://github.com/bazelbuild/bazel/commit/7149f578006a4ad0d51df69830a6986749b34df5:

1. API says the paths are relative to the working directory of the action execution.

2. Bazel set the working directory to null which means the working directory for action is input root.

3. input root is set to exec root.

4. output files are set to be relative to exec root, thus relative to input root.

How about we introduce the flag --incompatible_remote_output_paths_relative_to_input_root and make "relative to the working directory" a default behavior? After RBE works as specified, we can remote that flag.

Cheers,

Chi

[Lukács T. Berki]

On Wed, Apr 7, 2021 at 8:06 AM Chi Wang <chi...@google.com> wrote:

Hi Lukács,

I can work on this now but not sure whether we should introduce the flag --incompatible_remote_output_paths_relatitive_to_working_directory.

From the perspective of remote execution server, Bazel had been following the API spec before the change https://github.com/bazelbuild/bazel/commit/7149f578006a4ad0d51df69830a6986749b34df5:

1. API says the paths are relative to the working directory of the action execution.

2. Bazel set the working directory to null which means the working directory for action is input root.

3. input root is set to exec root.

4. output files are set to be relative to exec root, thus relative to input root.

How about we introduce the flag --incompatible_remote_output_paths_relative_to_input_root and make "relative to the working directory" a default behavior? After RBE works as specified, we can remote that flag.

That would a more principled solution, but wouldn't that break RBE right now? (AFAIU it doesn't support the correct behavior at the moment)

 

Cheers,

Chi

On Tue, Apr 6, 2021 at 10:42 PM Lukács T. Berki <lbe...@google.com> wrote:

Hey Chi,

As a followup to https://github.com/bazelbuild/bazel/issues/13188, it looks like that there are numerous breakages with Bazel that result from the fact that Bazel specifies the paths of action outputs relative to the root directory of the action (as RBE behaves) and not relative to the working directory (as specified).

Presumably, all the other remote execution engines work as specified. RBE will, eventually, and then we'll have to change Bazel anyway, but in the meantime, we could unblock users of other execution engines by implementing the --incompatible_remote_output_paths_relatitive_to_working_directory command line option now.

WDYT? Do you have capacity for that these days?

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[Chi Wang]

Yes, HEAD will break RBE after the change but users using 4.0.0 or below are not affected. Whichever Bazel release in the future contains this change should also mention this flag in the release note if RBE hasn't worked as specified as that time.

[Lukács T. Berki]

Let's not have a Bazel@HEAD that doesn't work with RBE, even if it's due to a bug in RBE.

[Chi Wang]

Ok.

How about change to use exec root as input root by default and only use the parent directory if --experimental_sibling_repository_layout is enabled?

So by default, --experimental_sibling_repository_layout is disabled, both RBE and other remote execution engines work. When --experimental_sibling_repository_layout is enabled, RBE will not work but others remain work. Enabling --incompatible_remote_output_paths_relative_to_input_root to work with RBE but will break others.

[Lukács T. Berki]

On Wed, Apr 7, 2021 at 8:54 AM Chi Wang <chi...@google.com> wrote:

Ok.

How about change to use exec root as input root by default and only use the parent directory if --experimental_sibling_repository_layout is enabled?

So by default, --experimental_sibling_repository_layout is disabled, both RBE and other remote execution engines work. When --experimental_sibling_repository_layout is enabled, RBE will not work but others remain work. Enabling --incompatible_remote_output_paths_relative_to_input_root to work with RBE but will break others.

So you are proposing to make the input root $EXECROOT/... only if --experimental_sibling_repository_layout is enabled? That would work, although it feels strange to introduce a coupling between the RBE interface and external repositories. What's its advantage over just implementing --incompatible_remote_output_paths_relative_to_input_root ?

[Chi Wang]

On Wed, Apr 7, 2021 at 3:02 PM Lukács T. Berki <lbe...@google.com> wrote:

On Wed, Apr 7, 2021 at 8:54 AM Chi Wang <chi...@google.com> wrote:

Ok.

How about change to use exec root as input root by default and only use the parent directory if --experimental_sibling_repository_layout is enabled?

So by default, --experimental_sibling_repository_layout is disabled, both RBE and other remote execution engines work. When --experimental_sibling_repository_layout is enabled, RBE will not work but others remain work. Enabling --incompatible_remote_output_paths_relative_to_input_root to work with RBE but will break others.

So you are proposing to make the input root $EXECROOT/... only if --experimental_sibling_repository_layout is enabled? That would work, although it feels strange to introduce a coupling between the RBE interface and external repositories. What's its advantage over just implementing --incompatible_remote_output_paths_relative_to_input_root ?

Just implementing --incompatible_remote_output_paths_relative_to_input_root implies paths relative to the working directory by default and will break RBE at HEAD. So I assume the question is about --incompatible_remote_output_paths_relative_to_working_directory.

Well, it also feels strange that Bazel lands a breaking change (change paths relative to input root instead of working directory) without a flag, requires users use a flag to be backwards compatible and after a period of time, the flag is removed, the breaking change is removed. However, if we can make sure these changes only live in HEAD, not in a release, the community should be fine with that.

[Lukács T. Berki]

On Wed, Apr 7, 2021 at 9:14 AM Chi Wang <chi...@google.com> wrote:

On Wed, Apr 7, 2021 at 3:02 PM Lukács T. Berki <lbe...@google.com> wrote:

On Wed, Apr 7, 2021 at 8:54 AM Chi Wang <chi...@google.com> wrote:

Ok.

How about change to use exec root as input root by default and only use the parent directory if --experimental_sibling_repository_layout is enabled?

So by default, --experimental_sibling_repository_layout is disabled, both RBE and other remote execution engines work. When --experimental_sibling_repository_layout is enabled, RBE will not work but others remain work. Enabling --incompatible_remote_output_paths_relative_to_input_root to work with RBE but will break others.

So you are proposing to make the input root $EXECROOT/... only if --experimental_sibling_repository_layout is enabled? That would work, although it feels strange to introduce a coupling between the RBE interface and external repositories. What's its advantage over just implementing --incompatible_remote_output_paths_relative_to_input_root ?

Just implementing --incompatible_remote_output_paths_relative_to_input_root implies paths relative to the working directory by default and will break RBE at HEAD. So I assume the question is about --incompatible_remote_output_paths_relative_to_working_directory.

Yep, sorry about that, I meant --incompatible_remote_output_paths_relative_to_working_directory .

 

Well, it also feels strange that Bazel lands a breaking change (change paths relative to input root instead of working directory) without a flag, requires users use a flag to be backwards compatible and after a period of time, the flag is removed, the breaking change is removed. However, if we can make sure these changes only live in HEAD, not in a release, the community should be fine with that.

The main reason why I'm opposed to breaking RBE-as-it-is at Bazel HEAD is twofold: first, it would by default break rolling releases and would impose a timing constraint because a new Bazel release could then only happen if RBE fixes the issue and rolls out the fix, which is not something under our control.

[Ed Schouten]

Op di 6 apr. 2021 om 16:42 schreef Lukács T. Berki <lbe...@google.com>:

> Presumably, all the other remote execution engines work as specified.

Honest opinion: I think that RBE's current semantics are the right
one. REv2 as a specification picked the wrong solution. Letting output
paths be relative to the working directory makes implementations more
complex, as you now also need to support output paths with leading
".." components. If output paths were relative to the input root, we
could require that they are normalized.

An alternative would be to add a boolean option to Command:
uses_sane_output_path_semantics. Existing clients announce it as
false, while Bazel can announce it as true. Slowly the world will
migrate to this, allowing to phase out the old semantics in REv3.

--
Ed Schouten <e...@nuxi.nl>