[Proposal] Remote Output Service: place bazel-out/ on a FUSE file system

[Ed Schouten]

Hello fellow Bazel users,

The other day I submitted a PR for a new proposal, named "Remote
Output Service: place bazel-out/ on a FUSE file system". To summarize,
the goal of this proposal is to be able to get Remote Builds without
the Bytes, but still be able to access build outputs. The full text of
this proposal can be found here:

https://github.com/bazelbuild/proposals/blob/master/designs/2021-02-09-remote-output-service.md

I would hereby like to invite you to give it a read and share your thoughts!

--
Ed Schouten <e...@nuxi.nl>

[Brentley Jones]

Hi Ed,

I really like this idea. I've had many issues with Remote Builds without the Bytes, some of which you cover nicely in your proposal. In our exploration with remote execution, especially with engineers working from home, we find that downloading all outputs is a non-starter.

The only sticking point for us currently is we use macOS, and the performance of the current implementation (either in bazel or bb_clientd, not sure which) makes this unusable, with performance worse than a normal build. I hope that it would at least be better than a normal build, if not very similar to a --remote_download_minimal build.

[Ed Schouten]

Hi Brentley,

Op vr 12 feb. 2021 om 17:29 schreef 'Brentley Jones' via bazel-dev
<baze...@googlegroups.com>:

> The only sticking point for us currently is we use macOS, and the performance of the current implementation (either in bazel or bb_clientd, not sure which) makes this unusable, with performance worse than a normal build. I hope that it would at least be better than a normal build, if not very similar to a --remote_download_minimal build.

Yes, this is unfortunate. It would have been nice if macFUSE/OSXFUSE
was both as stable and as performant as Linux FUSE. It no longer being
Open Source also makes it unlikely we're able to address this.

My hope is that we can at some point extend bb_clientd to also support
other virtual file system protocols, such as NFS. I think that work in
this area is independent from any changes we need to make on the Bazel
side.

--
Ed Schouten <e...@nuxi.nl>

[Janak Ramakrishnan]

+Alex Jurkowski, +Chi Wang  

Very interesting proposal, Ed. I was curious about the difficulties mentioned in the design doc around the extra ActionInputMap in terms of memory usage and incremental builds. Are there tracking issues for those problems? Google's internal implementation of "build without the bytes" doesn't suffer from those problems, I think. There's been talk of open-sourcing it, although it's a large-ish project.

--
You received this message because you are subscribed to the Google Groups "bazel-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-dev/CABh_MKnPg2_z4CHiNRU_%3DjVZr9%3DdwC6AQDjapejRJPpfo9KOOg%40mail.gmail.com.

[Janak Ramakrishnan]

Got it, thanks. It sounds like the slow incremental build issue is confined to post-restart builds, not ones in the same server. Sent a small PR to clarify the design doc.

On Mon, Feb 22, 2021 at 9:47 AM Ed Schouten <e...@nuxi.nl> wrote:

Hi Janak,

Op do 18 feb. 2021 om 18:07 schreef Janak Ramakrishnan <jan...@google.com>:

> Very interesting proposal, Ed. I was curious about the difficulties mentioned in the design doc around the extra ActionInputMap in terms of memory usage and incremental builds. Are there tracking issues for those problems?

With regards to the memory use, I don't think there are any tracking
bugs for that. With regards to the inability to do incremental builds,
I think that's covered by:

https://github.com/bazelbuild/bazel/issues/8248

Which is linked from:

https://github.com/bazelbuild/bazel/issues/6862

That said, the FUSE file system I've developed addresses this issue
adequately, while also speeding up builds significantly.

Best regards,
--
Ed Schouten <e...@nuxi.nl>

[Ed Schouten]

Hi Janak,

Op do 18 feb. 2021 om 18:07 schreef Janak Ramakrishnan <jan...@google.com>:

> Very interesting proposal, Ed. I was curious about the difficulties mentioned in the design doc around the extra ActionInputMap in terms of memory usage and incremental builds. Are there tracking issues for those problems?

[Alex Jurkowski]

Hello Ed,

First of all, hello and thank you for the excellent contribution! Both the document and the PR is very high quality! I would like to apologize for the delay and let you know that I would love to work together on this change. I would like to kick off the discussion here and share some initial thoughts.

I really like how your change integrates with Bazel -- OutputService is definitely a good place to plug the new feature in.

One high-level issue I would like to discuss with you is local execution. What I mean is builds like:

bazel build --strategy=foo=standalone --strategy=bar=remote //foo //bar

The problem here is that are runners like standalone, as you mentioned in the document, will write directly under blaze-out, which means that the FUSE/NFS/ProjFS implementation will need to support writes. I was wondering if it would be desirable to be able to mount the output tree as a read-only file system (OS, not Bazel's FileSystem class)? It seems that if we restricted the strategies to sandboxed ones, it could be feasible to make the runners go through OutputService for their outputs -- that would be trading implementing the write part of the OS file system API for an API to inject the files over gRPC. Problems I can see with this:

• It looks like sandboxing is not supported on Windows.
• Custom actions writing to outputs directly like the build info action (technically virtualization at the level of Bazel FileSystem could solve that, but that's extra work)
• Implementing writes at the level of OS file system may not be that much more work than adding support for outputs injection.

Probably, these reasons are good enough to say no to that, maybe it's worth adding a footnote to the doc? Do you have more thoughts on that?

Thank you

Alex

[lar...@google.com]

Hi Alex, Ed, interesting discussion. Just wanted to add that not only is sandboxing not supported on Windows, it's very slow on Mac (see https://github.com/bazelbuild/bazel/issues/8230).

-Lars

[Rob Sayre]

On Wed, Apr 21, 2021 at 10:40 AM 'lar...@google.com' via bazel-dev <baze...@googlegroups.com> wrote:

Hi Alex, Ed, interesting discussion. Just wanted to add that not only is sandboxing not supported on Windows, it's very slow on Mac (see https://github.com/bazelbuild/bazel/issues/8230).

Not to distract from the work here, but in case anyone else hits this thread in a search: I've started mounting a Linux machine over sshfs. I edit on the Mac, and run the Bazel commands on the Linux machine. Turns out Gigabit ethernet is fine for editing text files.

thanks,

Rob

 

[Ed Schouten]

Hi Alex,

Op di 20 apr. 2021 om 21:15 schreef 'Alex Jurkowski' via bazel-dev
<baze...@googlegroups.com>:

> First of all, hello and thank you for the excellent contribution! Both the document and the PR is very high quality! I would like to apologize for the delay and let you know that I would love to work together on this change. I would like to kick off the discussion here and share some initial thoughts.

Awesome! Looking forward to collaborating with you on this. Would it
make sense to have a meeting next week or so to come up with a full
list of action items? Be sure to share your availability and I'll
organise something.

> I really like how your change integrates with Bazel -- OutputService is definitely a good place to plug the new feature in.
>
> One high-level issue I would like to discuss with you is local execution. What I mean is builds like:
>
> bazel build --strategy=foo=standalone --strategy=bar=remote //foo //bar
>
> The problem here is that are runners like standalone, as you mentioned in the document, will write directly under blaze-out, which means that the FUSE/NFS/ProjFS implementation will need to support writes. I was wondering if it would be desirable to be able to mount the output tree as a read-only file system (OS, not Bazel's FileSystem class)? It seems that if we restricted the strategies to sandboxed ones, it could be feasible to make the runners go through OutputService for their outputs -- that would be trading implementing the write part of the OS file system API for an API to inject the files over gRPC.

Oh, man. I'd love that. Right now the FUSE file system needs to be
pretty complex because of all the POSIXness you need to provide
(renames, hardlinks, random access writes, appending to files, etc.).
Because Bazel is also oblivious of what changes, we need to provide
things like ModifiedFileSet.

Also in terms of snapshotting/persisting data in the FUSE file system
things can be complex. For example, I've noticed that files like
stable-status.txt and volatile-status.txt are updated by doing an
open(O_CREAT|O_TRUNC) instead of unlink();open(O_CREAT|O_EXCL),
meaning you can't just let those be read-only files backed by some
snapshot. Letting file writes go through gRPC would make that easier,
as you could translate it to a removal/replacement under the hood.

> Problems I can see with this:
>
> It looks like sandboxing is not supported on Windows.
> Custom actions writing to outputs directly like the build info action (technically virtualization at the level of Bazel FileSystem could solve that, but that's extra work)
> Implementing writes at the level of OS file system may not be that much more work than adding support for outputs injection.

I'm not sure, but it may also require us to make some changes to
stdout & stderr handling. I've observed that temporary files for those
are written into bazel-out/_tmp/.

> Probably, these reasons are good enough to say no to that, maybe it's worth adding a footnote to the doc? Do you have more thoughts on that?

Yeah, unfortunately I also need support for non-sandboxed actions for
my intended use case for now.

You know what? Let's not add it initially, but at least describe that
we should investigate it in the future. A transition to it could look
like this:

1. We add dedicated RPCs to let Bazel write files straight into the
FUSE file system.
2. We add a boolean to StartBuildRequest that Bazel can set as a hint,
indicating that the remote output service is permitted to offer a
read-only output path.

I'll send out a PR for this, hopefully later today.

Thanks,
--
Ed Schouten <e...@nuxi.nl>

[Alex Jurkowski]

Just a quick comment here -- that would require some refactoring in Bazel. I could see 2 different areas here:

1. Remote executor -- that could be the easy part -- no eager downloads when present.
2. Sandboxed runners -- in here, we have a distinctive moment when we are moving the outputs out of the sandbox -- that could be replaced with the call into the FUSE.

I think the main question is what the desired long-term state is. If that includes local runners support, the payoff of the extra work may be lower. I am not saying that would be a bad change -- I just don't want to put that on the critical path of your effort, especially if it was not to buy us that much in the long run.

I have never implemented a FUSE file system, but at least on the surface what we want seems simple -- it's almost a writable file system with an overlay to register extra read-only files (almost because you can have a build with a remote input followed by a local one in the same place).

I think that is a great idea to schedule a meeting for that.

Thank you

Alex

[Alex Jurkowski]

Ed, for some reason the emails here come with significant latency. Anyway, I started discussing the change on github instead and did a pass on it a month ago -- we can probably just take that from there.

Thank you

Alex