Bazel clean hook?

[Robert Cauble]

Hi,

Was wondering whether there's some way to get a callback when a "clean" is performed to do something extra?

For context, this is part of my overall exploration of migrating from maven to bazel. One of our sub-modules is an external project which is itself quite large -- too large for us to bazel-fy initially and we would like to take a piecemeal approach to this so that we can prove out bazel before we start making deeper changes. So I'm looking for an approach that'll allow us to treat this sub-module as a "black box" for the time being.

The sub-module's build is also docker-based and for [reasons] it ends up creating root-owned files so they can't simply be deleted without running "clean" via the docker image as well. At present we have this wrapped in maven and it's mostly de-coupled from our build except that the maven build outputs the docker image hash which we then use in our integration tests, so that we're using the same image as was built in the tree.

I think I can almost make this work in bazel via custom rule that does something like:

1) Define sources to be all the sources of the sub-module

2) execute by invoking the sub-module build

3) Define an output to the a file containing the image hash

The part that doesn't work, IIUC, is the "clean" step since that'll only delete the one image hash file rather than performing the deeper clean. 

Any suggestions?

Thanks,

Rob

[Brian Silverman]

Could you add a step to your custom rule to change the file ownership?
I think you'll have trouble with the sandbox cleanup if you don't do
that, before worrying about clean.

I don't think there's a way to directly add hooks to clean, but you
can do it via tools/bazel. If tools/bazel exists, the bazel launcher
will execute that as the main bazel binary (this is how things like
bazelisk work). You could make it a custom script that does some
limited command-line parsing to determine if a clean is being
executed, and do your cleanup then before or after calling the real
bazel binary. It's not particularly clean, but you should be able to
do what you're looking for there.

> --
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/35e8a901-de1f-492d-9d27-d1910f06556bn%40googlegroups.com.

[Robert Cauble]

File ownership change requires a sudo and I’m not sure that’ll work within a custom rule? To clarify, these are intermediate output files within a sub directory. I am assuming bazel won’t know about these files since they are not registered as outputs?

tools/bazel sounds good. I assume that’s a workspace-relative path? Do you have an example of how to call the real bazel from within the script? Seems like it would need to call bazel again but tell it not to use the script override.

> On Sep 17, 2022, at 2:38 AM, Brian Silverman <bsilve...@gmail.com> wrote:
>
> Could you add a step to your custom rule to change the file ownership?

[Brian Silverman]

On Sat, Sep 17, 2022 at 1:16 AM Robert Cauble <robert...@gmail.com> wrote:
>
> File ownership change requires a sudo and I’m not sure that’ll work within a custom rule? To clarify, these are intermediate output files within a sub directory. I am assuming bazel won’t know about these files since they are not registered as outputs?

Bazel still has to remove the whole temporary directory though. I'm
thinking you could use sudo from within the docker container that's
creating them in the first place? Or another docker container you run
afterwards?

> tools/bazel sounds good. I assume that’s a workspace-relative path? Do you have an example of how to call the real bazel from within the script? Seems like it would need to call bazel again but tell it not to use the script override.

Yes it's workspace-relative. If you're using bazelisk or something
similar, just call it. Otherwise, take a look at /usr/bin/bazel (or
equivalent for your OS), it's the script that looks for tools/bazel.
The logic that uses after not finding a tools/bazel is pretty similar
to what you're looking for. Basically you want to take over the
process of finding a real bazel binary and exec it, not the wrapper
script again.

[Robert Cauble]

Sent from my iPad

> On Sep 17, 2022, at 3:20 AM, Brian Silverman <bsilve...@gmail.com> wrote:

>
> On Sat, Sep 17, 2022 at 1:16 AM Robert Cauble <robert...@gmail.com> wrote:
>>
>> File ownership change requires a sudo and I’m not sure that’ll work within a custom rule? To clarify, these are intermediate output files within a sub directory. I am assuming bazel won’t know about these files since they are not registered as outputs?
>
> Bazel still has to remove the whole temporary directory though. I'm
> thinking you could use sudo from within the docker container that's
> creating them in the first place? Or another docker container you run
> afterwards?

Short answer is that it’s a bit of a mess and probably not very easy to do. Long answer is these files live in a several sub trees (roughly 20) of the source tree. We don’t actually know all their locations at the top level as they may change over time independent of our top level build as the sub module is not controlled by our group. We just know to invoke “clean” within the sub module build.

But I also am confused about how their presence could interfere with bazel deleting its tempdir since they are not children of the tempdir.

[Brian Silverman]

On Sat, Sep 17, 2022 at 2:59 AM Robert Cauble <robert...@gmail.com> wrote:
> But I also am confused about how their presence could interfere with bazel deleting its tempdir since they are not children of the tempdir.

Never mind then, I was envisioning something different.

[Robert Cauble]

Doesn’t look like it picks up tools/bazel. I am using bazelisk. When I tried debugging it, I found that “bazel” goes through a chain of asdf scripts which I couldn’t follow easily. So I ended up going with a top level “clean” script that calls “bazel clean” plus my own clean. Not too terrible — not like we should be cleaning all the time.

> On Sep 17, 2022, at 2:38 AM, Brian Silverman <bsilve...@gmail.com> wrote:
>
> Could you add a step to your custom rule to change the file ownership?