How to "bazel run" build file maintenance tools

[Jay Conrod]

Short version: is it possible for a tool built and executed with "bazel run" to retrieve information from Bazel (e.g., "bazel query") and modify WORKSPACE and BUILD files? We are getting the error "Another command (pid=31898) is running.  Waiting for it to complete..." because the parent bazel process is still running. I know bazel-run.sh is the standard solution for this, but I don't want to require our users to install or check in an extra script.

Background: I'm planning some changes to Gazelle, a tool that generates and updates BUILD files based on Go source files. It's important that the version of Gazelle being used matches the version of @io_baze_rules_go. We'd like our users to be able to "bazel run //:gazelle" where //:gazelle is a macro that generates a wrapper script and wraps it in an sh_binary. Gazelle is a dependency, and Bazel will build it before the script runs.

I'm planning some changes to Gazelle that will let it add and update external repositories in WORKSPACE. In order to do this efficiently, Gazelle will need to run commands like "bazel fetch" and "bazel query" in the same workspace that it's invoked from.

There are two problems we're running into.

* "bazel run" executes binaries in the execroot; we'd like this to run from either the repository root or the user's working directory. Some workarounds for this are possible.

* Bazel holds a lock while the binary is running. This is the real problem. Even commands like "bazel version" which should not need to access the local workspace block forever.

Is there any way to instruct Bazel to drop the lock when executing a binary? If not, is this something that could be changed in the future? Or is there a better way to do what we're doing?

[Jay Conrod]

Friendly ping? Let me know if I can clarify anything.

[Andrew Allen]

Just to get the ball rolling on this because I'm interested in the outcome.

Do you know what happens if you deamonize a process in Bazel? If I remember correctly when you daemonize your originally launched process is able to quit. Maybe it is possible for you to escape the lock when your top level process quits.

Another idea would be to make a sh_binary with a "seriously, I know what I'm doing" flag that wouldn't set the lock when it's running.

What other pie in the sky ideas are there for escaping the build lock?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGCADbYP1RfgTnjJKqA6TJogztDPiSkxfvg822NKQazAzDjp4A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Jay Conrod]

I'm sure it's possible to daemonize, wait for `bazel run` to quit, then do work in the background. From the user's perspective, that might be pretty bad though. If they ran another Bazel command, it would be blocked without any apparently reason why. Gazelle might take a long time if it has to `bazel fetch` something. It would also have no way of interacting with the user.

Having a flag on binary rules seems like an ideal solution to me. Something that says: drop the lock after building and run in the user's working directory. These kinds of binaries are written with the intent of running in the user's workspace, so it makes sense this would be a property of the binary rule, rather than a command line option.

On Wed, Sep 13, 2017 at 10:24 PM, Andrew Allen <m...@andrewzallen.com> wrote:

[Damien Martin-Guillerez]

Sorry for no answer but it is not possible to drop the lock but it is something that we want to change in the future. However it is not high priority and likely won't change before 1.0. What about running Gazelle outside of Bazel? You can eventually add a wrapper script that people can run that would do a bazel build && ./..../gazelle).

On Thu, Sep 14, 2017 at 4:28 PM 'Jay Conrod' via bazel-discuss <bazel-...@googlegroups.com> wrote:

I'm sure it's possible to daemonize, wait for `bazel run` to quit, then do work in the background. From the user's perspective, that might be pretty bad though. If they ran another Bazel command, it would be blocked without any apparently reason why. Gazelle might take a long time if it has to `bazel fetch` something. It would also have no way of interacting with the user.

Having a flag on binary rules seems like an ideal solution to me. Something that says: drop the lock after building and run in the user's working directory. These kinds of binaries are written with the intent of running in the user's workspace, so it makes sense this would be a property of the binary rule, rather than a command line option.

On Wed, Sep 13, 2017 at 10:24 PM, Andrew Allen <m...@andrewzallen.com> wrote:

On Wed, Sep 13, 2017 at 12:29 PM, 'Jay Conrod' via bazel-discuss <bazel-...@googlegroups.com> wrote:

Friendly ping? Let me know if I can clarify anything.

On Tue, Sep 12, 2017 at 2:13 PM, Jay Conrod <jayc...@google.com> wrote:

Short version: is it possible for a tool built and executed with "bazel run" to retrieve information from Bazel (e.g., "bazel query") and modify WORKSPACE and BUILD files? We are getting the error "Another command (pid=31898) is running.  Waiting for it to complete..." because the parent bazel process is still running. I know bazel-run.sh is the standard solution for this, but I don't want to require our users to install or check in an extra script.

Background: I'm planning some changes to Gazelle, a tool that generates and updates BUILD files based on Go source files. It's important that the version of Gazelle being used matches the version of @io_baze_rules_go. We'd like our users to be able to "bazel run //:gazelle" where //:gazelle is a macro that generates a wrapper script and wraps it in an sh_binary. Gazelle is a dependency, and Bazel will build it before the script runs.

I'm planning some changes to Gazelle that will let it add and update external repositories in WORKSPACE. In order to do this efficiently, Gazelle will need to run commands like "bazel fetch" and "bazel query" in the same workspace that it's invoked from.

There are two problems we're running into.

* "bazel run" executes binaries in the execroot; we'd like this to run from either the repository root or the user's working directory. Some workarounds for this are possible.

* Bazel holds a lock while the binary is running. This is the real problem. Even commands like "bazel version" which should not need to access the local workspace block forever.

Is there any way to instruct Bazel to drop the lock when executing a binary? If not, is this something that could be changed in the future? Or is there a better way to do what we're doing?

Just to get the ball rolling on this because I'm interested in the outcome.

Do you know what happens if you deamonize a process in Bazel? If I remember correctly when you daemonize your originally launched process is able to quit. Maybe it is possible for you to escape the lock when your top level process quits.

Another idea would be to make a sh_binary with a "seriously, I know what I'm doing" flag that wouldn't set the lock when it's running.

What other pie in the sky ideas are there for escaping the build lock?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGCADbYP1RfgTnjJKqA6TJogztDPiSkxfvg822NKQazAzDjp4A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGCADbaii8t6SS-8CTtoj%2B0dNAthwjAiNFbicqo7sVWzJCvrGA%40mail.gmail.com.

[Ian Cottrell]

If it's not going to happen soon, how about we (for now at least) add bazel-run.sh to your path in the installer so there is at least a simple mechanism we can recommend to people for now.

For bonus points, add a tag to indicate it should be run outside bazel (kind of an extreme form of "local")  and have the normal run mechanism print a message that you should use "bazel-run" instead of "bazel run" when you try to use it on one of those rules.

This would at least work for now, and have a clean upgrade path to when bazel run could cope with those rules directly.

[Damien Martin-Guillerez]

We could also change the shell launcher that we install. Caveat: we have to modify several package manager to handle it.

[Jay Conrod]

+1 for adding bazel-run.sh to the installer so that it appears in PATH without any additional set-up. That would be a simple short-term solution.

I've prototyped a wrapper script that builds and runs Gazelle. It requires users to build it, copy to their workspace, and probably check it in. More steps than I would like.

On Thu, Sep 14, 2017 at 10:53 AM, Damien Martin-Guillerez <dmar...@google.com> wrote:

We could also change the shell launcher that we install. Caveat: we have to modify several package manager to handle it.

On Thu, Sep 14, 2017 at 4:48 PM Ian Cottrell <ianco...@google.com> wrote:

If it's not going to happen soon, how about we (for now at least) add bazel-run.sh to your path in the installer so there is at least a simple mechanism we can recommend to people for now.

For bonus points, add a tag to indicate it should be run outside bazel (kind of an extreme form of "local")  and have the normal run mechanism print a message that you should use "bazel-run" instead of "bazel run" when you try to use it on one of those rules.

This would at least work for now, and have a clean upgrade path to when bazel run could cope with those rules directly.

On Thu, 14 Sep 2017 at 10:31 Damien Martin-Guillerez <dmar...@google.com> wrote:

Sorry for no answer but it is not possible to drop the lock but it is something that we want to change in the future. However it is not high priority and likely won't change before 1.0. What about running Gazelle outside of Bazel? You can eventually add a wrapper script that people can run that would do a bazel build && ./..../gazelle).

On Thu, Sep 14, 2017 at 4:28 PM 'Jay Conrod' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:

I'm sure it's possible to daemonize, wait for `bazel run` to quit, then do work in the background. From the user's perspective, that might be pretty bad though. If they ran another Bazel command, it would be blocked without any apparently reason why. Gazelle might take a long time if it has to `bazel fetch` something. It would also have no way of interacting with the user.

Having a flag on binary rules seems like an ideal solution to me. Something that says: drop the lock after building and run in the user's working directory. These kinds of binaries are written with the intent of running in the user's workspace, so it makes sense this would be a property of the binary rule, rather than a command line option.

On Wed, Sep 13, 2017 at 10:24 PM, Andrew Allen <m...@andrewzallen.com> wrote:

On Wed, Sep 13, 2017 at 12:29 PM, 'Jay Conrod' via bazel-discuss <bazel-discuss@googlegroups.com> wrote:

Friendly ping? Let me know if I can clarify anything.

On Tue, Sep 12, 2017 at 2:13 PM, Jay Conrod <jayc...@google.com> wrote:

Short version: is it possible for a tool built and executed with "bazel run" to retrieve information from Bazel (e.g., "bazel query") and modify WORKSPACE and BUILD files? We are getting the error "Another command (pid=31898) is running.  Waiting for it to complete..." because the parent bazel process is still running. I know bazel-run.sh is the standard solution for this, but I don't want to require our users to install or check in an extra script.

Background: I'm planning some changes to Gazelle, a tool that generates and updates BUILD files based on Go source files. It's important that the version of Gazelle being used matches the version of @io_baze_rules_go. We'd like our users to be able to "bazel run //:gazelle" where //:gazelle is a macro that generates a wrapper script and wraps it in an sh_binary. Gazelle is a dependency, and Bazel will build it before the script runs.

I'm planning some changes to Gazelle that will let it add and update external repositories in WORKSPACE. In order to do this efficiently, Gazelle will need to run commands like "bazel fetch" and "bazel query" in the same workspace that it's invoked from.

There are two problems we're running into.

* "bazel run" executes binaries in the execroot; we'd like this to run from either the repository root or the user's working directory. Some workarounds for this are possible.

* Bazel holds a lock while the binary is running. This is the real problem. Even commands like "bazel version" which should not need to access the local workspace block forever.

Is there any way to instruct Bazel to drop the lock when executing a binary? If not, is this something that could be changed in the future? Or is there a better way to do what we're doing?

Just to get the ball rolling on this because I'm interested in the outcome.

Do you know what happens if you deamonize a process in Bazel? If I remember correctly when you daemonize your originally launched process is able to quit. Maybe it is possible for you to escape the lock when your top level process quits.

Another idea would be to make a sh_binary with a "seriously, I know what I'm doing" flag that wouldn't set the lock when it's running.

What other pie in the sky ideas are there for escaping the build lock?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGCADbYP1RfgTnjJKqA6TJogztDPiSkxfvg822NKQazAzDjp4A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

[Jay Conrod]

I wanted to make sure we have a plan for Gazelle going forward.

Long term, we'd still like some way to tell Bazel to run a binary in the user's current directory without holding the lock. I understand it's a low priority, but would it be feasible for us to take on that work? It it something that we could do in a week or so? Would it interfere with anything else happening before 1.0?

As a short term solution, having bazel-run.sh be somewhere in PATH would help us a lot. Can it be part of the standard installation?

I'd hesitate to add this functionality to the launcher script; it doesn't seem like a good place for CLI changes, and ideally, we shouldn't need any if there is some tag or annotation on the target rule itself.

[Damien Martin-Guillerez]

On Mon, Sep 18, 2017 at 10:40 PM Jay Conrod <jayc...@google.com> wrote:

I wanted to make sure we have a plan for Gazelle going forward.

Long term, we'd still like some way to tell Bazel to run a binary in the user's current directory without holding the lock. I understand it's a low priority, but would it be feasible for us to take on that work? It it something that we could do in a week or so? Would it interfere with anything else happening before 1.0?

It is unfortunately not simple +Ulf Adams to comment because he had a plan IIRC.

 

As a short term solution, having bazel-run.sh be somewhere in PATH would help us a lot. Can it be part of the standard installation?

Certainly, but you have to understand that there is many many installer...

 

I'd hesitate to add this functionality to the launcher script; it doesn't seem like a good place for CLI changes, and ideally, we shouldn't need any if there is some tag or annotation on the target rule itself.

Sorry I do not understand what is about the tag/annotation? The launcher script has the advantage of being shipped in all the installation :)

 

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGCADbYP1RfgTnjJKqA6TJogztDPiSkxfvg822NKQazAzDjp4A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

[Jay Conrod]

Certainly, but you have to understand that there is many many installer... 

 

Ah, are the packaging scripts all separate? I was hoping they all used some common source, and it would just be a matter of adding another file to a list.

 

Sorry I do not understand what is about the tag/annotation? The launcher script has the advantage of being shipped in all the installation :)

My reasoning was that tools that need to be run in the user's workspace are written with that purpose in mind, so whatever option is used to run this way should be part of the binary rule, not on the command line (though an option recognized by "run" would be useful, too). Something like this:

go_binary(

    name = "gazelle",

    ...

    tags = ["run_in_workspace"],

)

There are some special tags like "manual", and "local". "local" prevents actions from executing remotely or in a sandbox; maybe it makes sense to extend that one.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGCADbYP1RfgTnjJKqA6TJogztDPiSkxfvg822NKQazAzDjp4A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

[Ulf Adams]

There's this pull request, which I assume you're aware of:

https://github.com/bazelbuild/bazel/pull/3635

On Mon, Sep 18, 2017 at 11:16 PM, Jay Conrod <jayc...@google.com> wrote:

Certainly, but you have to understand that there is many many installer... 

 

Ah, are the packaging scripts all separate? I was hoping they all used some common source, and it would just be a matter of adding another file to a list.

I am against adding an intermediate solution to work around this. It just creates more work for us when we eventually remove the intermediate 'fix', and if we're already so overloaded that we can't do the proper solution, then we're just increasing the overload. Talk about vicious cycles.

 

Sorry I do not understand what is about the tag/annotation? The launcher script has the advantage of being shipped in all the installation :)

My reasoning was that tools that need to be run in the user's workspace are written with that purpose in mind, so whatever option is used to run this way should be part of the binary rule, not on the command line (though an option recognized by "run" would be useful, too). Something like this:

go_binary(

    name = "gazelle",

    ...

    tags = ["run_in_workspace"],

)

There are some special tags like "manual", and "local". "local" prevents actions from executing remotely or in a sandbox; maybe it makes sense to extend that one.

From my work on this over the past few years, I believe that the changes needed to drop the lock for an individual binary on bazel run are basically the same changes needed to drop the lock in general. It simply isn't safe to drop the lock, because doing so can cause internal corruption of Bazel's data structures, regardless of what your binary may or may not do.

You might think that bazel run runs the subprocess as the last possible thing and there's nothing that could possibly happen afterwards to change Bazel's data structures, and you'd be unfortunately incorrect.

[Eric Fellheimer]

We have discussed changing the implementation of "blaze run" such that the command line is sent back to the client to execute. That would solve problems like stdin not being connected, and make it easier to ensure that we are no longer holding the blaze lock.

Eric Fellheimer
fe...@google.com

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CALS-RZ%2BfBA7Fxc1%3DJzys1PaoUeL-VqwBq1yff_KTLz0022vm2w%40mail.gmail.com.

[Jay Conrod]

That sounds promising. The client is likely a better place to run these kinds of tools. Would there be any negative impact to normal binaries (I guess they would still hold the lock)? How complicated would it be to make the change?

On Tue, Sep 19, 2017 at 3:52 PM, Eric Fellheimer <fe...@google.com> wrote:

We have discussed changing the implementation of "blaze run" such that the command line is sent back to the client to execute. That would solve problems like stdin not being connected, and make it easier to ensure that we are no longer holding the blaze lock.

Eric Fellheimer
fe...@google.com

[Eric Fellheimer]

This change would not be trivial, but I think would mostly be a lot of threading new features around the codebase. We would have to add some mechanism to pass back a command line request to the client as part of the client/server protocol. We would want to make sure such a feature carried its weight and wasn't needlessly complex.

Not sure exactly what you mean by "normal binaries", but I think in general there would be little impact.

Eric Fellheimer
fe...@google.com

On Wed, Sep 20, 2017 at 10:41 AM, 'Jay Conrod' via bazel-discuss <bazel-...@googlegroups.com> wrote:

That sounds promising. The client is likely a better place to run these kinds of tools. Would there be any negative impact to normal binaries (I guess they would still hold the lock)? How complicated would it be to make the change?

On Tue, Sep 19, 2017 at 3:52 PM, Eric Fellheimer <fe...@google.com> wrote:

We have discussed changing the implementation of "blaze run" such that the command line is sent back to the client to execute. That would solve problems like stdin not being connected, and make it easier to ensure that we are no longer holding the blaze lock.

Eric Fellheimer
fe...@google.com

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAGCADbZnNM37e%3Dwpk0V9JBpq9PGvVi%2BcBtNUFL%3DcNXH6Be%3DdWA%40mail.gmail.com.

[Lukács T. Berki]

On 20 September 2017 at 19:28, Eric Fellheimer <fe...@google.com> wrote:

This change would not be trivial, but I think would mostly be a lot of threading new features around the codebase. We would have to add some mechanism to pass back a command line request to the client as part of the client/server protocol. We would want to make sure such a feature carried its weight and wasn't needlessly complex.

Not sure exactly what you mean by "normal binaries", but I think in general there would be little impact.

FWIW, I think once we replace batch mode with one-shot server mode, this will become much simpler because then we'll have to implement this in the client-server protocol only once instead of twice. I don't particularly mind adding special-case features for this in the gRPC protocol; after all, the whole point of said protocol is to facilitate communication between the client and the server!

 

Eric Fellheimer
fe...@google.com

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891