Needing to use Batch mode, but is deprecated.

[Katie Lucas]

I'm currently running a build which needs "--batch"

The reason it needs it is that after the build is complete, Bazel is
occupying all my resource limit on the machine. (It's a shared
machine, I'm not root there.)

My shell can't fork at that point -- it is literally scrolling "fork:
retry: Resource temporarily unavailable" up the terminal -- so even if
could type "bazel shutdown", it can't execute. (I also can't login,
can't use any other shells to run a shutdown, can't run "ps" to find
process IDs even if my shell would let me type kill
commands.... Basically my safetynet is having a "top" already running
because that can find the process ID and send it a kill.[1])

So I'm actually using the behaviour that bazel shuts itself down,
freeing up the resources. Which leaves me wondering how long that
"deprecated" behaviour is going to be viable & if there's a planned
replacement... Otherwise I'm going to be stuck for running these
builds.

Are there any current plans/timescales to remove "--batch"? Or can I
just ignore the deprecation warning for a while..?

Katie



[1] As an aside, this only seems to be a problem Bazelling Boost C++
code. Python builds run without using my whole ulimit!)

[Julio Merino]

On Mar 29, 2019, at 11:58 AM, 'Katie Lucas' via bazel-discuss <bazel-...@googlegroups.com> wrote:
>
> I'm currently running a build which needs "--batch"
>
> The reason it needs it is that after the build is complete, Bazel is
> occupying all my resource limit on the machine. (It's a shared
> machine, I'm not root there.)

Do you know if you are running in a container? Bazel had been abusing resources within containers because it wasn't detecting their limits properly. I'm told that's fixed now, but maybe not completely (or maybe not in the release you are using).

What are your soft and hard resource limits? (ulimit -S -a and ulimit -H -a)?

[markus....@gmail.com]

We are running into some issues sometimes as well with bazel shutdown in our ci environment. We are using the standard circleci setup, which runs in docker with 4gb of memory. While we do intend to provide more resources eventually this is a tricky setup currently where runs fail sometimes because we run out of memory. To help with that I started adding bazel shutdown between commands and sometimes ci fails on the shutdown command. I'll see that I can get the exact error message but it says something along the lines that the bazel server did not shutdown on sigkill.

I have not considered using batch mode yet but might actually give that a try.

[markus....@gmail.com]

Oh yes and we are on bazel 0.23 which has jdk 11 which came with the docker fixes. That did seem to help but not remove the issue entirely.

[László Csomor]

Katie, use "bazel --max_idle_secs=1 build //foo" to get the same effect as --batch:

https://docs.bazel.build/versions/master/command-line-reference.html#flag--max_idle_secs  

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

On Sun, Mar 31, 2019 at 9:59 AM <markus....@gmail.com> wrote:

Oh yes and we are on bazel 0.23 which has jdk 11 which came with the docker fixes. That did seem to help but not remove the issue entirely.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/cde4aeec-5b5d-42ac-b3e5-16a1eb40dfaf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[markus....@ecosia.org]

Interesting, I will give that a try. But I also wonder if '--batch' mode is potentially slightly more memory efficient in a very memory constraint environment as then there is just one process running rather than two, but either way I will give this a try first.

Also I wonder if that creates a potential "race"? What we do now is in ci, run a query command that is very memory expensive, then run bazel shutdown to flush the memory and then run some further bazel commands. So if the further commands run within the one second it would still not turn off, right?

On Monday, April 1, 2019 at 8:55:03 AM UTC+2, László Csomor wrote:
> Katie, use "bazel --max_idle_secs=1 build //foo" to get the same effect as --batch:
> https://docs.bazel.build/versions/master/command-line-reference.html#flag--max_idle_secs  
>
>
>
>
>
>
>
> --
> László Csomor | Software Engineer | laszlo...@google.com
>
> Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
> Registergericht und -nummer: Hamburg, HRB 86891
> Sitz der Gesellschaft: Hamburg
> Geschäftsführer: Paul Manicle, Halimah DeLaine Prado
>
>
>
> On Sun, Mar 31, 2019 at 9:59 AM <markus....@gmail.com> wrote:
> Oh yes and we are on bazel 0.23 which has jdk 11 which came with the docker fixes. That did seem to help but not remove the issue entirely.
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.

[markus....@ecosia.org]

Hmm strange, just testing bazel --max_idle_secs=1 query //:all locally, then wating a few seconds, running the command again and I do not always see 'Starting local Bazel server and connecting to it...' printed. I suppose bazel is still doing something in the background?

But that does seem to not make it reliable for our ci case.

[László Csomor]

Hm, I revoke my claim, "--batch" and "--max_idle_secs=1" are not the same.

I don't know how "--max_idle_secs" is implemented. Possibly just a best-effort self-shutdown within the timeout.

"--batch" uses execv(3) to replace the Bazel client with the JVM, which indeed yields fewer processes. Call stack:

https://github.com/bazelbuild/bazel/blob/bc90f9e20e13eb4a8a4adbebd02b3003e7647903/src/main/cpp/blaze_util_posix.cc#L285 

https://github.com/bazelbuild/bazel/blob/bc90f9e20e13eb4a8a4adbebd02b3003e7647903/src/main/cpp/blaze.cc#L764  

https://github.com/bazelbuild/bazel/blob/bc90f9e20e13eb4a8a4adbebd02b3003e7647903/src/main/cpp/blaze.cc#L1600 

 

--
László Csomor | Software Engineer | laszlo...@google.com

Google Germany GmbH | Erika-Mann-Str. 33 | 80636 München | Germany
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Geschäftsführer: Paul Manicle, Halimah DeLaine Prado

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/f4b8de9d-b9c1-40da-b115-8d1334db89e4%40googlegroups.com.

[markus....@ecosia.org]

Thanks for clarifying. Also here is the error message we are seeing in our ci environment occasionally on bazel shutdown:

WARNING: Waiting for server process to terminate (waited 5 seconds, waiting at most 60)
WARNING: Waiting for server process to terminate (waited 10 seconds, waiting at most 60)
WARNING: Waiting for server process to terminate (waited 30 seconds, waiting at most 60)
INFO: Waited 60 seconds for server process (pid=243) to terminate.
WARNING: Waiting for server process to terminate (waited 5 seconds, waiting at most 10)
WARNING: Waiting for server process to terminate (waited 10 seconds, waiting at most 10)
INFO: Waited 10 seconds for server process (pid=243) to terminate.
FATAL: Attempted to kill stale server process (pid=243) using SIGKILL, but it did not die in a timely fashion.
Exited with code 36

I have not given --batch a try but will do soon.

[katie...@fetch.ai]

On Monday, April 1, 2019 at 7:55:03 AM UTC+1, László Csomor wrote:
> Katie, use "bazel --max_idle_secs=1 build //foo" to get the same effect as --batch:

Ah -- this might indeed be a solution. I'll give it a whirl this evening and report back.