Status on --experimental_spawn

Mark Sulkowski

unread,

Jun 23, 2020, 10:06:22 AM6/23/20

to bazel-discuss

I'm interested to know about any bugs or issues that may pertain to the use of --experimental_spawn_scheduler.

How mature is this feature, and what problems might I expect in using it?

Julio Merino

unread,

Jun 23, 2020, 10:55:57 AM6/23/20

to Mark Sulkowski, bazel-discuss

On 6/23/20 10:06 AM, 'Mark Sulkowski' via bazel-discuss wrote:

I'm interested to know about any bugs or issues that may pertain to the use of --experimental_spawn_scheduler.

How mature is this feature, and what problems might I expect in using it?

Hello Mark,

I'd say that the feature is pretty mature. We use it regularly at Google and don't experience random bugs. That isn't to say it's perfect. There are plenty of things to improve (you can query our open issues under team-Local-Exec), of course.

The big caveat here is that we use the dynamic scheduler against our internal client of remote execution... which is unfortunately very different to the external one. So it's possible that some issues remain in the latter that haven't been exposed due to reduced usage.

The things you have to be aware if you want to try this out are:

* Ensuring that the remote and local environments are equivalent. This is critical. Otherwise, mixing outputs between them could lead to strange build problems.

* Watching out for which workers you enable. Bazel doesn't have good resource tracking for workers, so if you end up enabling too many worker types during a build, you'll likely render your system unusable.

* Tuning --local_cpu_resources. We have found that 0.75*HOST_CPUS works best for us, but that will depend on your build behavior and whether you use workers or not.

I wrote a series of blog posts detailing what our work has been in this area, starting here: https://jmmv.dev/2019/12/bazel-dynamic-execution-introduction.html and a recent post on the upcoming version of the scheduler here: https://jmmv.dev/2020/06/shipping-bazel-new-dynamic-scheduler.html

Cheers

David Sanderson

unread,

Sep 29, 2020, 7:27:55 PM9/29/20

to bazel-discuss

On Tuesday, June 23, 2020 at 10:55:57 AM UTC-4 JMMV wrote:

I wrote a series of blog posts detailing what our work has been in this area, starting here: https://jmmv.dev/2019/12/bazel-dynamic-execution-introduction.html and a recent post on the upcoming version of the scheduler here: https://jmmv.dev/2020/06/shipping-bazel-new-dynamic-scheduler.html

Has the upcoming version of the scheduler described in https://jmmv.dev/2020/06/shipping-bazel-new-dynamic-scheduler.html made its way into a public bazel release? If so, which one?

Julio Merino

unread,

Sep 29, 2020, 8:58:42 PM9/29/20

to David Sanderson, bazel-discuss

It has been out there for a while. On mobile now so I cannot check easily. But it’s still disabled by default.

Cheers

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/b5fdf0ca-9ccd-41c0-85d8-132a921c3f4bn%40googlegroups.com.

--

jmmv.dev

David Sanderson

unread,

Sep 30, 2020, 12:31:25 AM9/30/20

to bazel-discuss

On Tuesday, September 29, 2020 at 8:58:42 PM UTC-4 JMMV wrote:

It has been out there for a while. On mobile now so I cannot check easily. But it’s still disabled by default.

Ah, thank you! I delved into the code a bit. If I'm understanding correctly, the basic way to try out the new

dynamic execution mechanism is to use:

--internal_spawn_scheduler

--spawn_strategy=dynamic

--legacy_spawn_scheduler=false

with the possible additions of

--experimental_local_lockfree_output

--experimental_local_execution_delay=1000

I tried this out with bazel 3.5.0 with buildfarm as the remote executor. Sadly, every attempt so far has

failed due to missing directories -- apparently race conditions of some sort.

The old dynamic execution mechanism (plain --experimental_spawn_scheduler) also exhibits these

races in bazel 3.5.0.

For what it's worth, the most recent version of bazel where we've been able to use

--experimental_spawn_scheduler with buildfarm is bazel 3.1.0.

Lars Clausen

unread,

Sep 30, 2020, 4:09:11 AM9/30/20

to David Sanderson, bazel-discuss

Hi David,

I'm working on this area, and would like to hear more about how it fails. Could you share the failure messages and what your setup is like?

Thanks,

-Lars

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a633648d-35c1-444c-8d15-0309eae408e1n%40googlegroups.com.

David Sanderson

unread,

Sep 30, 2020, 3:55:39 PM9/30/20

to bazel-discuss

On Wednesday, September 30, 2020 at 4:09:11 AM UTC-4 Lars wrote:

Hi David,

I'm working on this area, and would like to hear more about how it fails. Could you share the failure messages and what your setup is like?

I'll be happy to!
We are using Ubuntu 18.04 on 64-bit Intel hardware.
We are currently using buildfarm 1.2.0 (https://github.com/bazelbuild/bazel-buildfarm/releases/tag/1.2.0)
When executing the following script with "bash -x":

bazelrc=bazel.rc

cat $bazelrc

startup="--bazelrc=$bazelrc"

./bazel.3.5.0 $startup clean

./bazel.3.5.0 $startup build --verbose_failures --config=remote --config=dynamic-execution1 //atg/hello_world/cc:hello_world_test

we see output similar to the following:

+ bazelrc=bazel.rc

+ cat bazel.rc

build --extra_toolchains=//tools/toolchain:cc-toolchain-linux_k8_clang

build --compiler=clang

build --host_compiler=clang

build --crosstool_top=//tools/toolchain:cc-toolchain

build --host_crosstool_top=//tools/toolchain:host-cc-toolchain

build --platforms=//tools/toolchain:target_platform

build --host_platform=//tools/toolchain:host_platform

build --action_env=BAZEL_DO_NOT_DETECT_CPP_TOOLCHAIN=1

build --features=-static_libgcc

build --incompatible_strict_action_env

build --curses=no

build --color=no

build:remote --remote_executor=grpc://buildfarm-prd-1.aws.uberatc.net

build:remote --jobs=128

# Dynamic execution

#

# --config=dynamic-execution0 --experimental_spawn_scheduler with --local_cpu_resources=HOST_CPUS*0.75 --local_ram_resources=HOST_RAM*0.75

# --config=dynamic-execution1 Use the new dynamic scheduler

# --config=dynamic-execution2 Use the new dynamic scheduler and --experimental_local_lockfree_output

# --config=dynamic-execution3 Use the new dynamic scheduler and --experimental_local_lockfree_output and --experimental_local_execution_delay=1000

#

# Note that units of --experimental_local_execution_delay are milliseconds.

build:dynamic-execution0 --local_cpu_resources=HOST_CPUS*0.75

build:dynamic-execution0 --local_ram_resources=HOST_RAM*0.75

build:dynamic-execution0 --internal_spawn_scheduler

build:dynamic-execution0 --spawn_strategy=dynamic

build:dynamic-execution1 --config=dynamic-execution0

build:dynamic-execution1 --legacy_spawn_scheduler=false

build:dynamic-execution2 --config=dynamic-execution1

build:dynamic-execution2 --experimental_local_lockfree_output

build:dynamic-execution3 --config=dynamic-execution2

build:dynamic-execution3 --experimental_local_execution_delay=1000

+ startup=--bazelrc=bazel.rc

+ ./bazel.3.5.0 --bazelrc=bazel.rc clean

INFO: Starting clean (this may take a while). Consider using --async if the clean takes more than several minutes.

+ ./bazel.3.5.0 --bazelrc=bazel.rc build --verbose_failures --config=remote --config=dynamic-execution1 //atg/hello_world/cc:hello_world_test

INFO: Invocation ID: 2b7f70c8-860d-4a12-9cc9-de9e35f371d7

Loading:

Loading: 0 packages loaded

Analyzing: target //atg/hello_world/cc:hello_world_test (1 packages loaded, 0 targets configured)

INFO: Analyzed target //atg/hello_world/cc:hello_world_test (37 packages loaded, 4313 targets configured).

INFO: Found 1 target...

[0 / 1] [Prepa] BazelWorkspaceStatusAction stable-status.txt

WARNING: Reading from Remote Cache:

/home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/gperftools/_objs/tcmalloc/malloc_extension.pic.d.tmp (No such file or directory)

WARNING: Reading from Remote Cache:

/home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/gperftools/_objs/tcmalloc/thread_cache.pic.d.tmp (No such file or directory)

WARNING: Reading from Remote Cache:

/home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/gperftools/_objs/tcmalloc/raw_printer.pic.d.tmp (No such file or directory)

WARNING: Reading from Remote Cache:

/home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/gperftools/_objs/tcmalloc/symbolize.pic.d.tmp (No such file or directory)

WARNING: Reading from Remote Cache:

/home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/gperftools/_objs/tcmalloc/central_freelist.pic.d.tmp (No such file or directory)

WARNING: Reading from Remote Cache:

/home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/com_google_googletest/_objs/gtest/gtest-port.pic.d.tmp (No such file or directory)

ERROR: /home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/external/gperftools/BUILD.bazel:10:11: C++ compilation of rule '@gperftools//:tcmalloc' failed (Exit 34): java.io.FileNotFoundException: /home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/gperftools/_objs/tcmalloc/sysinfo.pic.d.tmp (No such file or directory)

at com.google.devtools.build.lib.unix.NativePosixFiles.lstat(Native Method)

at com.google.devtools.build.lib.unix.UnixFileSystem.statInternal(UnixFileSystem.java:186)

at com.google.devtools.build.lib.unix.UnixFileSystem.stat(UnixFileSystem.java:176)

at com.google.devtools.build.lib.vfs.Path.stat(Path.java:418)

at com.google.devtools.build.lib.vfs.FileSystemUtils.moveFile(FileSystemUtils.java:454)

at com.google.devtools.build.lib.remote.RemoteCache.moveOutputsToFinalLocation(RemoteCache.java:415)

at com.google.devtools.build.lib.remote.RemoteCache.download(RemoteCache.java:374)

at com.google.devtools.build.lib.remote.RemoteSpawnRunner.downloadAndFinalizeSpawnResult(RemoteSpawnRunner.java:440)

at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:266)

at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:240)

at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:132)

at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy.runRemotely(DynamicSpawnStrategy.java:397)

at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy.access$400(DynamicSpawnStrategy.java:64)

at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy$2.callImpl(DynamicSpawnStrategy.java:298)

at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy$Branch.call(DynamicSpawnStrategy.java:471)

at com.google.devtools.build.lib.dynamic.DynamicSpawnStrategy$Branch.call(DynamicSpawnStrategy.java:408)

at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)

at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)

at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

at java.base/java.lang.Thread.run(Unknown Source)

. Note: Remote connection/protocol failed with: execution failed java.io.FileNotFoundException: /home/dws/.cache/bazel/_bazel_dws/71d2c1360d0922685900f5826e324a56/execroot/__main__/bazel-out/k8-fastbuild/bin/external/gperftools/_objs/tcmalloc/sysinfo.pic.d.tmp (No such file or directory)