Action level parallelism & build perf

[Dave Lee]

We've come up against a build parallelization and performance issue for Bazel and Swift.

For local dev builds on our project, Xcode's build is by default faster than Bazel's.

My understanding is action parallelism works like this:

• Bazel allocates workers/slots to manage up to N actions running in parallel (using --jobs or --local_cpu_resources)
• Each of these actions are assumed to use one core (or alternatively, 1/Nth of cpu)

This seems good for the general case, but some actions can use more cores, in our case swiftc. In Swift, the module is the compilation unit, a single swiftc invocation is invoked with all source files in the module. Internally though, the compilation work can be more granular and divided into parallel jobs.

In some dependency graphs, the structure contains bottleneck sections of the build that cannot use the full amount of cpu available to Bazel. For example a small number of targets happens to be a bottleneck to the rest of the build graph. During bottleneck sections, the cpu is being underutilized, but it doesn't need to be that way because swiftc could be capitalizing on the full cpu availability using its own parallelism. By manually increasing swiftc parallelism for such targets, the target (and the build) finish much faster.

Have there been any discussion about "lending parallelism" to actions when Bazel sees the build graph is bottlenecked?

As I mentioned, actions can take matters into their own hands, by specifying increased parallelism via copts. However, this requires manual analysis of builds to identify which targets to optimize.

In my investigation, Xcode uses a constant -j8 (on my machine) for all module compilation. Xcode appears to dynamically control the number of parallel actions, but all swiftc actions use -j8. By manually specifying -j8 for some targets, the Bazel build is almost on par with Xcode. Manually curating which modules to hard code increased parallelism is a maintenance we'd rather avoid.

Note that this also affects the tail end of the build. Even if your build graph has no interior bottlenecks, the the last target(s), with few or no reverse dependencies, can also benefit from "borrowing" Bazel's unused parallelism, to complete the build sooner.

In the absence of Bazel support for sharing parallelism, I haven't seen any documentation or tooling to support finding and maintaining optimal settings for builds that have these outer/inner parallelism complexities.

Has action parallelism has discussed for Bazel? There doesn't seem to be anything on the roadmap related to this, are no other languages or rules finding this balance tricky?

thanks,

Dave

[Tobias Werth]

+Julio Merino 

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/6372df6d-758d-49dd-86ff-4a98546baa57%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Julio Merino]

Yes, that makes a lot of sense. It'd be great if Bazel could tell actions what their resources allowance is based on what's available at that moment (not as a fixed, predefined quantity). Bazel has that information already and, in fact, it's already doing the scheduling based on available local resources.

A tricky problem is how to propagate the information to the actions though: how can we do so without affecting command lines or the environment, which in turn affects cache keys.

We have discussed this at some point and I was convinced there was an open bug for it... but cannot find it right now. Mind filing a new feature request please?

Thanks

--

Julio Merino / go/jmmv / Blaze

[alle...@google.com]

Since we have rich Args objects now to construct action command lines, would it be crazy to allow rule authors to tag particular args as not affecting the output (just as Bazel can already do for its own command line flags)? Putting that capability in the hands of the user could be a potential foot-gun, but if it's something they have to explicitly opt-in to, it hopefully wouldn't be misused (intentionally or unintentionally).

Then, perhaps some sort of placeholder can be introduced that would be populated at the time the action executes. The following isn't a serious API suggestion, but just to express the concept:

args.add("-j{%available_cpus_when_this_action_runs%}", affects_output = False)

On Wednesday, June 19, 2019 at 8:20:42 AM UTC-7, Julio Merino wrote:
> Yes, that makes a lot of sense. It'd be great if Bazel could tell actions what their resources allowance is based on what's available at that moment (not as a fixed, predefined quantity). Bazel has that information already and, in fact, it's already doing the scheduling based on available local resources.
>
>
> A tricky problem is how to propagate the information to the actions though: how can we do so without affecting command lines or the environment, which in turn affects cache keys.
>
>
>
> We have discussed this at some point and I was convinced there was an open bug for it... but cannot find it right now. Mind filing a new feature request please?
>
>
> Thanks
>
>
> On Wed, Jun 19, 2019 at 2:38 AM Tobias Werth <twe...@google.com> wrote:
>
> +Julio Merino 
>
>
> 'Dave Lee' via bazel-discuss <bazel-...@googlegroups.com> schrieb am Mi., 19. Juni 2019, 00:04:
>
> We've come up against a build parallelization and performance issue for Bazel and Swift.
>
>
>
> For local dev builds on our project, Xcode's build is by default faster than Bazel's.
>
>
> My understanding is action parallelism works like this:
>
>

> Bazel allocates workers/slots to manage up to N actions running in parallel (using --jobs or --local_cpu_resources)Each of these actions are assumed to use one core (or alternatively, 1/Nth of cpu)

> This seems good for the general case, but some actions can use more cores, in our case swiftc. In Swift, the module is the compilation unit, a single swiftc invocation is invoked with all source files in the module. Internally though, the compilation work can be more granular and divided into parallel jobs.
>
>
> In some dependency graphs, the structure contains bottleneck sections of the build that cannot use the full amount of cpu available to Bazel. For example a small number of targets happens to be a bottleneck to the rest of the build graph. During bottleneck sections, the cpu is being underutilized, but it doesn't need to be that way because swiftc could be capitalizing on the full cpu availability using its own parallelism. By manually increasing swiftc parallelism for such targets, the target (and the build) finish much faster.
>
>
> Have there been any discussion about "lending parallelism" to actions when Bazel sees the build graph is bottlenecked?
>
>
>
> As I mentioned, actions can take matters into their own hands, by specifying increased parallelism via copts. However, this requires manual analysis of builds to identify which targets to optimize.
>
>
> In my investigation, Xcode uses a constant -j8 (on my machine) for all module compilation. Xcode appears to dynamically control the number of parallel actions, but all swiftc actions use -j8. By manually specifying -j8 for some targets, the Bazel build is almost on par with Xcode. Manually curating which modules to hard code increased parallelism is a maintenance we'd rather avoid.
>
>
>
> Note that this also affects the tail end of the build. Even if your build graph has no interior bottlenecks, the the last target(s), with few or no reverse dependencies, can also benefit from "borrowing" Bazel's unused parallelism, to complete the build sooner.
>
>
> In the absence of Bazel support for sharing parallelism, I haven't seen any documentation or tooling to support finding and maintaining optimal settings for builds that have these outer/inner parallelism complexities.
>
>
>
> Has action parallelism has discussed for Bazel? There doesn't seem to be anything on the roadmap related to this, are no other languages or rules finding this balance tricky?
>
>
> thanks,
> Dave
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
>

> To unsubscribe from this group and stop receiving emails from it, send an email to bazel-...@googlegroups.com.

[Dave Lee]

I've followed up with a Github issue: https://github.com/bazelbuild/bazel/issues/10443