Add rust_rmeta tool to support build pipelining

[Collin Baker]

Hi all, I want to propose adding a `rust_rmeta` tool type for toolchain definitions, and an associated target type such as `rust_metadata`.

The addition would allow us to experiment with Rust build pipelining: rlib targets may depend on a library's .rmeta instead of the complete .rlib, which in theory should be available earlier and unblock dependent build steps faster. This could enable more build parallelism even though the complete .rlibs are needed eventually (when linking a complete artifact like a binary).

Cargo uses pipelining, but it is tightly integrated with rustc. It asks rustc to pass messages over stdout for each output file, so it can call rustc once per library but start further invocations as soon as an .rmeta file is output. This approach unambiguously has build time benefits, but I don't think it's possible with ninja, or any make-like build system for that matter.

A build pipelining approach by invoking rustc twice per library may still have some benefit. The proposed GN change would enable such an experiment.

Does this sound reasonable?

[Aaron Wood]

Adding Tyler, David, and Erick, since this is something we're interested in within the Fuchsia build, as well.

To unsubscribe from this group and stop receiving emails from it, send an email to gn-dev+un...@chromium.org.

[Dirk Pranke]

It seems fine to work on something like that to me. I am not a Rust person, so I have no particular insight into these things, I'd defer to the people that actually know the tooling.

-- Dirk

[danakj]

As a person somewhat familiar with the tooling, I encouraged Collin to reach out here, and I think this is a good idea to try pursuing, and a good way to do so.

[Philipp Wollermann]

Hi Collin,

That sounds like an interesting idea to me. It reminds me of the "Java header compilation" that Bazel supports, which was an improvement over the previous "interface jar" approach. It also results in a higher overall CPU time spent on the build, but reduces wall time[1] by allowing more parallelization of build actions and can reduce unnecessary recompilation during incremental builds (as we only need to recompile something if the public API of a dependency changed).

[1] If the metadata / header compilation step is faster than the actual compilation step and you have enough compute capacity to actually do all that work in parallel.

There's an old, detailed design doc about Java header compilation that might still be interesting to read. I think it was never published externally, but Googlers feel free to ping me and I can send you the link and also connect you with the folks who worked on it. (Or just search the intranet for "Java header compilation" :))

Cheers,

Philipp

[Alfred Zien]

Hi!

I just stumbled upon this thread. Could we make it a more generic solution? We're currently investigating a similar build pipeline for Swift, aiming to depend only on the interface, not on the compilation, as Apple does in xcode.

Thanks,

Alfred