Bazel C++ rules cause major overbuild for incremental builds

[Konstantin]

Here is an example of the problem:

Imagine we have a folder with 1000 C++ source files and 1000 C++ header files. Each source includes just a few headers, not all 1000 of them.

BUILD file:

cc_library(

    name = "lib",

    srcs = glob(["*.cpp"]),

    hdrs = glob(["*.h"]),

)

Change in any header file causes recompilation of all 1000 sources although only a few actually include the changed header. Bummer.

What options do we have?

Headers scanning?

unused_inputs_list on ctx.actions.run?

Existing build system (CMake + Ninja) uses ".d" files generated by the compiler to build more precise dependency graph and only rebuild the sources which include the changed header. 

Help?

Konstantin

[Konstantin]

Further experiments discovered that on Linux Bazel C++ rules do the right thing and only recompile sources which include changed header, but on Windows all sources are recompiled. We really need to achieve the same behavior on Windows.

[Filip Filmar]

On Tue, Mar 28, 2023 at 8:14 PM Konstantin Erman <kon...@ermank.com> wrote:

Yes, potentially we can split cc_library to 1000 separate libraries with one object file in each (ugly!), but then we would need to specify “hdrs” for each of those libraries individually with only what that source actually includes (transitively!) which does not look like an easy feat at all ☹

You could auto-generate the BUILD file. It might be laborious, but on the flip side ostensibly you do the bulk of the generation once, and reuse the result many times. So the amortized cost seems low.

This still seems like an improvement over globbing thousands of files together.

You could also try to factor out most frequently changed files first, and then maybe handle the long tail, or maybe not.

F

[Konstantin Erman]

Unfortunately it does not look like “generate once – use many times”. As soon as there is any change in what sources include what headers that BUILD file needs to be regenerated. It is a hindrance, but more importantly I don’t see a way to produce transitive list of included headers for each source file shorter than compile that source.

[Lukács T. Berki]

+Yun (the person who knows most about Windows)

This looks like https://github.com/bazelbuild/bazel/issues/14947 .

Yun, what are the chances of being able to take a look at this soon-ish?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/BY5PR01MB563524D9B9AD1FF23EF556F6A3899%40BY5PR01MB5635.prod.exchangelabs.com.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[Yun Peng]

Hi Konstantin,

Feel free to submit a PR to fix this, the change in the issue description looks good to me.

After this is merged, we can mark it for cherry pick for Bazel 6.2 and hopefully it will unblock your migration.

Cheers,

Yun

[Konstantin]

Could somebody please merge 17928 ?

Thank you!

Konstantin