Private dependencies in cc_library?

[Magnus Andersson]

The cc_library(...) and cc_binary(...) rules only have one dependency list (the "deps" parameter). Some other build systems have two dependency lists - one public and one private. The public dependencies are the dependencies for the public .h-files. The private dependencies are the dependencies for private .h-files and .c/.cpp-files. The benefit with private dependencies is that .h-files that only your implementation needs are not exposed to libraries that depend on you. Or to put it in another way: Private dependencies only adds -I flags when you build your library. Public dependencies adds -I flags when you build your library, and everything that depends on your library.

One example of a build system that supports private dependencies is GN, the build system in Chromium (which generates ninja files from BUILD.gn files which are very similar to Bazel BUILD files).

What is the reason why Bazel lacks private dependencies? Are there any plans to add this feature?

[Luis Fernando Pino Duque]

+hlopko, plf

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/04b2c790-6629-4e18-9036-4978c360490a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Luis F. Pino | Software Engineer | lp...@google.com 

[Marcel Hlopko]

To express that a header is private we put it into srcs. That signals layering check that other libraries should not include them. Layering check currently depends on C++ modules, and I'm not aware of any external project that uses them unfortunately.

For Bazel, cc_library is only a bunch of sources that it compiled. Depending on the toolchain we create a static library from those objects, sometimes also shared library. But sometimes just objects. It's the cc_binary that links everything into an executable or shared library. So whereas private deps would make sense for headers, and therefore compilation, they won't make sense for linking.

Were you talking about deps attribute in gn? That looks like it is doing the same thing, propagating link dependency, but not propagating headers.

It can also be that I'm just obtuse after whole day of work :)

On Tue, May 29, 2018 at 6:58 PM Luis Fernando Pino Duque <lp...@google.com> wrote:

+hlopko, plf

On Tue, May 29, 2018 at 2:57 PM, Magnus Andersson <magnus.a...@gmail.com> wrote:

The cc_library(...) and cc_binary(...) rules only have one dependency list (the "deps" parameter). Some other build systems have two dependency lists - one public and one private. The public dependencies are the dependencies for the public .h-files. The private dependencies are the dependencies for private .h-files and .c/.cpp-files. The benefit with private dependencies is that .h-files that only your implementation needs are not exposed to libraries that depend on you. Or to put it in another way: Private dependencies only adds -I flags when you build your library. Public dependencies adds -I flags when you build your library, and everything that depends on your library.

One example of a build system that supports private dependencies is GN, the build system in Chromium (which generates ninja files from BUILD.gn files which are very similar to Bazel BUILD files).

What is the reason why Bazel lacks private dependencies? Are there any plans to add this feature?

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/04b2c790-6629-4e18-9036-4978c360490a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Luis F. Pino | Software Engineer | lp...@google.com 

--

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

[Magnus Andersson]

Sorry, my question was too vague. Let me try to clarify it.

Say that we have the following libraries:

cc_library(
     name = "libA",
     srcs = ["a.c"],
     deps = [":libB"],
)


cc_library(
    name ="libB",
    hdrs = ["b.h"],
    srcs = ["b.c"]
    deps = [":libC"],
)


cc_library(
    name = "libC",
    hdrs = ["c.h"],
)

If "b.h" includes "c.h", I call it a public dependency. The dependency is needed both when compiling "libB", and all libraries that depend on it ("libA" in this case).

If "b.c" includes "c.h", I call it a private dependency. The dependency is only needed when compiling "libB".

So what is the point in supporting private dependencies? The answer it that unnecessary -I flags will be passed to the compiler. You can include "c.h" from "a.c" and build. The dependency from "libB" to "libC" was only needed to compile "libB"s sources, but it leaked to "libA" as well.

I know that the recommendation is to only use direct dependencies. Then "libA" would not be allowed to use "libC" without an explicit dependency to "libC". But since Bazel does not check that this recommendation is enforced, users can include headers that they are not supposed to see by mistake.

As to my comparison to GN: If I have understood it correctly, libraries in GN has both "public_deps" and "deps" parameters. "public_deps" are public dependencies, which are used both for compiling and linking. "deps" are private dependencies, which only are used for linking.

[George Gensure]

Interface deps were previously available in bazel. Since I have similar needs to control header inclusion, I've incorporated changes in this branch: https://github.com/werkt/bazel/tree/impl-deps into my builds. That branch is up to date with master as of today.

Per your example, with that branch and the use of impl_deps as shown below in libB, an inclusion by a.c of c.h will result in a file not found under sandboxing/remote execution, and a header inclusion rule check violation in standalone.

cc_library(
     name = "libA",
     srcs = ["a.c"],
     deps = [":libB"],
)


cc_library(
    name ="libB",
    hdrs = ["b.h"],
    srcs = ["b.c"]

    impl_deps = [":libC"],

)


cc_library(
    name = "libC",
    hdrs = ["c.h"],
)

This is also a part of the larger discussion around control for C++ linking and has gotten a little less attention than the linking control, but was still critically important for my group, as we had an out of control header problem at one point.

Hope that helps, lmk if you use it, I'll get around to putting up a PR!

-George

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/a43e6303-e1ef-4c70-8acb-6924abe7fe4b%40googlegroups.com.

[Magnus Andersson]

Interesting!

I think that this feature is required for my group to accept Bazel. Our users have spent quite some time to get all dependencies right in our current build system (a makefile generator which reads build configuration files that support private/implementation dependencies).

/ Magnus

[Magnus Andersson]

"Interface deps were previously available in bazel."

Do you know why interface deps were removed from Bazel?

Den tisdag 29 maj 2018 kl. 22:00:38 UTC+2 skrev George Gensure:

[Marcel Hlopko]

+Manuel Klimek 

+Ilya Biryukov 

+Ulf Adams 

This is the cl which removed them: https://github.com/bazelbuild/bazel/commit/27ea5f8a71fe5be350466a817ad5afce2f8500f9

The idea was that bazel can compute the list of private deps automatically. But today that only works internally because:

* we have include scanning which scans headers and computes the list of used headers (this is only slightly related because we could get to that information from .d files too if we tried hard and didn't care about performance).

* we have modules enabled

* we have layering_check

It's unfortunate that so much of this is missing externally, but there's ongoing work in this area:

* ulfjack@ has been decoupling include scanning from our internal remote execution system (they were very strongly coupled).

* klimek@ and ibiryukov@ have been working on providing a clang bazel toolchain that will ideally support modules.

Feel free to open a feature request to track this, or comment on this doc: https://docs.google.com/document/d/1WW4iY3-74DVSX9wD3TiznrUIs7vybrxkj1U22JmWIrI/edit (although it's a bit dated). 

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/07a58bee-892c-431f-a420-de5b95585e9d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

[Ulf Adams]

Hi Marcel,

I'm afraid I don't understand how this relates to include scanning. Can you elaborate?

Personally, I am strongly in favor of open sourcing our current include scanning code, even if only so that Bazel contributors can see what it's doing. If it's actually useful, then that's all the more reason for open sourcing it.

Thanks,

-- Ulf

[Magnus Andersson]

Hi Ulf,

A side question about include scanning:

In one of your talks you said that the (or at least a) purpose of include scanning was to minimize the number of headers that need to be copied to the remote host, when using remote execution.

Does the current remote execution implementation copy all public headers transitively to the remote host? Then I would say that include scanning is not only useful, but required for us to get decent build performance with remote execution. We have long dependency graph branches, and ususally only a fraction of all these .h-files are necessary to compile a specific .c-file.

/ Magnus

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/07a58bee-892c-431f-a420-de5b95585e9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ulf Adams]

On Mon, Jun 4, 2018 at 11:01 AM Magnus Andersson <magnus.a...@gmail.com> wrote:

Hi Ulf,

A side question about include scanning:

In one of your talks you said that the (or at least a) purpose of include scanning was to minimize the number of headers that need to be copied to the remote host, when using remote execution.

Does the current remote execution implementation copy all public headers transitively to the remote host?

That's correct.

 

Then I would say that include scanning is not only useful, but required for us to get decent build performance with remote execution. We have long dependency graph branches, and ususally only a fraction of all these .h-files are necessary to compile a specific .c-file.

The protocol cost per file is fairly low; include scanning may be more expensive than just posting the additional files in the protocol (assuming the remote executor caches input files so you don't have to upload the files all the time) until you get to a very large number of files.

I don't know where the crossover point is that makes include scanning a net win. It is possible that very small builds would already benefit. There are also some additional drawbacks of include scanning including higher local CPU utilization as well as more expensive locally clean, remotely cached builds. Our include scanning implementation is also not perfect, and sometimes requires "include hints" to find all the necessary files.

As an example from Google, we have cases where include scanning reduces the number of inputs to the protocol from ~hundreds of millions to ~tens of millions for a build of a single target (not a single action).

I'll also say that we initially didn't want to open source include scanning because we were hoping to be able to remove it while still maintaining good performance. None of those plans have panned out, however, and I currently believe that include scanning is here to stay.

/ Magnus

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/07a58bee-892c-431f-a420-de5b95585e9d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Marcel Hlopko | Software Engineer | hlo...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Geschäftsführer: Paul Manicle, Halimah DeLaine Prado | Registergericht und -nummer: Hamburg, HRB 86891

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAFXA0Af6TcEo4jQBkoomvc19T%3D7bJ4H-n36fGvKiMgrDOoa8nw%40mail.gmail.com.

[Manuel Klimek]

+djasper 

On Mon, Jun 4, 2018 at 6:46 AM Ulf Adams <ulf...@google.com> wrote:

On Mon, Jun 4, 2018 at 11:01 AM Magnus Andersson <magnus.a...@gmail.com> wrote:

Hi Ulf,

A side question about include scanning:

In one of your talks you said that the (or at least a) purpose of include scanning was to minimize the number of headers that need to be copied to the remote host, when using remote execution.

Does the current remote execution implementation copy all public headers transitively to the remote host?

That's correct.

 

Then I would say that include scanning is not only useful, but required for us to get decent build performance with remote execution. We have long dependency graph branches, and ususally only a fraction of all these .h-files are necessary to compile a specific .c-file.

The protocol cost per file is fairly low; include scanning may be more expensive than just posting the additional files in the protocol (assuming the remote executor caches input files so you don't have to upload the files all the time) until you get to a very large number of files.

I don't know where the crossover point is that makes include scanning a net win. It is possible that very small builds would already benefit. There are also some additional drawbacks of include scanning including higher local CPU utilization as well as more expensive locally clean, remotely cached builds. Our include scanning implementation is also not perfect, and sometimes requires "include hints" to find all the necessary files.

As an example from Google, we have cases where include scanning reduces the number of inputs to the protocol from ~hundreds of millions to ~tens of millions for a build of a single target (not a single action).

I'll also say that we initially didn't want to open source include scanning because we were hoping to be able to remove it while still maintaining good performance. None of those plans have panned out, however, and I currently believe that include scanning is here to stay.

We also use include scanning to prune dependencies for module builds.

[fredrik...@gmail.com]

This issue seems to relate to https://github.com/bazelbuild/bazel/issues/2670, "Add an option to `cc_*` rules that lets one specify include directories only for the current rule". That issue suggests adding `impl_includes`.

What is the status of the include scanning? Is `impl_deps` the way to go? The implementation of `impl_deps` seems to be ready in https://github.com/werkt/bazel/commit/06f81cd32265ad0ca1f0010266ac038cabdf7521.

Will any of the solutions be part of https://github.com/bazelbuild/bazel/issues/4570, "Implement C++ Skylark API"?

Best regards,
Fredrik

[Peter Henell]

I also think that impl_deps would be very useful to get into bazel.