How to debug 'missing dependency declarations'

[James Philbin]

Hi,

I'm trying to debug an 'undeclared inclusion' error. AFAICT all the header dependencies are in fact declared. So how can I:

- Print out which headers a rule actually thinks are declared as an inclusion.

- Turn off the inclusion check for a rule so I can at least make forward progress for now.

Thanks!

James

[Brian Silverman]

Unfortunately, the best way I've found to do is by setting VALIDATION_DEBUG in src/main/java/com/google/devtools/build/lib/rules/cpp/CppCompileAction.java to 2 and recompiling from source. That will give you lots of information about what files Bazel thinks are being #included and which ones are allowed.

In my experience, building with sandboxing makes debugging this kind of issue much easier because you get a "include file not found" error from the C++ compiler and --sandbox_debug will show you all of the inputs which are allowed. Is there some reason you don't have sandboxing?

Bazel does not have a way to turn off this checking because skipping the check would result in incorrect incremental rebuilds.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/dbf03c1e-b7a0-4214-a280-b516e5933cf8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Philbin]

So in this case (bizarrely), the compile works without problems when
sandboxing is switched on. Only when sandboxing is switched off do we
get errors. I've managed to trace it down to an incorrect path being
generated by nvcc in the depfile (.d) of a CUDA target. All paths are
relative to the bazel output_base except for two which nvcc has for
some reason generated the full path for (starting
/home/$USER/.cache/bazel/...). One thing I could do is rewrite the
depfile in my crosstool wrapper although it feels pretty gross.

Thanks,
James

[Brian Silverman]

Those paths would be the problem. I bet the compiler is finding a shorter path by not using the symlinks, and the sandbox gets built up without those so it's a non-issue there.

Newer GCC has that exact behavior without -fno-canonical-system-headers (which is specified in newer Bazel CROSSTOOLs); maybe that flag will work for nvcc too or you can get it to pass that flag through to the host compiler somehow?

[James Philbin]

Good call, that fixed it! What a strange default behavior from gcc/nvcc...

If you're wondering why we're not using sandboxing, we've actually found that the overhead is very large (total compile times are ~2.2x slower with sandboxing enabled). It would be great if that could be improved!

Thanks,

James

[Austin Schuh]

Hi James,

Do you have a repository and/or environment you can share with the bazel developers to help identify the slowdown with sandboxing?  We are seeing the same issue, and Phil has been having trouble reproducing it.

Austin

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/50960f99-fec9-4c22-ab5e-88c990bab568%40googlegroups.com.

[James Philbin]

Unfortunately not: this is an internal (and fairly large) code base.

When I turn on sandbox_debug I do see an *enormous* number of files
getting mounted, so i'm not surprised that we may be hitting
performance problems. It occurs to me that one optimization may be to
mount at the directory level rather than individual files just to
reduce the number of mount points: consider an external depedency like
boost - it has something like 10K header files which will be each
individually mounted for every target that uses that library.

Thanks,
James

[Brian Silverman]

Austin and I have a similar issue (~3K files from C++ toolchains which are getting mounted). Mounting whole directories is complicated because you don't always want all the files.

However, now that I think about it, most of those files are in directories with everything included, so that would be a good optimization... For your situation, are most of the files in directories which are completely mounted?

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAA43iPKf7NF9w3ogUFhisuVMeJZfXDs6GGAMMUPmz366UQn6Rw%40mail.gmail.com.

[James Philbin]

Yes, typically they are 'external' dependencies installed through
apt-get and referenced using a shim in the WORKSPACE like:
===
...
new_local_repository(
name = "boost_repo",
path = "/usr/",
build_file = "build/BUILD.boost",
)
...
===

=== BUILD.boost ===
cc_library(
name = "boost",
srcs = [
"lib/x86_64-linux-gnu/libboost_date_time.so.1.54.0",
"lib/x86_64-linux-gnu/libboost_iostreams.so.1.54.0",
...
],
hdrs = glob([
"include/boost/**/*.hpp",
]),
linkstatic = 1,
)
===

So I believe that if we changed hdrs to equal
glob(["include/boost/**"]) then it would be safe to mount the whole of
'/usr/include/boost/' - is that correct?

Thanks,
James

On Tue, Mar 15, 2016 at 10:27 PM, Brian Silverman

[Brian Silverman]

'include/boost/**/*' and 'include/boost/**' should both be fine for what I'm envisioning. I might look into implementing that in the next few weeks, because it sounds fairly simple and potentially very effective.

[Austin Schuh]

It would be safe to mount the directory if bazel was clever enough to track it.

Austin

[James Philbin]

Sounds great! Once you have something working let me know and I can test internally.

Thanks!

James