How to make "Bazel run" and "Bazel test" (more) hermetic?

[Konstantin Erman]

As part of our C++ build on Linux we download (http_archive) several external shared libraries (.so) that carry dependency on libstdc++.so. We have custom clang toolchain and fully control which libstdc++.so is used to build our own binaries, but external ones carry dependency on /lib64/libstdc++.so

As a result, the build is pretty hermetic, but when we try to run tests the outcome very much depends on what each particular machine has installed at /lib64 and if the version is not fresh enough we get bogus test failures. Therefore the question: how to make Bazel run and test commands ignore the actual sysroot and instead use the one we supply as another http_archive. 

I learned about LD_LIBRARY_PATH and it seems to help, but unfortunately, I only could make it work with the absolute path to the libraries folder, which is different from machine to machine. I was hoping to specify the path relative to execroot, but could not make any form of the relative path work. 

Running the tests in isolation from the specific build machine configuration is very important and feels like a natural logical continuation of the hermetic build idea. 

Could somebody please advise on the recommended way to do it with Bazel?

Thank you!

Konstantin

[Herrmann, Andreas]

I can think of a few ways that might help to address this. Keep in mind that these will need to be adjusted to your setup and use-case.

- If you construct `LD_LIBRARY_PATH` at runtime, e.g. in a test script, then you can include `$PWD` to construct an absolute `LD_LIBRARY_PATH` that takes different execroot paths on different machines into account. E.g.
  ```
  LD_LIBRARY_PATH="$PWD/<relative_path>"
  ```
  This assumes that `<relative_path>` is valid within the Bazel execroot.

- When you fetch external binaries you can use [patchelf](https://github.com/NixOS/patchelf) to patch their `RUNPATH` or needed entries. E.g.
  ```
  RELPATH=$(realpath --relative-to=$(dirname <output-file>) <libdir>)
  patchelf --set-rpath "\$ORIGIN/$RELPATH:<other_rpath_entries>"
  ```
  The value `$ORIGIN` tells the dynamic loader to search relative to the binary, see `ld(1)`.

- Use [Nix](https://nixos.org/nix/) and [rules_nixpkgs](https://github.com/tweag/rules_nixpkgs) to pin all external dependencies including system libraries and tools instead of fetching external binaries using `http_archive`. In that case Nix would install external tools under paths like `/nix/store/<hash>-mytool` which would be the same across (similar) machines (paths would differ between, e.g., Linux and macOS).

I hope that helps.

Best, Andreas

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/3b2a56e6-3a84-4db9-b779-9a2674346aeb%40googlegroups.com.

[Konstantin Erman]

Thank you, Hermann! I will try it.