How to use system library

[lika...@gmail.com]

I am converting one of my projects from using CMake to bazel. The project uses FindBoost to lookup boost libraries and generates the correct CFLAGS and LDFLAGS/LIBS to compile and link to it. It supports finding custom boost library by providing BOOST_ROOT in command line.

How do I achieve similar thing in bazel? Boost is too large so I can't possibly include it into my source tree. new_http_archive is still overkill because it requires everyone with whom I shared the project download hundreds of megabytes.

Another possible way is setup a local repository one-time, just as https://github.com/mzhaom/trunk/blob/master/third_party/boost/BUILD, but people may prefer different locations to hold the local repository, therefore WORKSPACE file will get conflict when shared.

Another thing I thought was to create some cc_library's where srcs=[] but linkopts are things like [-lboost_system]. I never tried this approach, but this looks to be smallest overhead to me. Since I would like to support custom built boost, I'd like to have linkopts to be something like [-L$(BOOST_ROOT)/lib -lboost_system], but does cc_library rule supports environment variable in linkopts, include, hdrs, etc? I don't see any BUILD Encyclopedia covers such use case, so I'd like to confirm.

Thanks please advice.

Another thing was about libraries that support pkg-config, is there a way to generate cc_library target by running pkg-config and retrieve the correct compile and link flags?

[Damien Martin-guillerez]

Hello Likan,

Unfortunately, we have no way to do that easily for now and we are working on making that kind of autoconfiguration possible. A current workaround for now is to generate a WORKSPACE file using a custom shell script that discover the location of the boost library.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/ebc6ac51-d9b8-4c06-83b7-10a4677a6773%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Damien

[lika...@gmail.com]

Thanks Damien for confirming it, is there a thread discussing about the work you mentioned (making that kind of autoconfiguration possible)? I'd like to follow and get some idea on what will happen. And is there some roadmap or timeline for this? I probably need to estimate how long I need to keep any hacky work around. For example, if the thing won't happen in near term, I probably need to invest more work on the workaround to make it better to use.

[lika...@gmail.com]

Damien,

Can you give more details on how the workaround you suggested? It is ambiguous to me. There are three possible ways:

1. Generate a WORKSPACE file containing local_repository (or new_local_repository) that points to a local directory of boost source tree and some BUILD files.

This may work for boost as it may not require complicated configure step, but won't work nicely for other libraries. For example, it is difficult to make bazel to build even glog, gflags, etc. because they require running autoreconf and/or configure. The configure script generates installed header files including config.h, which defines macros indicating whether certain functions are available on the build machine, or whether the function behaves differently, or whether certain third-party libraries can be found, or lots of other hacks specific to build machine.

I saw in mazhaom's github repository he checked in all headers generated by configure, but that won't work nicely if it is shared between developers on different machines, where the config.h may vary. His current approach is to put these generated files under different directory and uses `config_settings` and `select` to choose, but that only works for things like x86_64 vs x86, not for whether a library is available or a function is available, and won't scale if some day bazel supports select by whether a library or a function is available.

I considered making genrules to run autoreconf and configure, but autoreconf generates configure in the source directory, which I guess won't work with bazel. Also it is not very elegant to cd to source directory as bazel doesn't provide any make variable like SOURCE_DIR. Moreover, there is no way to glob the generated .h and .cpp files so it has to be listed in BUILD, such a list has to be re-generated and maintained when the library is updated. Even if all thsse things can be worked around or tolerated, the major issue is it requires deep understanding of the third-party library, and every library requires different hacks, which doesn't really scale.


2. Generate WORKSPACE file to define make variables such as BOOST_CFLAGS, BOOST_LDFLAGS, etc. so in BUILD I can simply define cc_library(...., copts=$(BOOST_CFLAGS), linkopts=$(BOOST_LDFLAGS), ...) but I don't see any document saying it is possible to define make variable in WORKSPACE.


3. Generate BUILD file by some script. The script generates BUILD targets such cc_library(..., linkopts=["-L/path/to/boost/lib", "-lboost_system", "-Wl,-rpath=/path/to/boost/lib"], ...). The problem for this approach is, linkstatic is not going to work.


Any suggestions? Thanks.

On Friday, June 5, 2015 at 9:15:52 AM UTC-7, Damien Martin-guillerez wrote:

[Ming Zhao]

You can run autoconf with genrule, that's not a problem. autoconf is
relocatable so you can just run configure within the GENDIR so the
generated headers will just live inside gendir, a cmd may look like
this(I haven't verified it so it might need some change):

cmd = "TOPDIR=$$PWD; cd $(@D); $$TOPDIR/$(location :configure)"

But I strongly suggest you not doing that because of the following issues:
1. It introduces significant amount of overhead into the build process
because usually configure is slow, and it's on the critical path.

2. You start losing the "reproducible" attribute of bazel. bazel
defined a very high bar for "reproducible", it means for a given state
of the source tree and bazel options(-c dbg, etc), it should always
produce the same output. But that does come for free. It means bazel
has to be able to track all the inputs that may affect the outputs, it
also means for the command run by bazel, it should produce the same
output if the arguments remain the same.

So in the case of using autoconf, the assumption is the configure
results that will affect bazel build process should be stable no
matter where it runs, so that it can be checked into the source tree.
The generated header might vary depends on targeted cpu, in that case,
multiple headers should be generated and selected using select
function.

In our company's use case, all the sources(and precompiled libraries,
only .a not .so) are checked into the source tree, and we ensure all
the devlopers use the same version of compiler, that guarantees each
developer's workstation can be used the produce the same binary. The
benefit of doing that is the final binary produced only depends
dynamic libraries from glibc and libstdc++, which makes the server
side deployment very convenient because you don't have to worry about
dynamic libraries dependency(simply because you don't use any).

> To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/3e99be6a-4341-4302-a211-490e59191027%40googlegroups.com.

[lika...@gmail.com]

autoreconf and configure are separate. Autoreconf is not relocatable and always generates configure in the source tree, while bazel will not look into the source tree to find `outs`.

Configure script can't be run inside genrule, because genrule clears all the environment. Even if I run `CC=$(CC) configure`, it is not going to work either because gcc requires PATH to be set to find ld, and there is no option to let gcc know where ld is, put aside the fact $(LD) is not recognized by bazel. Also $(location :configure) is not going to work unless you create a target named "configure".

So there are tons of issues to be fixed to be able to do this (or it will never happen because bazel doesn't think it is the use case).

I agree with you that bazel works well for corp environment where everyone has the same environment setup, such as Google. Looks like by design, bazel is not going to work well for open source projects where contributors have different environment setup.