Creating and using Dynamic Libraries in Bazel

[Austin Hendrix]

By default, it appears that bazel creates static libraries and statically links those libraries into binaries. For most binaries that are fully-specified at build time, this is fine.

We're building a binary (cc_binary) which loads dynamic libraries (also cc_binary) as plugins, and we're also building those plugins in bazel. The binary and the plugins both depend on a few libraries that provide common functionality.

Our common libraries are originally from an ecosystem when they were dynamically linked, and they have some global state that is expected to be shared at runtime. Unfortunately by statically linking multiple plugins and the binary against this common library, we end up with separate copies of this global state in each cc_binary.

I think I want to compile my common libraries as shared libraries, but I haven't found any good documentation about how to do this. I've tried using a cc_binary (with linkshared=1) in the srcs of a cc_library, but this doesn't pass the headers, defines or linkopts through to the cc_library that other components are linking against.

Is there a way for force a cc_library to be linked as a dynamic library only, or another way to create a shared library that I haven't found yet?

Thanks,

-Austin

[Austin Hendrix]

After much aimless wandering through the source code for bazel, I've found that adding linking_mode_flags { mode: DYNAMIC } to our CROSSTOOL file allows bazel to do dynamic linking, and the linkstatic option in my cc_library rules seems to have some affect now.

Are there any caveats to enabling this option that I should be aware of?

Thanks,

-Austin

[Austin Hendrix]

After much continued tinkering, I'm still stuck on this.

I've found that with the previously mentioned updates to my CROSSTOOL, I can specify `linkstatic = 0` in my cc_binary and get a dynamically linked binary, but I haven't found a way to specify within my cc_library that it may only be dynamically linked, or any way to propagate the dynamic linking requirement to cc_binary rules that depend on my library.

As I continue to think about this, I think the only binaries that need to be dynamically linked are those that depend on the plugin loader library.

Is there a way to specify dynamic linking in a cc_library, and have that inherited by all other cc_* rules that depend it?

Thanks,

-Austin

[Damien Martin-guillerez]

Hi Austin,

Sorry for taking so long to answer, most of the team was out for FOSDEM. And our C++ expert is on vacation.

I believe that simply adding "-llib" to the linker_flags part of the CROSSTOOL should work. The tricky thing is that this crosstool file is generated in Bazel and not easily editable. Maybe adding it to the linkopts would work. +Lukács T. Berki is also probably more compenent than I am to answer those question. 

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/7d61e258-3821-46f4-b408-f2ce2e7c542c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lukács T. Berki]

Hey there,

As the ersatz C++ expert, I'm afraid I don't have good answer to you; we don't have a good story on controlling what code goes into which dynamic library and which dynamic library is linked against which other one. Bazel is pretty much designed for static linking at this point. I'd love to change that, but it's a significant amount of design + coding work and we don't have time allocated for that yet.

George is struggling with something similar ( https://github.com/bazelbuild/bazel/pull/2249 ), but I don't have a nice principled solution to that, either :(

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/7d61e258-3821-46f4-b408-f2ce2e7c542c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN3hOS8O4RxAoLvhksksCb0eizywtL1qdxXe%3DJTF%3Da5Li3LKeA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Lukács T. Berki | Software Engineer | lbe...@google.com | 

Google Germany GmbH | Erika-Mann-Str. 33  | 80636 München | Germany | Geschäftsführer: Matthew Scott Sucherman, Paul Terence Manicle | Registergericht und -nummer: Hamburg, HRB 86891

[dav...@graphcore.ai]

I have been having this problem with bazel and tensorflow.

for reasons I'm not fully aware of, I need to link the protobuf library into my .so plugin statically, but our own libraries dynamically. there isn't enough control in bazel to do this. as described above, you can go fully dynamic (where protobuf is linked dynamically too, which causes some error in protobuf), or static, where it fails to link because it uses -lpoplar on the command line to trry to statically link our .so file. I can remove our library altogether, importing a header only rule for the interface. This links successfully but doesn't include a reference to our .so files in the final output.

for what it is worth, i have a workaroud, which is to soft link the dynamic library with a static library name (libxxx.a). This works on os/x, but i have not tried it elsewhere.

[Marcel Hlopko]

Thanks David for letting us know. I hope (and I'm working on it so :) Bazel will be able to express your use case smoothly soon.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/c1da38c8-a768-4de5-93a8-933b562d2be0%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-- 

Marcel Hlopko | Software Engineer | hlo...@google.com | 

[David Norman]

Thanks.  Please let me know when it is released.

cheers

---

From: Marcel Hlopko <hlo...@google.com>
Sent: 21 March 2017 13:41:56
To: David Norman; bazel-discuss
Subject: Re: Creating and using Dynamic Libraries in Bazel

 

[Marcel Hlopko]

Hi David,

would you mind taking a look (and commenting) on bazel transitive libraries design doc to make sure it fully supports your use case?

Thanks!

[dav...@graphcore.ai]

yes - i believe that is right.  the runtime_deps, as a way of introducing a runtime only dependency (presumably on our external .so file).   the link line then includes the full path to the .so file which was the dynamic target and so the correct .so loading tag is introduced to the ELF file?

or - -lpoplar and -L...path/to/poplar is included, for the same result.