Re: [protobuf-team] Protobuf and Abseil in Debian/Ubuntu

546 views
Skip to first unread message

Feng Xiao

unread,
Dec 20, 2017, 1:58:43 PM12/20/17
to Josh Haberman, László Böszörményi, abse...@googlegroups.com, Protocol Buffers
+abse...@googlegroups.com, +prot...@gogolegroups.com

CCed public google groups so László can reply to the group. BCCed internal mailing lists.

On Tue, Dec 19, 2017 at 8:24 PM, Josh Haberman <habe...@google.com> wrote:
Hi all,

I wanted to kick off a conversation about the future of Protobuf and Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the Protobuf package, and has also expressed interest in packaging Abseil.

Debian currently distributes a binary shared library for protobuf (libprotobuf.so). Debian policy requires that the SONAME is bumped every time there is an ABI-incompatible change to the library:


Since protobuf does not guarantee any kind of binary compatibility, we have traditionally updated our SONAME for every minor release of protobuf. We are currently at major version 15:


This has worked reasonably well so far. But we want to start depending on Abseil (https://abseil.io) which does not do releases at all (so doesn't even have a major version number to track). How would this work for Debian?

Abseil's philosophy is that users will build from source. But Debian is inherently a binary distribution. We have to figure out how to reconcile these two philosophies. I don't know the answer, but I'm going to throw out a few straw man ideas as a starting point for discussion.

1. Distribute libprotobuf and libabsl as static libraries only.

Debian policy says that libraries can be distributed in static form only if that is what is intended by upstream:


Since both libprotobuf and libabsl are disavowing any binary compatibility guarantees, that implies that they should not be distributed as .so's. Any Debian packages or user programs that want to use protobuf have to link it statically.

The tricky part is that libprotobuf.a and libabsl.a would need to match each other exactly. They would need to be compiled from exactly the same ABSL headers. This implies that libprotobuf-dev would need to pin itself to one specific libabsl-dev version -- the one it was compiled against. Other libraries that depend on ABSL (like TensorFlow) would also need to be pinned to exactly the same libabsl-dev version, otherwise it would be impossible to install them all at the same time. So realistically all packages that depend on ABSL would need to be updated at the same time.

2. Distribute libprotobuf and libabsl as shared libraries.

This is against the philosophy of ABSL. But if Debian updated its ABSL infrequently enough, it could just treat each ABSL update as a separate major version. Sort of like we do with protobuf now.

The tricky part is if two different libraries pass ABSL types across library boundaries. For example, imagine that a user program was developing against libabsl.so.2 but also using libprotobuf.so.15 that was linked against libabsl.so.1. If a user got an absl::string_view from protobuf, this could be unsafe for the user to pass directly to ABSL functions. The representation of absl::string_view could have changed between the two versions of libabsl.

The all-static option (1) seems to solve this problem by pinning all of the ABSL dependencies so that they all rev together. But this solution could become unwieldy if too many packages depend on ABSL. Also any library that exposes ABSL types in its API would have to be part of this lock-step ABSL upgrade.

3. Remove protobuf from Debian completely, ask users to build from source.

Since the philosophy of ABSL is that users will always build from source, we could ask that users do this if they want to develop against protobuf. Debian packages that depend on protobuf could build protobuf (and ABSL) and statically link against it to build runnable binaries. It would not be possible to have library packages in Debian that depend on protobuf.

Thoughts?
Josh

Jisi Liu

unread,
Dec 20, 2017, 2:24:02 PM12/20/17
to Feng Xiao, Josh Haberman, László Böszörményi, abse...@googlegroups.com, Protocol Buffers
There's also header dependency issue, i.e. libprotobuf-dev may distribute headers which may transitively include headers from libabseil-dev (currently we duplicate atomicops, platform macros, etc). This means the protobuf dev package needs to pin with a specific libabseil-dev, even if we go with the shared library approach. One way to solve the problem is avoid transitive include of abseil headers in protobuf headers. 

Josh Haberman

unread,
Dec 20, 2017, 2:26:09 PM12/20/17
to Feng Xiao, László Böszörményi, abse...@googlegroups.com, Protocol Buffers
Thanks Feng and sorry for the confusion everyone!

Feng quoted my original email, but I'll repeat it here for clarity. Hopefully everyone will be able to reply to messages properly this time. :)

--

Feng Xiao

unread,
Dec 20, 2017, 2:40:39 PM12/20/17
to Josh Haberman, László Böszörményi, abse...@googlegroups.com, Protocol Buffers
On Wed, Dec 20, 2017 at 11:25 AM, Josh Haberman <habe...@google.com> wrote:
Thanks Feng and sorry for the confusion everyone!

Feng quoted my original email, but I'll repeat it here for clarity. Hopefully everyone will be able to reply to messages properly this time. :)

--

Hi all,

I wanted to kick off a conversation about the future of Protobuf and Abseil/ABSL in Debian/Ubuntu. Laszlo is the Debian maintainer for the Protobuf package, and has also expressed interest in packaging Abseil.

Debian currently distributes a binary shared library for protobuf (libprotobuf.so). Debian policy requires that the SONAME is bumped every time there is an ABI-incompatible change to the library:


Since protobuf does not guarantee any kind of binary compatibility, we have traditionally updated our SONAME for every minor release of protobuf. We are currently at major version 15:


This has worked reasonably well so far. But we want to start depending on Abseil (https://abseil.io) which does not do releases at all (so doesn't even have a major version number to track). How would this work for Debian?

Abseil's philosophy is that users will build from source. But Debian is inherently a binary distribution. We have to figure out how to reconcile these two philosophies. I don't know the answer, but I'm going to throw out a few straw man ideas as a starting point for discussion.

1. Distribute libprotobuf and libabsl as static libraries only.
If protobuf is static only, could other .so libraries still link against protobuf? If multiple .so libraries has linked-in protobuf statically, would it cause symbol conflicts when they are used together? It seems to me if protobuf goes static only, every other library that depends on protobuf must be static only as well.
 

Debian policy says that libraries can be distributed in static form only if that is what is intended by upstream:


Since both libprotobuf and libabsl are disavowing any binary compatibility guarantees, that implies that they should not be distributed as .so's. Any Debian packages or user programs that want to use protobuf have to link it statically.

The tricky part is that libprotobuf.a and libabsl.a would need to match each other exactly. They would need to be compiled from exactly the same ABSL headers. This implies that libprotobuf-dev would need to pin itself to one specific libabsl-dev version -- the one it was compiled against. Other libraries that depend on ABSL (like TensorFlow) would also need to be pinned to exactly the same libabsl-dev version, otherwise it would be impossible to install them all at the same time. So realistically all packages that depend on ABSL would need to be updated at the same time.

2. Distribute libprotobuf and libabsl as shared libraries.

This is against the philosophy of ABSL. But if Debian updated its ABSL infrequently enough, it could just treat each ABSL update as a separate major version. Sort of like we do with protobuf now.

The tricky part is if two different libraries pass ABSL types across library boundaries. For example, imagine that a user program was developing against libabsl.so.2 but also using libprotobuf.so.15 that was linked against libabsl.so.1. If a user got an absl::string_view from protobuf, this could be unsafe for the user to pass directly to ABSL functions. The representation of absl::string_view could have changed between the two versions of libabsl.
I'm not so worried about this because this problem is not new. With or without absl, there are libraries depending on different versions of protobuf.so and there must already exists an established solution to deal with this. Adding absl to the picture won't make it any trickier than it is before.
 

The all-static option (1) seems to solve this problem by pinning all of the ABSL dependencies so that they all rev together. But this solution could become unwieldy if too many packages depend on ABSL. Also any library that exposes ABSL types in its API would have to be part of this lock-step ABSL upgrade.

3. Remove protobuf from Debian completely, ask users to build from source.
It seems very unlikely we can take down all the protobuf.so distributed on the internet... We can make it harder but people will get a way around it.
Reply all
Reply to author
Forward
0 new messages