The linking of libstdc++ (again, sorry)

[Manuel Klimek]

Hi builders,

one thing that came up in the manylinux discussions, but I don't think I fully understand yet, is the requirement to link to the system libstdc++.

Thus, I wanted to ask what people think would be the main problems if TF didn't link to the system libstdc++ in the pip package, and e.g. statically link it, as that would solve a ton of problems for us regarding toolchain flexibility, and given that we don't support things outside of our control calling our C++ API anyway, the question is what the downsides would be.

Does anybody know of something that would break?

Thanks!

/Manuel

[Mihai Maruseac]

OSSFuzz links against libc++ so we cannot statically link libstdc++
directly if we want to not break that.

On the other hand, OSSFuzz doesn't use the pip package, so this is
only a datapoint against linking libstdc+ statically _everywhere_.

> --
> You received this message because you are subscribed to the Google Groups "SIG Build" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to build+un...@tensorflow.org.
> Visit this group at https://groups.google.com/a/tensorflow.org/group/build/.



--
"Problems worthy of attack
Prove their worth by fighting back"
Piet Hein

[jwa...@redhat.com]

Yes.

I don't think this solves anything, does it?

If you statically link the pip package to an old (pre-gcc-5) libstdc++ and then try to use that pip package in an application that dynamically links to a newer (post-gcc-5) libstdc++.so then you have a mix of old and new libstdc++ symbols in the same process. Calls that should resolve to the new symbols in the dynamic library might actually use old statically linked symbols, which do something different, or expect a type to have a different layout.

Trying to use versions of GCC before 5.1 to compile C++11 code is not going to give you portable packages that can be used on systems with libstdc++ from a newer GCC. Linking statically doesn't change that as far as I can see.

If you need to use C++11 then my advice is to either stick to versions of GCC where the C++11 support is finished and has a stable ABI, or always recompile everything from source and don't expect binaries to be portable to different systems.

[Günhan Gülsoy]

Wanted to add some more information to Manuel's query.

For TF, we would like to use a newer compiler, but maintain compatibility to older systems. At the moment, we see two options. One is using CentOS plus devtoolset 7. The other option we see is statically linking libstdc++.

[Manuel Klimek]

On Wed, Mar 20, 2019 at 11:27 PM Mihai Maruseac <mihaim...@google.com> wrote:

OSSFuzz links against libc++ so we cannot statically link libstdc++
directly if we want to not break that.

If we're already having a mismatch in standard library, why would static linking change anything?

[Manuel Klimek]

On Thu, Mar 21, 2019 at 12:47 AM <jwa...@redhat.com> wrote:

On Wednesday, 20 March 2019 21:10:54 UTC, Manuel Klimek wrote:

Hi builders,

one thing that came up in the manylinux discussions, but I don't think I fully understand yet, is the requirement to link to the system libstdc++.

Thus, I wanted to ask what people think would be the main problems if TF didn't link to the system libstdc++ in the pip package, and e.g. statically link it, as that would solve a ton of problems for us regarding toolchain flexibility, and given that we don't support things outside of our control calling our C++ API anyway, the question is what the downsides would be.

Does anybody know of something that would break?

Yes.

I don't think this solves anything, does it?

If you statically link the pip package to an old (pre-gcc-5) libstdc++ and then try to use that pip package in an application that dynamically links to a newer (post-gcc-5) libstdc++.so then you have a mix of old and new libstdc++ symbols in the same process. Calls that should resolve to the new symbols in the dynamic library might actually use old statically linked symbols, which do something different, or expect a type to have a different layout.

The idea is to not expose the symbols of the statically linked libstdc++ - thus, TF code would always use the one it was statically linked with, and other code that links against the system one would use the system one.

This would break if we had calls across the boundary of code that is linked with the system lib and code that is statically linked, but I'm not aware of any of this for tensorflow.

 

Trying to use versions of GCC before 5.1 to compile C++11 code is not going to give you portable packages that can be used on systems with libstdc++ from a newer GCC. Linking statically doesn't change that as far as I can see.

I'm not sure what you mean by using versions of GCC before 5.1 - I want to be able to use *newer* GCC versions (or other compilers / C++ standard libraries), and deploy to systems with any libstdc++.

If we hide the libstdc++ symbols that are statically linked into TF, no other part of the app could find / reference them.

 

If you need to use C++11 then my advice is to either stick to versions of GCC where the C++11 support is finished and has a stable ABI, or always recompile everything from source and don't expect binaries to be portable to different systems. 

[Uwe L. Korn]

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Uwe

[jwa...@redhat.com]

On Thursday, 21 March 2019 10:32:38 UTC, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 12:47 AM <jwa...@redhat.com> wrote:

On Wednesday, 20 March 2019 21:10:54 UTC, Manuel Klimek wrote:

Hi builders,

one thing that came up in the manylinux discussions, but I don't think I fully understand yet, is the requirement to link to the system libstdc++.

Thus, I wanted to ask what people think would be the main problems if TF didn't link to the system libstdc++ in the pip package, and e.g. statically link it, as that would solve a ton of problems for us regarding toolchain flexibility, and given that we don't support things outside of our control calling our C++ API anyway, the question is what the downsides would be.

Does anybody know of something that would break?

Yes.

I don't think this solves anything, does it?

If you statically link the pip package to an old (pre-gcc-5) libstdc++ and then try to use that pip package in an application that dynamically links to a newer (post-gcc-5) libstdc++.so then you have a mix of old and new libstdc++ symbols in the same process. Calls that should resolve to the new symbols in the dynamic library might actually use old statically linked symbols, which do something different, or expect a type to have a different layout.

The idea is to not expose the symbols of the statically linked libstdc++ - thus, TF code would always use the one it was statically linked with, and other code that links against the system one would use the system one.

This would break if we had calls across the boundary of code that is linked with the system lib and code that is statically linked, but I'm not aware of any of this for tensorflow.

 

Trying to use versions of GCC before 5.1 to compile C++11 code is not going to give you portable packages that can be used on systems with libstdc++ from a newer GCC. Linking statically doesn't change that as far as I can see.

I'm not sure what you mean by using versions of GCC before 5.1 - I want to be able to use *newer* GCC versions (or other compilers / C++ standard libraries), and deploy to systems with any libstdc++.

You have the same problem if calls in the application that should resolve to symbols in the old system libstdc++.so actually resolve to statically linked symbols from the new libstdc++, and the symbols are not compatible.

If you ensure the statically linked symbols are not global that might work, if no C++11 types are used in module boundaries.

[Manuel Klimek]

On Thu, 21 Mar 2019, 11:39 Uwe L. Korn, <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Thanks! I thought that the only supported way to do this would be to build all of the C++ parts within the same bazel invocation, as afaiu TF doesn't make any guarantees about C++ ABI stability.

[Manuel Klimek]

Yes, the idea would be to hide all the libstdc++ symbols.

[Manuel Klimek]

On Thu, Mar 21, 2019 at 2:25 PM Manuel Klimek <kli...@google.com> wrote:

On Thu, 21 Mar 2019, 11:39 Uwe L. Korn, <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Thanks! I thought that the only supported way to do this would be to build all of the C++ parts within the same bazel invocation, as afaiu TF doesn't make any guarantees about C++ ABI stability.

And just to make sure my understanding is correct - as long as this cross-calling happens, a single package switching to devtoolset7 will also break ABI compatibility and lead to crashes, right?

[Uwe L. Korn]

On Thu, Mar 21, 2019, at 2:31 PM, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 2:25 PM Manuel Klimek <kli...@google.com> wrote:

On Thu, 21 Mar 2019, 11:39 Uwe L. Korn, <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Thanks! I thought that the only supported way to do this would be to build all of the C++ parts within the same bazel invocation, as afaiu TF doesn't make any guarantees about C++ ABI stability.

And just to make sure my understanding is correct - as long as this cross-calling happens, a single package switching to devtoolset7 will also break ABI compatibility and lead to crashes, right?

 

That should work as it still is using the old CXXABI?

[Manuel Klimek]

I thought it was linker-script-patching symbols so that the common symbols with the old libstdc++ come from the .so, while the symbols not available are statically linked. Are we sure that flipping the C++ standard wouldn't introduce any of those symbols in the C++ API?

[jwa...@redhat.com]

On Thursday, 21 March 2019 13:31:55 UTC, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 2:25 PM Manuel Klimek <kli...@google.com> wrote:

On Thu, 21 Mar 2019, 11:39 Uwe L. Korn, <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Thanks! I thought that the only supported way to do this would be to build all of the C++ parts within the same bazel invocation, as afaiu TF doesn't make any guarantees about C++ ABI stability.

And just to make sure my understanding is correct - as long as this cross-calling happens, a single package switching to devtoolset7 will also break ABI compatibility and lead to crashes, right?

If the other parts use a version of GCC older than 5 and use C++11, yes. In general, you cannot mix C++11 code compiled with GCC 4.x and C++11 code compiled with later versions. Using devtoolset doesn't change that.

Some specific uses of C++11 might work between GCC 4.x and later releases, if the relevant components didn't change between the 4.x version and the 5+ versions that officially support C++11, but other uses don't work. We don't have a complete list of what changed and what didn't, and what works and what doesn't (because the party line is just "it's not supported, don't do it") but https://gcc.gnu.org/wiki/Cxx11AbiCompatibility mentions a few specific incompatibilities.

[Manuel Klimek]

On Thu, Mar 21, 2019 at 3:43 PM <jwa...@redhat.com> wrote:

On Thursday, 21 March 2019 13:31:55 UTC, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 2:25 PM Manuel Klimek <kli...@google.com> wrote:

On Thu, 21 Mar 2019, 11:39 Uwe L. Korn, <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Thanks! I thought that the only supported way to do this would be to build all of the C++ parts within the same bazel invocation, as afaiu TF doesn't make any guarantees about C++ ABI stability.

And just to make sure my understanding is correct - as long as this cross-calling happens, a single package switching to devtoolset7 will also break ABI compatibility and lead to crashes, right?

If the other parts use a version of GCC older than 5 and use C++11, yes. In general, you cannot mix C++11 code compiled with GCC 4.x and C++11 code compiled with later versions. Using devtoolset doesn't change that.

Yea, the transition in question is from 4.x to something with C++14 support.

 

Some specific uses of C++11 might work between GCC 4.x and later releases, if the relevant components didn't change between the 4.x version and the 5+ versions that officially support C++11, but other uses don't work. We don't have a complete list of what changed and what didn't, and what works and what doesn't (because the party line is just "it's not supported, don't do it") but https://gcc.gnu.org/wiki/Cxx11AbiCompatibility mentions a few specific incompatibilities.

Thanks for that link, this is super helpful!

[jwa...@redhat.com]

I don't think you mean CXXABI. That's just a symbol version tag that is always present in libstdc++.so.6.

If you mean devtoolset-7 uses the old gcc4-compatible ABI for strings and lists (i.e. _GLIBCXX_USE_CXX11_ABI=0 is defined), that's right. But that doesn't solve all problems. Other parts of libstdc++ were still in flux for gcc 4.x releases and are not compatible with the versions in later releases (including the release used by devtoolset-7).

For example, the "Maps, sets and trees" section at https://gcc.gnu.org/wiki/Cxx11AbiCompatibility mentions an alignment bug which affects some specializations of std::set on 32-bit x86. If you compile code using GCC 4.8 and use std::set<long long> with -std=c++0x then you'll get code which has an alignment bug, and so is not compatible with -std=c++11 code in later releases and not even compatible with -std=c++98 using the same GCC 4.8 version. There are bugs in the C++11 support prior to GCC 5. Trying to use C++11 prior to GCC 5 is unsupported. You either need to compile everything from source with the same compiler version (and maybe even the same CXXFLAGS) or just stick to GCC 5+ for C++11 code.

It's possible you can make this work with static linking tricks and non-global symbols, but I am going to keep repeating that you should really consider not trying to use GCC 4.x for C++11 code.

If the TF packages and related packages are built with devtoolset, and the code linking to them doesn't use C++11 at all, everything should work OK. The only object files using C++11 would be the ones inside the TF packages, which are all built with a consistnte version.

If TF etc. are built with devtoolset and the code linking to them uses C++11, the code linking to them should really use GCC 5+ as well, to avoid mixing potentially-incompatible 4.x era C++11 code with the code inside the TF packages.

[Manuel Klimek]

Coming back to this.

On Thu, Mar 21, 2019 at 11:39 AM Uwe L. Korn <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

It seems like tensorflow-io doens't have a large number of users:

[1] 169 downloads last month; this is 0.005% of tensorflow; does anybody know the impact that breaking this would have?

I checked out ray, and it doesn't seem to be using any C++ API from tf - is there something missing?

[1] https://pypistats.org/packages/tensorflow-io

[jwa...@redhat.com]

Right.

Are we sure that flipping the C++ standard wouldn't introduce any of those symbols in the C++ API?

I'm not sure I understand exactly what you're asking here, but in devtoolset the new cxx11 ABI (i.e. new definitions of std::string and std::list, and a few other changes) is completely disabled. Using -std=c++11 or -std=c++14 with devtoolset doesn't introduce the new cxx11 ABI types. (It doesn't introduce them even with non-devtoolset compilers either, because the choice of old or new ABI is orthogonal to the choice of -std, as https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html says, but with non-devtoolset compilers you can explicitly choose the ABI via a macro, but that macro does nothing for devtoolset).

[Manuel Klimek]

On Thu, Mar 21, 2019 at 4:02 PM <jwa...@redhat.com> wrote:

On Thursday, 21 March 2019 14:05:10 UTC, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 2:35 PM Uwe L. Korn <ma...@uwekorn.com> wrote:

On Thu, Mar 21, 2019, at 2:31 PM, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 2:25 PM Manuel Klimek <kli...@google.com> wrote:

On Thu, 21 Mar 2019, 11:39 Uwe L. Korn, <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Thanks! I thought that the only supported way to do this would be to build all of the C++ parts within the same bazel invocation, as afaiu TF doesn't make any guarantees about C++ ABI stability.

And just to make sure my understanding is correct - as long as this cross-calling happens, a single package switching to devtoolset7 will also break ABI compatibility and lead to crashes, right?

 

That should work as it still is using the old CXXABI?

I thought it was linker-script-patching symbols so that the common symbols with the old libstdc++ come from the .so, while the symbols not available are statically linked.

Right.

Are we sure that flipping the C++ standard wouldn't introduce any of those symbols in the C++ API?

I'm not sure I understand exactly what you're asking here, but in devtoolset the new cxx11 ABI (i.e. new definitions of std::string and std::list, and a few other changes) is completely disabled.

So if I have gcc 4 based C++11 enabled binaries that do not use the cxx11 ABI and use them with devtoolset7, you're saying the diffs in ABI you link above don't apply? What you said above read to me like the opposite, but I'm probably thoroughly confused now :)

A different problem I see is that the TF C++ types at the call boundaries themselves might change if we flip the C++ standard to C++14.

 

Using -std=c++11 or -std=c++14 with devtoolset doesn't introduce the new cxx11 ABI types. (It doesn't introduce them even with non-devtoolset compilers either, because the choice of old or new ABI is orthogonal to the choice of -std, as https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html says, but with non-devtoolset compilers you can explicitly choose the ABI via a macro, but that macro does nothing for devtoolset).

--

[jwa...@redhat.com]

On Thursday, 21 March 2019 15:10:00 UTC, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 4:02 PM <jwa...@redhat.com> wrote:

On Thursday, 21 March 2019 14:05:10 UTC, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 2:35 PM Uwe L. Korn <ma...@uwekorn.com> wrote:

On Thu, Mar 21, 2019, at 2:31 PM, Manuel Klimek wrote:

On Thu, Mar 21, 2019 at 2:25 PM Manuel Klimek <kli...@google.com> wrote:

On Thu, 21 Mar 2019, 11:39 Uwe L. Korn, <ma...@uwekorn.com> wrote:

We have a dependency between `pyarrow` and tensorflow-io at the moment for https://github.com/tensorflow/io/tree/master/tensorflow_io/arrow and https://github.com/tensorflow/io/tree/master/tensorflow_io/parquet There we are calling the C++ API of both.

Also I think that the Ray project also calls the C++ API of both, Arrow and Tensorflow. This is how most of this discussion started.

Thanks! I thought that the only supported way to do this would be to build all of the C++ parts within the same bazel invocation, as afaiu TF doesn't make any guarantees about C++ ABI stability.

And just to make sure my understanding is correct - as long as this cross-calling happens, a single package switching to devtoolset7 will also break ABI compatibility and lead to crashes, right?

 

That should work as it still is using the old CXXABI?

I thought it was linker-script-patching symbols so that the common symbols with the old libstdc++ come from the .so, while the symbols not available are statically linked.

Right.

Are we sure that flipping the C++ standard wouldn't introduce any of those symbols in the C++ API?

I'm not sure I understand exactly what you're asking here, but in devtoolset the new cxx11 ABI (i.e. new definitions of std::string and std::list, and a few other changes) is completely disabled.

So if I have gcc 4 based C++11 enabled binaries that do not use the cxx11 ABI and use them with devtoolset7, you're saying the diffs in ABI you link above don't apply?

No!

Devtoolset disables the https://gcc.gnu.org/onlinedocs/libstdc++/manual/using_dual_abi.html features. It doesn't remove ABI bugs in the experimental C++11 features in GCC 4.x.

What you said above read to me like the opposite, but I'm probably thoroughly confused now :)

A different problem I see is that the TF C++ types at the call boundaries themselves might change if we flip the C++ standard to C++14.

That should not happen (as long as you stick to GCC releases that actually support C++14, i.e. not GCC 4.x)

See https://stackoverflow.com/questions/46746878/is-it-safe-to-link-c17-c14-and-c11-objects/49119902#49119902

[Manuel Klimek]

I might be missing something - it seems pretty straight forward to define an object with a different size either ifdef'ing on the C++ standard or using type traits that return different values to trigger different object layouts?

[Yong Tang]

The package in tensorflow-io is part of the effort for a modular tensorflow. Prior to TF 2.0, tensorflow bundles many functionalities in tf.contrib that requires lots of third-party library linkages (e.g. Kafka, AWS, Apache Arrow, Apache Ignite, etc.). That makes tensorflow’s build time very long (several hours), as many third-party libraries have a big C++/C code base.

Once TF 2.0 is released, the tf.contrib will be removed completely, and many functionalities are shifted to different SIG's repo such as tensorflow-io, addon, or sig networking. For tensorflow-io, it is not in high usage yet as most of the features could still be used from tf.contrib (as part of the tensorflow pip package). However, once TF 2.0 is released users will need to change from tf.contrib (part of tensorflow pip package) to tensorflow-io. One example is the tensorflow-datasets package, at the moment tensorflow-datasets uses tf.contrib.lmdb for LMDB format support. That will need to be re-pointed to tensorflow_io.lmdb in TF 2.0.

On a separate note, I think for extension packages of tensorflow such as tensorflow-io (and to an extent, tensorflow-addon and others), could always try to build and match exactly the same C++ compiler used by tensorflow itself (we use gcc 4.8 and on ubuntu 14.04 in tensorflow-io pip package for that exact reason). It might post some challenges to require all repos (inside tensorflow org, or outside) linking against tensorflow with exactly the same version as used by tensorflow core repo.

(I could imagine MKL may prefer to build with intel C++ compiler?).

A C API (with all interface struct consisting only POD members) exposed by tensorflow core repo could be a solution I assume. Then linking of libstdc++ statically might be possible. It may take some time to have the C API ready though. The related RFC is located:

https://github.com/tensorflow/community/pull/77

Thanks

Yong

---

From: 'Manuel Klimek' via SIG Build <bu...@tensorflow.org>
Sent: Thursday, March 21, 2019 8:00 AM
To: Uwe L. Korn
Cc: SIG Build
Subject: Re: The linking of libstdc++ (again, sorry)

 

[Manuel Klimek]

On Thu, Mar 21, 2019 at 4:55 PM Yong Tang <yong...@hotmail.com> wrote:

The package in tensorflow-io is part of the effort for a modular tensorflow. Prior to TF 2.0, tensorflow bundles many functionalities in tf.contrib that requires lots of third-party library linkages (e.g. Kafka, AWS, Apache Arrow, Apache Ignite, etc.). That makes tensorflow’s build time very long (several hours), as many third-party libraries have a big C++/C code base.

One other question is: if these are pip packages, I assume they come with a Python API and thus a C level API; wondering whether we could go through a C API to these deps in principle.

 

Once TF 2.0 is released, the tf.contrib will be removed completely, and many functionalities are shifted to different SIG's repo such as tensorflow-io, addon, or sig networking. For tensorflow-io, it is not in high usage yet as most of the features could still be used from tf.contrib (as part of the tensorflow pip package). However, once TF 2.0 is released users will need to change from tf.contrib (part of tensorflow pip package) to tensorflow-io. One example is the tensorflow-datasets package, at the moment tensorflow-datasets uses tf.contrib.lmdb for LMDB format support. That will need to be re-pointed to tensorflow_io.lmdb in TF 2.0.

On a separate note, I think for extension packages of tensorflow such as tensorflow-io (and to an extent, tensorflow-addon and others), could always try to build and match exactly the same C++ compiler used by tensorflow itself (we use gcc 4.8 and on ubuntu 14.04 in tensorflow-io pip package for that exact reason). It might post some challenges to require all repos (inside tensorflow org, or outside) linking against tensorflow with exactly the same version as used by tensorflow core repo.

Yea, for our own packages I'm not concerned - the problem is managing dependencies we call into and still being able to control our toolchain fate.

[Manuel Klimek]

so, to conclude all this

If I wanted to do something that has the chance of minimal breakage, I'd probably want to try devtoolset-7.

Given that TF releases are currently ubuntu 14.04 based, and devtoolset-7 seems to be a centos thing, what's the best way to approach this? Will compiling with devtoolset-7 on a newer centos lead to backwards-compatible-enough binaries?

[Yong Tang]

The third-party dependencies  (e.g. Kafka, AWS, Apache Arrow, Apache Ignite, etc.) are mostly C++ code as they need to be built into the kernel ops in tensorflow graph. Some of the packages (like Kafka: librdkafka) exposes a C API. Apache Arrow is pretty much C++ (it also depends on boost C++ which is a big dependency itself). AWS SDK is C++ only.

Haven't looked into details about Apache Ignite (Anthony may provide more information I think as he is more familiar with this part).

It might be difficult to wrap some of the dependencies into C API. For example, AWS SDK and Apache Arrow have many features that are too bundled into C++. A C API wrapper for AWS SDK or Apache Arrow requires lots of work.

Thanks

Yong

---

From: Manuel Klimek <kli...@google.com>
Sent: Thursday, March 21, 2019 9:07 AM
To: Yong Tang
Cc: Uwe L. Korn; SIG Build

[Gunhan Gulsoy]

On Thu, Mar 21, 2019 at 9:28 AM 'Manuel Klimek' via SIG Build <bu...@tensorflow.org> wrote:

so, to conclude all this

If I wanted to do something that has the chance of minimal breakage, I'd probably want to try devtoolset-7.

Given that TF releases are currently ubuntu 14.04 based, and devtoolset-7 seems to be a centos thing, what's the best way to approach this? Will compiling with devtoolset-7 on a newer centos lead to backwards-compatible-enough binaries?

TF team will explore this as soon as possible.

[Jonathan Wakely]

So don't do that then? :-)

I misunderstood and thought your concern was that the C++ standard
library types would change when using a different -std option. THey
could, but we make sure they don't.

Types in TF itself could certainly change if they are written to do
so. But I'd just consider that a bug in TF that should be fixed.

[Manuel Klimek]

Makes sense, thanks! 

[Manuel Klimek]

Following up on this, I have a TF build in a centos7/devtoolset-7 setup (at least I hope that I'm correctly using devtoolset-7 :)

I ran it through auditwheel and get:

$ auditwheel show tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl        

tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-
provided shared libraries: libc.so.6 with versions {'GLIBC_2.9',
'GLIBC_2.10', 'GLIBC_2.7', 'GLIBC_2.3.4', 'GLIBC_2.4', 'GLIBC_2.16',
'GLIBC_2.14', 'GLIBC_2.11', 'GLIBC_2.3.3', 'GLIBC_2.3.2',
'GLIBC_2.17', 'GLIBC_2.3', 'GLIBC_2.15', 'GLIBC_2.2.5', 'GLIBC_2.6'},
libgcc_s.so.1 with versions {'GCC_3.0'}, libm.so.6 with versions
{'GLIBC_2.2.5'}, libpthread.so.0 with versions {'GLIBC_2.3.4',
'GLIBC_2.4', 'GLIBC_2.3.3', 'GLIBC_2.3.2', 'GLIBC_2.12',
'GLIBC_2.2.5'}, libdl.so.2 with versions {'GLIBC_2.2.5'},
libstdc++.so.6 with versions {'CXXABI_1.3.3', 'GLIBCXX_3.4.14',
'GLIBCXX_3.4.17', 'GLIBCXX_3.4.19', 'GLIBCXX_3.4', 'CXXABI_1.3',
'CXXABI_1.3.2', 'CXXABI_1.3.7', 'GLIBCXX_3.4.9', 'GLIBCXX_3.4.15',
'GLIBCXX_3.4.11', 'CXXABI_1.3.5', 'GLIBCXX_3.4.18'}, libcudart.so.10.0
with versions {'libcudart.so.10.0'}, librt.so.1 with versions
{'GLIBC_2.2.5'}, libcusparse.so.10.0 with versions
{'libcusparse.so.10.0'}, libcusolver.so.10.0 with versions
{'libcusolver.so.10.0'}

Is this expected? I thought devtoolset-7 would limit the number of symbols from libc/libstdc++ to earlier versions - am I potentially holding devtoolset-7 wrong?

Thanks!

/Manuel

On Thu, Mar 21, 2019 at 11:04 PM Jonathan Wakely <jwa...@redhat.com> wrote:

[Jonathan Helmus]

Manuel,

This output indicates that the symbol versions are being limited by the devtoolset toolchain. Devtoolset-7 will limit the symbol versions to those which are available in the system libraries.  For CentOS 7 these are:

* GLIBC_2.17 from the glibc library on the system
* CXXABI_3.4.19 and GLIBCXX_1.3.7 from libstdc++
* GCC_4.8.0 from libgcc_s

Note that libstdc++ symbols version are from the libraries corresponding to the GCC which shipped with the original CentOS 7 release (4.8.5) not those in devtoolset-7 (7.3.1).  Newer symbols would be exported if this toolchain were not configured to use a linker script to statically link newer symbols.  Without this configuration the library could potentially contain symbols up to and including GLIBCXX_3.4.24 and CXXABI_1.3.11.

The GCC manual [1] contains tables with the symbol versioning for current and past releases.

Cheers,

    - Jonathan Helmus

[1] https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html

[Yong Tang]

Those glibc symbols (e.g., GLIBC_2.9,  GLIBC_2.10) are from system gcc (4.8 it think) and is outside of devtoolset. There is not a lot you could do to control the version of system glibc version.

It is, theoretically, possible to force linking to an old version of GLIBC on a newer system. So the binary will be backward-compatible with old system.

Something like `__asm__(".symver realpath,realpath@GLIBC_2.2.5")` to all the functions I remember.

However,  this approach is not practical in reality (unless you have a small program).

Thanks

Yong

---

From: 'Manuel Klimek' via SIG Build <bu...@tensorflow.org>
Sent: Tuesday, March 26, 2019 9:16 AM
To: Jonathan Wakely

Cc: SIG Build
Subject: Re: The linking of libstdc++ (again, sorry)

[Jonathan Wakely]

It has done so. Which symbol versions are you concerned about?

GLIBCXX_3.4.18 and CXXABI_1.3.7 are the versions in the CentOS 7
libstdc++.so and GLIBC_2.17 is the version in the CentOS 7 glibc.

[Jonathan Wakely]

On 26/03/19 16:44 +0000, Yong Tang wrote:
>Those glibc symbols (e.g., GLIBC_2.9, GLIBC_2.10) are from system gcc (4.8 it think)

No, they're from the system glibc, not gcc. Glibc is not part of gcc.

>and is outside of devtoolset.

That part is true.

>There is not a lot you could do to control the version of system glibc version.
>
>It is, theoretically, possible to force linking to an old version of GLIBC on a newer system. So the binary will be backward-compatible with old system.
>
>Something like `__asm__(".symver realpath,realpath@GLIBC_2.2.5")` to all the functions I remember.
>
>However, this approach is not practical in reality (unless you have a small program).

Agreed.

[Manuel Klimek]

Ah, ok, and that's why even with devtoolset7 I won't get a manylinux1 compliant wheel.

Why can't I install a dev version of an older glibc and link against that?

Thanks everyone, this is insightful :)

[Jay Furmanek]

Ah, ok, and that's why even with devtoolset7 I won't get a manylinux1 compliant wheel.

Why can't I install a dev version of an older glibc and link against that?

Thanks everyone, this is insightful :)

Sure. You wouldn't want to "install" it in the usual sense, but the toolchain package could include an older glibc library packed in, and then

use that to build against to ensure compatibility with back to that level of glibc.

I believe Anaconda's toolchain does that very thing, so there is precedence for it.

[Gunhan Gulsoy]

From TF side, our goal is to use centos6 + devtoolset7, and then dlopen cuda if available to satisfy all manylinux2010 constraints.

I do not think we have a chance to satisfy manylinux1 constraints.

[Manuel Klimek]

On Tue, Mar 26, 2019 at 10:18 PM 'Gunhan Gulsoy' via SIG Build <bu...@tensorflow.org> wrote:

From TF side, our goal is to use centos6 + devtoolset7, and then dlopen cuda if available to satisfy all manylinux2010 constraints.

I do not think we have a chance to satisfy manylinux1 constraints.

Ok, so from what I understand to satisfy manylinux1 constraints we'd need a devtoolset-7 that has a libstdc++_nonshared that would work with a libstdc++ 4.2, which we don't have. My confusion came from that manylinux-docker ships with gcc 4.8, but that already does the same trick devtoolset-7 does.

My main concern with using centos6 as a base is that that's already the oldest version nvidia supports for their docker images, and I'm somewhat afraid what will happen once they drop support for that. But I guess we could find a way to cut our C++ ABI dependencies and free ourselves from this whole mess before that happens :)

[Uwe L. Korn]

On Wed, Mar 27, 2019, at 2:03 PM, 'Manuel Klimek' via SIG Build wrote:

On Tue, Mar 26, 2019 at 10:18 PM 'Gunhan Gulsoy' via SIG Build <bu...@tensorflow.org> wrote:

From TF side, our goal is to use centos6 + devtoolset7, and then dlopen cuda if available to satisfy all manylinux2010 constraints.

I do not think we have a chance to satisfy manylinux1 constraints.

Ok, so from what I understand to satisfy manylinux1 constraints we'd need a devtoolset-7 that has a libstdc++_nonshared that would work with a libstdc++ 4.2, which we don't have. My confusion came from that manylinux-docker ships with gcc 4.8, but that already does the same trick devtoolset-7 does.

Then you would need a devtoolset-7 that is built for CentOS 5. The GCC 4.8 from the manylinux1 docker is already devtoolset-2 which is to my knowledge the latest devtoolset for CentOS 5. Moving forward, it is important to have the manylinux2010/2014 variants that update the base CentOS image. I don't think it is worth the effort to backport it to CentOS 5 just to fulfill manylinux1.

Uwe

[Gunhan Gulsoy]

More than just having centos5 + devtoolset7, we also need a docker image we can build TF on that has NVIDIA libraries. CentOS5 is EOL, so there will be no NVIDIA docker images based on that.

When I discussed with NVIDIA they told me they would release docker images until the OS goes EOL, so that is why I think we need to push for manylinux2014 that is based off of centos7 before we come up with a futureproof solution.

--

[Manuel Klimek]

Trying to build a centos6 based image. Unfortunately this is not possible from my debian workstation so I have to jump through hoops to push it all onto some non-debian cloud instance :(

[Manuel Klimek]

On Thu, Mar 28, 2019 at 11:00 AM Manuel Klimek <kli...@google.com> wrote:

Trying to build a centos6 based image. Unfortunately this is not possible from my debian workstation so I have to jump through hoops to push it all onto some non-debian cloud instance :(

Which brings me back to that it  would be really great to have a devtoolset-7 that *runs* on a more recent OS but links against older glibc/libstdc++/etc libraries, basically "cross-compiler for older systems" style.

[Jonathan Wakely]

On 28/03/19 11:00 +0100, 'Manuel Klimek' via SIG Build wrote:
>Trying to build a centos6 based image. Unfortunately this is not possible
>from my debian workstation so I have to jump through hoops to push it all
>onto some non-debian cloud instance :(

I've used virt-manager to install centos6 in a VM for such situations.

[Manuel Klimek]

Yea, I can deal with it, my main problem is that putting that into the workflow required to update the image will not be well received by other engineers :(

I'll need to look into whether I can build something devtoolset-7-like enough on debian.

[Jonathan Wakely]

On 28/03/19 11:04 +0100, 'Manuel Klimek' via SIG Build wrote:
>On Thu, Mar 28, 2019 at 11:00 AM Manuel Klimek <kli...@google.com> wrote:
>
>> Trying to build a centos6 based image. Unfortunately this is not possible
>> from my debian workstation so I have to jump through hoops to push it all
>> onto some non-debian cloud instance :(
>>
>
>Which brings me back to that it would be really great to have a
>devtoolset-7 that *runs* on a more recent OS but links against older
>glibc/libstdc++/etc libraries, basically "cross-compiler for older systems"
>style.

There's no technical reason that can't be done, but it would be quite
a lot of work to maintain. Our team at Red Hat spends considerable
resources on the devtoolset packages, I don't think you want to try to
replicate that to reproduce it as a cross-compiler.

I think a much simpler solution is a docker image using centos6 (or
centos7, which isn't going EOL any time soon) and devtoolset. Anybody
that can run docker (or a compatible OCI container runtime) could use
that image to build in a container.

[Manuel Klimek]

That was my plan, but centos6 not running by default on debian docker is a big hurdle to jump through for folks.

I do run the team that manages (some of :) our OSS toolchains, so I'll at least investigate what the effort would be - the upside is that thanks to the great work the devtoolset-7 folks are doing we might be able to build on top of devtoolset-7's changes with a compatible toolchain (but I'll need to look into it). In the end, I think that would strengthen the ecosystem.

[Manuel Klimek]

And just to help folks understand where we're coming from:

The problem is that each new system / architecture we have to deal with significantly increases the effort to maintain a piece of software we have.

It's much more economic to provide a single toolchain that's managed by a central team to produce all our software with.

We have bet the company on clang & cuda-clang & live-at-head; cuda-clang is already a well-tested setup that everybody is knowledgeable about and comfortable with.

Thus, for us, investing into creating a built-from-head cuda-clang that's compatible with manylinuxes would significantly reduce the number of systems/toolchains people need to interact with on a daily basis, which would significantly improve overall productivity for our toolchain users.

[Manuel Klimek]

Hm, my wheel built off of a centos6/devtoolset-7 seems to include symbols outside the manylinux10 spec. glibc looks fine, but libstdc++ is not.

PEP 571 says:

GLIBC_2.12

CXXABI_1.3.3

GLIBCXX_3.4.13

GCC_4.3.0

auditwheel says I'm using GLIBCXX_3.4.19 and CXXABI_1.3.5.

$ auditwheel show tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl

tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-

provided shared libraries: libc.so.6 with versions {'GLIBC_2.7',
'GLIBC_2.17', 'GLIBC_2.3.3', 'GLIBC_2.2.5', 'GLIBC_2.3.4',
'GLIBC_2.3', 'GLIBC_2.3.2', 'GLIBC_2.4', 'GLIBC_2.16', 'GLIBC_2.14',
'GLIBC_2.11', 'GLIBC_2.9', 'GLIBC_2.15', 'GLIBC_2.6', 'GLIBC_2.10'},
librt.so.1 with versions {'GLIBC_2.2.5'}, libgcc_s.so.1 with versions

{'GCC_3.0'}, libm.so.6 with versions {'GLIBC_2.2.5'}, libpthread.so.0

with versions {'GLIBC_2.3.3', 'GLIBC_2.2.5', 'GLIBC_2.3.4',
'GLIBC_2.3.2', 'GLIBC_2.4', 'GLIBC_2.12'}, libdl.so.2 with versions
{'GLIBC_2.2.5'}, libstdc++.so.6 with versions {'CXXABI_1.3.2',
'CXXABI_1.3.3', 'GLIBCXX_3.4.15', 'GLIBCXX_3.4.18', 'GLIBCXX_3.4',
'GLIBCXX_3.4.17', 'CXXABI_1.3.7', 'CXXABI_1.3.5', 'GLIBCXX_3.4.11',
'GLIBCXX_3.4.14', 'GLIBCXX_3.4.19', 'CXXABI_1.3', 'GLIBCXX_3.4.9'},

libcudart.so.10.0 with versions {'libcudart.so.10.0'},

libcusparse.so.10.0 with versions {'libcusparse.so.10.0'},
libcusolver.so.10.0 with versions {'libcusolver.so.10.0'}

This constrains the platform tag to "linux_x86_64". In order to
achieve a more compatible tag, you would need to recompile a new wheel
from source on a system with earlier versions of these libraries, such
as a recent manylinux image.

[Jonathan Helmus]

That looks like a wheel built off CentOS 7 and devtoolset-7.  Are you sure you used CentOS 6?

    - Jonathan Helmus

[Manuel Klimek]

Aaaand it's pebcak - copying the dockerfiles around led to me actually not having a centos6 based image :(

[Manuel Klimek]

On Thu, Mar 28, 2019 at 4:48 PM Jonathan Helmus <jhe...@anaconda.com> wrote:

That looks like a wheel built off CentOS 7 and devtoolset-7.  Are you sure you used CentOS 6?

Yea, sorry :( Inadvertently proved that this setup is too complex for myself.

[Manuel Klimek]

Ok, the next problem that I should have foreseen is that we can't remote build with centos6, because the RBE machines run some form of debian :(

[Manuel Klimek]

On Thu, Mar 28, 2019 at 8:33 PM Manuel Klimek <kli...@google.com> wrote:

Ok, the next problem that I should have foreseen is that we can't remote build with centos6, because the RBE machines run some form of debian :(

That one might have been too early - the autoconfig doesn't work (that usually runs locally) but I can run it on a remote machine and copy stuff back. We'll see whether the actions actually all work remotely.

[Manuel Klimek]

On Thu, Mar 28, 2019 at 9:01 PM Manuel Klimek <kli...@google.com> wrote:

On Thu, Mar 28, 2019 at 8:33 PM Manuel Klimek <kli...@google.com> wrote:

Ok, the next problem that I should have foreseen is that we can't remote build with centos6, because the RBE machines run some form of debian :(

That one might have been too early - the autoconfig doesn't work (that usually runs locally) but I can run it on a remote machine and copy stuff back. We'll see whether the actions actually all work remotely.

Next stone in the way is that the bazel distribution doesn't work on centos6, so I'd need to create my own bazel for the autoconfig to run at all.

[Manuel Klimek]

After bootstrapping my own bazel on centos6, I got a whl file that seems to be good (minus cuda) based on centos6/devtoolset-7.

Thanks everybody for the help!

$ auditwheel show tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl

tensorflow-1.13.1-cp36-cp36m-linux_x86_64.whl is consistent with the
following platform tag: "linux_x86_64".

The wheel references external versioned symbols in these system-

provided shared libraries: libc.so.6 with versions {'GLIBC_2.9',
'GLIBC_2.3.4', 'GLIBC_2.11', 'GLIBC_2.3.3', 'GLIBC_2.4', 'GLIBC_2.3',
'GLIBC_2.2.5', 'GLIBC_2.10', 'GLIBC_2.3.2', 'GLIBC_2.7', 'GLIBC_2.6'},

librt.so.1 with versions {'GLIBC_2.2.5'}, libgcc_s.so.1 with versions
{'GCC_3.0'}, libm.so.6 with versions {'GLIBC_2.2.5'}, libpthread.so.0

with versions {'GLIBC_2.3.4', 'GLIBC_2.3.3', 'GLIBC_2.4',
'GLIBC_2.2.5', 'GLIBC_2.12', 'GLIBC_2.3.2'}, libdl.so.2 with versions
{'GLIBC_2.2.5'}, libstdc++.so.6 with versions {'GLIBCXX_3.4.11',
'CXXABI_1.3', 'GLIBCXX_3.4.9', 'CXXABI_1.3.2', 'GLIBCXX_3.4.10',
'CXXABI_1.3.3', 'GLIBCXX_3.4'}, libcudart.so.10.0 with versions

{'libcudart.so.10.0'}, libcusparse.so.10.0 with versions
{'libcusparse.so.10.0'}, libcusolver.so.10.0 with versions
{'libcusolver.so.10.0'}

This constrains the platform tag to "manylinux2010_x86_64". In order

to achieve a more compatible tag, you would need to recompile a new
wheel from source on a system with earlier versions of these
libraries, such as a recent manylinux image.

[Manuel Klimek]

Ok, one more question:

In what ways is using devtoolset-7 less risky than using the same libstdc++ version as devtoolset-7 with  _GLIBCXX_USE_CXX11_ABI set to 0 and statically linking libstdc++?

[Subin Modeel]

FWIW

I was able to get the manylinux2010_x86_64 compliant wheel built on centos6+devtoolset-7

The image I used was quay.io/aicoe/manylinux2010_x86_64 .

Thanks for your help Manuel.

bash-4.1# auditwheel show /input/tensorflow-1.13.1-cp27-none-linux_x86_64.whl

tensorflow-1.13.1-cp27-none-linux_x86_64.whl is consistent with the

following platform tag: "manylinux2010_x86_64".

The wheel references external versioned symbols in these system-

provided shared libraries: libc.so.6 with versions {'GLIBC_2.2.5',

'GLIBC_2.4', 'GLIBC_2.10', 'GLIBC_2.6', 'GLIBC_2.3.2', 'GLIBC_2.9',

'GLIBC_2.11', 'GLIBC_2.3.4', 'GLIBC_2.7', 'GLIBC_2.3'}, libstdc++.so.6

with versions {'GLIBCXX_3.4.9', 'CXXABI_1.3.3', 'CXXABI_1.3.2',

'GLIBCXX_3.4.11', 'GLIBCXX_3.4.10', 'CXXABI_1.3', 'GLIBCXX_3.4'},

libpthread.so.0 with versions {'GLIBC_2.3.3', 'GLIBC_2.12',

'GLIBC_2.2.5', 'GLIBC_2.3.2'}, libm.so.6 with versions

{'GLIBC_2.2.5'}, libgcc_s.so.1 with versions {'GCC_3.0', 'GCC_3.3'},

libdl.so.2 with versions {'GLIBC_2.2.5'}, librt.so.1 with versions

{'GLIBC_2.2.5'}

This constrains the platform tag to "manylinux2010_x86_64". In order

to achieve a more compatible tag, you would need to recompile a new

wheel from source on a system with earlier versions of these

libraries, such as a recent manylinux image.

Thanks & Regards

Subin M

[Manuel Klimek]

Great! To follow up, I also successfully got a devtoolset cross-compile from ubuntu and have both ubuntu16.04/devtoolset-7/glibc2.12/nvcc and ubuntu18.04/devtoolset-8/glibc2.12/cuda-clang toolchains working that produce wheels that both pass auditwheel and actually work on centos6. Thanks everybody for the insights! :)