[boost] C++03 and C++11 ABI compatibility for compiled libraries

[Andrey Semashev]

Hi,

I would like to raise the issue of binary compatibility of compiled Boost
libraries between C++03 and C++11. I faced this problem with gcc on Linux,
but other compilers and platforms may be affected as well.

To summarize the problem:

* Most, if not all Linux distributions build packages in C++03 mode. This
is also the default in gcc.
* Some Boost libraries conditionally use C++11 features in their binary
interfaces (see [1] as an example). This makes C++03 and C++11 versions of
the library binary incompatible.
* Given that, users are unable to use C++11 in their code that uses Boost
because of linker failures.

I don't have the complete list of the offending C++11 features that break
ABI, but it would seem that at least scoped enums and rvalue references are
among them. Are there any more?

I would like to hear opinions on the possible course of action to tackle
this problem. Currently, I see 2 choices:

1. Avoid using C++11 features in binary interfaces. This may also mean
avoiding STL types in binary interfaces, if they differ between C++03 and
C++11 (I didn't verify that). This is my suggested hotfix solution for
Boost.Filesystem [1] and the approach I took in Boost.Log. Obviously, this
cripples code and error prone, but it keeps a single binary that can be
used from any flavor of C++ on the user's side.
2. Compile different versions of Boost libraries, for each supported C++
version. The C++ version should be encoded into the library file names, so
that different versions can coexist. This would require changes in
Boost.Build and autolinking support code, but I think this would be a
better long-term solution.

Opinions?

[1] https://svn.boost.org/trac/boost/ticket/6779

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

[Mathias Gaunard]

On 13/05/13 14:08, Andrey Semashev wrote:

> I would like to hear opinions on the possible course of action to tackle
> this problem. Currently, I see 2 choices:
>
> 1. Avoid using C++11 features in binary interfaces. This may also mean
> avoiding STL types in binary interfaces, if they differ between C++03 and
> C++11 (I didn't verify that). This is my suggested hotfix solution for
> Boost.Filesystem [1] and the approach I took in Boost.Log. Obviously, this
> cripples code and error prone, but it keeps a single binary that can be
> used from any flavor of C++ on the user's side.
> 2. Compile different versions of Boost libraries, for each supported C++
> version. The C++ version should be encoded into the library file names, so
> that different versions can coexist. This would require changes in
> Boost.Build and autolinking support code, but I think this would be a
> better long-term solution.
>
> Opinions?
>
> [1] https://svn.boost.org/trac/boost/ticket/6779

I prefer [1].

[Michael Marcin]

On 5/13/2013 10:09 AM, Mathias Gaunard wrote:
> On 13/05/13 14:08, Andrey Semashev wrote:
>
>> I would like to hear opinions on the possible course of action to tackle
>> this problem. Currently, I see 2 choices:
>>
>> 1. Avoid using C++11 features in binary interfaces. This may also mean
>> avoiding STL types in binary interfaces, if they differ between C++03 and
>> C++11 (I didn't verify that). This is my suggested hotfix solution for
>> Boost.Filesystem [1] and the approach I took in Boost.Log. Obviously,
>> this
>> cripples code and error prone, but it keeps a single binary that can be
>> used from any flavor of C++ on the user's side.
>> 2. Compile different versions of Boost libraries, for each supported C++
>> version. The C++ version should be encoded into the library file
>> names, so
>> that different versions can coexist. This would require changes in
>> Boost.Build and autolinking support code, but I think this would be a
>> better long-term solution.
>>
>> Opinions?
>>
>> [1] https://svn.boost.org/trac/boost/ticket/6779
>
> I prefer [1].
>

Does boost ever claim support for abi compatibility with different
compiler settings? I thought not.

If there needs to be a change beyond users just building the version
they actually need, I would prefer 2.

Binding Boost to C++03 seems counter to Boost's stated goals.

[Klaim - Joël Lamotte]

On Mon, May 13, 2013 at 5:29 PM, Michael Marcin <mike....@gmail.com>wrote:

> Does boost ever claim support for abi compatibility with different
> compiler settings? I thought not.

I thought boost explicitely, somewhere, stated that binary compatibility is
not a requirement for libraries.
I remember it being the main argument for developing "booster" which is a
"fixed" set of boost libraries used in CPPCMS.

Joel Lamotte

[Andrey Semashev]

On Monday 13 May 2013 10:29:47 Michael Marcin wrote:
>
> Does boost ever claim support for abi compatibility with different
> compiler settings? I thought not.
>
> If there needs to be a change beyond users just building the version
> they actually need, I would prefer 2.
>
> Binding Boost to C++03 seems counter to Boost's stated goals.

Building and distributing custom builds of libraries with applications is
typical in Windows but not in Linux. And with the current state of affair
applications cannot use Boost libraries in C++11 mode, which also contradicts
Boost goals.

I'm not advocating for solution (1) here, I'm just saying that custom builds
are not the answer in this case, IMHO.

[Andrey Semashev]

On Monday 13 May 2013 18:26:38 Klaim - Joël Lamotte wrote:
> On Mon, May 13, 2013 at 5:29 PM, Michael Marcin
<mike....@gmail.com>wrote:
> > Does boost ever claim support for abi compatibility with different
> > compiler settings? I thought not.
>
> I thought boost explicitely, somewhere, stated that binary compatibility is
> not a requirement for libraries.
> I remember it being the main argument for developing "booster" which is a
> "fixed" set of boost libraries used in CPPCMS.

I think that was about binary compatibility between different releases of
Boost or its libraries. It's a bit different from compatibility between single
release builds for different C++ versions.

Speaking of different releases, Boost does mangle library names with the
release version, so linked applications will always use the correct binaries.
This counts in favor for the solution (2), with mangling library names
differently in C++03 and C++11.

[Jürgen Hunold]

Hi Andrey,

I jsut focus on the Boost.Build issues.

On Monday, 13. May 2013 16:08:31 Andrey Semashev wrote:
> 2. Compile different versions of Boost libraries, for each supported C++
> version. The C++ version should be encoded into the library file names, so
> that different versions can coexist.

Theoretically, yes.

> This would require changes in Boost.Build

This is mostly a gcc (and compatible compiler, aka clang, icc) question.
Technically, you have to provide a Boost.Build feature (e.g. " feature std :
c++98 c++11 ;") to distinguish between the standards. Then you can change
library names in boostcpp.jam easily. The tricky part is to figure out which
gcc enables c++11 support with which switches and hardcode this depending on
the real gcc version into gcc.jam. There are discussion about this on the ML,
too.

> and autolinking support code
Autolinking is a msvc (and compatible, aka intel) solution. (Un-)fortunately,
msvc just enables c++11 features by default without any switches. But as the
msvc runtimes are ABI-incompatible from version to version, this should be a
non-issue for msvc users. mingw is another story, sadly.

>, but I think this would be a better long-term solution.

The long-term solution is to get gcc (or the distros) to switch to c++11 as
default. Your solution is a mid-term solution. But I fear that similar
problems will occur with the next standard version, too.
I am just not sure whether this justifies the work needed here.

Yours,

Jürgen
--
* Dipl.-Math. Jürgen Hunold !
* voice: ++49 4257 300 ! Fährstraße 1
* fax : ++49 4257 300 ! 31609 Balge/Sebbenhausen
* jhu...@gmx.eu ! Germany

[Stephan T. Lavavej]

[Jürgen Hunold]

> (Un-)fortunately, msvc just enables c++11 features by default without any switches.

That's definitely a "fortunately". If VC's compiler or STL had C++03 modes, that'd be yet another thing for you to worry about. This is a deliberate design choice.

> But as the msvc runtimes are ABI-incompatible from version to version, this
> should be a non-issue for msvc users.

Yep. VC's STL absolutely forbids mixing different major versions.

Stephan T. Lavavej
Visual C++ Libraries Developer

[Michael Marcin]

On 5/13/13 12:21 PM, Andrey Semashev wrote:
> On Monday 13 May 2013 18:26:38 Klaim - Joël Lamotte wrote:
>> On Mon, May 13, 2013 at 5:29 PM, Michael Marcin
> <mike....@gmail.com>wrote:
>>> Does boost ever claim support for abi compatibility with different
>>> compiler settings? I thought not.
>>
>> I thought boost explicitely, somewhere, stated that binary compatibility is
>> not a requirement for libraries.
>> I remember it being the main argument for developing "booster" which is a
>> "fixed" set of boost libraries used in CPPCMS.
>
> I think that was about binary compatibility between different releases of
> Boost or its libraries. It's a bit different from compatibility between single
> release builds for different C++ versions.
>
> Speaking of different releases, Boost does mangle library names with the
> release version, so linked applications will always use the correct binaries.
> This counts in favor for the solution (2), with mangling library names
> differently in C++03 and C++11.
>

There are also parameters where the names do not encoding important
settings. If the code is compiled for x86 or x64 for instance. The user
is just expected to build and use the libraries they need.

[Andrey Semashev]

Libraries for different architectures are installed into different system
folders, so it's not a problem.

Surely there are compiler options that affect ABI and do not get encoded in
the file names. Things like stack alignment and calling conventions are far
from being commonly changed in projects, and if they are people know the
caveats. C++ flavor is another kind of option. You should typically _expect_
more and more people enable C++11, despite the fact it's not the default on
current systems. And, IMHO, Boost should honor and help that trend.

[Vicente J. Botet Escriba]

Le 13/05/13 14:08, Andrey Semashev a écrit :

I'm completely against option 1, it would kill Boost.
Best,
Vicente

[Charles_...@dell.com]

> I would like to hear opinions on the possible course of action to tackle this problem. Currently, I see 2 choices:

> 1. Avoid using C++11 features in binary interfaces.

> 2. Compile different versions of Boost libraries, for each supported C++ version.

> Opinions?

We use option 2 and rely on the names being well-formed.

Charles Wilson
Senior Software Development Engineer
Dell | Enterprise Solutions Group

[Mathias Gaunard]

On 13/05/13 22:35, Michael Marcin wrote:

> There are also parameters where the names do not encoding important
> settings. If the code is compiled for x86 or x64 for instance. The user
> is just expected to build and use the libraries they need.

A linker will automatically reject linking together code that was built
for two different incompatible architectures

[Mathias Gaunard]

On 13/05/13 23:26, Vicente J. Botet Escriba wrote:

> I'm completely against option 1, it would kill Boost.

Why would it kill Boost?
Binary interfaces are purely implementation details. There is no problem
lowering them to C++03 while still exposing C++11 features to the user
or using C++11 internally.

[Vicente J. Botet Escriba]

Le 14/05/13 00:01, Mathias Gaunard a écrit :

> On 13/05/13 23:26, Vicente J. Botet Escriba wrote:
>
>> I'm completely against option 1, it would kill Boost.
>
> Why would it kill Boost?
> Binary interfaces are purely implementation details. There is no
> problem lowering them to C++03 while still exposing C++11 features to
> the user or using C++11 internally.
>
>

Maybe kill Boost was too strong. The binaries not only can not use c++11
features in their interface, them can not use any class or function that
can have a different form on C++11 or C++03. This seems to me more than
an implementation detail.

I would say that it could be good if a library author follows this
advice, but I would not require it.

Best,
Vicente

[Mathias Gaunard]

On 14/05/13 00:23, Vicente J. Botet Escriba wrote:
> The binaries not only can not use c++11
> features in their interface, them can not use any class or function that
> can have a different form on C++11 or C++03.

They can as long as it's not on the ABI boundary.

> I would say that it could be good if a library author follows this
> advice, but I would not require it.

Of course this would not need to apply to libraries that are C++11-only.

[Michael Marcin]

On 5/13/13 3:47 PM, Andrey Semashev wrote:
>
> Libraries for different architectures are installed into different system
> folders, so it's not a problem.
>

Not on windows.

[Andrey Semashev]

On Monday 13 May 2013 19:14:16 Michael Marcin wrote:
> On 5/13/13 3:47 PM, Andrey Semashev wrote:
> > Libraries for different architectures are installed into different system
> > folders, so it's not a problem.
>
> Not on windows.

There is no common place for libraries on Windows, so you can arrange them the
way you like for your project.

[Paul A. Bristow]

> -----Original Message-----
> From: Boost [mailto:boost-...@lists.boost.org] On Behalf Of Klaim - Joël Lamotte
> Sent: Monday, May 13, 2013 5:27 PM
> To: Boost Developers List
> Subject: Re: [boost] C++03 and C++11 ABI compatibility for compiled libraries
>
> On Mon, May 13, 2013 at 5:29 PM, Michael Marcin <mike....@gmail.com>wrote:
>
> > Does boost ever claim support for abi compatibility with different
> > compiler settings? I thought not.

> I thought boost explicitely, somewhere, stated that binary compatibility is not a requirement for
> libraries.

Boost has *never* sought binary compatibility - it is too much of a brake on change.

Paul

---
Paul A. Bristow,
Prizet Farmhouse, Kendal LA8 8AB UK
+44 1539 561830 07714330204
pbri...@hetp.u-net.com

[Mathias Gaunard]

On 14/05/13 05:21, Andrey Semashev wrote:
> On Monday 13 May 2013 19:14:16 Michael Marcin wrote:
>> On 5/13/13 3:47 PM, Andrey Semashev wrote:
>>> Libraries for different architectures are installed into different system
>>> folders, so it's not a problem.
>>
>> Not on windows.
>
> There is no common place for libraries on Windows, so you can arrange them the
> way you like for your project.

There is a common place for dlls, and it is indeed two different
directories for 32 and 64 bit.

[Lars Viklund]

On Tue, May 14, 2013 at 11:06:34AM +0200, Mathias Gaunard wrote:
> On 14/05/13 05:21, Andrey Semashev wrote:
>> On Monday 13 May 2013 19:14:16 Michael Marcin wrote:
>>> On 5/13/13 3:47 PM, Andrey Semashev wrote:
>>>> Libraries for different architectures are installed into different system
>>>> folders, so it's not a problem.
>>>
>>> Not on windows.
>>
>> There is no common place for libraries on Windows, so you can arrange them the
>> way you like for your project.
>
> There is a common place for dlls, and it is indeed two different
> directories for 32 and 64 bit.

Noting of course that system32 and syswow64 are completely off limits to
end users and end user applications.

Anyone who puts anything in there in this day and age I will personally
keelhaul for polluting system directories.

It's bad enough that people pollute the %PATH% with DLLs, keep them out
of the system directories.

The recommended way to deploy third party libraries is to bundle them in
the application directory and keep them out of %PATH%.

--
Lars Viklund | z...@acc.umu.se

[Andrey Semashev]

On Tue, May 14, 2013 at 1:06 PM, Mathias Gaunard <
mathias...@ens-lyon.org> wrote:

> On 14/05/13 05:21, Andrey Semashev wrote:
>
>> On Monday 13 May 2013 19:14:16 Michael Marcin wrote:
>>
>>> On 5/13/13 3:47 PM, Andrey Semashev wrote:
>>>
>>>> Libraries for different architectures are installed into different
>>>> system
>>>> folders, so it's not a problem.
>>>>
>>>
>>> Not on windows.
>>>
>>
>> There is no common place for libraries on Windows, so you can arrange
>> them the
>> way you like for your project.
>>
>
> There is a common place for dlls, and it is indeed two different
> directories for 32 and 64 bit.
>

I was referring to the libraries you would link your application with (i.e.
*.lib).

Regarding built dlls I mentioned earlier that on Windows it is typical to
distribute applications with their own set of libraries. So on Windows the
ABI incompatibility between C++03 and C++11 is mostly non-issue. The
exception is Cygwin, which still needs different library names.

[Petr Machata]

I maintain Boost in Fedora, and C++03/11 incompatibility has been on my
list of things to look into for some time now.

Andrey Semashev <andrey....@gmail.com> writes:
> 1. Avoid using C++11 features in binary interfaces.

This is attractive from the downstream maintainer point of view, but
would likely be fragile. Also, avoiding use of ABI-incompatible STL
types seems like a serious limitation. This doesn't seem like it could
work.

> 2. Compile different versions of Boost libraries

I'm not sure how this fares with respect to autotools (by any name). We
might need to update some automation to adapt to the changes in soname
mangling, and to take any C++11-enabling switches into account. But I
didn't look closely. Covering autotools and cmake would take us a long
way towards the goal, which seems doable, so maybe #2 is the way to go.

Thanks,
PM

[Andrey Semashev]

On Mon, May 20, 2013 at 3:44 PM, Petr Machata <pmac...@redhat.com> wrote:

> I maintain Boost in Fedora, and C++03/11 incompatibility has been on my
> list of things to look into for some time now.
>

> > 2. Compile different versions of Boost libraries
>
> I'm not sure how this fares with respect to autotools (by any name). We
> might need to update some automation to adapt to the changes in soname
> mangling, and to take any C++11-enabling switches into account. But I
> didn't look closely. Covering autotools and cmake would take us a long
> way towards the goal, which seems doable, so maybe #2 is the way to go.
>

Great, nice to hear from package maintainers. I got the same impression
from the discussion, so maybe I'll try to implement a Boost.Build property
to select C++ flavor.

However, I don't know much about autotools or cmake internals, so I can't
tell how it will affect them.

[Lars Hagström]

Did you ever get anywhere with this? This issue has bitten me now, and
I would really like to avoid having to build custom boost versions on
Linux. The ease of installation when using the distro repos is *so*
nice.

Cheers
Lars

[Andrey Semashev]

On Wed, Jan 22, 2014 at 12:16 PM, Lars Hagström <la...@foldspace.nu> wrote:
> On Mon, May 20, 2013 at 2:15 PM, Andrey Semashev
> <andrey....@gmail.com> wrote:
>> On Mon, May 20, 2013 at 3:44 PM, Petr Machata <pmac...@redhat.com> wrote:
>>
>>> I maintain Boost in Fedora, and C++03/11 incompatibility has been on my
>>> list of things to look into for some time now.
>>>
>>> > 2. Compile different versions of Boost libraries
>>>
>>> I'm not sure how this fares with respect to autotools (by any name). We
>>> might need to update some automation to adapt to the changes in soname
>>> mangling, and to take any C++11-enabling switches into account. But I
>>> didn't look closely. Covering autotools and cmake would take us a long
>>> way towards the goal, which seems doable, so maybe #2 is the way to go.
>>>
>>
>> Great, nice to hear from package maintainers. I got the same impression
>> from the discussion, so maybe I'll try to implement a Boost.Build property
>> to select C++ flavor.
>>
>> However, I don't know much about autotools or cmake internals, so I can't
>> tell how it will affect them.
>>

> Did you ever get anywhere with this? This issue has bitten me now, and
> I would really like to avoid having to build custom boost versions on
> Linux. The ease of installation when using the distro repos is *so*
> nice.

No, not really. I had a few quick attempts that failed because of my
incompetence with Boost.Build and then I got distracted.

[Lars Hagström]

Ok, I didn't really think so, since there is a distinct lack of
further discussions of the issue on the mailing lists.
Unfortunately this issue probably stops me from using C++11 at all, at
least until GCC changes their default and some of the larger distros
have started using that version of GCC.
So C++11 is at least a few years away for me, it seems

:-(

/Lars

On Wed, Jan 22, 2014 at 9:22 AM, Andrey Semashev

[Paul A. Bristow]

> -----Original Message-----
> From: Boost [mailto:boost-...@lists.boost.org] On Behalf Of Lars Hagström
> Sent: Friday, January 24, 2014 8:12 AM
> To: bo...@lists.boost.org
> Subject: Re: [boost] C++03 and C++11 ABI compatibility for compiled libraries
>

> Ok, I didn't really think so, since there is a distinct lack of further discussions of the issue
on the mailing
> lists.
> Unfortunately this issue probably stops me from using C++11 at all, at least until GCC changes
their
> default and some of the larger distros have started using that version of GCC.

> So C++11 is at least a few years away for me, it seems.

Well the compiler boys are making good progress with C++11:

http://clang-developers.42468.n3.nabble.com/Heads-up-C-11-switch-mid-week-td4037247.html

so you may not have that long to wait?

Paul

---
Paul A. Bristow,
Prizet Farmhouse, Kendal LA8 8AB UK
+44 1539 561830 07714330204
pbri...@hetp.u-net.com