[boost] Linking libboost_python.so to libpython
161 views
Skip to first unread message
Jonathan Wakely via Boost
Apr 3, 2020, 11:43:25 AM4/3/20
to bo...@lists.boost.org, Jonathan Wakely
tools/build/src/tools/python.jam says:
# On *nix, we do not want to link either Boost.Python or Python
extensions
# to libpython, because the Python interpreter itself provides all those
# symbols. If we linked to libpython, we would get duplicate symbols. So
# declare two targets -- one for building extensions and another for
# embedding.
Why would that give duplicate symbols? Wouldn't ELF symbol interposition
mean that only one is used? Why would it be a problem to link to libpython?
In Fedora's Boost RPM packages we *do* link libboost_python.so to the
system libpython (and we don't get any problems). But to do this, we have
these two local patches:
https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
we have this patch to pass the "m" into the Boost build:
https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
I'd prefer not to need these local hacks. Would the upstream Boost.Python
consider making changes so that libboost_python.so can be (optionally)
linked to libpython?
Is there a cleaner way to do this than those patches?
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
# On *nix, we do not want to link either Boost.Python or Python
extensions
# to libpython, because the Python interpreter itself provides all those
# symbols. If we linked to libpython, we would get duplicate symbols. So
# declare two targets -- one for building extensions and another for
# embedding.
Why would that give duplicate symbols? Wouldn't ELF symbol interposition
mean that only one is used? Why would it be a problem to link to libpython?
In Fedora's Boost RPM packages we *do* link libboost_python.so to the
system libpython (and we don't get any problems). But to do this, we have
these two local patches:
https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
we have this patch to pass the "m" into the Boost build:
https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
I'd prefer not to need these local hacks. Would the upstream Boost.Python
consider making changes so that libboost_python.so can be (optionally)
linked to libpython?
Is there a cleaner way to do this than those patches?
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Stefan Seefeld via Boost
Apr 3, 2020, 11:54:00 AM4/3/20
to bo...@lists.boost.org, Stefan Seefeld
Hi Jonathan,
On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:
> tools/build/src/tools/python.jam says:
>
> # On *nix, we do not want to link either Boost.Python or Python
> extensions
> # to libpython, because the Python interpreter itself provides all those
> # symbols. If we linked to libpython, we would get duplicate symbols. So
> # declare two targets -- one for building extensions and another for
> # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?
Do all *nix variants use ELF these days ?
> In Fedora's Boost RPM packages we *do* link libboost_python.so to the
> system libpython (and we don't get any problems).
What is your rationale for this, given that (according to the above
reasoning) symbols will be resolved from the application that's loading
the module ?
> But to do this, we have
> these two local patches:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
>
> and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
> we have this patch to pass the "m" into the Boost build:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
>
> I'd prefer not to need these local hacks. Would the upstream Boost.Python
> consider making changes so that libboost_python.so can be (optionally)
> linked to libpython?
>
> Is there a cleaner way to do this than those patches?
I fully agree that whatever seems the right choice of build (including
link) process for a given platform, we (boost) should incorporate that
into our own infrastructure, so downstream package maintainers don't
need to patch it. I'll leave it to the Boost.Build maintainers to answer
the technical details of that process.
Thanks,
Stefan
--
...ich hab' noch einen Koffer in Berlin...
On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:
> tools/build/src/tools/python.jam says:
>
> # On *nix, we do not want to link either Boost.Python or Python
> extensions
> # to libpython, because the Python interpreter itself provides all those
> # symbols. If we linked to libpython, we would get duplicate symbols. So
> # declare two targets -- one for building extensions and another for
> # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?
> In Fedora's Boost RPM packages we *do* link libboost_python.so to the
> system libpython (and we don't get any problems).
reasoning) symbols will be resolved from the application that's loading
the module ?
> But to do this, we have
> these two local patches:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
>
> and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
> we have this patch to pass the "m" into the Boost build:
>
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
>
> I'd prefer not to need these local hacks. Would the upstream Boost.Python
> consider making changes so that libboost_python.so can be (optionally)
> linked to libpython?
>
> Is there a cleaner way to do this than those patches?
link) process for a given platform, we (boost) should incorporate that
into our own infrastructure, so downstream package maintainers don't
need to patch it. I'll leave it to the Boost.Build maintainers to answer
the technical details of that process.
Thanks,
Stefan
--
...ich hab' noch einen Koffer in Berlin...
Jonathan Wakely via Boost
Apr 3, 2020, 12:17:49 PM4/3/20
to Stefan Seefeld, Jonathan Wakely, bo...@lists.boost.org
On Fri, 3 Apr 2020 at 16:53, Stefan Seefeld <ste...@seefeld.name> wrote:
> Hi Jonathan,
> On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:
>
> tools/build/src/tools/python.jam says:
>
> # On *nix, we do not want to link either Boost.Python or Python
> extensions
> # to libpython, because the Python interpreter itself provides all those
> # symbols. If we linked to libpython, we would get duplicate symbols. So
> # declare two targets -- one for building extensions and another for
> # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?
>
> Do all *nix variants use ELF these days ?
>
All the ones in common usage, yes. And for my purposes, I only care about
> Hi Jonathan,
> On 2020-04-03 11:43 a.m., Jonathan Wakely via Boost wrote:
>
> tools/build/src/tools/python.jam says:
>
> # On *nix, we do not want to link either Boost.Python or Python
> extensions
> # to libpython, because the Python interpreter itself provides all those
> # symbols. If we linked to libpython, we would get duplicate symbols. So
> # declare two targets -- one for building extensions and another for
> # embedding.
>
> Why would that give duplicate symbols? Wouldn't ELF symbol interposition
> mean that only one is used? Why would it be a problem to link to libpython?
>
> Do all *nix variants use ELF these days ?
>
one specific OS (Fedora Linux) which is definitely ELF.
> In Fedora's Boost RPM packages we *do* link libboost_python.so to the
> system libpython (and we don't get any problems).
>
> What is your rationale for this, given that (according to the above
> reasoning) symbols will be resolved from the application that's loading the
> module ?
>
That links to https://svn.boost.org/trac10/ticket/2615 which suggests other
people agree that linking to libpython is correct. Dave Abrahams seems to
be the origin of the "linking will cause duplicate symbols" issue, which
might only be the case when the python interpreter is statically linked to
libpython.a. Since linux distros usually (maybe always?) dynamically link
the python interpreter to libpythonX.Ym.so, that isn't an issue, and
Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
distros having to kluge around it.
But to do this, we have
> these two local patches:
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.57.0-python-libpython_dep.patch
>
> and then to make it link to libpython3.7m.so (rather than libpython3.7.so)
> we have this patch to pass the "m" into the Boost build:
> https://src.fedoraproject.org/rpms/boost/blob/master/f/boost-1.66.0-python-abi_letters.patch
>
> I'd prefer not to need these local hacks. Would the upstream Boost.Python
> consider making changes so that libboost_python.so can be (optionally)
> linked to libpython?
>
> Is there a cleaner way to do this than those patches?
>
> I fully agree that whatever seems the right choice of build (including
> link) process for a given platform, we (boost) should incorporate that into
> our own infrastructure, so downstream package maintainers don't need to
> patch it. I'll leave it to the Boost.Build maintainers to answer the
> technical details of that process.
>
since 1.70.0 they break the build somehow, so that the mpi.so python
extensions are no longer built. I am unable to figure out the problem,
because I find jamfiles completely impenetrable. I'd love to try using
cmake instead, but that doesn't seem to be supported for Boost.MPI.
Boost 1.70.0 and 1.71.0 fail to build with:
...updating 24 targets...
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71.0
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71.0
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.so.1
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_serialization.a
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_wserialization.so.1.71
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1
ln-UNIX /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_graph_parallel.so.1.71
common.copy /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/libboost_mpi.a
...updated 24 targets...
+ mkdir -p /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost
+ touch /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/__init__.py
+ mv /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
mv: cannot stat
'/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
No such file or directory
And 1.72.0 fails even earlier, with what looks like a conflict between
building Boost.Serialization with "python" and "python-for-extensions"
properties (if I'm reading python.jam right, which is unlikely):
+ echo ============================= build openmpi-x86_64 ==================
+ ./b2 -d+2 -q -j8 --with-mpi --with-graph_parallel
--build-dir=openmpi-x86_64 variant=release threading=multi
debug-symbols=on pch=off python=3.8 stage
Performing configuration checks
- default address-model : 64-bit
- default architecture : x86
warning: non-free usage requirements <threading>multi ignored
warning: in main-target mpi at
/builddir/build/BUILD/boost_1_72_0/tools/build/src/user-config.jam:6
warning: non-free usage requirements <threading>multi ignored
warning: in main-target boost_mpi at libs/mpi/build/Jamfile.v2:120
warning: non-free usage requirements <threading>multi ignored
warning: in main-target boost_mpi_python at libs/mpi/build/Jamfile.v2:145
error: Name clash for
'<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
error:
error: Tried to build the target twice, with property sets having
error: these incompatible properties:
error:
error: - none
error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
error:
error: Please make sure to have consistent requirements for these
error: properties everywhere in your project, especially for install
error: targets.
Dimitri John Ledkov via Boost
Apr 3, 2020, 12:31:59 PM4/3/20
to bo...@lists.boost.org, Dimitri John Ledkov
Strong objection from me.
No python extensions nor applications that embed a python interpreter
should link with libpythonX.Y. The modules are effectively plugins,
and are compiled to be dlopened by the interpret and interpreter is
the one that must provide the symbols.
How an interpreter has been loaded, can be different from how an
application/extenion is linked, and thus symbols can be different.
Duplicate symbols are not a problem from ELF when they are the same,
it can become a problem when they are conflicting yet compatible. And
at that point modules can fail to load. The current schema allows
sharing compiled extensions across different builds of the python
interpret that may be shipped by different vendors.
Neither Ubuntu nor Debian links libpythonX.Y for any extension
modules, or apps. Please continue not linking against libpythonX.Y and
keep the binaries with unresolved python symbols.
For example, _ssl.so compiled extensions shipped by the python itself
has all the Py* symbols undefined, and is not linked against
libpythonX.Y.
Boost upstream, should follow Python upstream and do what they
recommend, which is not linking with libpythonX.Y.
--
Regards,
Dimitri.
No python extensions nor applications that embed a python interpreter
should link with libpythonX.Y. The modules are effectively plugins,
and are compiled to be dlopened by the interpret and interpreter is
the one that must provide the symbols.
How an interpreter has been loaded, can be different from how an
application/extenion is linked, and thus symbols can be different.
Duplicate symbols are not a problem from ELF when they are the same,
it can become a problem when they are conflicting yet compatible. And
at that point modules can fail to load. The current schema allows
sharing compiled extensions across different builds of the python
interpret that may be shipped by different vendors.
Neither Ubuntu nor Debian links libpythonX.Y for any extension
modules, or apps. Please continue not linking against libpythonX.Y and
keep the binaries with unresolved python symbols.
For example, _ssl.so compiled extensions shipped by the python itself
has all the Py* symbols undefined, and is not linked against
libpythonX.Y.
Boost upstream, should follow Python upstream and do what they
recommend, which is not linking with libpythonX.Y.
Regards,
Dimitri.
Peter Dimov via Boost
Apr 3, 2020, 1:07:36 PM4/3/20
to bo...@lists.boost.org, Peter Dimov
Jonathan Wakely wrote:
> Boost 1.70.0 and 1.71.0 fail to build with:
> ...updating 24 targets...
...
> Boost 1.70.0 and 1.71.0 fail to build with:
> ...updating 24 targets...
> ...updated 24 targets...
...
> + mv
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
> mv: cannot stat
> '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
> No such file or directory
This is not failure to build; the error comes from the script, it can't find
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
> mv: cannot stat
> '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
> No such file or directory
mpi.so.
> And 1.72.0 fails even earlier, with what looks like a conflict between
> building Boost.Serialization with "python" and "python-for-extensions"
> properties (if I'm reading python.jam right, which is unlikely):
> error: Name clash for
> '<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
> error:
> error: Tried to build the target twice, with property sets having
> error: these incompatible properties:
> error:
> error: - none
> error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
I don't think this is a conflict between `python` and
> '<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
> error:
> error: Tried to build the target twice, with property sets having
> error: these incompatible properties:
> error:
> error: - none
> error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
`python-for-extensions`. Rather, it is a conflict between
libboost_serialization.so by itself (without rpath), and
libboost_serialization.so built as a dependency of libboost_mpi.so (for
which someone tries to set rpath.)
What does user-config.jam contain?
(Also... before spending too much time on this, consider trying 1.73 as
well.)
Jonathan Wakely via Boost
Apr 3, 2020, 1:08:15 PM4/3/20
to Dimitri John Ledkov, Jonathan Wakely, bo...@lists.boost.org
On Fri, 3 Apr 2020 at 17:29, Dimitri John Ledkov <xn...@ubuntu.com> wrote:
> Strong objection from me.
>
> No python extensions nor applications that embed a python interpreter
> should link with libpythonX.Y. The modules are effectively plugins,
> and are compiled to be dlopened by the interpret and interpreter is
> the one that must provide the symbols.
>
> How an interpreter has been loaded, can be different from how an
> application/extenion is linked, and thus symbols can be different.
> Duplicate symbols are not a problem from ELF when they are the same,
> it can become a problem when they are conflicting yet compatible. And
> at that point modules can fail to load. The current schema allows
> sharing compiled extensions across different builds of the python
> interpret that may be shipped by different vendors.
>
> Neither Ubuntu nor Debian links libpythonX.Y for any extension
> modules, or apps. Please continue not linking against libpythonX.Y and
> keep the binaries with unresolved python symbols.
>
> For example, _ssl.so compiled extensions shipped by the python itself
> has all the Py* symbols undefined, and is not linked against
> libpythonX.Y.
>
> Boost upstream, should follow Python upstream and do what they
> recommend, which is not linking with libpythonX.Y.
>
That's fine as the default, why do you object to making it configurable, so
> Strong objection from me.
>
> No python extensions nor applications that embed a python interpreter
> should link with libpythonX.Y. The modules are effectively plugins,
> and are compiled to be dlopened by the interpret and interpreter is
> the one that must provide the symbols.
>
> How an interpreter has been loaded, can be different from how an
> application/extenion is linked, and thus symbols can be different.
> Duplicate symbols are not a problem from ELF when they are the same,
> it can become a problem when they are conflicting yet compatible. And
> at that point modules can fail to load. The current schema allows
> sharing compiled extensions across different builds of the python
> interpret that may be shipped by different vendors.
>
> Neither Ubuntu nor Debian links libpythonX.Y for any extension
> modules, or apps. Please continue not linking against libpythonX.Y and
> keep the binaries with unresolved python symbols.
>
> For example, _ssl.so compiled extensions shipped by the python itself
> has all the Py* symbols undefined, and is not linked against
> libpythonX.Y.
>
> Boost upstream, should follow Python upstream and do what they
> recommend, which is not linking with libpythonX.Y.
>
that distros that do link python modules to libpython can do so for
Boost.Python too?
On Fedora, I see
$ ldd /usr/lib64/python3.7/lib-dynload/_ssl.cpython-37m-x86_64-linux-gnu.so
linux-vdso.so.1 (0x00007ffde6583000)
libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007fb5deefb000)
libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007fb5dec1b000)
* libpython3.7m.so.1.0 => /lib64/libpython3.7m.so.1.0
(0x00007fb5de8cf000)*
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fb5de8ad000)
libc.so.6 => /lib64/libc.so.6 (0x00007fb5de6e4000)
libz.so.1 => /lib64/libz.so.1 (0x00007fb5de6ca000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007fb5de6c1000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007fb5de6bc000)
libm.so.6 => /lib64/libm.so.6 (0x00007fb5de576000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb5defed000)
Jonathan Wakely via Boost
Apr 3, 2020, 1:12:11 PM4/3/20
to bo...@lists.boost.org, Jonathan Wakely, Peter Dimov
On Fri, 3 Apr 2020 at 18:07, Peter Dimov via Boost <bo...@lists.boost.org>
wrote:
> Jonathan Wakely wrote:
> > Boost 1.70.0 and 1.71.0 fail to build with:
> > ...updating 24 targets...
> ...
> > ...updated 24 targets...
> ...
> > + mv
> >
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
> >
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
> > mv: cannot stat
> >
> '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
> > No such file or directory
>
> This is not failure to build; the error comes from the script, it can't
> find
> mpi.so.
>
Right, but mpi.so doesn't appear to get built. Maybe it did, and is just
not where our build script expects it, I should check that.
>
> > And 1.72.0 fails even earlier, with what looks like a conflict between
> > building Boost.Serialization with "python" and "python-for-extensions"
> > properties (if I'm reading python.jam right, which is unlikely):
> ...
> > error: Name clash for
> >
> '<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
> > error:
> > error: Tried to build the target twice, with property sets having
> > error: these incompatible properties:
> > error:
> > error: - none
> > error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
>
> I don't think this is a conflict between `python` and
> `python-for-extensions`. Rather, it is a conflict between
> libboost_serialization.so by itself (without rpath), and
> libboost_serialization.so built as a dependency of libboost_mpi.so (for
> which someone tries to set rpath.)
>
Ah, we also patch out the setting of rpaths, maybe we need to make that
happen in an additional place now. Thanks for the clues what to look at
next.
> What does user-config.jam contain?
>
>
import os ;
local RPM_OPT_FLAGS = [ os.environ RPM_OPT_FLAGS ] ;
local RPM_LD_FLAGS = [ os.environ RPM_LD_FLAGS ] ;
using gcc : : : <compileflags>$(RPM_OPT_FLAGS) <linkflags>$(RPM_LD_FLAGS) ;
using mpi ;
using python : 3.8 : /usr/bin/python3 : /usr/include/python3.8m : : : ;
> (Also... before spending too much time on this, consider trying 1.73 as
> well.)
>
>
I will, although I would be surprised if these problems have just gone away
by themselves, given that they seem to be related to downstream
customizations for Fedora, and so not reported upstream to Boost.
wrote:
> Jonathan Wakely wrote:
> > Boost 1.70.0 and 1.71.0 fail to build with:
> > ...updating 24 targets...
> ...
> > ...updated 24 targets...
> ...
> > + mv
> >
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so
> >
> /builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/python3.8/site-packages/openmpi/boost/
> > mv: cannot stat
> >
> '/builddir/build/BUILDROOT/boost-1.71.0-0.1.fc33.x86_64/usr/lib64/openmpi/lib/mpi.so':
> > No such file or directory
>
> This is not failure to build; the error comes from the script, it can't
> find
> mpi.so.
>
not where our build script expects it, I should check that.
>
> > And 1.72.0 fails even earlier, with what looks like a conflict between
> > building Boost.Serialization with "python" and "python-for-extensions"
> > properties (if I'm reading python.jam right, which is unlikely):
> ...
> > error: Name clash for
> >
> '<p/builddir/build/BUILD/boost_1_72_0/stage/lib>libboost_serialization.so.1.72.0'
> > error:
> > error: Tried to build the target twice, with property sets having
> > error: these incompatible properties:
> > error:
> > error: - none
> > error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
>
> I don't think this is a conflict between `python` and
> `python-for-extensions`. Rather, it is a conflict between
> libboost_serialization.so by itself (without rpath), and
> libboost_serialization.so built as a dependency of libboost_mpi.so (for
> which someone tries to set rpath.)
>
happen in an additional place now. Thanks for the clues what to look at
next.
> What does user-config.jam contain?
>
>
import os ;
local RPM_OPT_FLAGS = [ os.environ RPM_OPT_FLAGS ] ;
local RPM_LD_FLAGS = [ os.environ RPM_LD_FLAGS ] ;
using gcc : : : <compileflags>$(RPM_OPT_FLAGS) <linkflags>$(RPM_LD_FLAGS) ;
using mpi ;
using python : 3.8 : /usr/bin/python3 : /usr/include/python3.8m : : : ;
> (Also... before spending too much time on this, consider trying 1.73 as
> well.)
>
>
by themselves, given that they seem to be related to downstream
customizations for Fedora, and so not reported upstream to Boost.
Niall Douglas via Boost
Apr 3, 2020, 1:12:55 PM4/3/20
to bo...@lists.boost.org, Niall Douglas
On 03/04/2020 17:17, Jonathan Wakely via Boost wrote:
> The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667
>
> That links to https://svn.boost.org/trac10/ticket/2615 which suggests other
> people agree that linking to libpython is correct. Dave Abrahams seems to
> be the origin of the "linking will cause duplicate symbols" issue, which
> might only be the case when the python interpreter is statically linked to
> libpython.a. Since linux distros usually (maybe always?) dynamically link
> the python interpreter to libpythonX.Ym.so, that isn't an issue, and
> Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
> distros having to kluge around it.
Dave got that originally from me.
> The change was made for https://bugzilla.redhat.com/show_bug.cgi?id=1102667
>
> That links to https://svn.boost.org/trac10/ticket/2615 which suggests other
> people agree that linking to libpython is correct. Dave Abrahams seems to
> be the origin of the "linking will cause duplicate symbols" issue, which
> might only be the case when the python interpreter is statically linked to
> libpython.a. Since linux distros usually (maybe always?) dynamically link
> the python interpreter to libpythonX.Ym.so, that isn't an issue, and
> Boost's behaviour is wrong, and Fedora and RHEL are probably not the only
> distros having to kluge around it.
The problem was this:
1. Python extension A is built against Boost.Python v1.X.
2. Python extension B is built against Boost.Python v1.Y.
3. What happens if end user imports Python extensions A and B into the
same Python?
If Boost.Python insists on a Python link for Extension A, and a
different Python link for Extension B, now what happens?
The generally accepted hack was to load Python extensions using
dlopen(RTLD_LOCAL), which Python accepted as the least worst of all
options back in the early 2000s, and incorporated that into CPython as
the default semantic for loading extensions. Therefore there is no
global symbol resolution at play here, instead Boost.Python ends up
loading multiple copies of libpython.so, which is for obvious reasons
very bad. So we don't do that.
I have no idea what has happened since then, and all the above
information may be out of date. But that was the rationale ~20 years ago
or so. It made a lot of sense at that time because of the fundamental
brokenness that is ELF. Note that neither PE nor MachO have any of these
problems.
Since ~2002, I patched GCC with -fvisibility in ~2005, and Dave
committed a patchset for exporting visible symbols for a good chunk of
Boost. So it may now be possible to completely change how Boost.Python
does things, and meet your request.
Niall
Jonathan Wakely via Boost
Apr 3, 2020, 1:16:35 PM4/3/20
to bo...@lists.boost.org, Jonathan Wakely
On Fri, 3 Apr 2020 at 18:13, Niall Douglas via Boost <bo...@lists.boost.org>
wrote:
Additionally, in a distro like Fedora everything is built with the same
versions of Boost and Python anyway.
wrote:
versions of Boost and Python anyway.
Jonathan Wakely via Boost
Apr 3, 2020, 1:21:54 PM4/3/20
to bo...@lists.boost.org, Jonathan Wakely, Peter Dimov
On Fri, 3 Apr 2020 at 18:11, Jonathan Wakely <jwakel...@kayari.org>
wrote:
I should add that we build in stages, because we want to build Boost.MPI
twice, for openmpi and MPICH, so we do a build without MPI first:
./b2 -d+2 -q \
--without-mpi --without-graph_parallel --build-dir=serial \
./b2 -d+2 \
--with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
of properties.
wrote:
twice, for openmpi and MPICH, so we do a build without MPI first:
./b2 -d+2 -q \
--without-mpi --without-graph_parallel --build-dir=serial \
variant=release threading=multi debug-symbols=on pch=off \
python=3.8 \
stage
and then for each MPI backend we do:
python=3.8 \
stage
./b2 -d+2 \
--with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
variant=release threading=multi debug-symbols=on pch=off \
python=3.8 stage
so that's probably why libboost_serialization ends up with different sets
python=3.8 stage
of properties.
Peter Dimov via Boost
Apr 3, 2020, 2:14:10 PM4/3/20
to bo...@lists.boost.org, Peter Dimov
Jonathan Wakely wrote:
> > (Also... before spending too much time on this, consider trying 1.73 as
> > well.)
>
> I will, although I would be surprised if these problems have just gone
> away by themselves, given that they seem to be related to downstream
> customizations for Fedora, and so not reported upstream to Boost.
They won't... but new ones may appear :-) as we changed the directory for
> > (Also... before spending too much time on this, consider trying 1.73 as
> > well.)
>
> I will, although I would be surprised if these problems have just gone
> away by themselves, given that they seem to be related to downstream
> customizations for Fedora, and so not reported upstream to Boost.
mpi.so because it wasn't appropriate for Boost to install a file named
`mpi.so` into the default destination /usr/lib.
Peter Dimov via Boost
Apr 3, 2020, 2:24:33 PM4/3/20
to bo...@lists.boost.org, Peter Dimov
Jonathan Wakely wrote:
> I should add that we build in stages, because we want to build Boost.MPI
> twice, for openmpi and MPICH, so we do a build without MPI first:
>
> ./b2 -d+2 -q \
> --without-mpi --without-graph_parallel --build-dir=serial \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 \
> stage
>
> and then for each MPI backend we do:
>
> ./b2 -d+2 \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage
>
> so that's probably why libboost_serialization ends up with different sets
> of properties.
That's why you weren't getting the error before 1.72, but not the reason why
> I should add that we build in stages, because we want to build Boost.MPI
> twice, for openmpi and MPICH, so we do a build without MPI first:
>
> ./b2 -d+2 -q \
> --without-mpi --without-graph_parallel --build-dir=serial \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 \
> stage
>
> and then for each MPI backend we do:
>
> ./b2 -d+2 \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage
>
> so that's probably why libboost_serialization ends up with different sets
> of properties.
libboost_serialization ends up with the different property sets.
Peter Dimov via Boost
Apr 3, 2020, 2:27:11 PM4/3/20
to bo...@lists.boost.org, Peter Dimov
> That's why you weren't getting the error before 1.72, but not the reason
> why libboost_serialization ends up with the different property sets.
FWIW, my Travis build/install of --with-mpi works:
> why libboost_serialization ends up with the different property sets.
https://travis-ci.org/github/boostorg/boost_install/jobs/669821455
and I can't find the place that's setting <dll-path>/usr/lib
<dll-path>/usr/lib/python3.8/config.
Peter Dimov via Boost
Apr 3, 2020, 2:43:40 PM4/3/20
to Jonathan Wakely, bo...@lists.boost.org, Peter Dimov
Jonathan Wakely wrote:
> > error: - none
> > error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
...
> > error: - none
> > error: - <dll-path>/usr/lib <dll-path>/usr/lib/python3.8/config
> and then for each MPI backend we do:
>
> ./b2 -d+2 \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage
I think that what you need to do here is find the place setting <dll-path>
>
> ./b2 -d+2 \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage
on (I assume) libboost_mpi.so, remove it, and then pass dll-path on the
command line, so that it applies to the whole build:
> ./b2 -d+2 \
> --with-mpi --with-graph_parallel --build-dir=$MPI_COMPILER \
> variant=release threading=multi debug-symbols=on pch=off \
> python=3.8 stage
Jonathan Wakely via Boost
Apr 3, 2020, 2:52:38 PM4/3/20
to bo...@lists.boost.org, Jonathan Wakely, Peter Dimov
On Fri, 3 Apr 2020, 19:26 Peter Dimov via Boost, <bo...@lists.boost.org>
wrote:
> > That's why you weren't getting the error before 1.72, but not the reason
> > why libboost_serialization ends up with the different property sets.
>
> FWIW, my Travis build/install of --with-mpi works:
>
> https://travis-ci.org/github/boostorg/boost_install/jobs/669821455
>
> and I can't find the place that's setting <dll-path>/usr/lib
> <dll-path>/usr/lib/python3.8/config.
>
I think that's our local patch, I'll check later.
wrote:
> > That's why you weren't getting the error before 1.72, but not the reason
> > why libboost_serialization ends up with the different property sets.
>
> FWIW, my Travis build/install of --with-mpi works:
>
> https://travis-ci.org/github/boostorg/boost_install/jobs/669821455
>
> and I can't find the place that's setting <dll-path>/usr/lib
> <dll-path>/usr/lib/python3.8/config.
>
Reply all
Reply to author
Forward
0 new messages