-Wl,--as-needed considered possibly harmful
Simon Richter
due to the recent dpkg-shlibdeps changes, people have started adding
-Wl,--as-needed into their LDFLAGS.
THIS IS NOT A GOOD IDEA.
You are essentially telling gcc to pass an option to the linker without
understanding what it does, and, more specifically, an option that tells
the linker to second-guess the gcc compiler driver. This can introduce
really interesting and subtle bugs that will be difficult to find.
If there are broken scripts adding too many libraries then it is time to
fix those scripts, not employ an evil hack that makes the symptoms go away.
Simon
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Zack Weinberg
> -Wl,--as-needed into their LDFLAGS.
>
> THIS IS NOT A GOOD IDEA.
>
> You are essentially telling gcc to pass an option to the linker without
> understanding what it does, and, more specifically, an option that tells
> the linker to second-guess the gcc compiler driver. This can introduce
> really interesting and subtle bugs that will be difficult to find.
To first order, the only scenario I am aware of in which it can cause
problems is if someone uses a global variable with a C++ constructor
in a shared library, that constructor does critical work for the
application the library is linked into, and the application does not
reference any symbols whatsoever from the shared library. This is not
impossible, but it is so unlikely IMO that the possibliity can be
neglected.
I have in the past argued for --as-needed to be made the default in
upstream binutils; that's how safe I think it is. (Upstream
maintainers, conscious of the above and other (isomorphic) corner
cases, wanted a distribution to try it on a large scale first. Thus I
am very happy to see Debian experimenting with it.)
I'm curious what prompted your message. Did a program you use
actually break? What was the failure mode?
> If there are broken scripts adding too many libraries then it is time to
> fix those scripts, not employ an evil hack that makes the symptoms go away.
There are a *lot* of broken scripts. Would you like to mass-file bugs
on every package that contains a binary that links to libnsl? (this
iscommon, thanks to a buggy example in the autoconf manual, but
completely unnecessary under glibc for anything other than a small
handful of NIS-related programs, which probably have their own copies
of that code anyway) How about programs that link to libm, but every
symbol they use from libm happens to have been replaced by inline
code? (I have seen this happen in real life.) Libraries that are
linked against libpthread as a defensive measure for actual use of
threads by their users, but only need the stubs in libc for that?
(Can causes severe performance hits for single-threaded users of that
library.)
zw
Romain Beauxis
> I'm curious what prompted your message. Did a program you use
> actually break? What was the failure mode?
Yes, I agree too.
I've been using this flag recently, and in most cases it just does what's
required...
I think that before warning in capital letters it would be good to study and
advertise the case when it's potentially harmfull..
There's a good reference there:
http://www.gentoo.org/proj/en/qa/asneeded.xml
Romain
Josselin Mouette
> Hi,
>
> due to the recent dpkg-shlibdeps changes, people have started adding
> -Wl,--as-needed into their LDFLAGS.
>
> THIS IS NOT A GOOD IDEA.
Of course it is. This is the most reliable way of dropping unneeded
dependencies. All other ways will keep some of them.
If there are bugs in --as-needed, we need to fix them. It is much easier
than fixing thousands of packages. Or are you willing to drop
dh_fixperms and fix thousands of broken packages by hand as well?
> If there are broken scripts adding too many libraries then it is time to
> fix those scripts, not employ an evil hack that makes the symptoms go away.
Some of these scripts cannot be “fixed”. And for others, well, good luck
in getting libtool’s upstream to integrate the required changes. Maybe
they will be here in 5 years, and in 15 years all upstreams will have
migrated to a recent enough version.
--
.''`.
: :' : We are debian.org. Lower your prices, surrender your code.
`. `' We will add your hardware and software distinctiveness to
`- our own. Resistance is futile.
Bernhard R. Link
> > due to the recent dpkg-shlibdeps changes, people have started adding
> > -Wl,--as-needed into their LDFLAGS.
> >
> > THIS IS NOT A GOOD IDEA.
>
> > fix those scripts, not employ an evil hack that makes the symptoms go away.
>
> in getting libtool???s upstream to integrate the required changes. Maybe
> they will be here in 5 years, and in 15 years all upstreams will have
> migrated to a recent enough version.
Just curing the symptoms instead of the problems will not help to get
there any sooner. There might be stuff where the stuff at hand is not
easily fixed, but that is no reason to not consider it not worth doing.
(or to consider using -Wl,--as-needed as anything short of giving in
to a broken build system and putting the dirt under the carpet, so
people do no longer get dirty feet walking around most of the time).
Hochachtungsvoll,
Bernhard R. Link
Josselin Mouette
Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit :
> Just curing the symptoms instead of the problems will not help to get
> there any sooner.
What if there is no problem?
For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
-lglib-2.0 and -lm. And this is perfectly intentional. The former is
here because you can assume glib functions are available when doing
"#include <gtk/gtk.h>". The latter is here because some macros need math
functions. Still, many binaries using it don’t need glib, and most of
them don’t need libm. There is nothing that can be done to avoid that,
the only way is a check that removes the dependency if it isn’t actually
needed.
Kurt Roeckx
> Hi,
>
> Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit :
> > Just curing the symptoms instead of the problems will not help to get
> > there any sooner.
>
> What if there is no problem?
>
> For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
> -lglib-2.0 and -lm. And this is perfectly intentional. The former is
> here because you can assume glib functions are available when doing
> "#include <gtk/gtk.h>". The latter is here because some macros need math
> functions. Still, many binaries using it don't need glib, and most of
> them don't need libm. There is nothing that can be done to avoid that,
> the only way is a check that removes the dependency if it isn't actually
> needed.
I can't seem to find any example of a macro calling a math functions?
I also can't find any header that includes <math.h>. I do see
that libgtk-x11-2.0.so uses symbols from libm, like log10, atan2, sin,
cos, tan, pow, ceil. But I can't find anything in the header files
that would use them.
It's gdk-pixbuf-2.0.pc and cairo.pc that have -lm in them.
Anyway, I think it is bad style to use macro's in public header
files that call functions from another library. It's also easy
to replace them with real functions. If there are such functions
being called I suggest you replace them.
What might be harder is avoiding the includes from other libraries
because you use types from the other library. It would be great
if you didn't have to do that, and there are some ways to avoid that.
Kurt
Romain Beauxis
> Anyway, I think it is bad style to use macro's in public header
> files that call functions from another library. It's also easy
> to replace them with real functions. If there are such functions
> being called I suggest you replace them.
>
> What might be harder is avoiding the includes from other libraries
> because you use types from the other library. It would be great
> if you didn't have to do that, and there are some ways to avoid that.
Ok, let's consider another very simple case:
libshout allows to perform streaming of speex, vorbis, and theora formats.
Hence, when asking for the libs to link with, you got -lspeex and -ltheora
since it's needed to cover all build cases.
However, most of the applications don't need to link against theora and speex.
So, not only do I agree with Josselin, but I think that looking at each
project to fullfill any possible use case IS the problem itself. we can't
expect any upstream to commit and release every little hack that might solve
this or that issue.
The solution is, indeed, to link what's needed for each application and
nothing else.
Romain
Russ Allbery
> Ok, let's consider another very simple case:
> libshout allows to perform streaming of speex, vorbis, and theora formats.
>
> Hence, when asking for the libs to link with, you got -lspeex and -ltheora
> since it's needed to cover all build cases.
Why? Aren't you calling a function in libshout to stream such files?
libshout should then link against libseepx and libtheora, but why should
the *application* have to do so if it's using libshout?
This sounds broken to me.
--
Russ Allbery (r...@debian.org) <http://www.eyrie.org/~eagle/>
Nikita V. Youshchenko
> libshout allows to perform streaming of speex, vorbis, and theora formats.
>
> Hence, when asking for the libs to link with, you got -lspeex and -ltheora
> since it's needed to cover all build cases.
>
> However, most of the applications don't need to link against theora and
> speex.
Doesn't libshout reference symbols from libspeex and libtheora?
If it does, resulting binary must be linked against both these libraries.
Unresolved symbols within binaries are really bad, because they will once
be unexpectedly called (e.g. in some init or deinit or error-handling), and
binary will crash, leaving user data unsaved.
If it does not, then what for -lspeex and -ltheora are in the script output?
Kurt Roeckx
> > Ok, let's consider another very simple case:
> > libshout allows to perform streaming of speex, vorbis, and theora formats.
> >
> > Hence, when asking for the libs to link with, you got -lspeex and -ltheora
> > since it's needed to cover all build cases.
> >
> > However, most of the applications don't need to link against theora and
> > speex.
>
> Doesn't libshout reference symbols from libspeex and libtheora?
>
> If it does, resulting binary must be linked against both these libraries.
Where "binary" means libshout, yes. The application does not
directly use libspeex and libtheora so should not have any undefined
symbols that are defined in either libspeex or libtheora.
> Unresolved symbols within binaries are really bad, because they will once
> be unexpectedly called (e.g. in some init or deinit or error-handling), and
> binary will crash, leaving user data unsaved.
>
> If it does not, then what for -lspeex and -ltheora are in the script output?
That is only useful for linking to a static version of libshout.
Kurt
Romain Beauxis
> > However, most of the applications don't need to link against theora and
> > speex.
>
> Doesn't libshout reference symbols from libspeex and libtheora?
Yes
> If it does, resulting binary must be linked against both these libraries.
> Unresolved symbols within binaries are really bad, because they will once
> be unexpectedly called (e.g. in some init or deinit or error-handling), and
> binary will crash, leaving user data unsaved.
I think you are confusing the issue.
Yes, libshout uses symbols from libspeex:
objdump -x /usr/lib/libshout.so | grep NEEDED | grep speex
NEEDED libspeex.so.1
However, not all applications uses those symbols in their current code, so
they do not need to be linked against speex themselves.
Of course, correct dependencies will be pulled when requesting libshout.
Romain
Steve Langasek
> Yes
And therefore, the scripts generating dependency information for dynamically
linking to libshout should *not* list libspeex or libtheora, and
consequently there's no need for -Wl,--as-needed. (When statically linking,
-Wl,--as-needed is also irrelevant, because unused symbols from static libs
are already discarded.)
> Of course, correct dependencies will be pulled when requesting libshout.
No, currently the dependencies pulled in are overbroad. Please fix
/usr/lib/pkgconfig/shout.pc to list the dependent libraries in
"Requires.private", not in "Requires".
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slan...@ubuntu.com vor...@debian.org
Steve Langasek
> Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit :
> > Just curing the symptoms instead of the problems will not help to get
> > there any sooner.
> What if there is no problem?
> For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
> -lglib-2.0 and -lm. And this is perfectly intentional.
Just because it's intentional doesn't mean it isn't absurd and wrong.
> The former is here because you can assume glib functions are available
> when doing "#include <gtk/gtk.h>". The latter is here because some macros
> need math functions. Still, many binaries using it don’t need glib, and
> most of them don’t need libm. There is nothing that can be done to avoid
> that, the only way is a check that removes the dependency if it isn’t
> actually needed.
No, what can be done is to fix upstream's broken declaration that 'you can
assume glib functions are available when doing "#include <gtk/gtk.h>"'. It
doesn't follow that just because this works in practice, it should be a
supported usage.
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slan...@ubuntu.com vor...@debian.org
Steve Langasek
> > However, not all applications uses those symbols in their current code, so
> > they do not need to be linked against speex themselves.
> And therefore, the scripts generating dependency information for dynamically
> linking to libshout should *not* list libspeex or libtheora, and
> consequently there's no need for -Wl,--as-needed. (When statically linking,
> -Wl,--as-needed is also irrelevant, because unused symbols from static libs
> are already discarded.)
> > Of course, correct dependencies will be pulled when requesting libshout.
> No, currently the dependencies pulled in are overbroad. Please fix
> /usr/lib/pkgconfig/shout.pc to list the dependent libraries in
> "Requires.private", not in "Requires".
... also, -Wl,--as-needed is *not* a complete solution for the problems
caused by generating extra -l arguments. Every -lfoo option passed to the
linker requires the corresponding -dev package to be installed at build
time; while it is a bug if the -dev packages don't declare their
dependencies, the impact of such bugs (which do happen fairly often) would
be limited to static linking if appropriately-constructed .pc files were
available.
Felipe Sateler
> ... also, -Wl,--as-needed is *not* a complete solution for the problems
> caused by generating extra -l arguments. Every -lfoo option passed to the
> linker requires the corresponding -dev package to be installed at build
> time; while it is a bug if the -dev packages don't declare their
> dependencies, the impact of such bugs (which do happen fairly often) would
> be limited to static linking if appropriately-constructed .pc files were
> available.
Note that pkgconfig is not the only problem here. Some extra links arise
from the fact that several utilities or plugins for an application are
built with the same "environment" (ie: the same flags), and not all of them
require all the linked libraries. Of course, one could argue that fixing
the build script is the way to go, but sometimes it is not trivial to do
that (badly crafted or unmaintained build scripts tend to be quite ugly).
--
Felipe Sateler
Loïc Minier
> If there are broken scripts adding too many libraries then it is time to
> fix those scripts, not employ an evil hack that makes the symptoms go away.
We met actual cases were --as-needed breaks things; however, for pure C
program, I think combining it with "-z defs" should be safe:
LDFLAGS += -Wl,-z,defs -Wl,--as-needed
Sadly, there are many cases where "-z defs" will actually fail the
build, for example when creating Python bindings: these may not be
linked to libpython or to python as they might be loaded from either
one, and you don't want to load the other one (if you run within
python, you don't want to load libpython and vice-versa). In such
cases, -z defs will fail the build because of missing link flags for
the Python symbols.
I heartily agree with other people that fixing the build rules to not
add superfluous link flags is the best thing to do, but this ain't easy
in many cases, and --as-needed might allow us to benefit from shorter
Depends immediately. If fixing the build is too hard and your build
passes with -z defs, then you are probably safe with --as-needed.
NB: -z defs is the same as --no-undefined.
--
Loďc Minier
Loïc Minier
> I have in the past argued for --as-needed to be made the default in
> upstream binutils; that's how safe I think it is. (Upstream
> maintainers, conscious of the above and other (isomorphic) corner
> cases, wanted a distribution to try it on a large scale first. Thus I
> am very happy to see Debian experimenting with it.)
>
> I'm curious what prompted your message. Did a program you use
> actually break? What was the failure mode?
When this was added to pkg-gnome at large, we were hit by at least
toolchain issues on some arches (was it hppa?) -- now fixed -- and also
got some borken packages. I think gnome-panel / gnome-applets and
gedit were the two packages that were hit when we added --as-needed.
I think in both cases it was something like the main binary being
linked to many utility libraries (because it was easy to link it to
everything which configure found), and then the plugin calling directly
into these libs, but not being linked to them. If the main binary
didn't actually use the libs, these were stripped out, and then the
dlopening of the plugins would fail.
These were really issues in the build process, issues which would
have been caught with -z defs before --as-needed (which is why I'm
advocating usage of -z defs before --as-needed), but slipped silently
in the end package and actually hit users.
I don't think there's an insanely strong incentive to make --as-needed
the default or to not make it; distros such as Debian and Gentoo are
using it wildly at the moment; it just can point out some surprize
bugs in my experience.
HTH,
--
Loďc Minier
Sune Vuorela
> I think in both cases it was something like the main binary being
> linked to many utility libraries (because it was easy to link it to
> everything which configure found), and then the plugin calling directly
> into these libs, but not being linked to them. If the main binary
> didn't actually use the libs, these were stripped out, and then the
> dlopening of the plugins would fail.
> These were really issues in the build process, issues which would
> have been caught with -z defs before --as-needed (which is why I'm
> advocating usage of -z defs before --as-needed), but slipped silently
> in the end package and actually hit users.
Isn't the new dpkg-shlipdeps warning about symbols it can't find ?
/Sune
Loïc Minier
> Isn't the new dpkg-shlipdeps warning about symbols it can't find ?
Indeed; I was telling a story from the dark ages where .debs were
crafted with silex in your bare hands.
That said, -z defs is a good idea to check all intermediate objects and
to make the build actually fail on the first incorrect intermediate
object (dpkg-shlibdeps will only tell you at the end of the build and
only tell you about the installed final objects with missing NEEDED).
--
Loďc Minier
Josselin Mouette
> > For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
> > -lglib-2.0 and -lm. And this is perfectly intentional.
>
> Just because it's intentional doesn't mean it isn't absurd and wrong.
It may be absurd, but I don’t think it is wrong.
> No, what can be done is to fix upstream's broken declaration that 'you can
> assume glib functions are available when doing "#include <gtk/gtk.h>"'. It
> doesn't follow that just because this works in practice, it should be a
> supported usage.
When many of the types used by GTK+ are those provided by GLib, it
sounds wrong to ask developers to include the GLib headers to have these
types available.
Raphael Hertzog
> On 2007-12-10, Loïc Minier <lo...@dooz.org> wrote:
> > I think in both cases it was something like the main binary being
> > linked to many utility libraries (because it was easy to link it to
> > everything which configure found), and then the plugin calling directly
> > into these libs, but not being linked to them. If the main binary
> > didn't actually use the libs, these were stripped out, and then the
> > dlopening of the plugins would fail.
> > These were really issues in the build process, issues which would
> > have been caught with -z defs before --as-needed (which is why I'm
> > advocating usage of -z defs before --as-needed), but slipped silently
> > in the end package and actually hit users.
>
> Isn't the new dpkg-shlipdeps warning about symbols it can't find ?
Only if the library has a SONAME. This is intended so that
perl/python/apache modules do not generate bad warnings: they rely on the
binary dlopening them to provide the symbols (and it's okay that way).
So the problem might still be missed on some plugins without SONAME.
Cheers,
--
Raphaël Hertzog
Le best-seller français mis à jour pour Debian Etch :
http://www.ouaza.com/livre/admin-debian/
Neil Williams
> Le dimanche 09 décembre 2007 à 15:44 -0800, Steve Langasek a écrit :
>>> For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
I don't have a problem with libglib2.0-0 in gtk+2.0.pc - it may well be
correct to have that one in the pkgconfig because gtk headers define
variables in terms of Glib typedefs. (I have to do the same with libqof1).
The actual string is:
-lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 -lm
-lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl
-lglib-2.0
As I see it, there are a few problems here, one of which is in libglib2.0:
1. Glib: gmodule2.0 (and it's variants) shouldn't export -ldl. This is
minor because it's a libc6 lib anyway so it doesn't cause an extra
dependency but it will be the only "unwanted" link in the next version
of libqof1 now that I've fixed the others.
2. Gtk+2.0 really shouldn't export gmodule. Even if there is a reason
for lglib2.0, there is no reason for -lgmodule. It is trivial to omit
this linkage from the gtk+2.0.pc file.
3. Ditto for gobject although I might be wrong on that one.
4. Gtk+2: There is no need, as I see it, for atk, m, pangocairo, pango
or cairo to be Requires: in gtk+2.0.pc - these should be
Requires.private: instead.
This is the single largest cause of unwanted linking in GPE.
Sadly, it isn't possible to get lintian to check for these inter-package
problems but the basic check, IMHO, would be to check the headers of the
-dev package against the headers of the package being considered for
Requires: If none match, it should be Requires.private:
It should be possible to automate that in perl or python - gather the
declarations of your package into a hash, read in the declarations of
the package being considered for Requires:. No matches ->
Requires.private or omit altogether.
The problem is how many packages already have this assumption built-in
and will FTBFS when gtk+2.0.pc changes to drop the extra libraries.
Theoretically, other packages should have done things properly and
specified their -dev packages in full but I think that isn't going to be
the reality.
>>> -lglib-2.0 and -lm. And this is perfectly intentional.
>> Just because it's intentional doesn't mean it isn't absurd and wrong.
>
> It may be absurd, but I don’t think it is wrong.
>
>> No, what can be done is to fix upstream's broken declaration that 'you can
>> assume glib functions are available when doing "#include <gtk/gtk.h>"'. It
>> doesn't follow that just because this works in practice, it should be a
>> supported usage.
>
> When many of the types used by GTK+ are those provided by GLib, it
> sounds wrong to ask developers to include the GLib headers to have these
> types available.
>
Maybe so, but it doesn't excuse the rest.
--
Neil Williams
=============
http://www.data-freedom.org/
http://www.nosoftwarepatents.com/
http://www.linux.codehelp.co.uk/
Stephen Gran
> What happens for a user who (however absurd or insane he might be to
> try this with gtk+) tries to link his application statically?
>
> Perhaps the "absurd and wrong" part is that pkg-config does not
> provide a way to distinguish between use cases, and that the name for
> the current behavior should also be --static-libs rather than --libs,
> but there is a good reason to provide the transitive closure of
> dependencies for a package *somewhere* in pkg-config.
That is supported in pkg-config, but no upstream I've ever met seems to
understand it yet.
--
-----------------------------------------------------------------
| ,''`. Stephen Gran |
| : :' : sg...@debian.org |
| `. `' Debian user, admin, and developer |
| `- http://www.debian.org |
-----------------------------------------------------------------
Michael Poole
> On Sun, Dec 09, 2007 at 07:23:00PM +0100, Josselin Mouette wrote:
>
>> Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit :
>> > Just curing the symptoms instead of the problems will not help to get
>> > there any sooner.
>
>> What if there is no problem?
>
>> For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
>> -lglib-2.0 and -lm. And this is perfectly intentional.
>
> Just because it's intentional doesn't mean it isn't absurd and wrong.
What happens for a user who (however absurd or insane he might be to
try this with gtk+) tries to link his application statically?
Perhaps the "absurd and wrong" part is that pkg-config does not
provide a way to distinguish between use cases, and that the name for
the current behavior should also be --static-libs rather than --libs,
but there is a good reason to provide the transitive closure of
dependencies for a package *somewhere* in pkg-config.
Michael Poole
Pierre Habouzit
> Steve Langasek writes:
>
> > On Sun, Dec 09, 2007 at 07:23:00PM +0100, Josselin Mouette wrote:
> >
> >> Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit :
> >> > Just curing the symptoms instead of the problems will not help to get
> >> > there any sooner.
> >
> >> What if there is no problem?
> >
> >> For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
> >> -lglib-2.0 and -lm. And this is perfectly intentional.
> >
> > Just because it's intentional doesn't mean it isn't absurd and wrong.
>
> What happens for a user who (however absurd or insane he might be to
> try this with gtk+) tries to link his application statically?
>
> Perhaps the "absurd and wrong" part is that pkg-config does not
> provide a way to distinguish between use cases, and that the name for
Wrong, please read pkg-config(1) and think again.
$ pkg-config tokyocabinet --libs
-ltokyocabinet
$ pkg-config tokyocabinet --libs --static
-ltokyocabinet -lz -lpthread -lm -lc
$ grep Libs /usr/lib/pkgconfig/tokyocabinet.pc
Libs: -L${libdir} -ltokyocabinet
Libs.private: -lz -lpthread -lm -lc
pkg-config *is* perfectly suited for that matter actually, and
"broken" pkgconfig upstreams are this trivial to fix. My upstream for
tokyocabinet uses in a tokyocabinet.pc.in:
Libs: -L${libdir} -ltokyocabinet @LIBS@
the fix is just to put the @LIBS@ onto Libs.private and be done with it.
Upstreams using pkg-config are actually the first _easy_ targets to fix
the dependency issues _they_ generate in others binaries. And there is
no reason no to fix those when it's easy to do so.
--
·O· Pierre Habouzit
··O madc...@debian.org
OOO http://www.madism.org
Steve Langasek
> Steve Langasek writes:
> > On Sun, Dec 09, 2007 at 07:23:00PM +0100, Josselin Mouette wrote:
> >> Le dimanche 09 décembre 2007 à 19:11 +0100, Bernhard R. Link a écrit :
> >> > Just curing the symptoms instead of the problems will not help to get
> >> > there any sooner.
> >> What if there is no problem?
> >> For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
> >> -lglib-2.0 and -lm. And this is perfectly intentional.
> > Just because it's intentional doesn't mean it isn't absurd and wrong.
> What happens for a user who (however absurd or insane he might be to
> try this with gtk+) tries to link his application statically?
Not that you should link statically with gtk (this yields the classic
problem of internal ABI skew when statically linking with a library that
uses dlopen), but pkg-config --static does exactly this.
Michael Poole
> This one time, at band camp, Michael Poole said:
>> What happens for a user who (however absurd or insane he might be to
>> try this with gtk+) tries to link his application statically?
>>
>> Perhaps the "absurd and wrong" part is that pkg-config does not
>> provide a way to distinguish between use cases, and that the name for
>> the current behavior should also be --static-libs rather than --libs,
>> but there is a good reason to provide the transitive closure of
>> dependencies for a package *somewhere* in pkg-config.
>
> That is supported in pkg-config, but no upstream I've ever met seems to
> understand it yet.
Ah, I see. I overlooked the --static flag to pkg-config because (case
in point) "pkg-config --libs gtk+-2.0" only misses seven of the
libraries that the --static version uses. I would guess this is
because "Requires.private" is not mentioned in the pkg-config man
page. At least on my system, Libs.private seems to be used correctly
by most or all .pc files, but Requires.private use is spotty.
Thank you for highlighting that.
Michael Poole
Daniel Macks
> Steve Langasek writes:
> >>
> >> -lglib-2.0 and -lm. And this is perfectly intentional.
> >
> > Just because it's intentional doesn't mean it isn't absurd and wrong.
>
> What happens for a user who (however absurd or insane he might be to
> try this with gtk+) tries to link his application statically?
>
> Perhaps the "absurd and wrong" part is that pkg-config does not
> provide a way to distinguish between use cases, and that the name for
> the current behavior should also be --static-libs rather than --libs,
> but there is a good reason to provide the transitive closure of
> dependencies for a package *somewhere* in pkg-config.
pkg-config *does* have a --static flag, which does appear to cause
--libs to emit Libs.private and other things that would normally be
handled by the dynamic-linking of shared libs.
dan
--
Daniel Macks
dma...@netspace.org
http://www.netspace.org/~dmacks
Hendrik Sattler
> Stephen Gran writes:
> > This one time, at band camp, Michael Poole said:
> >> What happens for a user who (however absurd or insane he might be to
> >> try this with gtk+) tries to link his application statically?
> >>
> >> Perhaps the "absurd and wrong" part is that pkg-config does not
> >> provide a way to distinguish between use cases, and that the name for
> >> the current behavior should also be --static-libs rather than --libs,
> >> but there is a good reason to provide the transitive closure of
> >> dependencies for a package *somewhere* in pkg-config.
> >
> > That is supported in pkg-config, but no upstream I've ever met seems to
> > understand it yet.
>
> Ah, I see. I overlooked the --static flag to pkg-config because (case
> in point) "pkg-config --libs gtk+-2.0" only misses seven of the
> libraries that the --static version uses. I would guess this is
> because "Requires.private" is not mentioned in the pkg-config man
> page. At least on my system, Libs.private seems to be used correctly
> by most or all .pc files, but Requires.private use is spotty.
The flaw in the documentation is surely the part to blame here.
Requires.Private works but is not mentioned anywhere. Neither is the --static
option in the SYNOPSIS part of the man page.
Not many use undocumented stuff.
HS
Pierre Habouzit
You're kidding ? If someone that produces libraries stop at the SYNOPSIS
section while reading documentation, he's a moron. The pkg-config tool
has a very concise documentation that says it all, --static is
documented in the OPTIONS section, and Requires, Libs, Libs.private in
METADATA FILE SYNTAX where due. Calling it undocumented is just either
bad faith or well, I can't see what else it can be in fact.
Simon Richter
Josselin Mouette wrote:
>> No, what can be done is to fix upstream's broken declaration that 'you can
>> assume glib functions are available when doing "#include <gtk/gtk.h>"'. It
>> doesn't follow that just because this works in practice, it should be a
>> supported usage.
> When many of the types used by GTK+ are those provided by GLib, it
> sounds wrong to ask developers to include the GLib headers to have these
> types available.
If it is really expected that type declarations are to be shared between
a program and a library, then the program becomes dependent on the
library's ABI without this dependency being formally expressed in any
usable form, which is broken in and by itself.
GTK needs to provide its own definitions of types and not expose
interfaces it does not control.
Simon
Neil Williams
Pierre Habouzit <madc...@debian.org> wrote:
> > > > That is supported in pkg-config, but no upstream I've ever met seems to
> > > > understand it yet.
> > >
> > > Ah, I see. I overlooked the --static flag to pkg-config because (case
> > > in point) "pkg-config --libs gtk+-2.0" only misses seven of the
> > > libraries that the --static version uses. I would guess this is
> > > because "Requires.private" is not mentioned in the pkg-config man
> > > page. At least on my system, Libs.private seems to be used correctly
> > > by most or all .pc files, but Requires.private use is spotty.
> >
> > The flaw in the documentation is surely the part to blame here.
> > Requires.Private works but is not mentioned anywhere. Neither is the --static
> > option in the SYNOPSIS part of the man page.
> > Not many use undocumented stuff.
>
> You're kidding ? If someone that produces libraries stop at the SYNOPSIS
> section while reading documentation, he's a moron. The pkg-config tool
> has a very concise documentation that says it all, --static is
> documented in the OPTIONS section, and Requires, Libs, Libs.private in
> METADATA FILE SYNTAX where due. Calling it undocumented is just either
> bad faith or well, I can't see what else it can be in fact.
What *is* missing is documentation of Requires.private:
Pavlov Konstantin
> Steve Langasek wrote:
>
> > ... also, -Wl,--as-needed is *not* a complete solution for the problems
> > caused by generating extra -l arguments. Every -lfoo option passed to the
> > linker requires the corresponding -dev package to be installed at build
> > time; while it is a bug if the -dev packages don't declare their
> > dependencies, the impact of such bugs (which do happen fairly often) would
> > be limited to static linking if appropriately-constructed .pc files were
> > available.
>
> Note that pkgconfig is not the only problem here. Some extra links arise
> from the fact that several utilities or plugins for an application are
> built with the same "environment" (ie: the same flags), and not all of them
> require all the linked libraries. Of course, one could argue that fixing
> the build script is the way to go, but sometimes it is not trivial to do
> that (badly crafted or unmaintained build scripts tend to be quite ugly).
However, most of them are fixable. The only two problems we (ALT) had/have
with default --as-needed were mpeg4ip suit and net-snmp (though it's being
fixed already).
But again, not every upstream author accepts changes to .pc; but i've
never seen one not wanting to fix a build system.
--
ALT Linux Team,
ALT Linux Sisyphus project,
http://www.altlinux.com
Julien Cristau
> When many of the types used by GTK+ are those provided by GLib, it
> sounds wrong to ask developers to include the GLib headers to have these
> types available.
>
It sounds wrong not to ask them to use -lglib-2.0 themselves if they
call glib functions.
Cheers,
Julien
Julien Cristau
> On Mon, 10 Dec 2007, Sune Vuorela wrote:
> > Isn't the new dpkg-shlipdeps warning about symbols it can't find ?
>
> Only if the library has a SONAME. This is intended so that
> perl/python/apache modules do not generate bad warnings: they rely on the
> binary dlopening them to provide the symbols (and it's okay that way).
>
> So the problem might still be missed on some plugins without SONAME.
>
And for plugins/modules with an SONAME, it will be missed in the noise
of warnings about symbols provided by the main binary/library.
Cheers,
Julien
Josselin Mouette
> I don't have a problem with libglib2.0-0 in gtk+2.0.pc - it may well be
> correct to have that one in the pkgconfig because gtk headers define
> variables in terms of Glib typedefs. (I have to do the same with libqof1).
>
> The actual string is:
> -lgtk-x11-2.0 -lgdk-x11-2.0 -latk-1.0 -lgdk_pixbuf-2.0 -lm
> -lpangocairo-1.0 -lpango-1.0 -lcairo -lgobject-2.0 -lgmodule-2.0 -ldl
> -lglib-2.0
And it used to be much longer. We’ve been working on reducing it and it
is still work in progress.
> 1. Glib: gmodule2.0 (and it's variants) shouldn't export -ldl. This is
> minor because it's a libc6 lib anyway so it doesn't cause an extra
> dependency but it will be the only "unwanted" link in the next version
> of libqof1 now that I've fixed the others.
>
> 2. Gtk+2.0 really shouldn't export gmodule. Even if there is a reason
> for lglib2.0, there is no reason for -lgmodule. It is trivial to omit
> this linkage from the gtk+2.0.pc file.
Both sound right, although this is something that needs some more
checking.
> 3. Ditto for gobject although I might be wrong on that one.
Yes, GObject provides the object model and I don’t think we can omit it.
> 4. Gtk+2: There is no need, as I see it, for atk, m, pangocairo, pango
> or cairo to be Requires: in gtk+2.0.pc - these should be
> Requires.private: instead.
IIRC most of them are used in either macros or for the types they
provide. Maybe some of them could be omitted, but not all of them.
Romain Beauxis
> > Perhaps the "absurd and wrong" part is that pkg-config does not
> > provide a way to distinguish between use cases, and that the name for
>
> Wrong, please read pkg-config(1) and think again.
>
> $ pkg-config tokyocabinet --libs
> -ltokyocabinet
> $ pkg-config tokyocabinet --libs --static
> -ltokyocabinet -lz -lpthread -lm -lc
> $ grep Libs /usr/lib/pkgconfig/tokyocabinet.pc
> Libs: -L${libdir} -ltokyocabinet
> Libs.private: -lz -lpthread -lm -lc
>
>
> pkg-config *is* perfectly suited for that matter actually, and
> "broken" pkgconfig upstreams are this trivial to fix. My upstream for
> tokyocabinet uses in a tokyocabinet.pc.in:
> Libs: -L${libdir} -ltokyocabinet @LIBS@
> the fix is just to put the @LIBS@ onto Libs.private and be done with it.
> Upstreams using pkg-config are actually the first _easy_ targets to fix
> the dependency issues _they_ generate in others binaries. And there is
> no reason no to fix those when it's easy to do so.
Right, so now I now how to fix #395185 .. Seems I lacked some knowledge about
pkg-config :-)
Romain
Steve Langasek
> Le dimanche 09 décembre 2007 à 15:44 -0800, Steve Langasek a écrit :
> > > For example, pkg-config --libs gtk+-x11-2.0 will return, among others,
> > > -lglib-2.0 and -lm. And this is perfectly intentional.
> > Just because it's intentional doesn't mean it isn't absurd and wrong.
> It may be absurd, but I don’t think it is wrong.
> > No, what can be done is to fix upstream's broken declaration that 'you can
> > assume glib functions are available when doing "#include <gtk/gtk.h>"'. It
> > doesn't follow that just because this works in practice, it should be a
> > supported usage.
> When many of the types used by GTK+ are those provided by GLib, it
> sounds wrong to ask developers to include the GLib headers to have these
> types available.
Well, that part is fairly reasonable, I admit. What isn't reasonable is to
go from "including gtk.h lets you use glib types" to "calling pkg-config
--libs gtk+-2.0 lets you invoke glib functions". Yes, including gtk.h is
always going to be sufficient, in practice, to get the glib types; but a)
includes != pkg-config, b) since it isn't implied there really is no
obligation to support such an expectation wrt pkg-config use, aside from
upstream's apparent commitment to compatibility with foolishly-assembled
build systems.
Steve Langasek
>>> No, what can be done is to fix upstream's broken declaration that 'you can
>>> assume glib functions are available when doing "#include <gtk/gtk.h>"'. It
>>> doesn't follow that just because this works in practice, it should be a
>>> supported usage.
>> When many of the types used by GTK+ are those provided by GLib, it
>> sounds wrong to ask developers to include the GLib headers to have these
>> types available.
> If it is really expected that type declarations are to be shared between a
> program and a library, then the program becomes dependent on the library's
> ABI without this dependency being formally expressed in any usable form,
> which is broken in and by itself.
> GTK needs to provide its own definitions of types and not expose interfaces
> it does not control.
Fine in theory, in practice this can be a significant burden to the library
maintainer. FWIW, I don't see any reason why a library can't use
externally-defined types directly, /as long as/ they have some mechanism for
ensuring ABI consistency between the two libraries (i.e., an ABI change in
the underlying library is always accompanied by an soname change in the
dependent library).
(BTW, pkg-config upstream wrongly claims that exporting dependent libs in
the pkg-config --libs output provides this protection. :/)
--
Steve Langasek Give me a lever long enough and a Free OS
Debian Developer to set it on, and I can move the world.
Ubuntu Developer http://www.debian.org/
slan...@ubuntu.com vor...@debian.org
Josselin Mouette
> > When many of the types used by GTK+ are those provided by GLib, it
> > sounds wrong to ask developers to include the GLib headers to have these
> > types available.
>
> Well, that part is fairly reasonable, I admit. What isn't reasonable is to
> go from "including gtk.h lets you use glib types" to "calling pkg-config
> --libs gtk+-2.0 lets you invoke glib functions".
Most GTK+ data types are inherited from Glib, and you need at least to
free them with g_free, g_object_unref, g_slist_free, and so on.
I’ve tried to sum up in bug #456335 the garbage that remains in
GTK’s .pc files, so that in we end we can keep only the legitimate ones,
but I don’t think we can manage to drop things like glib and gobject.
Robert Collins
On Thu, 2007-12-20 at 00:51 -0800, Steve Langasek wrote:
> On Mon, Dec 10, 2007 at 09:47:36PM +0100, Simon Richter wrote:
>
> >>> No, what can be done is to fix upstream's broken declaration that 'you can
> >>> assume glib functions are available when doing "#include <gtk/gtk.h>"'. It
> >>> doesn't follow that just because this works in practice, it should be a
> >>> supported usage.
>
> >> When many of the types used by GTK+ are those provided by GLib, it
> >> sounds wrong to ask developers to include the GLib headers to have these
> >> types available.
>
> > If it is really expected that type declarations are to be shared between a
> > program and a library, then the program becomes dependent on the library's
> > ABI without this dependency being formally expressed in any usable form,
> > which is broken in and by itself.
>
> > GTK needs to provide its own definitions of types and not expose interfaces
> > it does not control.
>
> Fine in theory, in practice this can be a significant burden to the library
> maintainer. FWIW, I don't see any reason why a library can't use
> externally-defined types directly, /as long as/ they have some mechanism for
> ensuring ABI consistency between the two libraries (i.e., an ABI change in
> the underlying library is always accompanied by an soname change in the
> dependent library).
I'd make a stronger statement and say that reusing types is a positive
thing. If you are reusing a glib list in your pet library, creating your
own type makes your library incompatible with other libraries also using
glib lists and providing code to manipulate them... and if it doesn't
make it incompatible then its not going to achieve the ABI
considerations that are under discussion here.
So really I think each dependant library of a library that is
dynamically linked to falls into one of two categories:
- internal detail
- implementation detail resulting in exposed ABI
So if I link program X against libfoo which uses libbar, if libbar is an
internal detail of libfoo then:
libfoo -> libbar@some ABI
X -> libfoo@libfoo's ABI only.
And a rebuild of libfoo to link a new libbar ABI won't break X, nor will
it require a rebuild of X.
The conditions for internal detail only are very stringent: compiler
alignment, structure sizes etc etc for all non opaque (where opaque is
opaque to the /compiler/ not to the user's code!) exported types of
libfoo must not be dependent on libbar at all.
I doubt that many libraries will meet this requirement for all their
dependent libraries (and consider libc as just an extremely common case
of this).
For all other cases, when a used ABI gets exposed, while we may not want
X to depend (at the package level) on libbar; at an ABI level it does.
We need to decide how we represent that. We also have at least two
dimensions here:
runtime linker behaviour
package management behaviour
From the runtime linker we want:
* an error/warning/something when two instances of the same API with
different ABI's are loaded. This commonly occurs when libbar.1 and
libbar.2 are both loaded, but it can occur more generally when libbar is
renamed to libquux and then libbar.1 and libquux.2 are loaded (by X and
libfoo respectively).
* To be able to interrogate how something was linked sufficiently well
to generate packaging metadata automatically.
From the package manager we want:
* To know when we have to rebuild X because libbar has changed.
* To allow upgrades to libbar on users systems without changing X when
possible.
The case where libbar is an internal detail is trivial (detection isn't:
see abicheck and icheck - libraries really need icheck in the package
build rules IMNSHO); I'm going to ignore it here.
For the case where libbar is not an internal detail. Starting with
packaging as thats easiest:
* we should depend on a minimum version of the libbar we need (as we
already bump the package name when the ABI changes incompatibly in a
library this is straight forward) and gives the opportunity to upgrade
libbar within an ABI safely.
* we can do an rdepends on e.g. libbar2 when libbar's API changes from
2 to 3 to find packages we need to rebuild.
One defect here is when symbols are shared from within a non-library -
e.g. the case with some programs plugin's. Here, a plugin may use a
symbol from within the program X, which is ABI dependent all the way
back, so treating programs as libraries - that is having an ABI - might
make sense. I haven't considered this in detail...
The runtime loader is more tricky:
* If we link X to libfoo only:
- libfoo's soname has to be manually changed when libbar's changes or
hilarity results
- we don't have a robust way that I know of to interrogate X to find
out that we need a package dependency on libbar
+ we don't need to have libbar present when it is only optionally
used (that is by calling some libfoo feature that needs libbar)
* If we link X to libbar directly:
- because we can't tell how much of recent changes to libbar are
needed, any installation of a build of X will force an upgrade of libbar
to the version it was built against, unless a bunch of extra effort is
put in.
+ we can determine that X has a dependency on libbar specifically
- we always require libbar to be present (IIRC)
I don't think we have a good answer to this in general, and I think we
need to do some infrastructure work to be able to have one.
> (BTW, pkg-config upstream wrongly claims that exporting dependent libs
> in the pkg-config --libs output provides this protection. :/)
It doesn't? (I must be missing something as I thought it did, just over
aggressively).
-Rob
--
GPG key available at: <http://www.robertcollins.net/keys.txt>.
Steve Langasek
> > (BTW, pkg-config upstream wrongly claims that exporting dependent libs
> > in the pkg-config --libs output provides this protection. :/)
> It doesn't? (I must be missing something as I thought it did, just over
> aggressively).
Nope.
Consider two libraries, libfoo and libbar. libfoo depends on libbar,
references functions from it and uses some of libbar's types in its own
exported API.
We assume the Debian-style libbar-dev, which ensures that the libbar headers
match the version of libbar.so on the system.
The pkg-config technique guarantee that, when an application links against
libfoo, it is also linked against libbar. Because of the preceding, this
guarantees that the application is linked against the version of libbar
whose ABI matches that of the headers used when building the application.
But *nothing* here guarantees that the version of libbar the application is
linked against has the same ABI as the one libfoo itself linked against! If
libfoo linked against libbar1, the pkg-config approach only ensures that
when your application is built against the libbar2 version of libbar-dev,
the resulting binary will depend on both libbar1 and libbar2 (despite not
using any symbols from libbar2). All this does is increase the chances of
segfaults or bad runtime behavior as a result of symbol clobbering.
So the usage recommended by pkg-config upstream does nothing at all to solve
the problem it's supposed to, and instead causes other linkage problems.
Neil Williams
Steve Langasek <vor...@debian.org> wrote:
> Consider two libraries, libfoo and libbar. libfoo depends on libbar,
> references functions from it and uses some of libbar's types in its own
> exported API.
>
> We assume the Debian-style libbar-dev, which ensures that the libbar headers
> match the version of libbar.so on the system.
> The pkg-config technique guarantee that, when an application links against
> libfoo, it is also linked against libbar. Because of the preceding, this
> guarantees that the application is linked against the version of libbar
> whose ABI matches that of the headers used when building the application.
>
> But *nothing* here guarantees that the version of libbar the application is
> linked against has the same ABI as the one libfoo itself linked against!
... until both the application and libfoo are rebuilt. So the issue
here is triggering rebuilds of reverse dependencies of libbar? I can
see this getting to be a big problem if the "application" in this
example is actually another library. If libfoo actually doesn't build
against libbar2 because of the API changes, the whole thing grinds to
a halt. After all, libbar1->libbar2 could involve any level of changes
to the API. If the application uses 5% of the libbar functions and the
library uses 50%, it's easy to see that the library has a higher risk
of needing intrusive source changes to build against the new libbar.
The Gtk1->Gtk2 transition is still far from complete, KDE3->4 is
current.
Though the combination of libbar1 and libbar2 raises the chances of
problems, forcing rebuilds whether the problem appears or not would
seem to increase the problems of blocked transitions and hinder
releases even further.
> If
> libfoo linked against libbar1, the pkg-config approach only ensures that
> when your application is built against the libbar2 version of libbar-dev,
> the resulting binary will depend on both libbar1 and libbar2 (despite not
> using any symbols from libbar2). All this does is increase the chances of
> segfaults or bad runtime behavior as a result of symbol clobbering.
>
> So the usage recommended by pkg-config upstream does nothing at all to solve
> the problem it's supposed to, and instead causes other linkage problems.
Maybe I've got this wrong but there would appear to be a few methods to
fix this:
1. Incorporating the libbar SONAME into the libfoo package SONAME - as
Simon Richter recommended - this could quickly end up with
libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that
this would actually help anyone if, for example, libbaz0 migrates to
libbaz1 in a way that completely breaks libbar2.
2. A rebuild trigger mechanism that is separate from the library SONAME
and the pkg-config files - an automated version of what happens now for
requesting binNMU's ? Not as easy as it may sound - there's no
guarantee that libfoo *can* build against libbar2.
3. A build-time hook that checks the entire dependency chain for
duplicates and fails if a freshly built binary ends up linked to two
versions of one source? (i.e. requiring a bug in whichever package is
using libbar1 to upgrade to libbar2). That could lead to being unable
to upload packages to fix other bugs.
What is the correct solution?
Is there a way that the current pkg-config *can* provide what upstream
believe it already does? Even if pkg-config upstream are thinking of
gentoo-type distributions, you'll still end up with a prebuilt libfoo
depending on libbar1 and a freshly built application depending on
libbar2, even using libbar1-dev and libbar2-dev (although at least
that is more obvious).
Maybe debtree could be useful here - or maybe edos-debcheck could be
extended to look for problems like this. It should be apparent from the
apt cache that the application uses libbar2 and libfoo but that libfoo
depends on libbar1. Maybe there is a role here for deborphan or a
similar tool as well - to scan an installation for not just orphaned
libraries but duplicates as well.
Getting back to the original thread, if applications and libraries each
take more care not to link to spurious dependencies, it should be
easier to identify the real problems instead of chasing red herrings
where the application is linked to libbar2 and libfoo but actually only
uses symbols from libfoo.
I've just uploaded libqof1 that used the information from the new
dpkg-shlibdeps to trim the dependencies of libqof1, excluding some
libraries that were previously only brought in because of bugs in the
libgda pkg-config data. It meant changing the QOF upstream to check for
the particular functions that it needs instead of trusting the
pkg-config data from libgda so that it could identify the correct
dependency data for a public library provided by libgda but not exposed
in the pkg-config data. (libgdasql).
Steve Langasek
> On Thu, 20 Dec 2007 21:44:19 -0800
> Steve Langasek <vor...@debian.org> wrote:
> > Consider two libraries, libfoo and libbar. libfoo depends on libbar,
> > references functions from it and uses some of libbar's types in its own
> > exported API.
> > We assume the Debian-style libbar-dev, which ensures that the libbar headers
> > match the version of libbar.so on the system.
> > The pkg-config technique guarantee that, when an application links against
> > libfoo, it is also linked against libbar. Because of the preceding, this
> > guarantees that the application is linked against the version of libbar
> > whose ABI matches that of the headers used when building the application.
> > But *nothing* here guarantees that the version of libbar the application is
> > linked against has the same ABI as the one libfoo itself linked against!
> ... until both the application and libfoo are rebuilt. So the issue
> here is triggering rebuilds of reverse dependencies of libbar?
No. That doesn't cause previously released binaries to blink out of
existence.
What's required is to ensure that there's a package rename each time
libfoo's ABI changes, including when the libfoo ABI change is caused by type
changes in the underlying lib. This is essentially the general case of the
c102, c2, c2a, and ldbl transitions that we've been through for compiler
changes over the past few years; in theory we should be able to accomplish
these transitions for libraries with fewer reverse-dependencies than
libstdc++ with significantly less pain, as long as library maintainers are
forewarned.
But once the package name change is handled, the rebuild of reverse-deps
follows naturally.
> Maybe I've got this wrong but there would appear to be a few methods to
> fix this:
> 1. Incorporating the libbar SONAME into the libfoo package SONAME - as
> Simon Richter recommended - this could quickly end up with
> libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that
> this would actually help anyone if, for example, libbaz0 migrates to
> libbaz1 in a way that completely breaks libbar2.
Then either libbar2 becomes libbar3 at the same time, or libbar becomes
unsupported (and the stack above it has to be ported or else it's not
releasable), or libbaz needs to support symbol versioning to permit
coexistence of the two versions in the same address space.
> 2. A rebuild trigger mechanism that is separate from the library SONAME
> and the pkg-config files - an automated version of what happens now for
> requesting binNMU's ? Not as easy as it may sound - there's no
> guarantee that libfoo *can* build against libbar2.
Wrong for the reasons stated above.
> 3. A build-time hook that checks the entire dependency chain for
> duplicates and fails if a freshly built binary ends up linked to two
> versions of one source? (i.e. requiring a bug in whichever package is
> using libbar1 to upgrade to libbar2). That could lead to being unable
> to upload packages to fix other bugs.
Doesn't sound like it actually solves the problem that needs solving for the
same reasons as option 2. above, but could be usable if fleshed out
appropriately.
Neil Williams
> On Fri, Dec 21, 2007 at 10:55:32AM +0000, Neil Williams wrote:
> > On Thu, 20 Dec 2007 21:44:19 -0800
> > Steve Langasek <vor...@debian.org> wrote:
> > > But *nothing* here guarantees that the version of libbar the application is
> > > linked against has the same ABI as the one libfoo itself linked against!
>
> > ... until both the application and libfoo are rebuilt. So the issue
> > here is triggering rebuilds of reverse dependencies of libbar?
>
> No. That doesn't cause previously released binaries to blink out of
> existence.
>
> What's required is to ensure that there's a package rename each time
> libfoo's ABI changes, including when the libfoo ABI change is caused by type
> changes in the underlying lib. This is essentially the general case of the
> c102, c2, c2a, and ldbl transitions that we've been through for compiler
> changes over the past few years; in theory we should be able to accomplish
> these transitions for libraries with fewer reverse-dependencies than
> libstdc++ with significantly less pain, as long as library maintainers are
> forewarned.
>
> But once the package name change is handled, the rebuild of reverse-deps
> follows naturally.
That sounds a lot better than the original idea of incorporating the
entire dependency chain into the package name.
I think I'm doing this already - if libfoo1 implements and exports types
from libbar2 and libbar2 moves to libbar3, I would expect to have to
port libfoo to libbar3 and this would usually cause a SONAME bump in
libfoo.
So could I ask, from an upstream perspective, what kind of changes in
the underlying lib might *not* cause such a port and therefore end up
with libfoo1 still being buildable against libbar3 yet *still* require a
SONAME bump to accommodate the transition?
Have I got these possible scenarios correct:
1. libfoo depends on libbar2 without exporting libbar symbols. libfoo
can migrate to libbar3 with internal code changes and if these do not
require changing any libfoo interfaces, libfoo needs no SONAME bump.
pkgconfig does not export libbar in the libfoo --libs data.
If an application uses libbar and libfoo (for whatever reason), the
ported version of that application that uses libbar3 simply needs to
depend on the version of libfoo that depends on libbar3. In this case,
an automated rebuild of the application could miss this step, causing
breakage, yet forcing a change in the libfoo package name seems
unnecessary (especially if there are other applications that depend on
libfoo but not directly on libbar). Another problem is if the
application ports to libbar3 before libfoo - currently, there is no way
of reliably picking this up except the application maintainer being
particularly careful with the pbuilder logs and noticing that libbar2 is
still being installed during the build as well as libbar3 due to the
indirect dependency on libfoo. Maybe pbuilder could highlight such
incidences? (or even fail the build if it happens)? Maybe sbuild too?
However, this would again raise the spectre of being unable to fix
critical bugs in the application should libfoo be slow to port to
libbar3. Again, it's a balance of risk - problems are *possible* when
the app is linked against libbar3 and indirectly against libbar2 but in
the absence of real bugs, is it worth causing more delays to other bug
fixes?
Does libfoo need to use versioned symbols to cope with this or is it
simply down to those applications that use libfoo and libbar to specify
the right version of libfoo once the application itself has ported to
libbar3? (And wait until libfoo has also migrated?) What methods are
needed to ensure that the application does this? (e.g. if left without
changes, an application directly dependent upon libbar3 and libfoo could
be built for unstable and presumably migrate into testing before the
updated libfoo).
There needs to be some check on the application that compares the
dependency chain as well as the dependency list in the package itself.
lintian cannot do that.
2. libfoo depends on libbar2 and exports some libbar symbols in
libfoo-dev. libfoo can migrate to libbar3 with only internal code
changes because it only exports a portion of the libbar API that has not
actually been changed in the libbar2->libbar3 transition. libfoo needs
to export libbar in the libfoo --libs data. Is a SONAME bump needed in
libfoo even though *none* of the libfoo interfaces have changed? As
upstream for libfoo, I would not normally make a SONAME bump in this
situation. Would it be better to rename the Debian package without
bumping the SONAME, e.g. libfoo1-0->libfoo1-1 ? Wouldn't this cause
unnecessary rebuilds of applications that only depend on libfoo and not
libbar?
3. libfoo depends on libbar2 and needs to make changes to its own
interfaces to cope with the libbar transition or needs to export symbols
from the new libbar API and therefore makes a SONAME bump. No problem
here except ensuring that applications transition to both API's at the
same time. Can be enforced by making libbar-dev conflict with the old
libfoo-dev or making the new libfoo-dev depend on the new libbar-dev
version.
It is only in case 2: that I find confusion and a need for some kind of
Debian-only package name change in libfoo. Is that correct?
Now it may be that for case 2: to happen in real situations, libbar
would actually turn out to be a collection of libraries: libbar-x11.so
and libbar-sql.so. Say the libbar2->3 transition only affects libbar-x11
and it is libbar-x11 that is NEEDED by the application, not libbar-sql:
Old package: libbar-x11.so.2.1.3 libbar-sql.so.2.4.1
New package: libbar-x11.so.3.0.0 libbar-sql.so.2.5.0
A package split in libbar would actually seem to be the better solution,
yes?
Have the libbar source build an extra set of packages:
libbar-x113, libbar-x11-dev, libbar-x11-dbg, libbar-doc
libbar-sql2, libbar-sql-dev, libbar-sql-dbg
libbar-dev as a meta package depending on libbar-sql-dev and
libbar-x11-dev
pkgconfig data for libbar-x11 in bar-x11.pc
pkgconfig data for libbar-sql in bar-sql.pc
Remove bar.pc or fold bar-x11.pc and bar-sql.pc into a new bar.pc to
support those applications that use both. Get the whole thing done in
one spell in NEW.
To cope with this transition, libfoo could simply swap the pkg-config
call from:
pkg-config --libs bar
to
pkg-config --libs bar-sql
No SONAME bump, just bump the version of the libbar-dev dependency (or
preferably replace libbar-dev with libbar-sql-dev).
Would this be sufficiently beneficial that the libbar maintainer would
be considered remiss for not implementing such a split?
> > > If
> > > libfoo linked against libbar1, the pkg-config approach only ensures that
> > > when your application is built against the libbar2 version of libbar-dev,
> > > the resulting binary will depend on both libbar1 and libbar2 (despite not
> > > using any symbols from libbar2). All this does is increase the chances of
> > > segfaults or bad runtime behavior as a result of symbol clobbering.
>
> > Maybe I've got this wrong but there would appear to be a few methods to
> > fix this:
> > 1. Incorporating the libbar SONAME into the libfoo package SONAME - as
> > Simon Richter recommended - this could quickly end up with
> > libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that
> > this would actually help anyone if, for example, libbaz0 migrates to
> > libbaz1 in a way that completely breaks libbar2.
>
> Then either libbar2 becomes libbar3 at the same time, or libbar becomes
> unsupported (and the stack above it has to be ported or else it's not
> releasable), or libbaz needs to support symbol versioning to permit
> coexistence of the two versions in the same address space.
Even if the actual fix for libbar2 is only internal? Say a change in
libbaz meant that the core functionality of libbar fails. As "core
functionality", it would be used by libbar but not exported so libbar
would not export libbaz symbols; there is no need for libbar2 to become
libbar3.
Yet the problem isn't in libbaz either, so why force libbaz to implement
versioned symbols through (apparently) no fault of its own?
To me, this whole problem comes down to the applications, not the shared
libraries or pkgconfig. Problems only arise if one package is linked
directly and indirectly against different versions of the same object.
The problem should be solved at the point at which it arises - in
whichever package ends up with the double linkage. Changing all the
shared libraries names and causing lots more transitions with
unnecessary SONAME bumps seems the wrong approach, IMHO, because we end
up with more transitions, not easier transitions.
Lintian cannot check beyond the confines of the .changes file for one
source package so we need a supplementary check and one place for this
would appear to be all those scripts that build in a chroot.
What about making the version of libbar-dev that depends on libbar3,
conflict with libbar2? The application maintainer would soon discover
the problem because the application dependency on libfoo could not be
satisfied in a pbuilder environment until libfoo updates to libbar3
itself. This gives time for the application maintainer to consider an
interim release of the current (unported) version of the application
dependent on libfoo and libbar2, if it is possible to fix certain bugs
that way, and seek an upgrade of libfoo in the meantime. i.e. leave
pkgconfig out of it entirely.
This has the advantage that none of the applications that only depend on
libfoo (and libbar indirectly) would be affected - it is limited to
those who need both libfoo-dev (which brings in libfoo and libbar2) and
libbar-dev (which brings in libbar3 and complains about libbar2). It can
also be implemented entirely within the one library that is undergoing
the transition: libbar.
It also allows applications that still use libbar2 to coexist with
libbar3 - only when trying to build against libbar3 does it become a
problem.
The biggest problem with that approach is upstream. It complicates
porting the application itself - depending on how many other packages
would be removed by apt when trying to install libbar-dev on a real
system. There would be no point making libgtk+-3.0-dev conflict against
libgtk-2.0-0.
However, I think this is manageable for small transitions and less
common libraries - the very ones that would tend to get missed by the
current methods. Everyone knows when a gtk transition is taking place,
everyone is watching for situations where a package gets linked against
libgtk+2.0.so and libgtk+3.0.so and there is little chance that any
program would work with such duplication.
Maybe a check that can be confined to chroots is needed - a function
within pbuilder that checks the build dependencies being requested and
fails if libbar2 and libbar3 are being installed at the same time during
a build? (The mere fact both are needed in a pbuilder build should be
evidence enough that at least one object in the dependency list is going
to end up linked to both.)
It would seem relatively simple to implement: sort the fully parsed
dependency list before calling 'apt-get' to remove duplicates, strip the
SONAME from the list of dependencies, error if any of the remaining
strings are not unique.
Something like (although probably not needing tmpfiles):
#!/bin/sh -e
LIST="libfoo1 libbar2 libbar3 myprog libfoo1"
rm -f tmpfile
for PKG in $LIST; do
echo $PKG >> tmpfile
done
ULIST=`sort -u < tmpfile`
for PKG in $ULIST; do
PKG=`echo "$PKG" | sed -e 's/[0-9]$//'`
echo $PKG >> tmpfile2
done
ORIG=`grep -c '[a-z]' tmpfile`
SORT=`grep -c '[a-z]' tmpfile2`
if [ $ORIG -ne $SORT ]; then
echo -n "FAIL: "
echo $ULIST | sed -e 's/\n//'
exit 1
fi
echo "OK"
gives:
FAIL: libbar2 libbar3 libfoo1 myprog
I'm sure that can be improved to only show the problematic packages.
Robert Collins
On Wed, 2007-12-26 at 13:11 +0000, Neil Williams wrote:
>
> So could I ask, from an upstream perspective, what kind of changes in
> the underlying lib might *not* cause such a port and therefore end up
> with libfoo1 still being buildable against libbar3 yet *still* require
> a
> SONAME bump to accommodate the transition?
bar.h:
typedef unsigned int bar;
foo.h:
typedef struct {
bar a_bar;
} exported_foo_type;
Then this patch to bar.h:
@@ -0,1 +0,1 @@
-typedef unsigned int bar;
+typedef size_t bar;
is enough to cause an ABI break in libfoo, without needing 'porting'
work (though it may well cause print formatting failures on 64
environments and other side effects).
Steve Langasek
Succinctly addressed by Robert.
> Have I got these possible scenarios correct:
> 1. libfoo depends on libbar2 without exporting libbar symbols. libfoo
> can migrate to libbar3 with internal code changes and if these do not
> require changing any libfoo interfaces, libfoo needs no SONAME bump.
No, "exporting libbar symbols" is not the right line to draw. The right
line is the one I already identified, namely, "libfoo uses some of libbar's
types in its own exported ABI".
> Does libfoo need to use versioned symbols to cope with this
No, libbar does, as the library which has other libraries for
reverse-dependencies.
> There needs to be some check on the application that compares the
> dependency chain as well as the dependency list in the package itself.
> lintian cannot do that.
Not if libbar uses symbol versioning.
> 2. libfoo depends on libbar2 and exports some libbar symbols in
> libfoo-dev. libfoo can migrate to libbar3 with only internal code
> changes because it only exports a portion of the libbar API that has not
> actually been changed in the libbar2->libbar3 transition. libfoo needs
> to export libbar in the libfoo --libs data. Is a SONAME bump needed in
> libfoo even though *none* of the libfoo interfaces have changed? As
> upstream for libfoo, I would not normally make a SONAME bump in this
> situation. Would it be better to rename the Debian package without
> bumping the SONAME, e.g. libfoo1-0->libfoo1-1 ? Wouldn't this cause
> unnecessary rebuilds of applications that only depend on libfoo and not
> libbar?
If libfoo's exported ABI hasn't changed, then any package name or soname
change is gratuitous and should be avoided.
> 3. libfoo depends on libbar2 and needs to make changes to its own
> interfaces to cope with the libbar transition or needs to export symbols
> from the new libbar API and therefore makes a SONAME bump. No problem
> here except ensuring that applications transition to both API's at the
> same time. Can be enforced by making libbar-dev conflict with the old
> libfoo-dev or making the new libfoo-dev depend on the new libbar-dev
> version.
Requires disruptive name changes to libbar-dev which affect all packages
that build-depend on it, and is therefore again inferior to implementing
symbol versions up front.
> > > Maybe I've got this wrong but there would appear to be a few methods to
> > > fix this:
> > > 1. Incorporating the libbar SONAME into the libfoo package SONAME - as
> > > Simon Richter recommended - this could quickly end up with
> > > libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that
> > > this would actually help anyone if, for example, libbaz0 migrates to
> > > libbaz1 in a way that completely breaks libbar2.
> > Then either libbar2 becomes libbar3 at the same time, or libbar becomes
> > unsupported (and the stack above it has to be ported or else it's not
> > releasable), or libbaz needs to support symbol versioning to permit
> > coexistence of the two versions in the same address space.
> Even if the actual fix for libbar2 is only internal?
Then that would eliminate the first option, leaving only the other two.
> Yet the problem isn't in libbaz either, so why force libbaz to implement
> versioned symbols through (apparently) no fault of its own?
Because *all* libraries which have other libs as reverse-dependencies should
implement symbol versioning, precisely as the shortest path for reliably and
permanently addressing the various issues we're discussing.
It *is* libbaz's fault for having an ABI-breaking change. And based on
history, it is only reasonable to assume all such libraries are guilty, and
demand that they use symbol versioning in anticipation of such future ABI
changes.
> To me, this whole problem comes down to the applications, not the shared
> libraries or pkgconfig. Problems only arise if one package is linked
> directly and indirectly against different versions of the same object.
> The problem should be solved at the point at which it arises - in
> whichever package ends up with the double linkage.
No. You seem to agree that each soname change of a library near the bottom
of the stack should not propagate its way up the stack to affect other
libraries that themselves are not affected by the ABI change, but you again
overlook (or disdain) the impact that this has on partial upgrades if we
don't require those base libraries to use symbol versioning. Neither the
rebuild, nor the installation, of packages against the new version of libbar
is instantaneous.
Neil Williams
Steve Langasek <vor...@debian.org> wrote:
> On Wed, Dec 26, 2007 at 01:11:40PM +0000, Neil Williams wrote:
> > I think I'm doing this already - if libfoo1 implements and exports types
> > from libbar2 and libbar2 moves to libbar3, I would expect to have to
> > port libfoo to libbar3 and this would usually cause a SONAME bump in
> > libfoo.
> > Have I got these possible scenarios correct:
>
> > 1. libfoo depends on libbar2 without exporting libbar symbols. libfoo
> > can migrate to libbar3 with internal code changes and if these do not
> > require changing any libfoo interfaces, libfoo needs no SONAME bump.
>
> No, "exporting libbar symbols" is not the right line to draw. The right
> line is the one I already identified, namely, "libfoo uses some of libbar's
> types in its own exported ABI".
That makes more sense, thank you.
> > Does libfoo need to use versioned symbols to cope with this
>
> No, libbar does, as the library which has other libraries for
> reverse-dependencies.
(codehelp has a light bulb moment of clarity). :-)
> > 2. libfoo depends on libbar2 and exports some libbar symbols in
> > libfoo-dev. libfoo can migrate to libbar3 with only internal code
> > changes because it only exports a portion of the libbar API that has not
> > actually been changed in the libbar2->libbar3 transition. libfoo needs
> > to export libbar in the libfoo --libs data. Is a SONAME bump needed in
> > libfoo even though *none* of the libfoo interfaces have changed? As
> > upstream for libfoo, I would not normally make a SONAME bump in this
> > situation.
>
> If libfoo's exported ABI hasn't changed, then any package name or soname
> change is gratuitous and should be avoided.
Agreed.
> > 3. libfoo depends on libbar2 and needs to make changes to its own
> > interfaces to cope with the libbar transition or needs to export symbols
> > from the new libbar API and therefore makes a SONAME bump. No problem
> > here except ensuring that applications transition to both API's at the
> > same time. Can be enforced by making libbar-dev conflict with the old
> > libfoo-dev or making the new libfoo-dev depend on the new libbar-dev
> > version.
>
> Requires disruptive name changes to libbar-dev which affect all packages
> that build-depend on it, and is therefore again inferior to implementing
> symbol versions up front.
OK.
> Because *all* libraries which have other libs as reverse-dependencies should
> implement symbol versioning, precisely as the shortest path for reliably and
> permanently addressing the various issues we're discussing.
>
> It *is* libbaz's fault for having an ABI-breaking change. And based on
> history, it is only reasonable to assume all such libraries are guilty, and
> demand that they use symbol versioning in anticipation of such future ABI
> changes.
OK, I'm doing this upstream for one of my packages - following threads from:
http://lists.debian.org/debian-mentors/2007/05/msg00387.html
http://www.gnu.org/software/binutils/manual/ld-2.9.1/html_node/ld_25.html
That gave me a basic versioning addition to the library, a diff of
objdump -p shows:
+Version definitions:
+1 0x01 0x06077991 libqof.so.1
+2 0x00 0x0f307023 LIBQOF_0.7.3
+
(I'm not putting the history of the symbols into the file at this stage
- no new symbols are added in the new version so the libqof.ver file is
simply marking all as global. I could add historical data - is there any
benefit in that?)
That's upstream covered, it appears I also need debian/libqof1.symbols
from http://qa.debian.org/cgi-bin/mole/seedsymbols ? If I had done
symbol versioning correctly upstream, shouldn't dpkg-shlibdeps be able
to create the necessary data itself? I don't provide a .shlib file of
my own at this stage.
(A SONAME bump is due in this library fairly soon so I want to get
symbol versioning into libqof1 prior to the ABI change.)
> No. You seem to agree that each soname change of a library near the bottom
> of the stack should not propagate its way up the stack to affect other
> libraries that themselves are not affected by the ABI change, but you again
> overlook (or disdain) the impact that this has on partial upgrades if we
> don't require those base libraries to use symbol versioning.
It was an oversight. Sorry.
Steve Langasek
> > Because *all* libraries which have other libs as reverse-dependencies should
> > implement symbol versioning, precisely as the shortest path for reliably and
> > permanently addressing the various issues we're discussing.
> > It *is* libbaz's fault for having an ABI-breaking change. And based on
> > history, it is only reasonable to assume all such libraries are guilty, and
> > demand that they use symbol versioning in anticipation of such future ABI
> > changes.
> OK, I'm doing this upstream for one of my packages - following threads from:
> http://lists.debian.org/debian-mentors/2007/05/msg00387.html
> http://www.gnu.org/software/binutils/manual/ld-2.9.1/html_node/ld_25.html
> That gave me a basic versioning addition to the library, a diff of
> objdump -p shows:
> +Version definitions:
> +1 0x01 0x06077991 libqof.so.1
> +2 0x00 0x0f307023 LIBQOF_0.7.3
> +
> (I'm not putting the history of the symbols into the file at this stage
> - no new symbols are added in the new version so the libqof.ver file is
> simply marking all as global. I could add historical data - is there any
> benefit in that?)
No, I don't think there's any benefit in adding the historical data after
the fact.
> That's upstream covered, it appears I also need debian/libqof1.symbols
> from http://qa.debian.org/cgi-bin/mole/seedsymbols ? If I had done
> symbol versioning correctly upstream, shouldn't dpkg-shlibdeps be able
> to create the necessary data itself? I don't provide a .shlib file of
> my own at this stage.
The symbols files are orthogonal to symbol versioning. Symbols files are
basically "per-function shlibs", which are applicable even to libs that
don't use symbol versioning; their main benefit is for libraries which make
backwards-compatible ABI changes.
Cheers,
Neil Williams
> On Wed, Jan 02, 2008 at 11:53:16AM +0000, Neil Williams wrote:
> > That's upstream covered, it appears I also need debian/libqof1.symbols
> > from http://qa.debian.org/cgi-bin/mole/seedsymbols ? If I had done
> > symbol versioning correctly upstream, shouldn't dpkg-shlibdeps be able
> > to create the necessary data itself? I don't provide a .shlib file of
> > my own at this stage.
>
> The symbols files are orthogonal to symbol versioning. Symbols files are
> basically "per-function shlibs", which are applicable even to libs that
> don't use symbol versioning; their main benefit is for libraries which make
> backwards-compatible ABI changes.
In which case I should have been supporting symbols files since v0.6.0
because that is exactly what I've been doing in this package for the
last two years or more - adding new functions, deprecating the old ones,
collecting all the changes into one transition to the next SONAME. Not
sure why the sample symbols file showed the same version for all
functions - quite a lot were simply not present in 0.6.0. Ne'er mind.
It's a bit late now (I'm preparing for a SONAME bump where all those
backwards-compatible layers are removed) but I'll implement it in the
new version to allow tracking of future backwards-compatible ABI
changes.
Thanks.