Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

static linking broken

5,303 views
Skip to first unread message

phil-new...@ipal.net

unread,
Nov 4, 2005, 10:13:55 AM11/4/05
to
While trying to do a static compile of an application under a newly
installed Slackware 10.2 system, which consists of:

gcc (GCC) 3.3.6
GNU C Library stable release version 2.3.5, by Roland McGrath et al.

I am now getting the following warnings during linking steps:

=============================================================================
: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'endgrent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'endpwent' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getgrgid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getgrnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'gethostbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getprotobyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getpwnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getpwuid' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getservbyname' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'getspnam' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
: warning: Using 'initgroups' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
=============================================================================

WTF!?!?!

Static linking is supposed to be STATIC LINKING. Many reasons exist for
static linking, including:

1. Ensuring runability of critical tools during system maintenance which
may involve time when no libraries are configured or present.

2. Portability across multiple library versions.

This is telling me now that I have to link a different version of the program,
and distribute a different pre-linked binary copy, for every different version
of GLIBC that might be present. This is insane. This is moving backwards in
technology.

GLIBC FAQ question 2.22 also fails to address this since it is focused
strictly on the NSL stuff. The real problem here is that the GLIBC
developers have simply lost an understanding of what static linking
really means.

It will also be quite a mess to do all those builds, since it apparently would
require having present on one system, and selecting among them, every different
version of GLIBC and probably the linking tools as well.

1. Is there a way around this mess with the current GLIBC?

2. Do I need to back down to an older version of GLIBC? What version is
going to work?

3. Or should I switch to Dietlibc or uClibc?

FYI, another reason I need to solve this problem is because my practice is
to be sure my builds are 100% warning free. Many legitimate warnings are
genuine indications of real problems that while being technically legal and
compliance code, can, and in many cases actually have, resulted in program
malfunctions. This is particularly so with compiler warnings, but linker
warnings are held in similar regard. Once I enter a mode where I end up
ignoring warnings, then the danger is that other warnings will be missed
and an increased exposure to program malfunctions will exist. So please do
not suggest that these warnings can be ignored. I need to BOTH do static
linking (really static linking) _and_ eliminate the warning message.

--
-----------------------------------------------------------------------------
| Phil Howard KA9WGN | http://linuxhomepage.com/ http://ham.org/ |
| (first name) at ipal.net | http://phil.ipal.org/ http://ka9wgn.ham.org/ |
-----------------------------------------------------------------------------

John Reiser

unread,
Nov 4, 2005, 1:07:24 PM11/4/05
to
phil-new...@ipal.net wrote:
> While trying to do a static compile of an application under a newly
> installed Slackware 10.2 system, which consists of:
>
> gcc (GCC) 3.3.6
> GNU C Library stable release version 2.3.5, by Roland McGrath et al.
>
> I am now getting the following warnings during linking steps:
>
> =============================================================================
> : warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
...

> =============================================================================
>
> WTF!?!?!
>
> Static linking is supposed to be STATIC LINKING. Many reasons exist for
> static linking, including:
>
> 1. Ensuring runability of critical tools during system maintenance which
> may involve time when no libraries are configured or present.

Anything that calls dlopen() and friends violates this design goal.
Using dlopen() _assumes_ that some other file/library is present.
The design is broken before it ever begins implementation.

> 2. Portability across multiple library versions.

glibc's idea of portability is: a dynamically linked application will run the
same way under all future versions of glibc environment (shared libraries and
runtime supervisor rtld [ld-linux]). "Backwards portability" (running under
any previous version of glibc environment) is explicitly _not_ a goal, and
_has_ broken at least a few times in the last several years.
[There have been a couple glitches in forward compatibility, too.]

> This is telling me now that I have to link a different version of the program,
> and distribute a different pre-linked binary copy, for every different version
> of GLIBC that might be present. This is insane. This is moving backwards in
> technology.

Glibc does not have a stable ABI (application binary interface).
The Linux operating system and the C library change too often,
and a "bug" (usually a misunderstanding of a corner case in the specification,
but sometimes an outright error) to one application is a "feature" to another.
Do a "readelf --version-info <executable_or_shlib>" to see the different GLIBC_2.*.*
versions that exist. Sometimes there are multiple GLIBC_2.*.* versions for
the same symbol, for instance popen() has GLIBC_2.0 and GLIBC_2.1.
This is a case where the semantics of the specification changed.
Old applications that bind to popen@GLIBC_2.0 still run the same way
as when they were linked, new applications get popen@GLIBC_2.1 during
build, and new apps that want the old semantics can still get popen@GLIBC_2.0
if they really try hard.

> 1. Is there a way around this mess with the current GLIBC?

No.

> 2. Do I need to back down to an older version of GLIBC? What version is
> going to work?

None of them.

>
> 3. Or should I switch to Dietlibc or uClibc?

They are even less concerned with stable ABI than glibc, partly because
Dietlibc and uClibc are so young that they still have an order of magnitude
more bugs than glibc. Their coverage of the API, particularly
internationalization, is also much less than glibc.

> I need to BOTH do static linking (really static linking) _and_
> eliminate the warning message.

Glibc is not for you, if you want to statically link dlopen, getpwnam,
any nss* natural language function, etc. You must hunt down the *.a
verions of _everything_ that you could possibly need at runtime,
and put them _all_ into the app. Then if the environment changes,
the app will break. Dynamic linking usually accommodates such changes,
and thus is usually prefered. (If you only now realize the problem,
then dynamic linking probably is better for you.)


In practice: Dynamically link on a 1.5-year-old system that was upgraded
at most in the first 3 months after release. This will work for about 90%
of today's installed base.
For 97% coverage, dynamically link on a stock RedHat 7.3 system.
For 99% coverage, dynamically link on a stock RedHat 6.2 system.
[In all cases, statically link to libstdc++. Search Google groups for
Paul Pluzhnikov's recipe for doing this.]

--

"Nils O. Selåsdsal"

unread,
Nov 5, 2005, 12:56:20 PM11/5/05
to
Yes, well, some features in glibc relies on beeing able to dynamically
load libraries (using the dynamic linker). dlopen, naturally.
And all functions using the NSS, such as gethostbyname, and various others.


> 1. Ensuring runability of critical tools during system maintenance which
> may involve time when no libraries are configured or present.
>
> 2. Portability across multiple library versions.
>
> This is telling me now that I have to link a different version of the program,
> and distribute a different pre-linked binary copy, for every different version
> of GLIBC that might be present. This is insane. This is moving backwards in
> technology.

No. glibc is backwars compatible. So link to a sufficiently old glibc
*shrug*

Or use the LSB build tools.

> GLIBC FAQ question 2.22 also fails to address this since it is focused

Update it.


> strictly on the NSL stuff. The real problem here is that the GLIBC
> developers have simply lost an understanding of what static linking
> really means.

Bug them about it.

> It will also be quite a mess to do all those builds, since it apparently would
> require having present on one system, and selecting among them, every different
> version of GLIBC and probably the linking tools as well.
>
> 1. Is there a way around this mess with the current GLIBC?

Don't use the functions that is complained about above.

phil-new...@ipal.net

unread,
Nov 5, 2005, 7:34:19 PM11/5/05
to
On Fri, 04 Nov 2005 10:07:24 -0800 John Reiser <jre...@bitwagon.com> wrote:

| Anything that calls dlopen() and friends violates this design goal.
| Using dlopen() _assumes_ that some other file/library is present.
| The design is broken before it ever begins implementation.

If program that is statically linked is doing a dlopen() of a specific
library, then an assumption that _that_ specific library is present is
valid. Assuming any other library is present should not be done. Such
a program that is part of a package may validly assume that libraries
part of that package could be installed. It's also possible for some
application to consider something like a library as merely option and
just keep on running doing things other ways if it is absent.


|> 2. Portability across multiple library versions.
|
| glibc's idea of portability is: a dynamically linked application will run the
| same way under all future versions of glibc environment (shared libraries and
| runtime supervisor rtld [ld-linux]). "Backwards portability" (running under
| any previous version of glibc environment) is explicitly _not_ a goal, and
| _has_ broken at least a few times in the last several years.
| [There have been a couple glitches in forward compatibility, too.]

That would mean you need to build your dynamic executable using the OLDEST
version of library in the range of those you want it to work with.


|> This is telling me now that I have to link a different version of the program,
|> and distribute a different pre-linked binary copy, for every different version
|> of GLIBC that might be present. This is insane. This is moving backwards in
|> technology.
|
| Glibc does not have a stable ABI (application binary interface).
| The Linux operating system and the C library change too often,
| and a "bug" (usually a misunderstanding of a corner case in the specification,
| but sometimes an outright error) to one application is a "feature" to another.
| Do a "readelf --version-info <executable_or_shlib>" to see the different GLIBC_2.*.*
| versions that exist. Sometimes there are multiple GLIBC_2.*.* versions for
| the same symbol, for instance popen() has GLIBC_2.0 and GLIBC_2.1.
| This is a case where the semantics of the specification changed.
| Old applications that bind to popen@GLIBC_2.0 still run the same way
| as when they were linked, new applications get popen@GLIBC_2.1 during
| build, and new apps that want the old semantics can still get popen@GLIBC_2.0
| if they really try hard.

My issue here is that for a statically linked executable to be usable,
the user has to have the correct library (which one wouldn't even expect
a library to be needed at all). To support all versions, you have to
build the executables under each version, and distribute all those.


|> 1. Is there a way around this mess with the current GLIBC?
|
| No.
|
|> 2. Do I need to back down to an older version of GLIBC? What version is
|> going to work?
|
| None of them.

Old versions did work.


|> 3. Or should I switch to Dietlibc or uClibc?
|
| They are even less concerned with stable ABI than glibc, partly because
| Dietlibc and uClibc are so young that they still have an order of magnitude
| more bugs than glibc. Their coverage of the API, particularly
| internationalization, is also much less than glibc.

But for programs that need to be static, this does seem to work where
GLIBC won't work at all. That's a dramatic difference even if it is
not perfect.


|> I need to BOTH do static linking (really static linking) _and_
|> eliminate the warning message.
|
| Glibc is not for you, if you want to statically link dlopen, getpwnam,
| any nss* natural language function, etc. You must hunt down the *.a
| verions of _everything_ that you could possibly need at runtime,
| and put them _all_ into the app. Then if the environment changes,
| the app will break. Dynamic linking usually accommodates such changes,
| and thus is usually prefered. (If you only now realize the problem,
| then dynamic linking probably is better for you.)

Fortunately not too many apps need the static linking. But a few do.
This is the focus.

But consider the initial build environment of Linux From Scratch.
That's all statically linked. It is then used to build a dynamic
system. But even then, a few maintenance tools still need to be
statically linked. How do you do system maintenance that involves
repairing libraries or the ldconfig if your shell and tools are
dyanmically linked? Add sshd to that if logging in remotely.

Paul Pluzhnikov

unread,
Nov 5, 2005, 10:04:20 PM11/5/05
to
phil-new...@ipal.net writes:

> If program that is statically linked is doing a dlopen() of a specific
> library, then an assumption that _that_ specific library is present is
> valid. Assuming any other library is present should not be done.

So go design your own OS on which you can remove this assumption.
By what authority are you telling others what they should or should
not assume?

There are no UNIX systems I know of, which allow one to link
statically, use dlopen, and gurantee that your exe will still work
on newer OS releases. In fact, most UNIX systems guarantee backward
compatibility (older executable running on newer OS) *only* for
dynamically-linked executables.

> a program that is part of a package may validly assume that libraries
> part of that package could be installed. It's also possible for some
> application to consider something like a library as merely option and
> just keep on running doing things other ways if it is absent.

You could also write your own implementation of dlopen(), and
use *it* to load your binary code in whatever format you wish
(including ELF). But if you want to use glibc's dlopen(), then you
must satisfy its assumptions.

> | glibc's idea of portability is: a dynamically linked application will run the
> | same way under all future versions of glibc environment (shared libraries and
> | runtime supervisor rtld [ld-linux]).

> That would mean you need to build your dynamic executable using the OLDEST


> version of library in the range of those you want it to work with.

Exactly. Build on RedHat 6.2 (dynamically), and it will run everywhere.

> My issue here is that for a statically linked executable to be usable,
> the user has to have the correct library (which one wouldn't even expect
> a library to be needed at all). To support all versions, you have to
> build the executables under each version, and distribute all those.

IF you build statically, THEN you must build multiple versions
(which is one of the reasons why building statically is a bad idea (TM)).

I build *one* version on RedHat 6.2, and that one version runs
everywhere.

> But consider the initial build environment of Linux From Scratch.
> That's all statically linked. It is then used to build a dynamic
> system. But even then, a few maintenance tools still need to be
> statically linked.

The intersection of tools that must be statically linked and tools
that must be able to dlopen() is empty.

If your tool is in fact a proof positive that this intersection is
not empty, please provide convincing details.

Cheers,
--
In order to understand recursion you must first understand recursion.
Remove /-nsp/ for email.

Tauno Voipio

unread,
Nov 6, 2005, 3:39:40 AM11/6/05
to
phil-new...@ipal.net wrote:
> On Fri, 04 Nov 2005 10:07:24 -0800 John Reiser <jre...@bitwagon.com> wrote:
>
> | Anything that calls dlopen() and friends violates this design goal.
> | Using dlopen() _assumes_ that some other file/library is present.
> | The design is broken before it ever begins implementation.
>
> If program that is statically linked is doing a dlopen() of a specific
> library, then an assumption that _that_ specific library is present is
> valid. Assuming any other library is present should not be done. Such
> a program that is part of a package may validly assume that libraries
> part of that package could be installed. It's also possible for some
> application to consider something like a library as merely option and
> just keep on running doing things other ways if it is absent.

There is a techical reason why dlopen() implies dynamic linking:
To make any dynamic library work, your executable *must* have
the dynamic linker (ld.so) marked as the program interpreter
in the runnable image. On the other hand, ld.so itself is
usually a dynamic library itself, so you cannot use it in
a statically linked environment.

HTH

--

Tauno Voipio
tauno voipio (at) iki fi

Kasper Dupont

unread,
Nov 6, 2005, 4:16:13 AM11/6/05
to
"Nils O. Selåsdsal" wrote:
>
> Yes, well, some features in glibc relies on beeing able to dynamically
> load libraries (using the dynamic linker). dlopen, naturally.

Of course dlopen links libraries dynamically. But as long as
the requested libraries are part of the application and not
part of glibc, then it shouldn't worry about the version of
those.

> And all functions using the NSS, such as gethostbyname, and various others.

If the application references any of those, the necesarry
code should be linked statically. I guess this just shows
that the subject is true, at least as far as glibc is
concerned.

--
Kasper Dupont
Note to self: Don't try to allocate
256000 pages with GFP_KERNEL on x86.

"Nils O. Selåsdal"

unread,
Nov 6, 2005, 5:23:08 AM11/6/05
to
Kasper Dupont wrote:
> "Nils O. Selåsdsal" wrote:
>
>>Yes, well, some features in glibc relies on beeing able to dynamically
>>load libraries (using the dynamic linker). dlopen, naturally.
>
>
> Of course dlopen links libraries dynamically. But as long as
> the requested libraries are part of the application and not
> part of glibc, then it shouldn't worry about the version of
> those.
>
>
>>And all functions using the NSS, such as gethostbyname, and various others.
>
>
> If the application references any of those, the necesarry
> code should be linked statically. I guess this just shows
> that the subject is true, at least as far as glibc is
> concerned.
It's a bit hard for applications to guess which nss libraries
are needed, not all are part of glibc ...

Paul Pluzhnikov

unread,
Nov 6, 2005, 9:50:44 AM11/6/05
to
Tauno Voipio <tauno....@INVALIDiki.fi> writes:

> There is a techical reason why dlopen() implies dynamic linking:
> To make any dynamic library work, your executable *must* have
> the dynamic linker (ld.so) marked as the program interpreter
> in the runnable image.

That is incorrect. Glibc makes no such requirement:

$ cat foo.c
int foo() { return 1; }

$ gcc -g -fPIC -shared -o foo.so foo.c

$ cat dlmain.c
#include <dlfcn.h>
#include <stdio.h>

int main()
{
void *h = dlopen("./foo.so", RTLD_LAZY | RTLD_GLOBAL);
if (!h) {
puts(dlerror());
exit(1);
}
return 0;
}

$ gcc -static dlmain.c -ldl
/tmp/cc2AwFFx.o(.text+0x1e): In function `main':
dlmain.c:6: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

$ ./a.out && echo ok
ok

$ objdump -p a.out

a.out: file format elf32-i386

Program Header:
LOAD off 0x00000000 vaddr 0x08048000 paddr 0x08048000 align 2**12
filesz 0x00054f14 memsz 0x00054f14 flags r-x
LOAD off 0x00055000 vaddr 0x0809d000 paddr 0x0809d000 align 2**12
filesz 0x00000c48 memsz 0x00001904 flags rw-
NOTE off 0x000000b4 vaddr 0x080480b4 paddr 0x080480b4 align 2**2
filesz 0x00000020 memsz 0x00000020 flags r--
STACK off 0x00000000 vaddr 0x00000000 paddr 0x00000000 align 2**2
filesz 0x00000000 memsz 0x00000000 flags rw-

No PT_INTERP above ...

The ld-linux.so.2 *is* loaded at runtime by the statically linked
parts of libdl.a, and *it* (at runtime) must match the version
of libdl.a that was statically linked into the app.

Kasper Dupont <kas...@daimi.au.dk> writes:

> Of course dlopen links libraries dynamically. But as long as
> the requested libraries are part of the application and not
> part of glibc, then it shouldn't worry about the version of
> those.

It is warning about the fact that 'ld-linux.so.2' must match libdl.a
Since ld-linux *is* part of glibc, in practice that warning means
that the runtime glibc *must* match the linktime glibc, or the exe
will fail.

Tauno Voipio

unread,
Nov 6, 2005, 10:45:23 AM11/6/05
to

It is in the ELF specification. Good, if the Linux
implementation does not need it. I stand corrected.

phil-new...@ipal.net

unread,
Nov 6, 2005, 1:15:56 PM11/6/05
to
On Sat, 05 Nov 2005 19:04:20 -0800 Paul Pluzhnikov <ppluzhn...@charter.net> wrote:
| phil-new...@ipal.net writes:
|
|> If program that is statically linked is doing a dlopen() of a specific
|> library, then an assumption that _that_ specific library is present is
|> valid. Assuming any other library is present should not be done.
|
| So go design your own OS on which you can remove this assumption.
| By what authority are you telling others what they should or should
| not assume?

Because it makes common, and correct, sense. Static linking is to be sure
that the executable file contains all the code, and that no dynamic linking,
aside from what the program application itself may intentionally do, should
be involved.

It should work with _system_ dynamic libraries entirely missing or in other
state of non-function.


| There are no UNIX systems I know of, which allow one to link
| statically, use dlopen, and gurantee that your exe will still work
| on newer OS releases. In fact, most UNIX systems guarantee backward
| compatibility (older executable running on newer OS) *only* for
| dynamically-linked executables.

There is no such thing as a guarantee. Of course a statically linked
program will fail on a kernel version new enough that changes in the
system call ABI exist. But Linux changes sufficiently infrequently enough
that at most a small handful of executables would be needed to cover the
entire history of Linux. Requiring an executable for every version of
glibc, however, is absolutely insane.


|> a program that is part of a package may validly assume that libraries
|> part of that package could be installed. It's also possible for some
|> application to consider something like a library as merely option and
|> just keep on running doing things other ways if it is absent.
|
| You could also write your own implementation of dlopen(), and
| use *it* to load your binary code in whatever format you wish
| (including ELF). But if you want to use glibc's dlopen(), then you
| must satisfy its assumptions.

I could use dietlibc or some other. No need to re-invent the wheel.

Considering how long glibc let the strtold bug hang around, maybe it
is time to switch from that dinosaur.


|> | glibc's idea of portability is: a dynamically linked application will run the
|> | same way under all future versions of glibc environment (shared libraries and
|> | runtime supervisor rtld [ld-linux]).
|
|> That would mean you need to build your dynamic executable using the OLDEST
|> version of library in the range of those you want it to work with.
|
| Exactly. Build on RedHat 6.2 (dynamically), and it will run everywhere.

Why dynamically? I'm trying to make something that works when all dynamic
libraries are not (yet) present.


|> My issue here is that for a statically linked executable to be usable,
|> the user has to have the correct library (which one wouldn't even expect
|> a library to be needed at all). To support all versions, you have to
|> build the executables under each version, and distribute all those.
|
| IF you build statically, THEN you must build multiple versions
| (which is one of the reasons why building statically is a bad idea (TM)).

Yet it is most often that static executables work best, even in large
applications, where negative interactions between libraries still happen.
The Opera browser didn't work in dynamic form. It was believed that it
was because I had upgraded some libraries. Fortunately, they made a
static version which worked.


| I build *one* version on RedHat 6.2, and that one version runs
| everywhere.

OK, let's see if your dynamic build of a program will run on my "still
being installed" system, or on my "libraries under maintenance" system.
Build the latest version of sshd and let's see if that works to log in and
repair the system. Build the latest version of tar and see if that can
restore the libraries from backup.


|> But consider the initial build environment of Linux From Scratch.
|> That's all statically linked. It is then used to build a dynamic
|> system. But even then, a few maintenance tools still need to be
|> statically linked.
|
| The intersection of tools that must be statically linked and tools
| that must be able to dlopen() is empty.

I'm NOT focusing on dlopen(). There are many functions broken this way.

But, dlopen() surely is a special case. How does it succeed at loading the real dlopen() code from a dynamic library at run time, from within a statically linked execuable, unless the library loading code is already present? Something is especially goofy or strange there. But dlopen() by itself is not the issue. The issue is that if I ask the linker to put every function needed together into an executable, and assuming I have all those functions present at build time (which should be provided in the respective libraries as installed, e.g. the glibc functions when glibc is installed and .a libraries have been included), than that executable should work on the kernel versions that library code is compatible with. And given the wide range of kernel changes I have made on a system that stays with the same library version, that has a history of being rather wide.


| If your tool is in fact a proof positive that this intersection is
| not empty, please provide convincing details.

You're looking at the wrong intersection. Look at the FULL set of all
functions that glibc has implemented this way, not just dlopen().

phil-new...@ipal.net

unread,
Nov 6, 2005, 1:31:10 PM11/6/05
to

Explain why this is the case with a newer version of glibc, but not with
an older version? Glibc has apparently been changed in the approach it
uses to deal with linking. I'm saying it is a change for the worse for
cases of static linking. And I don't see how such a change can help any
for dynamic linking, since dynamic linking already allows you to match
up library and kernel (and this hasn't been an overwhelming issue, either).

All the logic that is needed to do the dynamic loading of a library can
be put into a set of functions. Indeed, ld.so itself could very well
have been statically linked with that library and actually use it to do
what it does (I don't know if that was the actual way it was implemented
for glibc, but I am saying it could have been ... more likely, ld.so has
a stripped down subset of libdl for the subset of things it needs to do).

Måns Rullgård

unread,
Nov 6, 2005, 2:24:05 PM11/6/05
to
phil-new...@ipal.net writes:

Indeed, XFree86 has its own loader, and dynamically loads all sorts of
things at runtime.

--
Måns Rullgård
m...@inprovide.com

Paul Pluzhnikov

unread,
Nov 6, 2005, 4:59:25 PM11/6/05
to
phil-new...@ipal.net writes:

> Explain why this is the case with a newer version of glibc, but not with
> an older version?

It was *always* the case with glibc2. The newer versions simply
added the warning. The older versions didn't warn, but failed in
exactly the same way.

> All the logic that is needed to do the dynamic loading of a library can
> be put into a set of functions.

I believe you grossly underestimate the complexities involved.

There are all kinds of issues (various kinds of relocations,
lazy symbol resolution, dladdr / dl_iterate_phdr interfaces, TLS,
interface with the debugger, etc. etc.) which make implementing it
in such a way that it works for both static and dynamic executables,
with and without "reasonable" runtime environment, prohibitively
complicated.

Of course you can prove me wrong: just write such a set of
routines and maybe people who must link static tools will use your
set instead of the one from glibc.

Paul Pluzhnikov

unread,
Nov 6, 2005, 6:58:15 PM11/6/05
to
phil-new...@ipal.net writes:

> | Exactly. Build on RedHat 6.2 (dynamically), and it will run everywhere.
>
> Why dynamically? I'm trying to make something that works when all dynamic
> libraries are not (yet) present.

That something then should *not* depend on dynamic linking.

> | I build *one* version on RedHat 6.2, and that one version runs
> | everywhere.
>
> OK, let's see if your dynamic build of a program will run on my "still
> being installed" system, or on my "libraries under maintenance" system.

My program (obviously) will not run. But, it doesn't make sense to
run my program on such a system. It isn't a system maintenance
utility.

> I'm NOT focusing on dlopen(). There are many functions broken this way.

Here is the complete list for glibc 2.3.3 (well it's a compile list
extracted from libc.a and libdl.a):

dlopen endaliasent endgrent endhostent endnetent endprotoent
endpwent endrpcent endservent endspent getaddrinfo getaliasbyname
getaliasent getaliasent_r getgrent getgrent_r getgrgid getgrgid_r
getgrnam getgrnam_r getgrouplist gethostbyaddr gethostbyname
gethostbyname2 gethostent gethostent_r getnetbyaddr getnetbyaddr_r
getnetbyname getnetbyname_r getnetent getnetent_r getprotobyname
getprotoent getprotoent_r getpwent getpwent_r getpwnam getpwnam_r
getpwuid getpwuid_r getrpcbyname getrpcbyname_r getrpcbynumber
getrpcent getrpcent_r getservbyname getservbyport getservent
getservent_r getspent getspent_r getspnam getspnam_r initgroups
setaliasent setgrent sethostent setnetent setprotoent setpwent
setrpcent setservent setspent

All of these (except for dlopen()) need to load other parts of
glibc due to /etc/nsswitch.conf, etc.

> But, dlopen() surely is a special case. How does it succeed
> at loading the real dlopen() code from a dynamic library at run
> time, from within a statically linked execuable, unless the library
> loading code is already present?

Indeed. My experimentation shows that the library loading code is
in fact present ...

I don't know why dlopen() itself issues a warning, as it doesn't
need any other DSOs. Perhaps you can simply ignore the warning.

Continuing my other post, at least the debugger interface *is*
broken in this scenario: gdb "info shared" replies with (incorrect)

No shared libraries loaded at this time.



> | If your tool is in fact a proof positive that this intersection is
> | not empty, please provide convincing details.
>
> You're looking at the wrong intersection. Look at the FULL set of all
> functions that glibc has implemented this way, not just dlopen().

Well, I did look at all the functions above.

They can't be implemented with the current glibc setup (where libnss*
is split into multiple DSOs) to work correctly (and respect
/etc/nsswitch.conf) *unless* compatible (i.e. same version) pieces
of glibc are present at runtime.

So what you are asking for can be made to work, if every executable
on the system is burdened with all versions of libnss* ...

Apparently glibc developers decided that the system-maintenance
utilities which must be linked statically can be bothered to be
build without these routines.

phil-new...@ipal.net

unread,
Nov 7, 2005, 3:24:52 PM11/7/05
to
On Sun, 06 Nov 2005 15:58:15 -0800 Paul Pluzhnikov <ppluzhn...@charter.net> wrote:
| phil-new...@ipal.net writes:
|
|> | Exactly. Build on RedHat 6.2 (dynamically), and it will run everywhere.
|>
|> Why dynamically? I'm trying to make something that works when all dynamic
|> libraries are not (yet) present.
|
| That something then should *not* depend on dynamic linking.

Exactly. Hence the effort to do static linking.


|> | I build *one* version on RedHat 6.2, and that one version runs
|> | everywhere.
|>
|> OK, let's see if your dynamic build of a program will run on my "still
|> being installed" system, or on my "libraries under maintenance" system.
|
| My program (obviously) will not run. But, it doesn't make sense to
| run my program on such a system. It isn't a system maintenance
| utility.

Several programs I was trying to link statically could be used in such
situations, both repair, and installation (both considered maintenance, I
suppose).


|> I'm NOT focusing on dlopen(). There are many functions broken this way.
|
| Here is the complete list for glibc 2.3.3 (well it's a compile list
| extracted from libc.a and libdl.a):
|
| dlopen endaliasent endgrent endhostent endnetent endprotoent
| endpwent endrpcent endservent endspent getaddrinfo getaliasbyname
| getaliasent getaliasent_r getgrent getgrent_r getgrgid getgrgid_r
| getgrnam getgrnam_r getgrouplist gethostbyaddr gethostbyname
| gethostbyname2 gethostent gethostent_r getnetbyaddr getnetbyaddr_r
| getnetbyname getnetbyname_r getnetent getnetent_r getprotobyname
| getprotoent getprotoent_r getpwent getpwent_r getpwnam getpwnam_r
| getpwuid getpwuid_r getrpcbyname getrpcbyname_r getrpcbynumber
| getrpcent getrpcent_r getservbyname getservbyport getservent
| getservent_r getspent getspent_r getspnam getspnam_r initgroups
| setaliasent setgrent sethostent setnetent setprotoent setpwent
| setrpcent setservent setspent
|
| All of these (except for dlopen()) need to load other parts of
| glibc due to /etc/nsswitch.conf, etc.

And some of these may be needed, at least in some non-NSS form (since even
/etc/nsswitch.conf can be absent during maintenance).


|> But, dlopen() surely is a special case. How does it succeed
|> at loading the real dlopen() code from a dynamic library at run
|> time, from within a statically linked execuable, unless the library
|> loading code is already present?
|
| Indeed. My experimentation shows that the library loading code is
| in fact present ...
|
| I don't know why dlopen() itself issues a warning, as it doesn't
| need any other DSOs. Perhaps you can simply ignore the warning.

Only to have the program crash when the dynamic library is not present ...
not the one being loaded by the dlopen() call, the one needed by glibc's
implementation of dlopen().


| They can't be implemented with the current glibc setup (where libnss*
| is split into multiple DSOs) to work correctly (and respect
| /etc/nsswitch.conf) *unless* compatible (i.e. same version) pieces
| of glibc are present at runtime.

Hence a static executable needs to match a single version of glibc. This
is regressive, not progressive.


| So what you are asking for can be made to work, if every executable
| on the system is burdened with all versions of libnss* ...
|
| Apparently glibc developers decided that the system-maintenance
| utilities which must be linked statically can be bothered to be
| build without these routines.

Not many of those functions are needed. The few that are only need basic
UNIX passwd association in the maintenance context. A kind of "null nss"
facility would be nice. Hopefully most of these programs will link OK
with dietlibc.

phil-new...@ipal.net

unread,
Nov 7, 2005, 3:28:56 PM11/7/05
to
On Sun, 06 Nov 2005 13:59:25 -0800 Paul Pluzhnikov <ppluzhn...@charter.net> wrote:
| phil-new...@ipal.net writes:
|
|> Explain why this is the case with a newer version of glibc, but not with
|> an older version?
|
| It was *always* the case with glibc2. The newer versions simply
| added the warning. The older versions didn't warn, but failed in
| exactly the same way.
|
|> All the logic that is needed to do the dynamic loading of a library can
|> be put into a set of functions.
|
| I believe you grossly underestimate the complexities involved.
|
| There are all kinds of issues (various kinds of relocations,
| lazy symbol resolution, dladdr / dl_iterate_phdr interfaces, TLS,
| interface with the debugger, etc. etc.) which make implementing it
| in such a way that it works for both static and dynamic executables,
| with and without "reasonable" runtime environment, prohibitively
| complicated.

But this is complication that means a lot of code, not a dynamic library.


| Of course you can prove me wrong: just write such a set of
| routines and maybe people who must link static tools will use your
| set instead of the one from glibc.

It needs a lot of code. I don't have time to write a lot of code. I just
wish wiser people hand been the ones to do so, and made it such that one
could _fully_ statically link every piece of code needed to do all that
needs to be done, but some means (maybe bu appending a special .a file).

What is the point of having -static as a linker option? What is the point
of having some library defeat it?

David Schwartz

unread,
Nov 10, 2005, 11:33:03 AM11/10/05
to

"Tauno Voipio" <tauno....@INVALIDiki.fi> wrote in message
news:gjjbf.31$j41...@read3.inet.fi...
> phil-new...@ipal.net wrote:

> There is a techical reason why dlopen() implies dynamic linking:
> To make any dynamic library work, your executable *must* have
> the dynamic linker (ld.so) marked as the program interpreter
> in the runnable image. On the other hand, ld.so itself is
> usually a dynamic library itself, so you cannot use it in
> a statically linked environment.

That really doesn't make any sense. If a statically-linked program calls
'dlopen', then it *is* a dynamic linker and should do the same job 'ld.so'
does (but with the necessary code linked in statically, of course).

DS


phil-new...@ipal.net

unread,
Nov 10, 2005, 2:17:21 PM11/10/05
to

Yes. However, everything needed to dynamically (at the time the application
calls dlopen(), not at the time the application executable is loaded) link a
library picked by the application should be linked into that executable, for
the target kernel the libraries being linked in understand how to call (ABI
version).

David Schwartz

unread,
Nov 10, 2005, 6:58:58 PM11/10/05
to

<phil-new...@ipal.net> wrote in message
news:dl06c...@news2.newsguy.com...

> On Thu, 10 Nov 2005 08:33:03 -0800 David Schwartz <dav...@webmaster.com>
> wrote:

> | That really doesn't make any sense. If a statically-linked program
> calls
> | 'dlopen', then it *is* a dynamic linker and should do the same job
> 'ld.so'
> | does (but with the necessary code linked in statically, of course).

> Yes. However, everything needed to dynamically (at the time the
> application
> calls dlopen(), not at the time the application executable is loaded) link
> a
> library picked by the application should be linked into that executable,
> for
> the target kernel the libraries being linked in understand how to call
> (ABI
> version).

Which should support any later kernel version unless an incompatible
change is forced, which should be very rare. Working on an older kernel
requires detection and fallback, which isn't always provided. Either way,
your code should work on a very large number of later kernel versions and,
with luck, quite a few earlier ones.

DS


phil-new...@ipal.net

unread,
Nov 14, 2005, 1:56:54 PM11/14/05
to

Which is actually a larger scope of compatibility than what is imposed by
glibc when it requires having present certain dynamic libraries under its
new "static" scheme.

0 new messages