Re: version krb5support_0_MIT not defined

[Jeff Blaine]

I'm still completely stumped by this in case anyone
has any suggestions. I cannot make any sense of it
and have tried everything I know:

/usr/rcf-krb5/lib/libcom_err.so.3: symbol krb5int_strlcpy,
version krb5support_0_MIT not defined in file
libkrb5support.so.0 with link time reference

/usr/rcf-krb5/lib/libcom_err.so.3 (1.9.2) is finding
/usr/lib/libkrb5support.so.0 (MIT 1.6.3, which we need
to stay there) for no good reason I understand.

Using 'readelf -d' on /usr/rcf-krb5/lib/libkrb5support.so,
/usr/rcf-krb5/lib/libcomm_err.so, and pam_krb5 indicate:

0x0000000f (RPATH) Library rpath:
[/usr/rcf-krb5/lib]
0x0000001d (RUNPATH) Library runpath:
[/usr/rcf-krb5/lib]

On 2/1/2012 5:58 PM, Jeff Blaine wrote:
> I'm having a bit of a linking dilemma while trying to build
> and use Russ Alberry's pam_krb5 module against our own MIT
> Kerberos build.
>
> The module reports (at load time via PAM):
>
> symbol krb5int_strlcpy, version krb5support_0_MIT not defined
> in file libkrb5support.so.0 with link time reference
>
> MIT Kerberos 1.9.2 was built as follows
> * RHEL 5.7 i686 with RHEL Kerberos installed and needed too
> * export LDFLAGS="-Xlinker -rpath -Xlinker /usr/rcf-krb5/lib"
> * ./configure --prefix=/usr/rcf-krb5
>
> pam_krb5 4.5 was built as follows:
> * RHEL 5.7 i686 (same host) with RHEL Kerberos installed and needed too
> * export LDFLAGS="-Xlinker -rpath -Xlinker /usr/rcf-krb5/lib"
> * ./configure --with-krb5=/usr/rcf-krb5
>
> Using 'readelf -d' on /usr/rcf-krb5/lib/libkrb5support.so,
> /usr/rcf-krb5/lib/libcomm_err.so, and pam_krb5 indicate:
>
> 0x0000000f (RPATH) Library rpath: [/usr/rcf-krb5/lib]
> 0x0000001d (RUNPATH) Library runpath:
> [/usr/rcf-krb5/lib]
>
> A functioning, but not acceptable, solution to the problem,
> is 'echo /usr/rcf-krb5/lib> /etc/ld.so.conf.d/krb5.conf;
> ldconfig' : pam_krb5 works fine then.
>
> Does anyone have any ideas?
> ________________________________________________
> Kerberos mailing list Kerb...@mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>

[Tom Yu]

Jeff Blaine <jbl...@kickflop.net> writes:

> I'm still completely stumped by this in case anyone
> has any suggestions. I cannot make any sense of it
> and have tried everything I know:
>
> /usr/rcf-krb5/lib/libcom_err.so.3: symbol krb5int_strlcpy,
> version krb5support_0_MIT not defined in file
> libkrb5support.so.0 with link time reference
>
> /usr/rcf-krb5/lib/libcom_err.so.3 (1.9.2) is finding
> /usr/lib/libkrb5support.so.0 (MIT 1.6.3, which we need
> to stay there) for no good reason I understand.
>
> Using 'readelf -d' on /usr/rcf-krb5/lib/libkrb5support.so,
> /usr/rcf-krb5/lib/libcomm_err.so, and pam_krb5 indicate:
>
> 0x0000000f (RPATH) Library rpath:
> [/usr/rcf-krb5/lib]
> 0x0000001d (RUNPATH) Library runpath:
> [/usr/rcf-krb5/lib]

I'm not quite sure what's going on here. Maybe krb5-1.9.2 is built
with library symbol versions while 1.6.3 is not?

Do the programs built as part of krb5-1.9.2 work correctly? Does
pam_krb5 depend on the system's libcom_err directly? This could be a
symptom of the lack of RTLD_GROUP in dlopen() in glibc.

[Greg Hudson]

On 02/07/2012 11:07 AM, Jeff Blaine wrote:
> /usr/rcf-krb5/lib/libcom_err.so.3: symbol krb5int_strlcpy,
> version krb5support_0_MIT not defined in file
> libkrb5support.so.0 with link time reference

Is LD_LIBRARY_PATH set in the environment in the case where you see
these PAM failures, and if so, what value is it set to?

[Jeff Blaine]

Not set that I can see. I've not touched "LD_LIBRARY_PATH"
for ~10 years in any environment. Grepping around shows
nothing.

The environment is "Stock RHEL 5.7 sshd calling (our)
pam_krb5.so, which has only runtime linker path info
for /usr/rcf-krb5/lib in it, ... for authentication".

Here's my understanding of what is happening:

sshd --> pam_krb5.so (*)
|
|--> /usr/rcf-krb5/lib/libcom_err.so.3 (**)
|
|--> /usr/lib64/libkrb5support.so.0 (***)

All built with LDFLAGS="-Xlinker -rpath -Xlinker /usr/rcf-krb5/lib"

* Our module, not stock RHEL.
** Being found properly as /usr/rcf-krb5/lib/libcom_err.so.3,
according to syslog line above.
*** WTF

I just spent another hour on this, and it just makes no
sense to me. I can see all of the proper runtime linker
path info right in the output of make for both MIT
Kerberos 1.9.2 and the PAM module. It all shows up in
'readelf -d' output too.

For some reason I cannot begin to explain, the MIT
libcom_err.so.3 is seeking libkrb5support.so.0 and
refuses to find it sitting right where it was told
to look.

The only thing that works toward this simple (or
so I thought) goal is putting "/usr/rcf-krb5/lib"
into /etc/ld.so.conf.d/krb5.conf and running ldconfig.
Everything works perfectly then.

[Marcus Watts]

> Date: Tue, 07 Feb 2012 17:28:18 EST
> To: Greg Hudson <ghu...@mit.edu>
> cc: kerb...@mit.edu
> From: Jeff Blaine <jbl...@kickflop.net>
> Subject: Re: version krb5support_0_MIT not defined

> ________________________________________________
> Kerberos mailing list Kerb...@mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos

bad ld.so.cache?
ok, you've run ldconfig. maybe not...

prelinking?
the "prelink" command has interesting options...

-Marcus Watts

[Tom Yu]

Jeff Blaine <jbl...@kickflop.net> writes:

> Here's my understanding of what is happening:
>
> sshd --> pam_krb5.so (*)
> |
> |--> /usr/rcf-krb5/lib/libcom_err.so.3 (**)
> |
> |--> /usr/lib64/libkrb5support.so.0 (***)

How did you determine which path was actually used to find
libkrb5support.so.0?

> All built with LDFLAGS="-Xlinker -rpath -Xlinker /usr/rcf-krb5/lib"
>
> * Our module, not stock RHEL.
> ** Being found properly as /usr/rcf-krb5/lib/libcom_err.so.3,
> according to syslog line above.
> *** WTF
>
> I just spent another hour on this, and it just makes no
> sense to me. I can see all of the proper runtime linker
> path info right in the output of make for both MIT
> Kerberos 1.9.2 and the PAM module. It all shows up in
> 'readelf -d' output too.

What are the direct dependencies listed by "readelf -d" for sshd and
pam_krb5.so? (the "NEEDED" lines)

> For some reason I cannot begin to explain, the MIT
> libcom_err.so.3 is seeking libkrb5support.so.0 and
> refuses to find it sitting right where it was told
> to look.

It's possible that setting the LD_DEBUG environment variable to "libs"
would give you more useful information about the shared library search
paths. You might need to point LD_DEBUG_OUTPUT somewhere useful so it
won't send the debug output to stderr.

[Jeff Blaine]

First, thanks to everyone helping unravel this mystery.

On 2/7/2012 6:52 PM, Tom Yu wrote:
> Jeff Blaine<jbl...@kickflop.net> writes:
>
>> Here's my understanding of what is happening:
>>
>> sshd --> pam_krb5.so (*)
>> |
>> |--> /usr/rcf-krb5/lib/libcom_err.so.3 (**)
>> |
>> |--> /usr/lib64/libkrb5support.so.0 (***)
>
> How did you determine which path was actually used to find
> libkrb5support.so.0?

Proven, no, but it was based on this:

-bash-3.2# ldconfig -p | grep krb5support | head -1
libkrb5support.so.0 (libc6,x86-64) =>
/usr/lib64/libkrb5support.so.0
-bash-3.2#

>> All built with LDFLAGS="-Xlinker -rpath -Xlinker /usr/rcf-krb5/lib"
>>
>> * Our module, not stock RHEL.
>> ** Being found properly as /usr/rcf-krb5/lib/libcom_err.so.3,
>> according to syslog line above.
>> *** WTF
>>
>> I just spent another hour on this, and it just makes no
>> sense to me. I can see all of the proper runtime linker
>> path info right in the output of make for both MIT
>> Kerberos 1.9.2 and the PAM module. It all shows up in
>> 'readelf -d' output too.
>
> What are the direct dependencies listed by "readelf -d" for sshd and
> pam_krb5.so? (the "NEEDED" lines)

-bash-3.2# readelf -d /lib64/security/pam_krb5.so | grep NEEDED
0x0000000000000001 (NEEDED) Shared library: [libkrb5.so.3]
0x0000000000000001 (NEEDED) Shared library: [libk5crypto.so.3]
0x0000000000000001 (NEEDED) Shared library: [libcom_err.so.3]
0x0000000000000001 (NEEDED) Shared library: [libkeyutils.so.1]
0x0000000000000001 (NEEDED) Shared library: [libresolv.so.2]
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED) Shared library: [libpam.so.0]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
-bash-3.2# readelf -d /usr/sbin/sshd | grep NEEDED
0x0000000000000001 (NEEDED) Shared library: [libwrap.so.0]
0x0000000000000001 (NEEDED) Shared library: [libpam.so.0]
0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
0x0000000000000001 (NEEDED) Shared library: [libselinux.so.1]
0x0000000000000001 (NEEDED) Shared library: [libaudit.so.0]
0x0000000000000001 (NEEDED) Shared library: [libfipscheck.so.1]
0x0000000000000001 (NEEDED) Shared library: [libcrypto.so.6]
0x0000000000000001 (NEEDED) Shared library: [libutil.so.1]
0x0000000000000001 (NEEDED) Shared library: [libz.so.1]
0x0000000000000001 (NEEDED) Shared library: [libnsl.so.1]
0x0000000000000001 (NEEDED) Shared library: [libcrypt.so.1]
0x0000000000000001 (NEEDED) Shared library: [libresolv.so.2]
0x0000000000000001 (NEEDED) Shared library: [libgssapi_krb5.so.2]
0x0000000000000001 (NEEDED) Shared library: [libkrb5.so.3]
0x0000000000000001 (NEEDED) Shared library: [libk5crypto.so.3]
0x0000000000000001 (NEEDED) Shared library: [libcom_err.so.2]
0x0000000000000001 (NEEDED) Shared library: [libnss3.so]
0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
-bash-3.2#

>> For some reason I cannot begin to explain, the MIT
>> libcom_err.so.3 is seeking libkrb5support.so.0 and
>> refuses to find it sitting right where it was told
>> to look.
>
> It's possible that setting the LD_DEBUG environment variable to "libs"
> would give you more useful information about the shared library search
> paths. You might need to point LD_DEBUG_OUTPUT somewhere useful so it
> won't send the debug output to stderr.

I'm going to fire up an sshd -d -o "Port 2222" with LD_DEBUG=libs
and will report back with any findings.

[Jeff Blaine]

> I'm going to fire up an sshd -d -o "Port 2222" with LD_DEBUG=libs
> and will report back with any findings.

Well, that was easy, except it doesn't tell me which
libkrb5support.so.0 that libcom_err.so.3 is using:

debug1: PAM: initializing for "jblaine"
28358:
28358: calling init: /lib64/security/pam_env.so
28358:
28358: find library=libcom_err.so.3 [0]; searching
28358: search
path=/usr/rcf-krb5/lib/tls/x86_64:/usr/rcf-krb5/lib/tls:/usr/rcf-krb5/lib/x86_64:/usr/rcf-krb5/lib
(RUNPATH from file /lib64/security/pam_krb5.so)
28358: trying file=/usr/rcf-krb5/lib/tls/x86_64/libcom_err.so.3
28358: trying file=/usr/rcf-krb5/lib/tls/libcom_err.so.3
28358: trying file=/usr/rcf-krb5/lib/x86_64/libcom_err.so.3
28358: trying file=/usr/rcf-krb5/lib/libcom_err.so.3
28358:
28358: /usr/rcf-krb5/lib/libcom_err.so.3: error: relocation
error: symbol krb5int_strlcpy, version krb5support_0_MIT not defined in
file libkrb5support.so.0 with link time reference (fatal)

Neither does strace -f:

[pid 28613] open("/usr/rcf-krb5/lib/libcom_err.so.3", O_RDONLY) = 8
[pid 28613] writev(2, [{" 28613:\t", 12},
{"/usr/rcf-krb5/lib/libcom_err.so."..., 33}, {": error: ", 9},
{"relocation error", 16}, {": ", 2}, {"symbol krb5int_strlcpy, version
"..., 114}, {" (", 2}, {"fatal", 5}, {")\n", 2}], 9 28613:
/usr/rcf-krb5/lib/libcom_err.so.3: error: relocation error: symbol

krb5int_strlcpy, version krb5support_0_MIT not defined in file

libkrb5support.so.0 with link time reference (fatal)

But a tiny C program that does a:

dlopen("/usr/rcf-krb5/lib/libcom_err.so.3", RTLD_NOW);

on the file (totally grasping in the dark here) shows:

-bash-3.2# env | grep LD
OLDPWD=/tmp
-bash-3.2# strace ./crap 2>&1 | grep open
...
open("/usr/rcf-krb5/lib/libcom_err.so.3", O_RDONLY) = 3
open("/usr/rcf-krb5/lib/tls/x86_64/libkrb5support.so.0", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/rcf-krb5/lib/tls/libkrb5support.so.0", O_RDONLY) = -1 ENOENT
(No such file or directory)
open("/usr/rcf-krb5/lib/x86_64/libkrb5support.so.0", O_RDONLY) = -1
ENOENT (No such file or directory)
open("/usr/rcf-krb5/lib/libkrb5support.so.0", O_RDONLY) = 3
...

[Tom Yu]

Jeff Blaine <jbl...@kickflop.net> writes:

> -bash-3.2# readelf -d /lib64/security/pam_krb5.so | grep NEEDED
> 0x0000000000000001 (NEEDED) Shared library: [libkrb5.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libk5crypto.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libcom_err.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libkeyutils.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libresolv.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libpam.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]

This tells me that pam_krb5.so directly lists libcom_err.so.3 as a
dependency, but not libkrb5_support.so.0. The syslog message could be
coming from the runtime linker as it tries to resolve strlcpy() in the
/usr/rcf-krb5/lib/libcom_err.so.3 that pam_krb5.so depends on. If the
runtime linker thinks that has already loaded a copy of
libkrb5_support.so.0, it might look in that copy rather than the
correct copy that would be found through the RPATH of libcom_err.so.3.

A lot of this mystery may revolve around whether the runtime linker
has a way to disambiguate different shared libraries that happen to
have the same SONAME but reside in different files and have different
contents. I've heard it claimed that the Solaris runtime linker
tracks pairs of device numbers with inode numbers to disambiguate
libraries that have the same SONAME, but I don't know if the Linux one
does, and skimming its source code did not make it obvious to me.

> -bash-3.2# readelf -d /usr/sbin/sshd | grep NEEDED
> 0x0000000000000001 (NEEDED) Shared library: [libwrap.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libpam.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libdl.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libselinux.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libaudit.so.0]
> 0x0000000000000001 (NEEDED) Shared library: [libfipscheck.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libcrypto.so.6]
> 0x0000000000000001 (NEEDED) Shared library: [libutil.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libz.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libnsl.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libcrypt.so.1]
> 0x0000000000000001 (NEEDED) Shared library: [libresolv.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libgssapi_krb5.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libkrb5.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libk5crypto.so.3]
> 0x0000000000000001 (NEEDED) Shared library: [libcom_err.so.2]
> 0x0000000000000001 (NEEDED) Shared library: [libnss3.so]
> 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
> -bash-3.2#

It appears that sshd also depends directly on libcom_err.so.2, which
is a different SONAME than the libcom_err.so.3 that pam_krb5.so
depends on. libcom_err.so.2 might or might not depend on
libkrb5_crypto.so.0, but it's possible that some other dependency path
involving the krb5 libraries used by sshd causes the runtime linker to
load the system copy of libkrb5_support.so.0 into the sshd process.

> I'm going to fire up an sshd -d -o "Port 2222" with LD_DEBUG=libs
> and will report back with any findings.

Thanks, that will give us information that we can use in attempts to
document this sort of behavior.

[Nico Williams]

On Wed, Feb 8, 2012 at 9:07 AM, Tom Yu <tl...@mit.edu> wrote:
> A lot of this mystery may revolve around whether the runtime linker
> has a way to disambiguate different shared libraries that happen to
> have the same SONAME but reside in different files and have different
> contents.  I've heard it claimed that the Solaris runtime linker
> tracks pairs of device numbers with inode numbers to disambiguate
> libraries that have the same SONAME, but I don't know if the Linux one
> does, and skimming its source code did not make it obvious to me.

Here's the OpenSolaris (last open build, 151a) RTLD function that
checks the st_dev and st_ino:

http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/sgs/rtld/common/analyze.c#1026
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/cmd/sgs/rtld/common/analyze.c#1259
http://src.opensolaris.org/source/s?refs=is_devinode_loaded&project=onnv

The glibc rtld does too, as you can see here:

http://sourceware.org/git/?p=glibc.git;a=blob;f=elf/dl-load.c;h=f3717ea58ca69dcd4dac0971ca794213f83d5943;hb=HEAD#l943

But this is almost certainly a red herring.  The main thing is to
prevent DLL hell.  The RTLD must be finding the SONAME (and probably
version number) listed as a dependency in is already on the main link
map, thus not even opening nor stat-ing the shared object ELF to find
that the shared object that really ought to be loaded is distinct from
the one on the link map that has the same SONAME.

Nico
--

[Russ Allbery]

Tom Yu <tl...@MIT.EDU> writes:

> This tells me that pam_krb5.so directly lists libcom_err.so.3 as a
> dependency, but not libkrb5_support.so.0.

This, in turn, would be because of krb5-config:

windlord:~> krb5-config --libs
-L/usr/lib/i386-linux-gnu -Wl,-z,relro -lkrb5 -lk5crypto -lcom_err

(ignore the first -L, which is an artifact of Debian multiarch and a minor
bug in Debian's krb5 packages, which I'll go file).

krb5-config lists -lcom_err as an explicit dependency, but not
-lkrb5support.

I don't recall if there are any Debian-specific patches in this area or if
this matches standard MIT Kerberos behavior.

--
Russ Allbery (r...@stanford.edu) <http://www.eyrie.org/~eagle/>

[Jeff Blaine]

>> This tells me that pam_krb5.so directly lists libcom_err.so.3 as a
>> dependency, but not libkrb5_support.so.0.
>
> This, in turn, would be because of krb5-config:
>
> windlord:~> krb5-config --libs
> -L/usr/lib/i386-linux-gnu -Wl,-z,relro -lkrb5 -lk5crypto -lcom_err

Confirming Russ:

rcf-linux-beta# /usr/rcf-krb5/bin/krb5-config --libs
-L/usr/rcf-krb5/lib -Wl,--enable-new-dtags -Wl,-rpath
-Wl,/usr/rcf-krb5/lib -Xlinker -rpath -Xlinker /usr/rcf-krb5/lib -lkrb5
-lk5crypto -lcom_err -lkeyutils -lresolv -ldl

[Jeff Blaine]

> rcf-linux-beta# /usr/rcf-krb5/bin/krb5-config --libs
> -L/usr/rcf-krb5/lib -Wl,--enable-new-dtags -Wl,-rpath
> -Wl,/usr/rcf-krb5/lib -Xlinker -rpath -Xlinker /usr/rcf-krb5/lib -lkrb5
> -lk5crypto -lcom_err -lkeyutils -lresolv -ldl

FWIW, tweaking krb5-config (just to see...) to add
"-lkrb5support" when asked for "--libs" does
not get pam_krb5.so any further.

Same PAM error when tickled.

pam-krb5-4.5:rcf-linux-beta# /usr/rcf-krb5/bin/krb5-config --libs

-L/usr/rcf-krb5/lib -Wl,--enable-new-dtags -Wl,-rpath
-Wl,/usr/rcf-krb5/lib -Xlinker -rpath -Xlinker /usr/rcf-krb5/lib -lkrb5

-lk5crypto -lcom_err -lkrb5support -lkeyutils -lresolv -ldl
pam-krb5-4.5:rcf-linux-beta#

[ reconfigure, rebuild and reinstall pam-krb5.so ]

Same deal.

/usr/rcf-krb5/lib/libcom_err.so.3: symbol krb5int_strlcpy,

[Russ Allbery]

Jeff Blaine <jbl...@kickflop.net> writes:

> FWIW, tweaking krb5-config (just to see...) to add
> "-lkrb5support" when asked for "--libs" does
> not get pam_krb5.so any further.

> Same PAM error when tickled.

Yeah, I think you have two libraries with different contents but the same
SONAME, at which point the run-time linker is only going to load the
library once (whichever one it finds first) into a given process space.

[Jeff Blaine]

> Yeah, I think you have two libraries with different contents but the same
> SONAME, at which point the run-time linker is only going to load the
> library once (whichever one it finds first) into a given process space.

This is a correct assessment.

-bash-3.2# readelf -d /usr/lib64/libcom_err.so
/usr/rcf-krb5/lib/libcom_err.so /usr/lib64/libkrb5support.so
/usr/rcf-krb5/lib/libkrb5support.so | grep SONAME
0x000000000000000e (SONAME) Library soname: [libcom_err.so.2]
0x000000000000000e (SONAME) Library soname: [libcom_err.so.3]
0x000000000000000e (SONAME) Library soname: [libkrb5support.so.0]
0x000000000000000e (SONAME) Library soname: [libkrb5support.so.0]
-bash-3.2#

It would appear that I cannot control library location
preference in this situation, but I am certainly open
to any last chance suggestions.

This raises the question to me (largely ignorant past this
point): Isn't this incorrect? I'd think that an incompatible
library issue like this means something was not incremented
when it should have been, no?

[Tom Yu]

Jeff Blaine <jbl...@kickflop.net> writes:

> -bash-3.2# readelf -d /usr/lib64/libcom_err.so
> /usr/rcf-krb5/lib/libcom_err.so /usr/lib64/libkrb5support.so
> /usr/rcf-krb5/lib/libkrb5support.so | grep SONAME
> 0x000000000000000e (SONAME) Library soname: [libcom_err.so.2]
> 0x000000000000000e (SONAME) Library soname: [libcom_err.so.3]
> 0x000000000000000e (SONAME) Library soname: [libkrb5support.so.0]
> 0x000000000000000e (SONAME) Library soname: [libkrb5support.so.0]
> -bash-3.2#
>
> It would appear that I cannot control library location
> preference in this situation, but I am certainly open
> to any last chance suggestions.

I think the solution that I would suggest at this point is using a
wrapper script to start sshd with a value of LD_LIBRARY_PATH that
includes /usr/rcf-krb5/lib, unless there's some specific reason that
you need sshd to use a different set of krb5 libraries than you want
pam_krb5.so to use.

In the long run, there are better ways to overcome the problems you're
experiencing, but they probably require adding new capabilities to the
glibc runtime linker:

* support for RTLD_GROUP

* distinguishing libraries that have the same SONAME but exist as
distinct files

Solaris supports these; I have heard that the glibc developers have
repeatedly resisted adding these capabilities.

You are expecting pam_krb5.so to resolve the symbols that it needs
using the libraries that it would find in its RPATH. On a Solaris
system, if sshd had called dlopen() on pam_krb5.so using the
RTLD_GROUP flag, this would be the case. On RHEL, using the glibc
runtime linker as it currently exists, that does not occur, and
pam_krb5.so will find the versions of the symbols that already exist
in the address space of the sshd process.

In this case on RHEL, the runtime linker actually loads
libcom_err.so.3 into the address space of the sshd process, but this
has minimal practical effect, because the symbols that pam_krb5.so
requires will already bind to the same ones found in the
libcom_err.so.2 that sshd already depended on. The runtime linker
will not map the /usr/rcf-krb5/lib copy of libkrb5.so.3, because it
has the same SONAME as the /usr/lib copy that sshd depended on, so
your efforts to change the behavior of pam_krb5.so by having it depend
on a more recent version of libkrb5 would not succeed.

> This raises the question to me (largely ignorant past this
> point): Isn't this incorrect? I'd think that an incompatible
> library issue like this means something was not incremented
> when it should have been, no?

Yes and no. Shared libraries present some fairly complex
interactions, and some difficult tradeoffs.

The SONAME (the name that the link editor records in dependent
executables and libraries) of the library is something that developers
typically don't increment unless an incompatible change in the
external interface of the library occurs. Typically this is something
like dramatically altering the interface of an API function, or
deleting interfaces from the library. Changing the implementation of
the library or adding a new API function aren't normally cause for
incrementing the SONAME. Changing the SONAME of a library tends to
require distributors of binary packages and operating systems to go
through a lot of effort to manage an orderly transition, so they don't
like library developers to make SONAME changes without a really good
reason.

Here, the OS provides libcom_err.so.2 (from the krb5-1.6.x series,
though a brief check of my sources indicates that we shipped
libcom_err.so.3 at least toward the end of krb5-1.6.x), but
pam_krb5.so declares an explicit dependency on libcom_err.so.3. The
runtime linker maps libcom_err.so.3 into the sshd address space, but
nothing will reference anything in it (unless something in pam_krb5.so
or its dependencies need to bind to a symbol that libcom_err.so.3
provides but libcom_err.so.2 does not).

I believe the error that you see involves the runtime linker
attempting to resolve all undefined references within libcom_err.so.3,
assuming the call that sshd made to dlopen() included the RTLD_NOW
flag, which forces all symbol binding in the opened shared object to
occur before returning. Here, libcom_err.so.3 has an undefined
reference to the function symbol "krb5int_strlcpy" with version
"krb5support_0_MIT". The copy of libkrb5_support.so.0 in /usr/lib
doesn't define this symbol, but the one in /usr/rcf-krb5/lib does.
The two libraries have the same SONAME, so the glibc runtime linker
(unlike Solaris) can't tell them apart when trying to map all the
dependencies. I believe the Solaris runtime linker would have mapped
both copies of libkrb5_support.so.0 in this case, but it would not
have generally searched the /usr/rcf-krb5/lib versions of the krb5
libraries for symbol resolution unless the call to dlopen() included
the RTLD_GROUP flag, or unless the /usr/lib versions lacked symbols
that the /usr/rcf-krb5/lib versions defined.

[Carson Gaspar]

[ much DLL hell deleted ]

In general, it is death to link any PAM module against non-system
libraries. Just try it on Solaris, and you'll see different (but still
fatal) errors (yes, the Solaris linker is better, but GIGO still applies).

If you _insist_ on using non-system krb5 libs for your pam_krb5 (and I'd
ask why...), you'll have to either:

- Replace the system libkrb5support.so.0 with your version (this is what
changing the ld.so search order is effectively doing)

- Change the SONAME of libkrb5support in your krb5 build (to something
like libkrb5support.so.0.rfc-krb5) and re-link everything against it.

--
Carson

[Jeff Blaine]

On 2/14/2012 2:41 AM, Carson Gaspar wrote:
> [ much DLL hell deleted ]
>
> In general, it is death to link any PAM module against non-system
> libraries. Just try it on Solaris, and you'll see different (but still
> fatal) errors (yes, the Solaris linker is better, but GIGO still applies).

Carson, we've had MIT Kerberos built in /usr/rcf-krb5 for
2 years now on Solaris 10 SPARC. Our pam_krb5 and pam_afs_session
modules link to it and work fine. Our OpenAFS client builds
against it as well.

This is why we sought to do the same for RHEL in order to
standardize on MIT Kerberos 1.9.2, our PAM modules linked to
it, and OpenAFS linked against it.

We've given up on that effort.

[Tom Yu]

Jeff Blaine <jbl...@kickflop.net> writes:

> On 2/14/2012 2:41 AM, Carson Gaspar wrote:
>> [ much DLL hell deleted ]
>>
>> In general, it is death to link any PAM module against non-system
>> libraries. Just try it on Solaris, and you'll see different (but still
>> fatal) errors (yes, the Solaris linker is better, but GIGO still applies).
>
> Carson, we've had MIT Kerberos built in /usr/rcf-krb5 for
> 2 years now on Solaris 10 SPARC. Our pam_krb5 and pam_afs_session
> modules link to it and work fine. Our OpenAFS client builds
> against it as well.

I thought libpam on Solaris did not use RTLD_GROUP to load modules,
but I could be mistaken. If it does, that could explain why pam_krb5
works there.

Some background for parts of the Solaris behavior is here:

https://blogs.oracle.com/rie/entry/loading_multiple_files_same_name

> This is why we sought to do the same for RHEL in order to
> standardize on MIT Kerberos 1.9.2, our PAM modules linked to
> it, and OpenAFS linked against it.
>
> We've given up on that effort.

In case you're interested in additional alternative solutions:

Further digging has turned up the -Bgroup option to GNU ld, which
might be useful if you build pam_krb5.so using it. It makes the
resulting module behave as if it were always dlopen()ed with
RTLD_GROUP, in theory. You will probably need to explicitly list all
the dependencies that pam_krb5.so might have been relying upon
locating in the sshd process, such as libc and possibly dependencies
of libc. I experimentally confirmed that it works for a minimal
loadable module test case on Ubuntu Lucid, but I did have to declare
an explicit dependency on libc and /lib64/ld-linux-x86-64.so.2, which
seems like a bug. It might not work for a more complex set of
dependencies than my test case uses.

It does seem like people occasionally run into these sorts of shared
library conflict problems, especially when loadable module frameworks
are involved, and I'd like to try to find a way to reduce this pain.

[Carson Gaspar]

On 2/14/12 7:35 AM, Jeff Blaine wrote:
> On 2/14/2012 2:41 AM, Carson Gaspar wrote:
>> [ much DLL hell deleted ]
>>
>> In general, it is death to link any PAM module against non-system
>> libraries. Just try it on Solaris, and you'll see different (but still
>> fatal) errors (yes, the Solaris linker is better, but GIGO still applies).
>
> Carson, we've had MIT Kerberos built in /usr/rcf-krb5 for
> 2 years now on Solaris 10 SPARC. Our pam_krb5 and pam_afs_session
> modules link to it and work fine. Our OpenAFS client builds
> against it as well.

When I tried it (long ago, back when Solaris was missing krb5 headers,
so it may behave differently now), I was trying to link sshd against MIT
GSSAPI libs. If the Solaris kerberos pam module was in my pam stack, it
crashed. (Note I was doing the opposite of what you were - app with
self-built libs, system pam with system libs).

--
Carson

[Nico Williams]

See the blog entry that Tom linked. You must have been using Solaris
8. The Solaris linker has changed quite a bit since then.

Nico
--

[Douglas E. Engert]

On 2/14/2012 10:51 AM, Tom Yu wrote:
> Jeff Blaine<jbl...@kickflop.net> writes:
>

>> On 2/14/2012 2:41 AM, Carson Gaspar wrote:
>>> [ much DLL hell deleted ]
>>>
>>> In general, it is death to link any PAM module against non-system
>>> libraries. Just try it on Solaris, and you'll see different (but still
>>> fatal) errors (yes, the Solaris linker is better, but GIGO still applies).

On Solaris 10, we use the Oracle provided Kerberos libs, pam_krb5 and applications
including ssh and sshd. Pam_afs_session and aklog are linked against the Oracle Kerberos.

But to test PKINIT with smart cards at the console, dtlogin is calling pam
were all the pam kerberos aware modules are using the MIT Kerberos libs. This
includes using Russ's pam_krb5 and a pam_afs_session linked against the MIT kerberos.
This avoids the "DLL hell".

pam_afs_session does an exec of aklog, but since aklog is running in a separate
process, there is no conflict between the two Kerberos implementations.

Where it could get messy is if two Kerberos aware pam modules complied
against different Kerberos implementations try and pass a Kerberos context.
So even if one can loading two implementations, as Tom discusses below,
there may still be problems.

>>
>> Carson, we've had MIT Kerberos built in /usr/rcf-krb5 for
>> 2 years now on Solaris 10 SPARC. Our pam_krb5 and pam_afs_session
>> modules link to it and work fine. Our OpenAFS client builds
>> against it as well.
>

> I thought libpam on Solaris did not use RTLD_GROUP to load modules,
> but I could be mistaken. If it does, that could explain why pam_krb5
> works there.
>
> Some background for parts of the Solaris behavior is here:
>
> https://blogs.oracle.com/rie/entry/loading_multiple_files_same_name
>
>> This is why we sought to do the same for RHEL in order to
>> standardize on MIT Kerberos 1.9.2, our PAM modules linked to
>> it, and OpenAFS linked against it.
>>
>> We've given up on that effort.
>
> In case you're interested in additional alternative solutions:
>
> Further digging has turned up the -Bgroup option to GNU ld, which
> might be useful if you build pam_krb5.so using it. It makes the
> resulting module behave as if it were always dlopen()ed with
> RTLD_GROUP, in theory. You will probably need to explicitly list all
> the dependencies that pam_krb5.so might have been relying upon
> locating in the sshd process, such as libc and possibly dependencies
> of libc. I experimentally confirmed that it works for a minimal
> loadable module test case on Ubuntu Lucid, but I did have to declare
> an explicit dependency on libc and /lib64/ld-linux-x86-64.so.2, which
> seems like a bug. It might not work for a more complex set of
> dependencies than my test case uses.
>
> It does seem like people occasionally run into these sorts of shared
> library conflict problems, especially when loadable module frameworks
> are involved, and I'd like to try to find a way to reduce this pain.

> ________________________________________________
> Kerberos mailing list Kerb...@mit.edu
> https://mailman.mit.edu/mailman/listinfo/kerberos
>
>

--

Douglas E. Engert <DEEn...@anl.gov>
Argonne National Laboratory
9700 South Cass Avenue
Argonne, Illinois 60439
(630) 252-5444