Interaction of LD_PRELOAD and RTLD_GROUP w/ dlopen (interposer)

[Tony]

Greetings,

I'm having a discussion with an OS vendor about the behavior of dlopen
when only the RTLD_GROUP mode is specified, and where LD_PRELOAD has
been set. In this case, an application to which I have no source is
using dlopen on a shared object and only setting the RTLD_GROUP mode.
My interposer wants to catch all calls to malloc, but the vendor
linker/loader sees the RTLD_GROUP mode, and doesn't search the
libraries specified via LD_PRELOAD. As a result, my interposer
doesn't well... interpose for calls to malloc in that shared object.

Can someone tell me if this is expected behavior (on an ELF system),
and if not point me to some documentation that would help me make my
case?

Thanks,
Tony

[Lowell, Randy]

RTLD_GROUP and LD_PRELOAD are vendor extensions. What you've
described below sounds like the behavior we would expect to
see on HP/UX. LD_PRELOAD dependencies are inserted in the
search order between that executable and its actual dependencies.
They are not searched to resolve relocations in libraries
dlopen'd with the RTLD_GROUP flag.

Randy

[Rod Evans]

On Solaris I'd expect the LD_PRELOAD to interpose on malloc.

There are a number of binding techniques, like RTLD_GROUP and
direct bindings, that change the symbol search model from the
default (ie. starting at the executable and inspecting each
dependency in the order it was loaded). However, when these
techniques are used, we still want to preserve the
traditional/expected behavior for things like LD_PRELOAD.

We've gone through a number of steps to distinguish between
"explicit" interposition, and "implicit" interposition. The
former is achieved through LD_PRELOAD or tagging individual
objects as interposers. Here our belief was that the user
wanted their interposer to get control no matter what type of
binding was called for. The latter is a result of objects
"accidentally" defining symbols that exist in other dependencies,
and interposition occurring as a result of the default symbol
search. These, in our experience, are the name-space pollution
problems that crop up from time to times and cause application
failure.

Techniques like RTLD_GROUP and direct binding protect against
implicit interposition. However, explicit interposition should
always get control.

Now, if you're running an older version of Solaris, you might find
some holes in the above explanation ... there's been a few twists in
the road as we nailed the corner cases to implement our intent :-)

--
Rod

[Dave Prosser]

Strictly speaking, there's nothing specified here, assuming
that by "ELF systems" you mean as described in the generic
ABI or something common to all processor-specific ELF
implementations.

The only dlopen() flags that fall into that category are
RTLD_LAZY, RLTD_NOW, RTLD_GLOBAL, and RTLD_LOCAL. The
LD_PRELOAD feature isn't there, either.

Until these are part of the gABI, there's likely to be
mild variations in behavior of these extensions on different
OS platforms.

--
Dave Prosser d...@sco.com (908)790-2358 The SCO Group, Murray Hill, NJ
SCO is a leading provider of UNIX-based solutions and mobile services.

[cburgess]

Hmmm, that's interesting. It's very hard to determine by the Linkers
and Libraries guide that this is actually the intent. :-)

I did try it out on an OpenSolaris distribution, however I guess I
messed up something in my testcase (maybe some cc options
wrong, I'm used to gcc).

So in the case of LD_PRELOAD, you put the list of preloaded objects
before the dso's, but after the excutable. What about a
DT_SYMBOLIC restriction - is the order dso excutable preload_objects
dso_dependencies?

Colin