dlopen() slowdown

[da...@trellisinc.com]

after loading a .so using dlopen(n,RTLD_NOW|RTLD_GLOBAL), i spawn a pthread
using one of the syms from this .so as the thread's function argument (is
there a proper term for this argument? the thread's main?). to be more
precise, the result of a successful dlsym() is used as the third parameter
to a pthread_create().

this works wonderfully. however, i've noted that if i unload the module
(cancel and join on the thread, then dlclose() the original reference)
and then load it again, the thread created runs much more (50-60%) slowly
than the first, ignoring black smoke around my app (ie, i've tried to keep
the system state as similar as possible while testing this behavior, but
i'm aware os scheduling issues, caching, etc may well be introducing a
widely different environment).

if a different .so is loaded the second time, no slowdown is noticed. this
remains true no matter how many different .so's are dlopen()ed and -close()d:
only a close and subsequent open of the same .so gives rise to this very
repeatable behavior.

i'm on linux 2.4.6, using glibc 2.2.3.

--
nicholas black (da...@trellisinc.com) http://trellisinc.com
"correct terminology...they are 'member selection operators'. if you call them
'dot' and 'arrow' at a c++ cocktail party, people will turn and stare. a glass
will drop to the floor in a remote corner." -alger, secrets of the c++ masters

[Nicholas Dronen]

da...@trellisinc.com wrote:

> after loading a .so using dlopen(n,RTLD_NOW|RTLD_GLOBAL), i spawn a pthread
> using one of the syms from this .so as the thread's function argument (is
> there a proper term for this argument? the thread's main?). to be more
> precise, the result of a successful dlsym() is used as the third parameter
> to a pthread_create().

> this works wonderfully. however, i've noted that if i unload the module
> (cancel and join on the thread, then dlclose() the original reference)
> and then load it again, the thread created runs much more (50-60%) slowly
> than the first, ignoring black smoke around my app (ie, i've tried to keep
> the system state as similar as possible while testing this behavior, but
> i'm aware os scheduling issues, caching, etc may well be introducing a
> widely different environment).

> if a different .so is loaded the second time, no slowdown is noticed. this
> remains true no matter how many different .so's are dlopen()ed and -close()d:
> only a close and subsequent open of the same .so gives rise to this very
> repeatable behavior.

> i'm on linux 2.4.6, using glibc 2.2.3.

This is just a guess: it sounds like dlopen() causes the symbols
to be remapped in the address space of the process, and while this
affects the function symbol you acquire with dlsym(), its addresses
have to be remapped as they are accessed -- not while dlopen()
executes -- resulting in slower execution for that function and
that function only.

Again, just a guess.

Regards,

Nicholas

[da...@trellisinc.com]

Nicholas Dronen <ndr...@io.frii.com> wrote:
> This is just a guess: it sounds like dlopen() causes the symbols
> to be remapped in the address space of the process, and while this
> affects the function symbol you acquire with dlsym(), its addresses
> have to be remapped as they are accessed -- not while dlopen()
> executes -- resulting in slower execution for that function and
> that function only.
> Again, just a guess.

i see two problems with this guess (thanks for the idea, though): 0) the
code in the .so is pulled in by dlopen(), and only references from within
the .so to the rest of the app, it seems, need to be mapped, and that
number is miniscule (references into the .so, of course, are mapped
at dlsym() time, but intra-.so addresses are at dlopen() time). 1) even
if some addresses must be bound outside of dl*()space, it would seem to
me that this binding need only be done during one iteration.