singletons, double-free errors, RTLD_GLOBAL, pycryptopp

[Zooko Wilcox-O'Hearn]

Folks:

Please see this comment in which I applied a certain patch from
Crypto++ to the copy of Crypto++ which is embedded into pycryptopp,
but much to my disappointment this patch did not make certain
automated tests under valgrind go from red to green:

https://tahoe-lafs.org/trac/pycryptopp/ticket/44#comment:14

Here is the text of that comment:

Well, this is pretty disappointing. I applied
http://bazaar.launchpad.net/~zooko/cryptopp/trunk/revision/466 as
https://github.com/zooko/pycryptopp/commit/95c78ce5f08b03be70e5d69e5eb09d7acc1ce420,
and it didn't change anything in terms of test results! In particular,
the two systems that had valgrind warnings with the old version have
the exact same warnings with the new version:
https://tahoe-lafs.org/buildbot-pycryptopp/builders/Ruben%20Fedora/builds/27
and https://tahoe-lafs.org/buildbot-pycryptopp/builders/Ruben%20Fedora/builds/27.

I had thought that those valgrind warnings were due to the problem of
handling global symbols and singletons. In fact, now that I think
about it, what exactly did I think was happening here? I guess on
Linux the module loading flags default to RTLD_LOCAL instead of
RTLD_GLOBAL, which I think should be causing a failure of cross-module
typeinfo comparison (e.g. a failure to catch an exception of a certain
type, raised in a different C++ module), but there isn't such a
failure demonstrated by the tests.

In contrast if the flag was set to RTLD_GLOBAL then I would have
expected exception-catching to work, but multiple libraries which
dynamically link to the Crypto++ dynamic link library would each think
they were solely responsible for managing the global symbols in that
library, including constructing and destructing singletons, which
should lead to some problems, most importantly a double-free problem.
I don't see that either, on the buildbots, although possibly that's
because none of the systems is using RTLD_GLOBAL now.

Okay, now what? The pressing questions are:

* what these valgrind warnings mean (pasted into this comment below),
* why Wei Dai wrote that patch,
* why we no longer get a problem where exceptions raised from one
module can't be caught (by their type) in another module,
* if the current version of pycryptopp triggers a double-free error
(or another critical problem) when it and another library (e.g.
another Python module) are loaded into the same process and linking to
the same Crypto++ DLL

Help!

==30887== Conditional jump or move depends on uninitialised value(s)
==30887== at 0x4017A16: index (in /lib64/ld-2.15.so)
==30887== by 0x4007927: expand_dynamic_string_token (in /lib64/ld-2.15.so)
==30887== by 0x40081BC: _dl_map_object (in /lib64/ld-2.15.so)
==30887== by 0x400171D: map_doit (in /lib64/ld-2.15.so)
==30887== by 0x400EB65: _dl_catch_error (in /lib64/ld-2.15.so)
==30887== by 0x4000EE3: do_preload (in /lib64/ld-2.15.so)
==30887== by 0x4003EB2: dl_main (in /lib64/ld-2.15.so)
==30887== by 0x4014E4A: _dl_sysdep_start (in /lib64/ld-2.15.so)
==30887== by 0x4004E71: _dl_start (in /lib64/ld-2.15.so)
==30887== by 0x4001537: ??? (in /lib64/ld-2.15.so)
==30887== by 0x1: ???
==30887== by 0x7FF000E96: ???

Regards,

Zooko

[Jeffrey Walton]

Hi Zooko,

On Jan 25, 8:24 am, "Zooko Wilcox-O'Hearn" <zo...@zooko.com> wrote:
> <SNIP>

>
> Help!
>
> ==30887== Conditional jump or move depends on uninitialised value(s)
> ==30887==    at 0x4017A16: index (in /lib64/ld-2.15.so)
> ==30887==    by 0x4007927: expand_dynamic_string_token (in /lib64/ld-2.15.so)
> ==30887==    by 0x40081BC: _dl_map_object (in /lib64/ld-2.15.so)
> ==30887==    by 0x400171D: map_doit (in /lib64/ld-2.15.so)
> ==30887==    by 0x400EB65: _dl_catch_error (in /lib64/ld-2.15.so)
> ==30887==    by 0x4000EE3: do_preload (in /lib64/ld-2.15.so)
> ==30887==    by 0x4003EB2: dl_main (in /lib64/ld-2.15.so)
> ==30887==    by 0x4014E4A: _dl_sysdep_start (in /lib64/ld-2.15.so)
> ==30887==    by 0x4004E71: _dl_start (in /lib64/ld-2.15.so)
> ==30887==    by 0x4001537: ??? (in /lib64/ld-2.15.so)
> ==30887==    by 0x1: ???
> ==30887==    by 0x7FF000E96: ???

This does not appear to be related to Crypto++.

Its not a dry, blow you off kind of answer - I've worked with Vagrind/
Helgrind for some time, including suppression files to squelch benign
messages (for example, related to RNGs and entropy incorporation).

Perhaps you could ask on the Valgrind User List (http://valgrind.org/
support/). There is lots of expert Valgrind knowledge there (they've
helped me on more than one occasion).

Jeff

[Jeffrey Walton]

Forgot to mention... install symbols for libc and libstdc++, and run
Valgrind with --track-origins=yes.

Jeff