errors with multiple loading cryptopp as shared lib on Linux

[David]

We experience errors when trying to use cryptopp as shared lib from
more than one other shared libs. the scenarios are as follows (all
libraries are shared libs):

scenario 1: working
library X --uses--> cryptopp
libraries A, B, C --use--> X
Apache --loads--> A, B, C

this works fine.

scenario 2: not working
library X --uses--> cryptopp
libraries A, B --use--> X
library C --uses--> cryptopp (directly!)
also C --uses--> X
Apache --loads--> A, B, C

We get an error (crash) on something with name-value pairs.
My hunch is that, even though the OS should load a shared lib only
once, some initialization occurs twice and fails on some static or
global update.

Is there some implicit initialization of cryptopp that shouls happen
only once? are there some statics/globals used?
Has anybody experienced this? I'd be grateful for any tips on this
problem.

my environment:

cryptopp: 5.5.2 (installed it with package manager, which brought only
the shared lib version)
OS: Linux bagvapp 2.6.18-164.2.1.el5.plus #1 SMP Fri Oct 9 12:34:43
EDT 2009 i686 i686 i386 GNU/Linux
(centOS package under VM)
GCC: gcc -v output:
Using built-in specs.
Target: i386-redhat-linux
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --
infodir=/usr/share/info --enable-shared --enable-threads=posix --
enable-checking=release --with-system-zlib --enable-__cxa_atexit --
disable-libunwind-exceptions --enable-libgcj-multifile --enable-
languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --
disable-dssi --enable-plugin --with-java-home=/usr/lib/jvm/java-1.4.2-
gcj-1.4.2.0/jre --with-cpu=generic --host=i386-redhat-linux
Thread model: posix
gcc version 4.1.2 20080704 (Red Hat 4.1.2-46)

thanks a lot!
david

[Zooko O'Whielacronx]

[cross-posted to the Crypto++ users mailing list and the Python C++-sig.]

David wrote on the Crypto++ users mailing list:

> We experience errors when trying to use cryptopp as shared lib from more than one other shared libs.

I struggled with this for a long time for the pycryptopp project and
finally gave up and just refactored pycryptopp so that there is only
one sharedlib that links to libcryptopp.so.

It kind of sucks because this means that if some *other* library gets
loaded into the same process as pycryptopp and that library links to
libcryptopp.so then it will crash the process. But, I couldn't figure
out any other way to solve it. Here is a letter I wrote at the time
outlining four possible solutions. I tried them all and ended up
using the one listed as "#2":

http://mail.python.org/pipermail/cplusplus-sig/2009-May/014531.html

I still kind of hope that some Crypto++ or Python hackers will figure
out how to implement the one listed as "#4", which is to mark the
appropriate symbols in the Crypto++ API as being satisfied by a
different DSO (mark them as 'U' -- undefined symbols) so that the
loader will do the right thing. See the end of that letter referenced
above for details. It kind of seems like gcc cannot be configured to
do the right thing here, although MSVC can.

Here is an earlier letter I wrote explaining how there are two
different problems, one that happens if you set RTLD_GLOBAL and one
that happens if you don't set RTLD_GLOBAL:

http://groups.google.com/group/cryptopp-users/msg/7a0dacd0a4be5e2d

Here is the ticket that we used to track this issue in pycryptopp
until it was solved by the "there can be only one user of
libcryptopp.so" solution:

http://allmydata.org/trac/pycryptopp/ticket/9

Regards,

Zooko

[Zooko O'Whielacronx]

[cross-posted to the Crypto++ users mailing list and the Python C++-sig.]

David wrote on the Crypto++ users mailing list:

> We experience errors when trying to use cryptopp as shared lib from more than one other shared libs.

[Jeffrey Walton]

Hi David,

> scenario 2: not working
> library X --uses--> cryptopp
> libraries A, B --use--> X
> library C --uses--> cryptopp (directly!)
> also C --uses--> X
> Apache --loads--> A, B, C

I was not able to duplicate the issue. My setup had a traditional EXE
using cryptopp.so directly, with 3 SOs (loaded by the EXE) that each
used cryptopp.so. I guess mine looked more like:
EXE -> cryptopp.so
EXE-> SO1
EXE-> SO2
EXE-> SO3
SO1 -> cryptopp.so
SO2 -> cryptopp.so
SO3 -> cryptopp.so

The EXE linked directly against cryptopp.so (-lcryptopp), but it
dynamically loaded the three test SOs (dlopen(), dlsym(), dlclose()).
The three SOs linked directly against cryptopp.so (-lcryptopp). The
EXE had 16 threads which round-robined one of the DSOs. The EXE and
SOs used the X917 generator, AES, and SHA. In each case, I was able to
verify that cryptopp.so was mapped in once, and the EXE and DSO all
claimed the same load address for Crypto++.

> We get an error (crash) on something with
> name-value pairs.

Two questions: (1) was the debug info stripped? (2) if not stripped,
what was the call stack?

> My hunch is that, even though the OS should load
> a shared lib only once, some initialization occurs twice
> and fails on some static or global update. Is there

> some implicit initialization of cryptopp that should
> happen only once?
Maybe Wei can correct me here, but most likely not. Crypto++ objects
were written to be re-entrant. But a call stack would be needed so the
object in question (AES, SHA1, etc) can be examined more closely.

I took the default library setup from Fedora for the tests. The test
setup above may (or may not be) different than what Zooko describes in
his notes, and what you're using for CentOS. For example, my
RTLD_GLOBAL setting is what ever the package maintainer made it.

Jeff

> [SNIP]