SSL_CTX_new race condition
Andrew Mann
come through the list yet and it was sent Friday, so I presume it's been
gobbled up somewhere :)
I'm working on an application that needs to work with a lot of
simultaneous SSL connections to the same remote host. It spawns a
thread for each connection and handles each connection entirely in that
thread. These connections each last anywhere from half a second to a
minute, and as soon as the connection is finished, it's closed and the
thread is terminated (and often another thread started to process
another request).
The OpenSSL library is initialized prior to any connection handling
thread starting. Apparently thread safety is a FAQ, but the answer on
the www.openssl.org FAQ and related links falls far short of helpful and
borders on just plain inaccurate.
Initially I tried using a single SSL_CTX for all connections, so that
while no SSL objects were being shared across threads, one SSL_CTX
object was being used to all calls to SSL_new(). This failed miserably.
SSL connections failed randomly and frequently during the negotation
phase. The SSL_CTX ended up getting freed on some test runs during the
middle of the run. Although there are suggestions that a single SSL_CTX
is meant to be shared between multiple connections, there isn't any
specific mention of wether that is supposed to work with connections
handled by different threads, so I suppose it's not (well it appears not
to at least).
I moved to making a new SSL_CTX for each thread as well. This had
somewhat better luck, but randomly during startup one of the calls to
SSL_CTX_new() would fail with "error:140A90A1:SSL
routines:SSL_CTX_new:library has no ciphers". After a bit of digging it
appears that SSL_CTX_new() is not really threadsafe either. It calls
ssl_create_cipher_list() which checks the status of an unmutexed global
variable init_ciphers (ssl/ssl_ciph.c). If the value is its
initialization value of 1, then load_ciphers() is called.
load_ciphers() sets init_ciphers to 0 and then procedes to load various
ciphers for the library to use.
Unfortunately, since this value isn't protected by a mutex it's not
unlikely that in the event of two threads "racing" through SSL_CTX_new()
that the lead thread will clear init_ciphers and then be interrupted
before it can actually perform any loading.
For the time being I've worked around this by creating a SSL_CTX prior
to starting any threads that has no point other than to prompt cipher
loading before any potential race conditions.
Fixes:
I'm not sure about the SSL_CTX shared between threads, but it would be
nice if the settings (CA files particularly) didn't have to be loaded
from disk for each thread spawned in this scenario.
Protect init_ciphers with a mutex. Or at least move init_ciphers=0 to
the end of the load_ciphers() function so that the worst case condition
becomes multiple calls to load_ciphers() instead of a tiny chance of
multiple calls and a larger chance of a"library has no ciphers" error.
Andrew
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openss...@openssl.org
Automated List Manager majo...@openssl.org
Emilio Arce
1. I'm using 0.9.6 so I don't know if what I write here applies to your
OpenSSL version. Initializing SSL_CTX once before spawning threads should
work. I have SSL_CTX be initialized in a static class so I'm sure SSL_CTX is
ready first and goes away when the process is shutdown (any lingering
threads will fail of course in this case). (I'm passing
SSLv23_client_method() to SSL_CTX_new call).
2. Also, if you are running on Solaris you will see that the default hash
id created for thread slots is not a very good one: it uses the pid which
will probably be the same for all threads :)
In your version of OpenSSl check if in cryptlib.c the CRYPTO_thread_id
function returns getpid() when in SOLARIS. You will notice that
id_callback will be NULL if you don't set an id_callback via
CRYPTO_set_id_callback.
If this is the case, then this is probably your problem.
Make sure that you provide a "sane" implementation to generate IDs by
calling CRYPT_set_id_callback function when you are initializing the SSL
context (you need to this only once per process obviously).
You can use pthreads_thread_id(void ); or solaris_thread_id(void ); (in
crypto/threads/th-locks.c ) functions as the callback functions to generate
the hash for thread slot ids.
For example:
// Set function to create hash for slot ID
CRYPTO_set_id_callback(solaris_thread_id);
I hope this helps
Emilio
Andrew Mann
Thanks for the suggestions. I'm sorry my initial message wasn't more
specific. I was trying to fit both a question and apparent bug into one
message - I should have sent two instead :)
I believe I have things working ok now. I don't seem to have any
memory corruption, crashing or connection failure issues at least :) I
may have a small memory leak, but that's not a concern at the moment.
I'm building/testing on two systems:
System #1
Dual Intel Xeon 2.4 ghz
RedHat Linux 8.0
openssl-0.9.6b-33 (I believe -33 is the 33rd RedHat package)
System #2
Single Intel Pentium 3 550 mhz
RedHat Linux 9
openssl-0.9.7a-5
And I'm using the source of openssl-0.9.7b from the downloads section
of www.openssl.org as a reference and alternate test library on System
#2 - though I'd like to be able to use a mainstream release through
RedHat so that updates don't require a manual rebuild.
I have the id callback set (it simply wraps pthread_self() ), as well
as the locking function. I don't handle dynamic locks at all mainly
because I can't find any documentation on them. In light of the comment
in the CHANGES file (Changes between 0.9.6h and 0.9.7 [31 Dec 2002]),
which sounds like dynamic locks will be required for at least some
functionality when 0.9.8 is released, I would have liked to have found
some documentation on how these should be implemented.
It sounds like you have a single SSL_CTX shared among all threads?
When I use that method several connections work fine, and I can in fact
run OK with about 3-4 threads at once. When I get up above that I start
getting TLS negotiation errors on the connections. If I use a separate
SSL_CTX for each thread (and each connection since the model here is 1
thread = 1 connection) then I've tested up to 140 simultaneous
connections without a problem (I haven't tested higher but I see no
reason it would fail with more barring limits on file descriptors, etc).
My failing setup is:
library_init()
{
...
SSL_load_error_strings();
SSL_library_init();
CRYPTO_set_id_callback(id_function);
CRYPTO_set_locking_callback(locking_function);
global.ssl_context=SSL_CTX_new(SSLv23_client_method());
SSL_CTX_load_verify_locations(...);
SSL_CTX_set_default_verify_paths(...);
SSL_CTX_set_verify(...);
SSL_CTX_set_verify_depth(...);
SSL_CTX_use_certificate_file(...);
SSL_CTX_use_PrivateKey_file(...);
SSL_CTX_check_private_key(...);
SSL_CTX_set_mode(...,SSL_MODE_AUTO_RETRY);
}
Followed by:
per_thread_processing()
{
... Create TCP connection to server
ssl_connection=SSL_new(global.ssl_context);
ERR_clear_error();
SSL_set_fd(...);
ERR_clear_error();
rval=SSL_connect(ssl_connection);
if (rval<=0)
{
... debugging gobbledygook
}
}
Everything is error checked, though stripped out above. I'm not sure
if the set_*_callback() functions should be called before or after the
SSL_library_init(), but I call them after. Everything in library_init()
is completed before any additional threads are kicked off.
When I share the SSL_CTX between threads, the SSL_connect() call
returns 0 on most threads (occassionally one or two succeed).
SSL_get_error(...) returns 1. e=ERR_get_error();
ERR_error_string_n(e,errorbuf,2048); fills errorbuf with
"error:1409441B:SSL routines:SSL3_READ_BYTES:tlsv1 alert decrypt error".
I'm also using a test application that's really hammering the server
and performs multiple serialized connections per thread. Each thread
connects, sends a request, waits for a response, and then closes the
connection and does it all over again immediately (with maybe a couple
ms delay for processing the returned data) - repeating for a specified
number of requests (usually 1000 or so).
Currently I have things working, but instead of calling SSL_CTX_new()
in the library_init() and using the returned SSL_CTX * in each thread, I
call SSL_CTX_new() in library_init() only to force cipher loading, and
then each thread has an init function that calls SSL_CTX_new() to create
its own SSL_CTX which it reuses for all future serial SSL_new() calls.
Hopefully I'm doing something wrong here that's causing the shared
SSL_CTX to act up. I figure there's a good bit of needless work being
done re-parsing the cafile (and the cert and key files, though I could
pre-load those if needed) for each thread.
Andrew
Lutz Jaenicke
> Emilio,
> Thanks for the suggestions. I'm sorry my initial message wasn't
> more specific. I was trying to fit both a question and apparent bug into
> one message - I should have sent two instead :)
I have opened a ticket in the request tracker on your behalf. Please send
any updates with respect to the bug to r...@openssl.org with the
[openssl.org #???] tag in the header line, such that the information
is recorded.
Best regards,
Lutz
--
Lutz Jaenicke Lutz.J...@aet.TU-Cottbus.DE
http://www.aet.TU-Cottbus.DE/personen/jaenicke/
BTU Cottbus, Allgemeine Elektrotechnik
Universitaetsplatz 3-4, D-03044 Cottbus