crash on rc1 with ssl

[b_tho...@lycos.com]

 

I'm running rc1, with libuv 0.10.28, and ssl. We are seeing a crash. It looks to be related to ssl/cpp-driver/libuv in some way.

It has the following backtrace:

#0  0x00007f0120679625 in raise () from /lib64/libc.so.6
#1  0x00007f012067ae05 in abort () from /lib64/libc.so.6
#2  0x00007f01206b7537 in __libc_message () from /lib64/libc.so.6
#3  0x00007f01206bce66 in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f01218ddc5d in CRYPTO_free () from /usr/lib64/libcrypto.so.10
#5  0x00007f012196a0ba in EVP_PKEY_CTX_free () from /usr/lib64/libcrypto.so.10
#6  0x00007f012119e316 in ssl3_send_client_verify () from /usr/lib64/libssl.so.10
#7  0x00007f01211a246b in ssl3_connect () from /usr/lib64/libssl.so.10
#8  0x00007f01211ab677 in ssl23_connect () from /usr/lib64/libssl.so.10
#9  0x00007f011c8c540d in cass::OpenSslSession::do_handshake() () from /usr/local/test/lib/libcassandra.so.1
#10 0x00007f011c8b53ce in cass::Connection::ssl_handshake() () from /usr/local/test/lib/libcassandra.so.1
#11 0x00007f011c8b4ad5 in cass::Connection::on_read_ssl(uv_stream_s*, long, uv_buf_t) () from /usr/local/test/lib/libcassandra.so.1
#12 0x00007f011ce07b3a in uv__read (stream=0x7effdc0255d8) at src/unix/stream.c:1025
#13 0x00007f011ce07f73 in uv__stream_io (loop=0x7f01141e3bd0, w=0x7effdc025648, events=1) at src/unix/stream.c:1121
#14 0x00007f011ce1075e in uv__io_poll (loop=0x7f01141e3bd0, timeout=424) at src/unix/linux-core.c:271
#15 0x00007f011cdfca96 in uv_run (loop=0x7f01141e3bd0, mode=UV_RUN_DEFAULT) at src/unix/core.c:317
#16 0x00007f011c86ad55 in cass::LoopThread::on_run_internal(void*) () from /usr/local/test/lib/libcassandra.so.1
#17 0x00007f011ce0f039 in uv__thread_start (ctx_v=0x7f00740014c0) at src/uv-common.c:322
#18 0x00007f0121c5c9d1 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f012072f8fd in clone () from /lib64/libc.so.6

Has anyone seen anything like this?

Thanks
Brenda

 

[b_tho...@lycos.com]

 

There was a  second backtrace that I forgot to add to the original email. Any help would be appreciated.

Thanks Brenda

#0  0x00007f7a19c62a0a in X509_NAME_cmp () from /usr/lib64/libcrypto.so.10
#1  0x00007f7a19bb4e99 in OBJ_bsearch_ex_ () from /usr/lib64/libcrypto.so.10
#2  0x00007f7a19c2b25c in ?? () from /usr/lib64/libcrypto.so.10
#3  0x00007f7a19c690c8 in ?? () from /usr/lib64/libcrypto.so.10
#4  0x00007f7a19c69189 in X509_OBJECT_retrieve_by_subject () from /usr/lib64/libcrypto.so.10
#5  0x00007f7a19c6925b in X509_STORE_get_by_subject () from /usr/lib64/libcrypto.so.10
#6  0x00007f7a19c693ee in X509_STORE_CTX_get1_issuer () from /usr/lib64/libcrypto.so.10
#7  0x00007f7a19c664d7 in X509_verify_cert () from /usr/lib64/libcrypto.so.10
#8  0x00007f7a1947cc43 in ssl3_output_cert_chain () from /usr/lib64/libssl.so.10
#9  0x00007f7a19471d71 in ssl3_send_client_certificate () from /usr/lib64/libssl.so.10
#10 0x00007f7a19476273 in ssl3_connect () from /usr/lib64/libssl.so.10
#11 0x00007f7a1947f677 in ssl23_connect () from /usr/lib64/libssl.so.10
#12 0x00007f7a14b9940d in cass::OpenSslSession::do_handshake() () from /usr/local/test/lib/libcassandra.so.1
#13 0x00007f7a14b893ce in cass::Connection::ssl_handshake() () from /usr/local/test/lib/libcassandra.so.1
#14 0x00007f7a14b88ad5 in cass::Connection::on_read_ssl(uv_stream_s*, long, uv_buf_t) () from /usr/local/test/lib/libcassandra.so.1
#15 0x00007f7a150dbb3a in uv__read (stream=0x7f78c003d5e8) at src/unix/stream.c:1025
#16 0x00007f7a150dbf73 in uv__stream_io (loop=0x7f7a0c21bd50, w=0x7f78c003d658, events=1) at src/unix/stream.c:1121
#17 0x00007f7a150e475e in uv__io_poll (loop=0x7f7a0c21bd50, timeout=1962) at src/unix/linux-core.c:271
#18 0x00007f7a150d0a96 in uv_run (loop=0x7f7a0c21bd50, mode=UV_RUN_DEFAULT) at src/unix/core.c:317
#19 0x00007f7a14b3ed55 in cass::LoopThread::on_run_internal(void*) () from /usr/local/test/lib/libcassandra.so.1
#20 0x00007f7a150e3039 in uv__thread_start (ctx_v=0x7f7950003500) at src/uv-common.c:322
#21 0x00007f7a19f309d1 in start_thread () from /lib64/libpthread.so.0
#22 0x00007f7a18a038fd in clone () from /lib64/libc.so.6

To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Michael Penick]

Hi Brenda,

Is this happening immediately when connecting a new session. What OS and version of OpenSSL are you running on? 

Are you able to share example code to reproduce the issue (or maybe the sequence of driver calls)?

Mike

[b_tho...@lycos.com]

hi Mike

Centos 6.5
OpenSSL 1.0.1e-fips 11 Feb 2013

It's unclear when it is happening. It occurs at random times. It's been seen on two different clusters. Multiple times on one of the clusters.

An effort is underway to consistently reproduce the issue. If we can gather more information I'll let you know. Any thoughts or suggestions you might have would be appreciated.

Thanks
Brenda