Segfault in metrics

[Jacek]

Hi,

with the release of 2.0.0 I've started to gather driver metrics (BTW: thanks for great feature) but got segfault in get_count_at_index():

Program terminated with signal 11, Segmentation fault.
#0 get_count_at_index (iter=0x7f7f135fd940) at /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/third_party/hdr_histogram/hdr_histogram.cpp:187
187 /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/third_party/hdr_histogram/hdr_histogram.cpp: No such file or directory.
in /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/third_party/hdr_histogram/hdr_histogram.cpp
(gdb) where
#0 get_count_at_index (iter=0x7f7f135fd940) at /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/third_party/hdr_histogram/hdr_histogram.cpp:187
#1 move_next (iter=0x7f7f135fd940) at /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/third_party/hdr_histogram/hdr_histogram.cpp:775
#2 0x00007f804152d7e8 in _basic_iter_next (iter=<value optimized out>) at /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/third_party/hdr_histogram/hdr_histogram.cpp:801
#3 0x00007f804152d528 in hdr_mean (h=0x7f803b016010) at /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/third_party/hdr_histogram/hdr_histogram.cpp:684
#4 0x00007f80414f7660 in get_snapshot (session=<value optimized out>, metrics=0x7f7f135fda80) at /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/metrics.hpp:312
#5 cass_session_get_metrics (session=<value optimized out>, metrics=0x7f7f135fda80) at /root/rpmbuild/BUILD/cpp-driver-2.0.0/src/session.cpp:103
#6 0x0000000000409134 in stats_send_to_cassandra (epoch=1431769934) at stats.c:281
#7 0x0000000000409e31 in stats_thread (arg=0x0) at stats.c:346
#8 0x00007f8041763d7f in uv__thread_start (arg=<value optimized out>) at src/unix/thread.c:49
#9 0x00007f8040ad2851 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f8040fd890d in clone () from /lib64/libc.so.6

I've got two instances running with metrics enabled but so far it happend only once in one of the instances.

-Jacek

[Michael Penick]

Thanks for the report. I will work to reproduce the issue.

What version of libuv are you using?

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Jacek]

I'm using libuv 1.5.0. The above trace comes from driver version 2.0.0. I've updated driver today.

In the night again weird thing happened. App died but not terminated - it stopped processing data. When I send TERM it core dumped. Main code was hanging in stats uv_mutex_lock while abort happened due to call to uv_mutex_destroy on Histogram locked mutex.

I'm dumping internal and driver stats in a dedicated thread that is waiting on condition variable. Timer kicks thread with given freq. In the above case it was handing on the lock due to pending lock from previous run as it not completed hanging in Histogram. I will post stack tomorrow.

Cheers,
-jacek

[Jacek]

Here's the stack trace:
#0 0x00007f713f28c8a5 in raise () from /lib64/libc.so.6
#1 0x00007f713f28e085 in abort () from /lib64/libc.so.6
#2 0x00007f713faccc77 in uv_mutex_destroy (mutex=<value optimized out>) at src/unix/thread.c:114
#3 0x00007f713f8667c2 in ~Histogram (this=<value optimized out>, ptr=0x1dad770) at /root/rpmbuild/BUILD/cpp-driver-2.0.1/src/metrics.hpp:293
#4 ~Metrics (this=<value optimized out>, ptr=0x1dad770) at /root/rpmbuild/BUILD/cpp-driver-2.0.1/src/metrics.hpp:46
#5 cass::DefaultDeleter<cass::Metrics>::operator() (this=<value optimized out>, ptr=0x1dad770) at /root/rpmbuild/BUILD/cpp-driver-2.0.1/src/scoped_ptr.hpp:29
#6 0x00007f713f863bcc in ~ScopedPtr (this=0x1dacdc0, __in_chrg=<value optimized out>) at /root/rpmbuild/BUILD/cpp-driver-2.0.1/src/scoped_ptr.hpp:49
#7 cass::Session::~Session (this=0x1dacdc0, __in_chrg=<value optimized out>) at /root/rpmbuild/BUILD/cpp-driver-2.0.1/src/session.cpp:151
#8 0x00007f713f863da9 in cass::Session::~Session (this=0x1dacdc0, __in_chrg=<value optimized out>) at /root/rpmbuild/BUILD/cpp-driver-2.0.1/src/session.cpp:151
#9 0x00007f713f861db4 in cass_session_free (session=0x1dacdc0) at /root/rpmbuild/BUILD/cpp-driver-2.0.1/src/session.cpp:42
#10 0x0000000000404578 in do_shutdown (exit_code=-1) at main.c:630
#11 0x0000000000403239 in sig_intterm_handler (signal=2) at main.c:101
#12 <signal handler called>
#13 0x00007f713ee43054 in __lll_lock_wait () from /lib64/libpthread.so.0
#14 0x00007f713ee3e388 in _L_lock_854 () from /lib64/libpthread.so.0
#15 0x00007f713ee3e257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#16 0x00007f713faccc49 in uv_mutex_lock (mutex=<value optimized out>) at src/unix/thread.c:119
#17 0x0000000000408631 in stats_expire_timer (handle=0x1fac198) at stats.c:192
#18 0x00007f713facd386 in uv__run_timers (loop=0x7f713fcd8ae0) at src/unix/timer.c:165
#19 0x00007f713fac371f in uv_run (loop=0x7f713fcd8ae0, mode=UV_RUN_DEFAULT) at src/unix/core.c:315
#20 0x00000000004061b9 in start_server_loop () at server.c:270
#21 0x0000000000404b30 in main (argc=5, argv=0x7fffec7bfb98) at main.c:772