Cpp driver crash when schema agreement not met within 10 seconds

[Robin Mahony]

So our application crashed recently and it seems to be related to schema agreement not being met within 10 seconds. We have issues with the creation of some tables being done by multiple nodes which we are already addressing; but still seems like this shouldn't cause a core dump? We are using cpp driver version 2.14.1.

log messages:

CassDriver: 1582339196.770 [WARN] (schema_agreement_handler.cpp:108:virtual void datastax::internal::core::SchemaAgreementHandler::on_error(datastax::internal::core::WaitForHandler::WaitForError, const String&)): No schema agreement on live nodes after 10000 ms. Schema may not be up-to-date on some nodes.

Feb 22 02:39:56 DC2-SN2-013-001-005-142 ADE: |12262193 0000000000 ???? ???? 2020-02-22T02:39:56.771857| CRITICAL 0141 ????: Assertion Failed

stack trace:

2020-02-22T02:39:56.802311| ERROR    0382 ????:  2# Signal_FatalHandler(int) in /usr/local/ldr/ldr

2020-02-22T02:39:56.802314| ERROR    0382 ????:  3# 0x00007F4D630D2730 in /lib/x86_64-linux-gnu/libpthread.so.0

2020-02-22T02:39:56.802316| ERROR    0382 ????:  4# datastax::internal::core::DefaultAddressFactory::create(datastax::internal::core::Row const*, datastax::internal::SharedRefPtr<datastax::internal::core::Host> const&, datastax::internal::core::Address*) in /usr/lib/libcassandra.so.2

2020-02-22T02:39:56.802320| ERROR    0382 ????:  5# datastax::internal::core::SchemaAgreementHandler::on_set(datastax::internal::SharedRefPtr<datastax::internal::core::ChainedRequestCallback> const&) in /usr/lib/libcassandra.so.2

2020-02-22T02:39:56.802324| ERROR    0382 ????:  6# datastax::internal::core::WaitForCallback::on_chain_set() in /usr/lib/libcassandra.so.2

2020-02-22T02:39:56.802326| ERROR    0382 ????:  7# datastax::internal::core::ChainedRequestCallback::maybe_finish() in /usr/lib/libcassandra.so.2

2020-02-22T02:39:56.802329| ERROR    0382 ????:  8# datastax::internal::core::ChainedRequestCallback::maybe_finish() in /usr/lib/libcassandra.so.2

2020-02-22T02:39:56.802331| ERROR    0382 ????:  9# datastax::internal::core::Connection::on_read(char const*, unsigned long) in /usr/lib/libcassandra.so.2

2020-02-22T02:39:56.802334| ERROR    0382 ????: 10# datastax::internal::core::SslSocketHandler::on_read(datastax::internal::core::Socket*, long, uv_buf_t const*) in /usr/lib/libcassandra.so.2

Cheers,

Robin M

[Michael Penick]

Hi Robin,

Thanks for the report. I'll work on reproducing/fixing later today and keep you posted.

Mike

[Robin Mahony]

Forgot the stacktrace:

#5  <signal handler called>

#6  0x00007f4d62d8e395 in datastax::internal::core::DefaultAddressFactory::create (this=<optimized out>, peers_row=0x7f4ca14e43b8, connected_host=..., output=0x7f4ca87702e0) at /build/src/cpp-driver-2.14.1/src/address_factory.cpp:25

#7  0x00007f4d62e7b380 in datastax::internal::core::SchemaAgreementHandler::on_set (this=0x7f4d2d639100, callback=...) at /build/src/cpp-driver-2.14.1/src/connection.hpp:208

#8  0x00007f4d62ee4ae9 in datastax::internal::core::WaitForCallback::on_chain_set (this=0x7f4d2b942000) at /usr/include/c++/8/bits/atomic_base.h:512

#9  0x00007f4d62e58869 in datastax::internal::core::ChainedRequestCallback::maybe_finish (this=0x7f4d2b942000) at /build/src/cpp-driver-2.14.1/src/request_callback.cpp:310

#10 0x00007f4d62e5877e in datastax::internal::core::ChainedRequestCallback::maybe_finish (this=0x7f4d2b80ad80) at /build/src/cpp-driver-2.14.1/src/request_callback.cpp:308

#11 0x00007f4d62dbce04 in datastax::internal::core::Connection::on_read (this=0x7f4ca14bcc00, buf=<optimized out>, size=475) at /build/src/cpp-driver-2.14.1/src/connection.cpp:274

#12 0x00007f4d62e8b542 in datastax::internal::core::SslSocketHandler::on_read (this=0x7f4ca1427380, socket=0x7f4ca142a700, nread=<optimized out>, buf=<optimized out>) at /build/src/cpp-driver-2.14.1/src/socket.cpp:213

#13 0x00007f4d623e7d09 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1

#14 0x00007f4d623e8928 in ?? () from /usr/lib/x86_64-linux-gnu/libuv.so.1

#15 0x00007f4d623ed4d0 in uv.io_poll () from /usr/lib/x86_64-linux-gnu/libuv.so.1

#16 0x00007f4d623def95 in uv_run () from /usr/lib/x86_64-linux-gnu/libuv.so.1

#17 0x00007f4d62e0ca61 in datastax::internal::core::EventLoop::handle_run (this=0x7f4cab1ea688) at /build/src/cpp-driver-2.14.1/src/event_loop.hpp:57

#18 0x00007f4d630c7fa3 in start_thread (arg=<optimized out>) at pthread_create.c:486

#19 0x00007f4d61c844cf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95