disconnection when waiting for futures

[Jean Giard]

Hi,
I have a strange driver behavior that happens in a test where I do a bunch of queries and then wait them all at once. At some point in the waiting loop I get the error "Session: No hosts available" from cass_future_error_code.

I tried to run the async example provided in the sources and it seems unstable as well (it works sometimes)

I ran the test in debug log level and get the following

1427108337.589 [DEBUG] (src/session.cpp:242:void cass::Session::connect_async(const cass::Config&, const string&, cass::Future*)): Issued connect event
1427108337.589 [INFO] (src/session.cpp:328:virtual void cass::Session::on_run()): Creating 1 IO worker threads
1427108337.590 [DEBUG] (src/connection.cpp:387:static void cass::Connection::on_connect(cass::Connector*)): Connected to host 127.0.0.1
1427108337.592 [DEBUG] (src/control_connection.cpp:194:void cass::ControlConnection::on_connection_ready(cass::Connection*)): Connection ready on host 127.0.0.1
1427108337.600 [DEBUG] (src/schema_metadata.cpp:305:void cass::SchemaMetadata::add_json_map_field(int, const cass::Row*, const string&)): Expected JSON object for column 'index_options' (probably null or empty)
[ repeats ]
empty)
1427108337.604 [DEBUG] (src/schema_metadata.cpp:305:void cass::SchemaMetadata::add_json_map_field(int, const cass::Row*, const string&)): Expected JSON object for column 'index_options' (probably null or empty)
1427108337.604 [INFO] (src/dc_aware_policy.cpp:32:virtual void cass::DCAwarePolicy::init(const cass::SharedRefPtr<cass::Host>&, const HostMap&)): Using 'datacenter1' for the local data center (if this is incorrect, please provide the correct data center)
1427108337.605 [INFO] (src/io_worker.cpp:130:void cass::IOWorker::add_pool(const cass::Address&, bool)): Adding pool for host 127.0.0.1:9042 io_worker(0x18da4a0)
1427108337.605 [DEBUG] (src/pool.cpp:69:void cass::Pool::connect()): Connect 127.0.0.1 pool(0x18ebda0)
1427108337.605 [INFO] (src/pool.cpp:236:void cass::Pool::spawn_connection()): Spawning new connection to host 127.0.0.1:9042
1427108337.605 [DEBUG] (src/connection.cpp:387:static void cass::Connection::on_connect(cass::Connector*)): Connected to host 127.0.0.1
1427108337.606 [DEBUG] (src/session.cpp:373:virtual void cass::Session::on_event(const cass::SessionEvent&)): Session is connected
1427108337.606 [DEBUG] (src/session.cpp:376:virtual void cass::Session::on_event(const cass::SessionEvent&)): Session pending pool count 0
Error: Cannot add existing keyspace "examples"
Error: Cannot add already existing column family "async" to keyspace "examples"
1427108337.608 [INFO] (src/pool.cpp:236:void cass::Pool::spawn_connection()): Spawning new connection to host 127.0.0.1:9042
1427108337.608 [DEBUG] (src/connection.cpp:387:static void cass::Connection::on_connect(cass::Connector*)): Connected to host 127.0.0.1
1427108337.608 [DEBUG] (src/pool.cpp:146:void cass::Pool::add_pending_request(cass::RequestHandler*)): 11 requests pending on 127.0.0.1 pool(0x18ebda0)
1427108337.608 [DEBUG] (src/pool.cpp:146:void cass::Pool::add_pending_request(cass::RequestHandler*)): 21 requests pending on 127.0.0.1 pool(0x18ebda0)
1427108337.608 [DEBUG] (src/pool.cpp:146:void
[ repeats with increments of 10 ]
1427108337.609 [DEBUG] (src/pool.cpp:146:void cass::Pool::add_pending_request(cass::RequestHandler*)): 241 requests pending on 127.0.0.1 pool(0x18ebda0)
1427108337.609 [DEBUG] (src/pool.cpp:146:void cass::Pool::add_pending_request(cass::RequestHandler*)): 251 requests pending on 127.0.0.1 pool(0x18ebda0)
waiting
Error: Session: No hosts available
Error: Session: No hosts available
[ repeats a variable number of times ]
Error: Session: No hosts available
Error: Session: No hosts available
1427108337.633 [DEBUG] (src/session.cpp:290:void cass::Session::internal_close()): Issued close
1427108337.633 [DEBUG] (src/pool.cpp:81:void cass::Pool::close(bool)): Closing pool(0x18ebda0)
1427108337.633 [DEBUG] (src/connection.cpp:422:static void cass::Connection::on_close(uv_handle_t*)): Connection to host 127.0.0.1 closed
1427108337.633 [DEBUG] (src/connection.cpp:422:static void cass::Connection::on_close(uv_handle_t*)): Connection to host 127.0.0.1 closed
1427108337.633 [INFO] (src/io_worker.cpp:195:void cass::IOWorker::notify_pool_closed(cass::Pool*)): Pool for host 127.0.0.1 closed: pool(0x18ebda0) io_worker(0x18da4a0)
1427108337.633 [DEBUG] (src/pool.cpp:57:cass::Pool::~Pool()): Pool dtor with 0 pending requests pool(0x18ebda0)
1427108337.633 [DEBUG] (src/io_worker.cpp:253:void cass::IOWorker::close_handles()): Active handles following close: 0
1427108337.633 [DEBUG] (src/session.cpp:382:virtual void cass::Session::on_event(const cass::SessionEvent&)): Session is disconnected
1427108337.634 [DEBUG] (src/connection.cpp:422:static void cass::Connection::on_close(uv_handle_t*)): Connection to host 127.0.0.1 closed

If I change to test to wait futures immediately everything works fine.
I'm using libuv1.4 with cassandra driver built from master branch.
Do you have any idea what might be causing this ?

note : It might be related to CPP-218

the json error at connection is likely unrelated to the issue and happens since cassandra upgrade from 2.0.11 to 2.1.3.

Thanks.

[Michael Penick]

Hi Jean,

That can be fixed by increasing some of the driver's settings. Your application has exceeded the default capacity of the driver to handle requests. There are a few setting you can increase to fix that. You may need to try a couple different configurations to get optimal performance. These are just example settings:

// This is will increase the number of threads handling IO to 2 (This alone might be enough to fix the issue).

cass_cluster_set_num_threads_io(cluster, 2); 

// Or you can also increase the number of connections per thread

cass_cluster_set_core_connections_per_host(cluster, 2);
cass_cluster_set_max_connections_per_host(cluster, 4);

// Or increase the number of request that will queue waiting for a thread.
cass_cluster_set_pending_requests_low_water_mark(cluster, 5000);
cass_cluster_set_pending_requests_high_water_mark(cluster, 10000);

I'm working on better ways for communicating how these settings affect performance in future documentation and possible API simplefications. I'm also going to add log warnings (as part of https://datastax-oss.atlassian.net/browse/CPP-188) when some of these thresholds are exceeded so that it's easier to see to see when these setting need to be tuned.

Hope this helps.

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Jean Giard]

Tweaking these settings indeed seems to solve my problem, mostly the pending requests one. I will experiment a bit with them.

Thanks !