Strange behaviour of DCAwarePolicy and offline local DC...

[Oliver Seiler]

Hi all,

I'm doing some debugging of DCAwarePolicy and when it (via DCQueryPlan) decides that there are no more hosts available.

More specifically, I am doing a test with a 2 DC cluster, 4 Cassandra nodes per DC, with all Cassandra nodes in the local DC down. We're using version 2.8.0. I have disabled token and latency awareness so the only thing involved should be DCAwarePolicy, which I have configured with used_hosts_per_remote_dc of 2 and allow_remote_dcs_for_local_cl of cass_false. We do a lot CL=1 and CL=2 queries and I'm looking at how cpp-driver acts when we do these queries in this sort of situation (keyspaces are all NetworkTopologyStrategy with 3 replicas per DC).

Strangely (to me), this is very sensitive to used_hosts_per_remote_dc: if I set it to 1, I get 100% failure rate *from cpp-driver* indicating no hosts are available (specifically I get the CASS_ERROR_LIB_NO_HOSTS_AVAILABLE from io_worker.cpp, IOWorker::retry); if I set it to 2 I get a ~60% failure rate; if I set it to 3 I get a 0% failure rate.

I don't really understand what's going on here, and I can't really get enough out of the driver via logging to tell. The only explanation I have for the error behaviour I see is that the driver doesn't think it can use any of the hosts that it knows about in the remote DC and that this becomes less likely as I increase used_hosts_per_remote_dc; again all nodes in the remote DC are up.

I have gone in with gdb to poke around at the internal state of DCAwarePolicy and DCQueryPlan instances. DCAwarePolicy::per_remote_dc_live_hosts_ doesn't have all the hosts in the remote DC, but I've seen it exceed used_hosts_per_remote_dc as well; this seems to be an artifact of how Session interacts with the policy and how DCAwarePolicy::distance is implemented. All the Host instances I've seen are marked as UP.

Working in gdb can be misleading, though, since it can lead to timeouts and disconnects that trigger changes to these objects, and it is the behaviour described above that I'm trying to understand (specifically why I'm seeing errors at all, and why increasing used_hosts_per_remote_dc would make them go away...)

Next steps would be to rebuild our cpp-driver with additional logging to see better what is going on with the state of DCAwarePolicy, but I wanted to ask here first to see if anyone had some insight into this.

Cheers
Oliver

[Oliver Seiler]

On Thursday, January 18, 2018 at 11:17:36 AM UTC-8, Oliver Seiler wrote:
>...

> Strangely (to me), this is very sensitive to used_hosts_per_remote_dc: if I set it to 1, I get 100% failure rate *from cpp-driver* indicating no hosts are available (specifically I get the CASS_ERROR_LIB_NO_HOSTS_AVAILABLE from io_worker.cpp, IOWorker::retry); if I set it to 2 I get a ~60% failure rate; if I set it to 3 I get a 0% failure rate.

After adding more debugging I've tracked this down to this logic in IOWorker::retry:

if (it != pools_.end() &&
it->second->is_ready() &&
it->second->write(request_execution)) {
return; // Successfully written or pending
}

Specifically it turns out the it == pools_end(), meaning that the DCAwareQueryPlan is returning a host that isn't in the PoolMap pools_. I couldn't tell you why at this point. So it winds up skipping all the hosts returned from DCAwareQueryPlan in these failure cases and returning CASS_ERROR_LIB_NO_HOSTS_AVAILABLE. I guess that by bumping up used_hosts_per_remote_dc high enough that at least one of the hosts returned is present in pools_.

At this point we're considering just making used_hosts_per_remote_dc very large (larger than any reasonable cluster size)...

[Michael Penick]

Maybe remote DC connections are not being properly created on the IO workers? At first sight this seems like a DC aware bug, but I'd need to investigate further. 

Thanks for the good information and debugging the issue. I'd be interested in hearing if you find anymore details.

I'll take a look at this later today.

Cheers!

Mike

--
You received this message because you are subscribed to the Google Groups "DataStax C++ Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-user+unsubscribe@lists.datastax.com.

[Oliver Seiler]

Not sure if it is related but I also see cass_session_prepare fail with CASS_ERROR_LIB_NO_HOSTS_AVAILABLE if we stop all nodes in the local site before the client prepares statements (but after the client has connected to the cluster and been issuing non-prepared CQL). Our application was doing statement preparation lazily (at the time of the first use of a prepared statement), and I chose to just move this forward to when we first connect to the cluster. This was happening *after* I'd already worked around the problem I first described by setting the used_hosts_per_remote_dc parameter to a largish number, where a CL=1 or CL=2 query seem to now have no problem finding and using a node in the remote DC, so perhaps there is something unique to how statements get prepared.

(Moving where we prepare statements forward greatly reduces the window where I might run into this, but I still find the behaviour perplexing...)

> To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Michael Penick]

I'm able to reproduce the issue, but I'm not sure what it is yet.

Mike

To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-user+unsubscribe@lists.datastax.com.

[Michael Penick]

Found the issue: https://datastax-oss.atlassian.net/browse/CPP-572

Mike

[Robin Mahony]

Awesome. Just want to confirm (on behalf of Oliver) that our workaround of just bumping that "used_hosts_per_remote_dc_" value to something larger than the number of nodes in our cluster is ok? Basically we want to enable scenarios such as this:

2 sites (A,B). 4 nodes per site. CPP driver instance (D1) connected at Site A, Site A goes down, D1 still can issues CL=TWO batch queries. CL=TWO has no locality requirements, hence fine to go to remote sites.

> To unsubscribe from this group and stop receiving emails from it, send an email to cpp-driver-us...@lists.datastax.com.

[Michael Fero]

Robin/Oliver,

Increasing the number of used hosts per remote DC in order to work around this issue for now should not cause any problems in the driver when using the DC Aware Policy. I looked through the policy code and bound checks are being performed when calculating the next host to be used. I wouldn't recommend this in a production environment however as it could produce unknown side effects.

~Fero

​Michael Fero​

[Oliver Seiler]

Thanks Mike!

On Friday, January 26, 2018 at 12:51:30 PM UTC-8, Michael Penick wrote: