v2.5 driver - Cassandra.NoHostAvailableException: None of the hosts tried for query are available (tried: )

[Zac Evans]

Cassandra.NoHostAvailableException: None of the hosts tried for query are available (tried: )

I've seen folks discuss the above error around previous versions of the driver, but not with 2.5.  The resolution for the aged conversations was to update the driver.

I will occasionally see this form (and the form that actually lists out the ups of the endpoints).  It seems to be volume related, but, I'm not finding anything obvious to suggest always failing.  I'm reading when this happens - at a decent pace, but nothing gigantic.  The c* nodes are running as a proof of concept on a cluster of win boxes.

[Jorge Bay Gondra]

Hi Zac,

Could you provide the stack trace, it could help.

Also, you can enable tracing to see what is going on.

Jorge

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.

[Zac Evans]

Cassandra.NoHostAvailableException: None of the hosts tried for query are available (tried: ) at Cassandra.RequestHandler`1.GetNextConnection(IStatement statement) at Cassandra.RequestHandler`1.TrySend() at Cassandra.TaskHelper.WaitToComplete(Task task, Int32 timeout) at Cassandra.Session.Prepare(String cqlQuery)

at ...my calling code... under a stack from the web.api pipeline

Working on getting tracing flipped on and dropped onto one of the web servers.

[Jorge Bay Gondra]

This usually happens when no host can be considered as UP by the driver, check the driver trace to see why it is happening.

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.

[Zac Evans]

Understood.  I can see that nodetool status says they are all UN.  Will post the trace as soon as I can snag it.

Looking at our code - as it gets hit in volume and rereading the do/nots for the driver, I realize I'm doing Prepare().Bind() on every attempted read/write in my data access code ... I'm presuming this is the exact opposite behavior expected.  I'm working on pushing out and retaining those prep'd statements in a few key spots to see if that is the culprit (while I wait for trace).

Any thoughts on the effect of preparing every time?  Would that exhibit this behavior?

ze

On Thursday, 12 March 2015 09:11:05 UTC-5, Zac Evans wrote:

[Jorge Bay Gondra]

That is not the underlying cause but preparing every time is anti-pattern, you should prepare once in your app lifetime, reuse that same prepared statement instance and bind different parameters each time.

Here you have a list of "rules" to follow when using the DataStax drivers: http://www.datastax.com/dev/blog/4-simple-rules-when-using-the-datastax-drivers-for-cassandra

The samples are written in java, but the same applies to C#.

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.

[Zac Evans]

I've uploaded a trace here ... 

Our servers are settled down a bit in the evenings, the errors are much fewer.  In this case, there's a pretty big gap in time between the previous request and this one.

Some topology if it is interesting: 10 web servers (5 each in two data centers), 10 nodes in a c* cluster (5 each in two data centers).  

I notice from the errors that the attempts to connect are only 5 of the 10 possible nodes - my presumption being that it's trying to stay near.  I will verify the ips and their C* DC affiliation.

Any thoughts on the attached trace and the behavior?

ze

[Jorge Bay Gondra]

Hi,

It looks like something is killing your network connections due to low traffic, to avoid that:

- Set to 1 core connection per host.

- Enable the Heartbeat feature (an artificial keep alive).

You can do it using the builder, on Cluster creation:

var poolingOptions = new PoolingOptions()

  .SetCoreConnectionsPerHost(1)

  .SetHeartBeatInterval(10000);

var cluster = Cluster

  .Builder()

  .AddContactPoints(hosts).

  .WithPoolingOptions(poolingOptions)

  .Build();

Hope it helps,

Jorge

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.