Cassandra Datastax C# driver inactivity throws NoHostAvailableException

[RAID]

I am using the Datastax C# driver (2.0.3) to access my Cassandra cluster (2.0.8) hosted on a VM in Azure.

I have a long running process with reads or writes statements using a session object. If 5 minutes passes without any statements to the open session, then the following statement will fail with a NoHostAvailableException.

This code reproduces the problem:

static void Main(string[] args)
    {
        // Arrange
        var _cluster = Cluster();
        var session = _cluster.Connect();
        var keyspace = "nohostavailabletest";
        session.CreateKeyspaceIfNotExists(keyspace);
        session.ChangeKeyspace(keyspace);
        session.Execute(@"CREATE TABLE IF NOT EXISTS test (customer_id text, PRIMARY KEY (customer_id));");
        session.Execute("INSERT INTO \"test\"(\"customer_id\") VALUES ('myCustomer');");

        // Act
        Console.WriteLine("Cassandra ready for time out test...");
        var query = "select * from test;";
        session.Execute(query);
        Console.WriteLine("Read from Cassandra");
        Thread.Sleep(5*60*1000);
        session.Execute(query); // This execution will throw an NoHostAvailableException
        Console.WriteLine("This point is never reach because of NoHostAvailableException");
    }

    private static Cluster Cluster()
    {
        return Cassandra.Cluster.Builder()
            .AddContactPoints(new[] { "insertYourClusterIpHere" })
            .WithPort(9042)
            .Build();
    }

Am I doing anything wrong? Or is there some kind of "keep alive" feature in the driver?

[Jorge Bay Gondra]

The driver should recycle the dead connections...

I will check using the 2.0 branch and the master branch.

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.

[Jorge Bay Gondra]

I'm not able to reproduce it on the 2.0 or the master branch... I created a test to cover on it.

You could try to get more info by enabling tracing.

Diagnostics.CassandraTraceSwitch.Level = TraceLevel.Info;

Trace.Listeners.Add(new ConsoleTraceListener());

Could you check using master branch?

(sample using command line, you could use visual studio instead)

$ if exist csharp-driver rmdir /S /Q csharp-driver
$ git clone https://github.com/datastax/csharp-driver.git --branch master --single-branch
$ msbuild.exe /v:m /property:Configuration=Release csharp-driver\src\Cassandra.sln

(and reference the newly compiled Cassandra.dll)

Thanks,

Jorge

[RAID]

This is the console output using master:

Cassandra.ControlConnection: 06-30-2014 15:33:18.683 +02:00 : Refreshing Control

Connection...

Cassandra.HostConnectionPool: 06-30-2014 15:33:18.702 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.ControlConnection: 06-30-2014 15:33:18.932 +02:00 : Refreshing NodeLis

t and TokenMap..

Cassandra.ControlConnection: 06-30-2014 15:33:19.150 +02:00 : NodeList and Token

Map have been successfully refreshed!

Cassandra.ControlConnection: 06-30-2014 15:33:19.151 +02:00 : ControlConnection

is fresh!

Cassandra.Cluster: 06-30-2014 15:33:19.151 +02:00 : Binary protocol version: [2]

Cassandra.HostConnectionPool: 06-30-2014 15:33:19.152 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.HostConnectionPool: 06-30-2014 15:33:19.271 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.Cluster: 06-30-2014 15:33:19.391 +02:00 : Session connected!

Cassandra.Session: 06-30-2014 15:33:20.158 +02:00 : Keyspace [nohostavailabletes

t] has been successfully CREATED.

Cassandra.Connection: 06-30-2014 15:33:20.218 +02:00 : Connection to host x.y

.z.214 switching to keyspace nohostavailabletest

Cassandra ready for time out test...

Read from Cassandra

Cassandra.Connection: 06-30-2014 15:38:42.171 +02:00 : Canceling pending operati

ons 1 and write queue 0

Cassandra.Session: 06-30-2014 15:38:42.174 +02:00 #WARNING: Setting host x.y

.z.214 as DOWN

Cassandra.ControlConnection: 06-30-2014 15:38:42.176 +02:00 : Refreshing Control

Connection...

Cassandra.ControlConnection: 06-30-2014 15:38:42.178 +02:00 #ERROR: ControlConne

ction is lost. Retrying..

Unhandled Exception: Cassandra.NoHostAvailableException: None of the hosts tried

 for query are available (tried: x.y.z.214)

   at Cassandra.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout) in c:\p

rojects\csharp-driver\src\Cassandra\TaskHelper.cs:line 80

   at Cassandra.Session.Execute(IStatement statement) in c:\projects\csharp-driv

er\src\Cassandra\Session.cs:line 205

   at Cassandra.Session.Execute(String cqlQuery) in c:\projects\csharp-driver\sr

c\Cassandra\Session.cs:line 213

   at CassandraKeepAlive.Program.Main(String[] args) in c:\Users\xxx\Documents\V

isual Studio 2013\Projects\CassandraKeepAlive\CassandraKeepAlive\Program.cs:line

 35

[Jorge Bay Gondra]

Thanks for the great detail!

I have some ideas on how to reproduce it now... I created the Jira ticket CSHARP-147 for follow up.

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.

[Jorge Bay Gondra]

CSHARP-147 is fixed, now you keep alive is set to on by default and the socketoptions are exposed via Cluster.Builder().WithSocketoptions().

As I explain in the ticket, this wont prevent all long running inactive tcp connections from dropping, as that is a hardware or software setting at infrastructure level. What can be done in the driver for the future, is try to recycle 1 connection per host or something like that to prevent having the pool from throwing NoHostAvailableException until the reconnection policy allows reconnection. I'll create a new ticket for it.

If you don't want to wait for the next release, you can access this setting in your current version (it is not "fluent",though):

var builder = Cluster.Builder()...//other options.

builder.SocketOptions.SetKeepAlive(true);

[RAID]

Hello

Thank you for your effort.

I have just tried again both for master and for 2.0. I have added builder.SocketOptions.SetKeepAlive(true) but I still have the NoHostAvailableException problem. 

This is the current console output (using the 2.0 branch):

Cassandra.ControlConnection: 06-30-2014 18:11:21.233 +02:00 #INFO: Refreshing Co

ntrolConnection...

Cassandra.Session: 06-30-2014 18:11:21.481 +02:00 #INFO: Allocated new connectio

n

Cassandra.ControlConnection: 06-30-2014 18:11:21.552 +02:00 #INFO: Refreshing No

deList and TokenMap..

Cassandra.ControlConnection: 06-30-2014 18:11:21.756 +02:00 #INFO: NodeList and

TokenMap have been successfully refreshed!

Cassandra.ControlConnection: 06-30-2014 18:11:21.758 +02:00 #INFO: ControlConnec

tion is fresh!

Cassandra.Cluster: 06-30-2014 18:11:21.760 +02:00 #INFO: Binary protocol version

: [2]

Cassandra.Session: 06-30-2014 18:11:21.944 +02:00 #INFO: Allocated new connectio

n

Cassandra.Cluster: 06-30-2014 18:11:21.945 +02:00 #INFO: Session connected!

Cassandra.RequestHandlers.RequestHandler: 06-30-2014 18:11:22.017 +02:00 #ERROR:

 ( Exception! Source

 Message: Keyspace nohostavailabletest already exists

 StackTrace:

 To display StackTrace, change Debugging.StackTraceIncluded property value to tr

ue. )

Cassandra.Session: 06-30-2014 18:11:22.049 +02:00 #INFO: Cannot CREATE keyspace:

  nohostavailabletest  because it already exists.

Cassandra.Session: 06-30-2014 18:11:22.112 +02:00 #INFO: Changed keyspace to [no

hostavailabletest]

Cassandra ready for time out test...

Read from Cassandra

Cassandra.RequestHandlers.RequestHandler: 06-30-2014 18:16:41.365 +02:00 #ERROR:

 ( Exception! Source

 Message: None of the hosts tried for query are available (tried: x.y.z.21

4)

 StackTrace:

 To display StackTrace, change Debugging.StackTraceIncluded property value to tr

ue. )

This is the stack trace:

   at Cassandra.RequestHandlers.RequestHandler.Connect(Session owner, Boolean moveNext, Int32& streamId) in c:\projects\csharp-driver\src\Cassandra\RequestHandlers\RequestHandler.cs:line 63

   at Cassandra.RequestHandlers.ExecuteQueryRequestHandler.Connect(Session owner, Boolean moveNext, Int32& streamId) in c:\projects\csharp-driver\src\Cassandra\RequestHandlers\ExecuteQueryRequestHandler.cs:line 49

   at Cassandra.Session.ExecConn(RequestHandler handler, Boolean moveNext) in c:\projects\csharp-driver\src\Cassandra\Session.cs:line 713

   at Cassandra.AsyncResultNoResult.End(IAsyncResult result, Object owner, String operationId) in c:\projects\csharp-driver\src\Cassandra\AsyncResultNoResult.cs:line 166

   at Cassandra.AsyncResult`1.End(IAsyncResult result, Object owner, String operationId) in c:\projects\csharp-driver\src\Cassandra\AsyncResult.cs:line 53

   at Cassandra.Session.EndQuery(IAsyncResult ar) in c:\projects\csharp-driver\src\Cassandra\Session.cs:line 819

   at Cassandra.Session.EndExecute(IAsyncResult ar) in c:\projects\csharp-driver\src\Cassandra\Session.cs:line 560

   at Cassandra.Session.Execute(IStatement query) in c:\projects\csharp-driver\src\Cassandra\Session.cs:line 590

   at Cassandra.Session.Execute(String cqlQuery) in c:\projects\csharp-driver\src\Cassandra\Session.cs:line 611

   at CassandraKeepAlive.Program.Main(String[] args) in c:\Users\Ope\Documents\Visual Studio 2013\Projects\CassandraKeepAlive\CassandraKeepAlive\Program.cs:line 35

   at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)

   at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)

   at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()

   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)

   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)

   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)

   at System.Threading.ThreadHelper.ThreadStart()

[Jorge Bay Gondra]

Hi,

As I state in the CSHARP-148, it is hard to reproduce. It is due to the infrastructure: a network appliance (ie: router) is actively dropping all the connections.

I included the possibility of "last chance" resurrection: if there is no other available host and the last socket exception "allows resurrection", it will retry to connect just 1 more time.

I added you as reviewer in CSHARP-148, could you check if that solves your problem?

Thanks,

Jorge

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.

[RAID]

I have just tried master and branch 2.0 again.

The problem remains. This is the output using master:

Cassandra.ControlConnection: 07-01-2014 12:36:41.037 +02:00 : Refreshing Control

Connection...

Cassandra.HostConnectionPool: 07-01-2014 12:36:41.083 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.ControlConnection: 07-01-2014 12:36:41.355 +02:00 : Refreshing NodeLis

t and TokenMap..

Cassandra.ControlConnection: 07-01-2014 12:36:41.591 +02:00 : NodeList and Token

Map have been successfully refreshed!

Cassandra.ControlConnection: 07-01-2014 12:36:41.592 +02:00 : ControlConnection

is fresh!

Cassandra.Cluster: 07-01-2014 12:36:41.592 +02:00 : Binary protocol version: [2]

Cassandra.HostConnectionPool: 07-01-2014 12:36:41.593 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.HostConnectionPool: 07-01-2014 12:36:41.690 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.Cluster: 07-01-2014 12:36:41.788 +02:00 : Session connected!

Cassandra.Session: 07-01-2014 12:36:41.868 +02:00 : Cannot CREATE keyspace:  noh

ostavailabletest  because it already exists.

Cassandra.Connection: 07-01-2014 12:36:41.918 +02:00 : Connection to host x.y.z

.214 switching to keyspace nohostavailabletest

Cassandra ready for time out test...

Read from Cassandra

Cassandra.Connection: 07-01-2014 12:42:01.089 +02:00 : Canceling pending operati

ons 1 and write queue 0

Cassandra.Session: 07-01-2014 12:42:01.092 +02:00 #WARNING: Setting host x.y.z

.214 as DOWN

Cassandra.ControlConnection: 07-01-2014 12:42:01.095 +02:00 : Refreshing Control

Connection...

Cassandra.ControlConnection: 07-01-2014 12:42:01.097 +02:00 #ERROR: ControlConne

ction is lost. Retrying..

Unhandled Exception: Cassandra.NoHostAvailableException: None of the hosts tried

 for query are available (tried: x.y.z.214)

   at Cassandra.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout) in c:\p

rojects\csharp-driver\src\Cassandra\TaskHelper.cs:line 80

   at Cassandra.Session.Execute(IStatement statement) in c:\projects\csharp-driv

er\src\Cassandra\Session.cs:line 205

   at Cassandra.Session.Execute(String cqlQuery) in c:\projects\csharp-driver\sr

c\Cassandra\Session.cs:line 213

   at CassandraKeepAlive.Program.Main(String[] args) in c:\Users\xxx\Documents\V

isual Studio 2013\Projects\CassandraKeepAlive\CassandraKeepAlive\Program.cs:line

 35

Cassandra.ControlConnection: 07-01-2014 12:42:03.104 +02:00 : Refreshing Control

Connection...

Cassandra.ControlConnection: 07-01-2014 12:42:19.355 +02:00 #ERROR: Unexpected e

rror occurred during ControlConnection refresh.

EXCEPTION:

 ( Exception! Source Cassandra

 Message: The task didn't complete before timeout.

 StackTrace:

 To display StackTrace, change Debugging.StackTraceIncluded property value to tr

ue. )

And output from 2.0:

Cassandra.ControlConnection: 07-01-2014 12:55:26.660 +02:00 #INFO: Refreshing Co

ntrolConnection...

Cassandra.Session: 07-01-2014 12:55:26.850 +02:00 #INFO: Allocated new connectio

n

Cassandra.ControlConnection: 07-01-2014 12:55:26.910 +02:00 #INFO: Refreshing No

deList and TokenMap..

Cassandra.ControlConnection: 07-01-2014 12:55:27.099 +02:00 #INFO: NodeList and

TokenMap have been successfully refreshed!

Cassandra.ControlConnection: 07-01-2014 12:55:27.100 +02:00 #INFO: ControlConnec

tion is fresh!

Cassandra.Cluster: 07-01-2014 12:55:27.100 +02:00 #INFO: Binary protocol version

: [2]

Cassandra.Session: 07-01-2014 12:55:27.274 +02:00 #INFO: Allocated new connectio

n

Cassandra.Cluster: 07-01-2014 12:55:27.275 +02:00 #INFO: Session connected!

Cassandra.RequestHandlers.RequestHandler: 07-01-2014 12:55:27.333 +02:00 #ERROR:

 ( Exception! Source

 Message: Keyspace nohostavailabletest already exists

 StackTrace:

 To display StackTrace, change Debugging.StackTraceIncluded property value to tr

ue. )

Cassandra.Session: 07-01-2014 12:55:27.352 +02:00 #INFO: Cannot CREATE keyspace:

  nohostavailabletest  because it already exists.

Cassandra.Session: 07-01-2014 12:55:27.402 +02:00 #INFO: Changed keyspace to [no

hostavailabletest]

Cassandra ready for time out test...

Read from Cassandra

Cassandra.RequestHandlers.RequestHandler: 07-01-2014 13:00:46.535 +02:00 #ERROR:

 ( Exception! Source

 Message: None of the hosts tried for query are available (tried: x.y.z.21

4)

 StackTrace:

 To display StackTrace, change Debugging.StackTraceIncluded property value to tr

ue. )

Cassandra.ControlConnection: 07-01-2014 13:00:46.540 +02:00 #INFO: Refreshing Co

ntrolConnection...

Cassandra.ControlConnection: 07-01-2014 13:00:46.541 +02:00 #ERROR: ( Exception!

 Source

 Message: None of the hosts tried for query are available (tried: )

 StackTrace:

 To display StackTrace, change Debugging.StackTraceIncluded property value to tr

ue.

)

Unhandled Exception: Cassandra.ControlConnection: 07-01-2014 13:00:46.543 +02:00

 #ERROR: ControlConnection is lost. Retrying..

Cassandra.NoHostAvailableException: None of the hosts tried for query are availa

ble (tried: x.y.z.214)

   at Cassandra.RequestHandlers.RequestHandler.Connect(Session owner, Boolean mo

veNext, Int32& streamId) in c:\projects\csharp-driver\src\Cassandra\RequestHandl

ers\RequestHandler.cs:line 63

   at Cassandra.RequestHandlers.ExecuteQueryRequestHandler.Connect(Session owner

, Boolean moveNext, Int32& streamId) in c:\projects\csharp-driver\src\Cassandra\

RequestHandlers\ExecuteQueryRequestHandler.cs:line 49

   at Cassandra.Session.ExecConn(RequestHandler handler, Boolean moveNext) in c:

\projects\csharp-driver\src\Cassandra\Session.cs:line 713

   at Cassandra.AsyncResultNoResult.End(IAsyncResult result, Object owner, Strin

g operationId) in c:\projects\csharp-driver\src\Cassandra\AsyncResultNoResult.cs

:line 166

   at Cassandra.AsyncResult`1.End(IAsyncResult result, Object owner, String oper

ationId) in c:\projects\csharp-driver\src\Cassandra\AsyncResult.cs:line 53

   at Cassandra.Session.EndExecute(IAsyncResult ar) in c:\projects\csharp-driver

\src\Cassandra\Session.cs:line 555

   at Cassandra.Session.Execute(String cqlQuery) in c:\projects\csharp-driver\sr

c\Cassandra\Session.cs:line 611

   at CassandraKeepAlive.Program.Main(String[] args) in c:\Users\xxx\Documents\V

isual Studio 2013\Projects\CassandraKeepAlive\CassandraKeepAlive\Program.cs:line

 35

Cassandra.ControlConnection: 07-01-2014 13:00:50.113 +02:00 #INFO: Refreshing Co

ntrolConnection...

(I am not able to comment directly on CSHARP-148)

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-user+unsub...@lists.datastax.com.

[Jorge Bay Gondra]

Could you pull the latest changes from master (again) and switch tracing to verbose?

Diagnostics.CassandraTraceSwitch.Level = TraceLevel.Verbose;

Thanks,

Jorge

[RAID]

Cassandra.ControlConnection: 07-01-2014 13:37:24.808 +02:00 : Refreshing Control

Connection...

Cassandra.HostConnectionPool: 07-01-2014 13:37:24.828 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.TcpSocket: 07-01-2014 13:37:24.890 +02:00 Socket connected, start read

ing using SocketEventArgs interface

Cassandra.Connection: 07-01-2014 13:37:24.897 +02:00 Sending #0 for StartupReque

st

Cassandra.Connection: 07-01-2014 13:37:24.961 +02:00 Read #0 for Opcode 2

Cassandra.Connection: 07-01-2014 13:37:24.975 +02:00 Sending #0 for RegisterForE

ventRequest

Cassandra.Connection: 07-01-2014 13:37:25.028 +02:00 Read #0 for Opcode 2

Cassandra.ControlConnection: 07-01-2014 13:37:25.033 +02:00 : Refreshing NodeLis

t and TokenMap..

Cassandra.Connection: 07-01-2014 13:37:25.034 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.112 +02:00 Read #0 for Opcode 8

Cassandra.Connection: 07-01-2014 13:37:25.133 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.184 +02:00 Read #0 for Opcode 8

Cassandra.ControlConnection: 07-01-2014 13:37:25.193 +02:00 : NodeList and Token

Map have been successfully refreshed!

Cassandra.ControlConnection: 07-01-2014 13:37:25.193 +02:00 : ControlConnection

is fresh!

Cassandra.Cluster: 07-01-2014 13:37:25.194 +02:00 : Binary protocol version: [2]

Cassandra.HostConnectionPool: 07-01-2014 13:37:25.194 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.TcpSocket: 07-01-2014 13:37:25.247 +02:00 Socket connected, start read

ing using SocketEventArgs interface

Cassandra.Connection: 07-01-2014 13:37:25.248 +02:00 Sending #0 for StartupReque

st

Cassandra.Connection: 07-01-2014 13:37:25.299 +02:00 Read #0 for Opcode 2

Cassandra.HostConnectionPool: 07-01-2014 13:37:25.300 +02:00 : Creating a new co

nnection to the host x.y.z.214

Cassandra.TcpSocket: 07-01-2014 13:37:25.348 +02:00 Socket connected, start read

ing using SocketEventArgs interface

Cassandra.Connection: 07-01-2014 13:37:25.349 +02:00 Sending #0 for StartupReque

st

Cassandra.Connection: 07-01-2014 13:37:25.397 +02:00 Read #0 for Opcode 2

Cassandra.Cluster: 07-01-2014 13:37:25.398 +02:00 : Session connected!

Cassandra.Connection: 07-01-2014 13:37:25.405 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.454 +02:00 Read #0 for Opcode 0

Cassandra.Session: 07-01-2014 13:37:25.474 +02:00 : Cannot CREATE keyspace:  noh

ostavailabletest  because it already exists.

Cassandra.Connection: 07-01-2014 13:37:25.476 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.524 +02:00 Read #0 for Opcode 8

Cassandra.Connection: 07-01-2014 13:37:25.526 +02:00 : Connection to host x.y.z

.214 switching to keyspace nohostavailabletest

Cassandra.Connection: 07-01-2014 13:37:25.527 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.576 +02:00 Read #0 for Opcode 8

Cassandra.Connection: 07-01-2014 13:37:25.577 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.627 +02:00 Read #0 for Opcode 8

Cassandra.Connection: 07-01-2014 13:37:25.627 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.677 +02:00 Read #0 for Opcode 8

Cassandra ready for time out test...

Cassandra.Connection: 07-01-2014 13:37:25.678 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:37:25.759 +02:00 Read #0 for Opcode 8

Read from Cassandra

Cassandra.Connection: 07-01-2014 13:42:25.760 +02:00 Sending #0 for QueryRequest

Cassandra.Connection: 07-01-2014 13:42:44.662 +02:00 Socket error TimedOut

Cassandra.Connection: 07-01-2014 13:42:44.662 +02:00 : Canceling pending operati

ons 1 and write queue 0

Cassandra.Session: 07-01-2014 13:42:44.663 +02:00 Socket error TimedOut

Cassandra.Session: 07-01-2014 13:42:44.664 +02:00 #WARNING: Setting host x.y.z

.214 as DOWN

Cassandra.ControlConnection: 07-01-2014 13:42:44.667 +02:00 : Refreshing Control

Connection...

Cassandra.ControlConnection: 07-01-2014 13:42:44.668 +02:00 #ERROR: ControlConne

ction is lost. Retrying..

Unhandled Exception: Cassandra.NoHostAvailableException: None of the hosts tried

 for query are available (tried: x.y.z.214)

   at Cassandra.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout) in c:\p

rojects\csharp-driver\src\Cassandra\TaskHelper.cs:line 80

   at Cassandra.Session.Execute(IStatement statement) in c:\projects\csharp-driv

er\src\Cassandra\Session.cs:line 205

   at Cassandra.Session.Execute(String cqlQuery) in c:\projects\csharp-driver\sr

c\Cassandra\Session.cs:line 213

   at CassandraKeepAlive.Program.Main(String[] args) in c:\Users\Ope\Documents\V

isual Studio 2013\Projects\CassandraKeepAlive\CassandraKeepAlive\Program.cs:line

 37

Cassandra.ControlConnection: 07-01-2014 13:43:33.581 +02:00 : Refreshing Control

Connection...

Cassandra.Connection: 07-01-2014 13:43:33.581 +02:00 Sending #0 for RegisterForE

ventRequest

[Levi Page]

I believe I am having the same problem. I have prepared statements that just start failing when trying to execute them that were working fine before. It seems to happen after inactivity. I can execute the same commands via DevCenter as it's failing and it works fine. I can also use cqlsh without any issues. By all indications the host is available. As soon as I restart my app and create a new connection it starts working again. Since I have a service that needs to run 24/7 this is not a reasonable option for me (restarting all the time). The JIRA ticket shows this was fixed, but maybe it's not or perhaps there is a similar issue specific to prepared statement use after inactivity. Any thoughts? I am using version 2.5.2 of the driver.

[Solvej Pedersen]

We are also having issues like this after rewriting all our queries to prepared statements. No problems before when using batch writes and simple statements for reads.
At some point in time all queries from that applikation times out, while other applikations, that use the same cluster, works fine.
Applikation restart is also bringing the applikation up and running again in our case.

We have yet to track down if this problem relates to a single prepared statement beeing inactive, but that could very well be the case.

To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.

[Levi Page]

Yeah when it happens to me, it seems to be a group of prepared statements that are failing while others are still working. It seems to be the group that has been the most inactive so to speak. If I don't run any queries at all, they all start failing with this error. Then as soon as I restart my application which causes the reconnect, they all are working again. When I have a constant load going to the database it seems to work fine.

[Jorge Bay Gondra]

Hi Levi,

You should enable tracing in the driver and check what is happening under the hood. If you need any help with the information gathered, please create a new thread.

Thanks,

Jorge

[Levi Page]

"   at Cassandra.RequestHandler`1.GetNextConnection(IStatement statement)\r\n   at Cassandra.RequestHandler`1.TrySend()\r\n   at Cassandra.Tasks.TaskHelper.WaitToComplete(Task task, Int32 timeout)\r\n   at Cassandra.Tasks.TaskHelper.WaitToComplete[T](Task`1 task, Int32 timeout)\r\n   at Cassandra.Session.Prepare(String cqlQuery)\r\n   at x.Data.Factory.PreparedStatementFactory.PrepareStatement(ISession session, String cqlQuery) in x.cs:line 44"

"None of the hosts tried for query are available (tried: myip:9042)"

Here is the driver trace:

Cassandra.Connection: 06/23/2015 19:20:33.509 -04:00 : Canceling pending operations 3 and write queue 0

Cassandra.Session: 06/23/2015 19:20:33.516 -04:00 : RequestHandler received exception System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Cassandra.Session: 06/23/2015 19:20:33.516 -04:00 : RequestHandler received exception System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Cassandra.Session: 06/23/2015 19:20:33.516 -04:00 : RequestHandler received exception System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond

Cassandra.ControlConnection: 06/23/2015 19:20:33.539 -04:00 #WARNING: Host 104.210.0.163:9042 considered as DOWN

Cassandra.ControlConnection: 06/23/2015 19:20:33.560 -04:00 #WARNING: Host 104.210.0.163:9042 considered as DOWN

Cassandra.ControlConnection: 06/23/2015 19:20:33.539 -04:00 #WARNING: Host 104.210.0.163:9042 considered as DOWN

Cassandra.ControlConnection: 06/23/2015 19:20:33.563 -04:00 #WARNING: Host 104.210.0.163:9042 used by the ControlConnection DOWN

Cassandra.ControlConnection: 06/23/2015 19:20:33.583 -04:00 #WARNING: Host 104.210.0.163:9042 used by the ControlConnection DOWN

Cassandra.ControlConnection: 06/23/2015 19:20:33.563 -04:00 #WARNING: Host 104.210.0.163:9042 used by the ControlConnection DOWN

Cassandra.ControlConnection: 06/23/2015 19:20:33.606 -04:00 : Trying to reconnect the ControlConnection

Cassandra.Session: 06/23/2015 19:20:33.628 -04:00 : Retrying request: ExecuteRequest

Cassandra.Session: 06/23/2015 19:20:33.629 -04:00 : Retrying request: ExecuteRequest

Cassandra.Session: 06/23/2015 19:20:33.629 -04:00 : Retrying request: PrepareRequest

A first chance exception of type 'Cassandra.NoHostAvailableException' occurred in Cassandra.dll

A first chance exception of type 'Cassandra.NoHostAvailableException' occurred in Cassandra.dll

A first chance exception of type 'Cassandra.NoHostAvailableException' occurred in Cassandra.dll

Cassandra.Session: 06/23/2015 19:20:33.669 -04:00 : RequestHandler received exception Cassandra.NoHostAvailableException: None of the hosts tried for query are available (tried: 104.210.0.163:9042)

   at Cassandra.RequestHandler`1.GetNextConnection(IStatement statement)

   at Cassandra.RequestHandler`1.TrySend()

Cassandra.Session: 06/23/2015 19:20:33.669 -04:00 : RequestHandler received exception Cassandra.NoHostAvailableException: None of the hosts tried for query are available (tried: 104.210.0.163:9042)

   at Cassandra.RequestHandler`1.GetNextConnection(IStatement statement)

   at Cassandra.RequestHandler`1.TrySend()

Cassandra.Session: 06/23/2015 19:20:33.669 -04:00 : RequestHandler received exception Cassandra.NoHostAvailableException: None of the hosts tried for query are available (tried: 104.210.0.163:9042)

   at Cassandra.RequestHandler`1.GetNextConnection(IStatement statement)

   at Cassandra.RequestHandler`1.TrySend()

A first chance exception of type 'System.AggregateException' occurred in mscorlib.dll

A first chance exception of type 'System.AggregateException' occurred in mscorlib.dll

A first chance exception of type 'Cassandra.NoHostAvailableException' occurred in Cassandra.dll

[Jorge Bay Gondra]

Hi Levi,

If you are experiencing this exceptions after periods of inactivity, you should enable heartbeat at connection level:

Some firewalls and network devices kill connections that have been idling for some time.
To avoid that, a heartbeat functionality should be implemented that makes a request to the Cassandra server after a configurable interval passed since the last request.

This functionality will be common in all DataStax drivers.

The setting is included in the PoolingOptions.

//Heartbeat after 50 seconds of idling
Cluster.Builder()
  .AddContactPoints(hosts)
  .WithPoolingOptions(
    new PoolingOptions()
    .SetHeartBeatInterval(50000));

[Jorge Bay Gondra]

Sorry, I just send it before finish writing the post...

On the other side, it looks that you are preparing your queries more than once. If that is the case, you should prepare each query once in your app lifetime (generally at startup).

If you are still experiencing issues after enabling the hearbeat functionality, please create a ticket on JIRA: 

https://datastax-oss.atlassian.net/projects/CSHARP/issues

Thanks,

Jorge 

[Levi Page]

Thanks Jorge. I'll give the heartbeat a shot! In regards to your other comment. I only prepare statements once. I have tested this several times even with 10000 async threads, but I'll double check my code again to make sure. 

I use a small class I wrote in case anyone is interested. There is also one that is task based used in the KillrVideo example from DataStax worth checking out:

  public class PreparedStatementFactory

    {

        private ConcurrentDictionary<string, Lazy<PreparedStatement>> _statements

            = new ConcurrentDictionary<string, Lazy<PreparedStatement>>();

        public PreparedStatementFactory()

        {

        }

        /// <summary>

        /// Gets a statement from the cache if it already exists, otherwise it adds it.

        /// </summary>

        /// <param name="key"></param>

        /// <returns></returns>

        public PreparedStatement GetOrAddStatement(ISession session, string cqlQuery)

        {

            // Preparing a statement is an expensive (slow) process. We use a factory to retrieve the statements, 

            // so we can avoid preparing the same statement (i.e. cql query ) more than once.

            // Furthermore we use Lazy<T> to prevent the statement from being prepared on multiple threads.

            PreparedStatement ps = _statements.GetOrAdd(

                cqlQuery,

                key => new Lazy<PreparedStatement>(() => PrepareStatement(session, cqlQuery))).Value;

            return ps;

        }

        private PreparedStatement PrepareStatement(ISession session, string cqlQuery)

        {

            Trace.TraceInformation("Preparing statement: " + cqlQuery);

            try

            {

                PreparedStatement ps = session.Prepare(cqlQuery).SetConsistencyLevel(ConsistencyLevel.LocalOne);

                return ps;

            }

            catch (Exception ex)

            {

                Trace.TraceError("Error preparing statement: " + cqlQuery + " with error: " + ex.Message);

                throw ex;

            }

        }

    }

[Solvej Pedersen]

Hello Levi

Your solution does not guarantee that each query only will be prepared once. You are relying on ConcurrentDictionary.GetOrAdd(key, Func). But ConcurrentDictionary does not guarantee that Func only will be called once. If two (or more) threads calls GetOrAdd at the same time with the same key (that's not yet in the dictionary), then Func will be called once for each thread.

Our experience is that Cassandra (or the driver throws timeouts) for all attempts if many threads are calling Prepare() at the same time.

[Levi Page]

That is what the lazy<t> is for. I can call this in a parallel loop 10000 times and use trace statements to veriify it only attemots to prepare the statements one time.

[Levi Page]

Thanks Jorge, the Heartbeat worked perfect.