Creating a session takes too long (20s+) - how/what can we test/investigate?

[Tom Kokoska]

Hello, we are using Cassandra as a data storage for some time without problems, but during testing we've found out that creating a session always takes very long time, always around 21s. We create one session per keyspace and cache result in memorycache, so after creating there's no trouble while cache holds.

We use CassandraCSharpDriver 3.19.5 within .NET 6 WebApi which provides access to stored data. We build a cluster as following (with 4 nodes):

        var queryOpts = new QueryOptions()
            .SetConsistencyLevel(ConsistencyLevel.Quorum);
        var socketOpts = new SocketOptions()
            .SetReadTimeoutMillis(90 * 1000)
            .SetConnectTimeoutMillis(30 * 1000);

        return Cluster.Builder()
            .AddContactPoints(nodes)
            .WithCredentials(_userName, _password)
            .WithQueryOptions(queryOpts)
            .WithQueryTimeout(90 * 1000)

            .WithSocketOptions(socketOpts);

All 4 nodes seems to be UN (Up & Normal) according to nodetool.

Both ConnectAsync() and Connect() method take almost the same time around 21s, even when cluster is already built and other sessions wotk properly.

I'm the developer of that backend webapi without direct access to production cassandra node servers. For local development I use single node cassandra cluster in docker and I experience session creation times around 200ms, which would be really nice to have in prodution environment.

What/how we should investigate/test to find out reason for such unexpected long times? It's almost  like there is some 5s timeout awaited for each node, before Session is connected, but I was unable to find any hint in documentation how to proceed.

Even pointing me in right direction would be a great help.

Thanks, Tom.

[Joao Reis]

Hi,

The driver opens connections to all nodes during initialization so it is possible that it is timing out when connecting to one or multiple nodes which causes the initialization to be slow. I would start by enabling driver logging at INFO level. DEBUG/TRACE level might have more info but it could be a lot of log spam, if your infrastructure can handle it then you can try log levels more verbose than INFO.

With driver logs you might be able to identify if a particular node is slowing down the initialization process or if there's something else that is causing it

	João Reis   w. www.datastax.com

--
You received this message because you are subscribed to the Google Groups "DataStax C# Driver for Apache Cassandra User Mailing List" group.
To unsubscribe from this group and stop receiving emails from it, send an email to csharp-driver-u...@lists.datastax.com.
To view this discussion on the web visit https://groups.google.com/a/lists.datastax.com/d/msgid/csharp-driver-user/d0ef17ac-f054-4b91-aa05-b71d0f6ad45fn%40lists.datastax.com.

[Tom Kokoska]

Hi Joao, thanks a million, we've finally made a breaktrough. Found a dead node, long gone from network, still somehow surviving in the cluster memory, not reported by nodetool status / describecluster. nodetool assassinate (!) fixed that and now we're making sessions in under 200ms.

Thanks again for speedy answer, we really appreciate you help.

Tom

Dne čtvrtek 14. prosince 2023 v 16:24:25 UTC+1 uživatel Joao Reis napsal: