Creating a session takes too long when a Cassandra node is shut down - Possible to Ignore Down Hosts?

[Dan]

Hello,

Would it be possible to add an option to Cluster.Build or Cluster.Connect which will ignore hosts that aren't considered 'Up' so that the connection pool warm-up is not considerably extended by hosts that are shutdown?

Situation:

A cluster with 4 nodes. One of the nodes is turned off (shutdown) for maintenance. (As oppose to the node being switched on, with the Cassandra service stopped)

Problem:

When we run the example code (below), with a node in the cluster turned off, we hit the default (5000ms) for SocketOptions.ConnectTimeoutMillis, once for each session. With some of our applications using 8 or more keyspaces and thus using multiple sessions, turning off a Cassandra node extends the warm up time for each application by 32 seconds. This same delay is not experienced if the node is up with the Cassandra service stopped.

Example Code:

using System;
using System.Collections.Concurrent;
using System.Diagnostics;
using System.Linq;
using Cassandra;
namespace ConsoleApp1
{
    class CassandraCluster
    {
        private static readonly Cluster Instance;
        static CassandraCluster()
        {
            Cluster.MaxProtocolVersion = 3;
            var thingy = Cluster.Builder()
                .WithLoadBalancingPolicy(new TokenAwarePolicy(new DCAwareRoundRobinPolicy("localcluster")))
                .AddContactPoints(new string[] {"cassandralocal1", "cassandralocal2", "cassandralocal3", "cassandralocal4"});
            var sw = new Stopwatch();
            sw.Start();
                Instance = thingy.WithQueryTimeout(1000).Build();
            sw.Stop();
            Console.WriteLine("Cluster build took {0} seconds", sw.Elapsed);
            sw.Reset();
        }
        private static ConcurrentDictionary<string, byte> _oneConnectionPerKeySpace = new ConcurrentDictionary<string, byte>();
        public static ISession CassandraConnectUnwrapped(string keySpace)
        {
            if (_oneConnectionPerKeySpace.TryAdd(keySpace, 0))
            {
                return Instance.Connect(keySpace);
            }
            throw new InvalidOperationException("You have attempted to open more than one connection per key space. Best practice is one connection per key space per app");
        }
    }
    class Program
    {
        static void Main(string[] args)
        {
            var sw = new Stopwatch();
            sw.Start();
            var session1 = CassandraCluster.CassandraConnectUnwrapped("keyspace1");
            Console.WriteLine("session 1 instantiated in {0} seconds", sw.Elapsed);
            sw.Restart();
            var session2 = CassandraCluster.CassandraConnectUnwrapped("keyspace2");
            Console.WriteLine("session 2 instantiated in {0} seconds", sw.Elapsed);
            Console.ReadLine();
        }
    }
}

Results when run normally:

Cluster build took 00:00:00.1049711 seconds
session 1 instantiated in 00:00:00.3510474 seconds
session 2 instantiated in 00:00:00.0045223 seconds

Results when run with a node shutdown:

Cluster build took 00:00:00.1361451 seconds
session 1 instantiated in 00:00:05.4229149 seconds
session 2 instantiated in 00:00:05.0161207 seconds


Thanks
Dan

[João Reis]

Hi,

I think it makes sense to have that improvement on the session warmup process, I've created CSHARP-899 to track this [1].

Note that this would not speed up the initialization of the first Session instance since the driver would not see any node as DOWN at that point. After the first session is initialized, then the driver would see shutdown nodes as DOWN and any session created afterwards will benefit from this improvement.

Have you considered creating those sessions in parallel with the Cluster.ConnectAsync method or is that not possible in your scenario?

Another possible workaround is to disable the session warmup [2] and create a similar process "manually" by sending a simple query with the Session.Execute API (which will skip nodes that the driver sees as DOWN).

Do you absolutely need to create a session per keyspace? I'm interested in knowing more about the use case behind this requirement.

Thanks,

João Reis

[1] https://datastax-oss.atlassian.net/browse/CSHARP-899

[2] https://docs.datastax.com/en/drivers/csharp/3.14/api/Cassandra.PoolingOptions.html#Cassandra_PoolingOptions_SetWarmup_System_Boolean_

[Dan]

Thank you João, helpful as always.

Using Cluster.ConnectAsync might be possible, as is the other suggested work around. We will do some testing with both these options.
I think the use case is simultaneous access to more than one keyspace[1]

Thanks again
Dan

[1] https://github.com/datastax/csharp-driver/blob/master/src/Cassandra/ISession.cs#L35

[João Reis]

Glad to know that it might be possible to use those work arounds.

I know that this would require a lot of work so I'm not suggesting you should do this just because of this issue but in the future you might want to consider setting the keyspace on the mapping configuration if you are using Linq2Cql or the Mapper to generate the statements or specifying the keyspace in the actual statement string if you are using prepared statements directly. This way you can avoid creating multiple sessions (with all the extra resources associated with new session instances like connections).

If your schema and queries are different between keyspaces, then it's just a matter of specifying the appropriate keyspace on each statement or table mapping configuration.

// prefer

var ps = session.Prepare("SELECT * FROM ks1.table1 WHERE id1 = ?, id2 = ?");

// instead of

var ps = session.Prepare("SELECT * FROM table1 WHERE id1 = ?, id2 = ?");

If you have similar schema and similar queries for different keyspaces then it's a bit more difficult but you can still do it. With Linq2Cql, you can specify the keyspace when creating the Table instance. With the Mapper, you can not specify the keyspace when creating the instance but you can create multiple MappingConfiguration instances with the same mappings and different keyspaces. Then you can create one mapper instance for each MappingConfiguration (and every mapper would reuse the same session). With prepared statements you could do string concatenation/interpolation and prepare every statement for each keyspace.

Again, I'm not suggesting you rewrite all of this code just to deal with this issue but it's something to keep in mind for the future.

Thanks,

João Reis