cannot connect to redis sentinel using Lettuce if one of the sentinels is down

[Tuco]

Hi, 

We are unable to connect to redis-sentinel using Lettuce if we provide three sentinels and one of them is down.

It defeats the purpose of having a sentinel because if one of the machine goes down, then lettuce is unable to connect to sentinel.

However, if the server is up and running, then it works well, and it seems to give timeouts for new connections.

Below is the code(IP1,IP2,IP3 are ip addresses of form A.B.C.D).

public static void main(String[] args) throws Exception {

      RedisURI uri = RedisURI.create("redis-sentinel://IP1,IP2,IP3?sentinelMasterId=mymaster&timeout=60s");

      RedisClient redisClient = RedisClient.create();

      StatefulRedisMasterSlaveConnection<String, String> connection = MasterSlave.connect(redisClient,  new Utf8StringCodec(), uri);

       connection.setReadFrom(ReadFrom.MASTER_PREFERRED);

       int j = 0;

while(true){

System.out.println("=====  " + j++ + " th iteration ======" );

for(int i = 0 ; i < 4 ; i++) {

System.out.println(new Date() + " : "  + i + " = " + connection.sync().get(String.valueOf(i)));

connection.sync().set(String.valueOf(i), "Value" + String.valueOf(i));

}

Thread.sleep(10000);

}

}

It gives the error

Exception in thread "main" com.lambdaworks.redis.RedisConnectionException: Unable to connect to /IP3:26379

at com.lambdaworks.redis.AbstractRedisClient.initializeChannel(AbstractRedisClient.java:248)

at com.lambdaworks.redis.RedisClient.connectStateful(RedisClient.java:449)

at com.lambdaworks.redis.RedisClient.connectPubSub(RedisClient.java:515)

at com.lambdaworks.redis.masterslave.SentinelTopologyRefresh.lambda$bind$432(SentinelTopologyRefresh.java:61)

at java.util.ArrayList.forEach(ArrayList.java:1249)

at com.lambdaworks.redis.masterslave.SentinelTopologyRefresh.bind(SentinelTopologyRefresh.java:59)

at com.lambdaworks.redis.masterslave.MasterSlave.connectSentinel(MasterSlave.java:171)

at com.lambdaworks.redis.masterslave.MasterSlave.connect(MasterSlave.java:100)

at redisloadtest.SentinelMain.main(SentinelMain.java:121)

[Mark Paluch]

That's a bug. Can you file an issue in https://github.com/mp911de/lettuce?

I see however one caveat: Creating a connection to a Redis Server while one Sentinel node is down is possible by using other Sentinel nodes but the failed connection will remain disconnected and won't recover as the Sentinel Pub/Sub connections are established only once during the connect phase.

[Tuco]

not sure if i understand correctly, would you please elaborate on the caveat.

I have created the issue here.

[Mark Paluch]

Let's first take a look how Sentinel works for Standalone connections:

1. lettuce connects (any) Sentinel to retrieve the master connection details

2. Sentinel connection is closed

3. lettuce connects to the Redis server using the connection details

... much later...

4. Redis connection gets interrupted

5. lettuce connects (any) Sentinel to retrieve the master connection details

6. resume with 2.

For Master/Slave connections, the Redis Sentinel connections are kept open using Pub/Sub and auto-reconnect. This is, to listen actively to configuration changes, e.g. if a slave node is added or removed, if the master is reconfigured or a slave is promoted to master. The Redis Sentinel connections are opened at the time of the connection (MasterSlave.connect(...)). 

In reality, when you supply 3 Sentinel addresses and one of them is down, only two Sentinels are used to receive Pub/Sub messages. This is still enough to receive configuration updates but I wanted to mention it.

Please don't hesitate to ask if you have further questions.

[Tuco]

Thanks for the detail explanation. 

when the 3rd sentinel, which was down comes up, and lettuce auto-reconnects with it, shouldn't it start getting the pub-sub updates from other sentinel nodes?

[Mark Paluch]

Good catch. I created https://github.com/mp911de/lettuce/issues/306 to investigate on the possibilities. 

The Redis documentation mentions the +sentinel Pub/Sub message which could be used to dynamically add Sentinel servers once a new Sentinel is discovered.