Redis cluster unavailable after slave goes away

55 views
Skip to first unread message

Nikos Pyrgiotis

unread,
Jun 29, 2021, 8:18:24 AM6/29/21
to Redis DB
Hello,

I have created a redis cluster in a testing environment with 3 nodes. Each node runs 3 instances of redis, one master and 2 slaves for each master, 9 in total with the following commands.

```
- Create cluster with 3 masters:<br>`redis-cli --cluster create 10.0.23.229:6379 10.0.23.241:6379 10.0.23.175:6379`
- Add slaves to node 1:
`redis-cli --cluster add-node --cluster-slave --cluster-master-id d83cb0988ae63cbd11868fae87aee4e66668b9da 10.0.23.241:6378 10.0.23.175:6379`
`redis-cli --cluster add-node --cluster-slave --cluster-master-id d83cb0988ae63cbd11868fae87aee4e66668b9da 10.0.23.229:6377 10.0.23.175:6379`
- Add slaves to node 2:
`redis-cli --cluster add-node --cluster-slave --cluster-master-id ea2c9f01271367404e5b02bb0646c1645a27a01f 10.0.23.175:6380 10.0.23.241:6379`
`redis-cli --cluster add-node --cluster-slave --cluster-master-id ea2c9f01271367404e5b02bb0646c1645a27a01f 10.0.23.229:6378 10.0.23.241:6379`
- Add slaves to node 3: <br>
`redis-cli --cluster add-node --cluster-slave --cluster-master-id 80cb4f04098723e15546d6ae3efa71c0bb49e47d 10.0.23.175:6381 10.0.23.229:6379`
`redis-cli --cluster add-node --cluster-slave --cluster-master-id 80cb4f04098723e15546d6ae3efa71c0bb49e47d 10.0.23.241:6380 10.0.23.229:6379

All redis-instances have similar configurations, they run on debian buster and redis version is 6.0.14.

When i want to perform maintenance on any node(reboot, upgrade of redis , etc ...) i failover the master role to a slave in one of the other 2 nodes.
The problem is when i stop any slave instance the whole redis cluster is shown as unavailable(offline) in redisinsight. I have the same problem with kamailio software that uses hiredis client library, when i stop a slave the client loses connectivity with the redis database. From my understanding redis cluster should be operational for reads and writes if all masters are up even if a slave goes away.

  • Are there any configuration options that need to be changed to enable the expecting behavior?
  • Might it be an issue with the redis cluster client implementation?

Any feedback would be appreciated. Thank you

Nikos
Reply all
Reply to author
Forward
0 new messages