Slave connected to the wrong master after fail-over through Redis Sentinel

[Sean Lo]

Hi All,

We met a weird situation in our Redis instances (v3.0.7). 

In normal cases, we have 9 Redis instances on 3 servers, and each server has 3 instance. And we use Redis Sentinel to ensure high availability for Redis. 

After all instances of Redis startup, we can see 3 masters, and each master has 2 slaves. But we found a weird situation... 

If 1 master crashed, we can see the fail-over process happened automatically, and 1 slave that belongs to the crashed master has been promoted to master, and the new master will connect to remained slave. 

However.. the new master not only connect to its own slave.. it also connect another 2 slaves from other masters. 

So, after fail-over happened, the new master will connect 3 slaves. Other masters only connect 1 slave. 

We can't find the root cause of this situation. Could anybody can help to explain it? 

Thanks.

[Greg Andrews]

Are you using the sharded Redis Cluster configuration, or are you using the non-sharded Redis configuration?  (3 masters is such a common configuration with the sharded Redis Cluster that it's good to double check)

If you're using the non-sharded Redis, what are the names you've given to the three Redis masters in your Sentinel configuration?

[Jose Moreira]

Perhaps the other slaves are all sharing the same sentinel "master name" which should be unique and it's getting confused when searching for the slaves. 

My understanding is if the sentinel conf file is correct and is listing 1 master, 2 slaves (the right ones) and a unique master name, it should never include another servers not listed on the file. But if so and you can reproduce it, then it's a bug. But I never heard that before...

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/2fb163b1-e674-47f2-a108-af71d2df338f%40googlegroups.com.

[Sean Lo]

Dear Greg,
Thanks for your reply. After double checked with my colleague. Seems we didn't use Sentinel in our environment. We use Redis Cluster to ensure HA. Sorry for the wrong question..

So... This situation is happened in Redis Cluster. Should I provide our configuration or any information to clerify the situation?

Very appreciated and sorry again for wrong question.

Thanks.

[Sean Lo]

Dear Jose,
Also sorry for asking wrong question. And very thankful for your reply.

Could I provide any information to clerify the situation?

Thanks.

[Greg Andrews]

The behavior you describe, where slaves of other shards are taken away from the other shards and configured to be slaves of this one shard, is very different from Redis Cluster.  But it's very similar to Sentinel.

The pattern here still matches up with a Sentinel interfering with Redis Cluster.  There really, truly are no Sentinels running at all in your network?

[Itamar Haber]

As a related remark, v3.0.7 is quite ancient and no longer maintained. Furthermore, numerous fixes were applied to the cluster since then. I strongly recommend an upgrade of that environment.

Cheers,

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/6e836153-4e61-4817-b488-6db5724a1144%40googlegroups.com.

--

Itamar Haber Technicalist Evangely Phone: +972.54.567.9692

[Sean Lo]

Dear Greg,

Sorry for the late reply. And my colleague confirmed again that we don't use Sentinels in our environment. 

Thanks.

Itamar Haber於 2019年9月7日星期六 UTC+8下午8時31分49秒寫道：

As a related remark, v3.0.7 is quite ancient and no longer maintained. Furthermore, numerous fixes were applied to the cluster since then. I strongly recommend an upgrade of that environment.

Cheers,

On Sat, Sep 7, 2019 at 3:20 PM Greg Andrews <hva...@gmail.com> wrote:

The behavior you describe, where slaves of other shards are taken away from the other shards and configured to be slaves of this one shard, is very different from Redis Cluster.  But it's very similar to Sentinel.

The pattern here still matches up with a Sentinel interfering with Redis Cluster.  There really, truly are no Sentinels running at all in your network?

On Friday, September 6, 2019 at 6:53:46 PM UTC-7, Sean Lo wrote:

Dear Greg,
Thanks for your reply. After double checked with my colleague. Seems we didn't use Sentinel in our environment. We use Redis Cluster to ensure HA. Sorry for the wrong question..

So... This situation is happened in Redis Cluster. Should I provide our configuration or any information to clerify the situation?

Very appreciated and sorry again for wrong question.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/6e836153-4e61-4817-b488-6db5724a1144%40googlegroups.com.

[Sean Lo]

Dear Itamar,

Thanks for the reply. We have proposed the upgrade plan for our superiors. But they hope us to find the root cause.

Thanks. 

Itamar Haber於 2019年9月7日星期六 UTC+8下午8時31分49秒寫道：

As a related remark, v3.0.7 is quite ancient and no longer maintained. Furthermore, numerous fixes were applied to the cluster since then. I strongly recommend an upgrade of that environment.

Cheers,

On Sat, Sep 7, 2019 at 3:20 PM Greg Andrews <hva...@gmail.com> wrote:

The behavior you describe, where slaves of other shards are taken away from the other shards and configured to be slaves of this one shard, is very different from Redis Cluster.  But it's very similar to Sentinel.

The pattern here still matches up with a Sentinel interfering with Redis Cluster.  There really, truly are no Sentinels running at all in your network?

On Friday, September 6, 2019 at 6:53:46 PM UTC-7, Sean Lo wrote:

Dear Greg,
Thanks for your reply. After double checked with my colleague. Seems we didn't use Sentinel in our environment. We use Redis Cluster to ensure HA. Sorry for the wrong question..

So... This situation is happened in Redis Cluster. Should I provide our configuration or any information to clerify the situation?

Very appreciated and sorry again for wrong question.

Thanks.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.

To unsubscribe from this group and stop receiving emails from it, send an email to redi...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/redis-db/6e836153-4e61-4817-b488-6db5724a1144%40googlegroups.com.