Multi-master redis cluster with zookeeper and redis_node_manager

[chithaluri kiran]

Hi All,

I am working on Redis-3.0.1 to configure redis in cluster mode. I was able to configure the cluster successfully. Below is my configuration with two linux servers(Server-A,Server-B)

Server-A:6379 Server-B:6384

Server-A:6380 Server-B:6385

Server-A:6381 Server-B:6386

Server-A:6382 Server-B:6387

Server-A:6383 Server-B:6388

I have used redis-trib.rb to create the cluster as shown below with one replica:

./redis-trib.rb  create --replicas 1 Server-A:6379,Server-A:6380,Server-A:6381,Server-A:6382,Server-A:6383,Server-B:6384,Server-B:6385,Server-B:6386,Server-B:6387,Server-B:6388

The above command has configured my cluster with 5 masters & 5 slaves.

Below are the masters:

Server-A:6379  Server-B:6384

Server-A:6380             Server-B:6385

                                         Server-B:6386

Cluster is up and running fine. 

Now , I have a zookeeper cluster up & running on another servers (Server-C,Server-D,Server-E)

My config.yml file content

---------------------------------------------

:max_failures: 2

:node_strategy: majority

:failover_strategy: latency

:required_node_managers: 3

:nodes:

  - Server-A:6379

  - Server-B:6384

  - Server-A:6380

  - Server-B:6385

  - Server-A:6381

  - Server-B:6386

  - Server-A:6382

  - Server-B:6387

  - Server-A:6383

  - Server-B:6388

:zkservers:

  - Server-C:5181

  - Server-D:5181

  - Server-E:5181

---------------------------------------------

I have installed redis_failover gem along with its dependencies.

When I try to configure redis_node_manager with the zookeeper , I am getting below error:

--> redis_node_manager -C config.yml

2015-05-28 22:21:44 UTC RedisFailover 31996 INFO: Redis Node Manager v1.0.2 starting (ruby 1.9.3p448 (2013-06-27 revision 41675) [x86_64-linux])

2015-05-28 22:21:44 UTC RedisFailover 31996 INFO: Monitoring redis nodes at (10.18.48.178:6379, 10.18.48.178:6380, 10.18.48.178:6381, 10.18.48.178:6382, 10.18.48.178:6383, 10.18.50.178:6384, 10.18.50.178:6385, 10.18.50.178:6386, 10.18.50.178:6387, 10.18.50.178:6388)

2015-05-28 22:21:44 UTC RedisFailover 31996 INFO: Waiting to become master Node Manager ...

2015-05-28 22:21:44 UTC RedisFailover 31996 INFO: Acquired master Node Manager lock.

2015-05-28 22:21:44 UTC RedisFailover 31996 INFO: Configured node strategy RedisFailover::NodeStrategy::Majority

2015-05-28 22:21:44 UTC RedisFailover 31996 INFO: Configured failover strategy RedisFailover::FailoverStrategy::Latency

2015-05-28 22:21:44 UTC RedisFailover 31996 INFO: Required Node Managers to make a decision: 3

2015-05-28 22:21:44 UTC RedisFailover 31996 WARN:  Failed to discover master node: #<RedisFailover::MultipleMastersError: Multiple nodes with master role: ["10.18.48.178:6379", "10.18.48.178:6380", "10.18.50.178:6384", "10.18.50.178:6385", "10.18.50.178:6386"]> In order to ensure a safe startup, redis_failover requires that all redis nodes be accessible, and only a single node indicating that it's the master. In order to fix this, you can perform a manual failover via redis_failover, or manually fix the individual redis servers. This discovery process will retry in 5s.

My Questions:-

Can't we keep multiple masters, and redis_node_manager select the current master for playing the master role?

What is the fix for this ?

Help is much appreciated. Thanks in advance.

Thanks,

Kiran

[Jan-Erik Rediger]

If you run it in Cluster mode, you don't need an external service
handling the failover. Redis Cluster will do this on its own.
Every external service will cause more harm then do good.

On Thu, May 28, 2015 at 04:11:16PM -0700, chithaluri kiran wrote:
> Hi All,
>
> I am working on Redis-3.0.1 to configure redis in cluster mode. I was able
> to configure the cluster successfully. Below is my configuration with two
> linux servers(Server-A,Server-B)
>

> *Server-A:6379 Server-B:6384*
> *Server-A:6380 Server-B:6385*
> *Server-A:6381 Server-B:6386*
> *Server-A:6382 Server-B:6387*
> *Server-A:6383 Server-B:6388*

>
> I have used redis-trib.rb to create the cluster as shown below with one
> replica:
>
> ./redis-trib.rb create --replicas 1
> Server-A:6379,Server-A:6380,Server-A:6381,Server-A:6382,Server-A:6383,Server-B:6384,Server-B:6385,Server-B:6386,Server-B:6387,Server-B:6388
>
> The above command has configured my cluster with 5 masters & 5 slaves.
> Below are the masters:
>

> *Server-A:6379 Server-B:6384*
> *Server-A:6380 Server-B:6385*
> * Server-B:6386*

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[Salvatore Sanfilippo]

All Jan-Erik said plus, a friendly advice: if you missed this about
Redis Cluster, you don't have enough information to successfully run
it in production.
Return back to the documentation and read everything in order to
understand what it is, how it works, ... otherwise you are guaranteed
to have issues.

Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - Pivotal http://pivotal.io

"If a system is to have conceptual integrity, someone must control the
concepts."
— Fred Brooks, "The Mythical Man-Month", 1975.