Sentinel problem

[danilo...@olxbr.com]

Good afternoon,

     Can anyone help me with the error below?

16385:X 11 Jul 17:13:27.661 # +new-epoch 11

16385:X 11 Jul 17:13:27.661 # +try-failover master mymaster 10.49.1.223 6389

16385:X 11 Jul 17:13:27.664 # +vote-for-leader a066b75b746744dca7263f39ef32765928baac2d 11

16385:X 11 Jul 17:13:38.452 # -failover-abort-not-elected master mymaster 10.49.1.223 6389

16385:X 11 Jul 17:13:38.518 # Next failover delay: I will not start a failover before Mon Jul 11 17:19:28 2016

Thanks!

[Shaun Davis]

How many sentinels are you using and with what quorum?  Also can you confirm that each of your sentinels are subscribing to your master / slave nodes?  

You should be able to see these by running CLIENT LIST in your redis-cli.

[Sumanth Kondaveeti]

Hi guys,

I am also having the same problem. I used two servers one master and I made redis in other server as slave. I configured 3 sentinels for Master. The Master Redis and 3 Sentinels are in same machine. Is the configuration what I have done is correct? 

1. I am using In Java Application with java-client lettuce. 

2. Master is conneced to slave, But the fail-over doesn't happen as I tested It by stopping the server one when I am running my Application.

3 can any one please help me out how to configure sentinel to make slave as master during fail-over.

Thanks,

Sumanth

[Shaun Davis]

Here is a script I hacked together that may or may not help you.  

https://gist.github.com/davissp14/a2ef2fb500ab73777619b0c7d777ba68

** Use at your own risk ** 

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/NDiVuW-2e_o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[The Real Bill]

You put the three sentinels on the same machine as the master? So when that system goes down no sentinels will be running to promote the slave. You really need to priced better information for anyone to help you.

Post the configs for each sentinel. Not the ones you made for it, but the file it created after it connected to the master.
Post the full lots of the event in question. If they are long put them in a gist on github and post the link. Don't assume you can smartly trim it down. If you knew what was important in the log you would have already properly analyzed them.

Cheers,
Bill

[Sumanth Kondaveeti]

Hi Guys,

Yes I know what I am doing Is not correct. I want to know the exact way how to configure sentinel. I read the sentinel documentation, but it could not help me and I am working on different things but now sentinel is priority. 

I am attaching my three conf files .

The senario is I have four application servers. The four server hits one Redis server. I am planning to have a backup redis server through sentinel. What is the best way to configure it. One slave is enough for  Master inorder sentinel to work. I am using redis 3.2 in windows machine.

In prod the servers used for redis in linux. Application servers are windows.

thanks

Sumanth.

On Monday, July 11, 2016 at 4:36:24 PM UTC-4, danilo...@olxbr.com wrote:

[Sumanth Kondaveeti]

Hey Bill, 

I am using lettuce as Java client In my Application.

Thanks

On Monday, July 11, 2016 at 4:36:24 PM UTC-4, danilo...@olxbr.com wrote:

[The Real Bill]

Ok, so your comfits are certainly messed up. Follow this procedure:

1. Stop all sentinels

2. The sentinel config files?  Wipe them out. All of them

3. Make the following your config for each: "port 26379" and nothing else.

4. Start them all up.

Now run:

redis-cli -h <ip of sentinel1> -p 26379 sentinel monitor myredis <ip-of-master> <port-of-master> 2

redis-cli -h <ip of sentinel2> -p 26379 sentinel monitor myredis <ip-of-master> <port-of-master> 2

redis-cli -h <ip of sentinel3> -p 26379 sentinel monitor myredis <ip-of-master> <port-of-master> 2

This will configure them to monitor the master.

Next check to see I they detect each other. If you cat/tail the config files you should see the known-sentinel entries. These entries should contain the IP and port of the other sentinels. You can also do:

redis-cli -h <ip-of-master> subscribe __sentinel__:hello.

And you should see each sentinel sending a ping command every couple seconds. This is how they discover each other. If this isn't happening you have a networking/firewall problem. If the above doesn't solve your problem it is at a system level. Unfortunately I can't help with Redis on Windows as I don't use Windows. The above will correctly configure the sentinels. Again, ensure they are *not* all running on the same machine as that is really pointless and will cause problems of its own as you'll have three processes assuming they have complete ownership of certain files.

Also when posting configs it is a good idea to strip out comments. ;)

Cheers,

Bill

[Sumanth Kondaveeti]

Thanks Bill. I will give a try and let you know.

On Monday, July 11, 2016 at 4:36:24 PM UTC-4, danilo...@olxbr.com wrote:

[Sumanth Kondaveeti]

Hey Bill,

I am planning to try with two Redis server and three Sentinels. For example: Two redis on two different VMs. Three sentinels on different VMs.

Example: 10.0.0.66(Redis1) , 10.0.0.67(Redis2)  and Sentinels on three different VMs(examples: 10.0.0.66(sentinel1), 10.0.0.67(sentinel2), 10.0.0.68(sentinel3) )

Is it correct way of configuring redis with sentinel.

1) I have a question, In the above example should Redis1 and Redis2 should be Master-slave or It should be independent Redis servers without any Master-Slave configurations?

2) or Should Redis2 should be Slaveof Redis1 Instance.

This Is the major doubt I have as different blogs say differently.I want to know the high-level-Architecure before I start as we have four servers In production.  I hope you understood my question. 

Thanks,

Sumanth

On Monday, July 11, 2016 at 4:36:24 PM UTC-4, danilo...@olxbr.com wrote:

[Bill Anderson]

1) I would avoid having sentinels on the same nodes as Redis instances. This is known to have odd issues where it fails to failover under circumstances which haven't quite been pinned down beyond "well it worked for the first failover, but ...".

2) It doesn't matter which is master and which is slave. Just point the sentinels to whichever one you make the master. This means 5 servers: 3 sentinels + 2 Redis.

So assuming :

 a) redis instances are .66 and .67, 

 b) sentinels are .68-.70, 

 c) picking .66 as master

 d) redis password is to be: dontusethis

 e) ports are all default

1.   Start everything up with no custom configs other than password and any needed bind/port statements

2.   redis-cli -h 10.0.0.67 -a dontusethis config set masterauth dontusethis

3.   redis-cli -h 10.0.0.67 -a dontusethis slaveof 10.0.0.66 6379

4.   verify it connects and becomes a slave

5.   redis-cli -h 10.0.0.68  -a -p 26379  sentinel monitor pod-name 10.0.0.66 6379 2

6.   redis-cli -h 10.0.0.68  -a -p 26379  sentinel set pod-name auth-pass dontusethis

7.   redis-cli -h 10.0.0.69  -a -p 26379  sentinel monitor pod-name 10.0.0.66 6379 2

8.   redis-cli -h 10.0.0.69  -a -p 26379  sentinel set pod-name auth-pass dontusethis

9.   redis-cli -h 10.0.0.70  -a -p 26379  sentinel monitor pod-name 10.0.0.66 6379 2

10. redis-cli -h 10.0.0.70  -a -p 26379  sentinel set pod-name auth-pass dontusethis

This will configure the system properly. It will expect a quorum of two, each sentinel will discover via the master's channel each other and the slave.  Be sure that none of your configuration management tries to manage the redis.conf or the sentinel.conf files. This is doubly true for the sentinel.conf files.

That is as simple as it gets. If the above doesn't work it has nothing to do w/sentinel or Redis. 

Cheers,

Bill

--

[Sumanth Kondaveeti]

Thankyou Bill. I will let you know If I have any issues.

Thanks,

Sumanth.

On Monday, July 11, 2016 at 4:36:24 PM UTC-4, danilo...@olxbr.com wrote:

[sys-ops]

Hi The Real Bill,

I have posted my question in
http://redis-db.2338650.n4.nabble.com/Redis-Sentinel-cluster-failover-results-in-failover-abort-not-elected-master-td7523.html
which seams to be related or the same as this one.

Could you please take a look at it. 3 redis instances and 3 sentinels, all
on separate hosts with no firewall between them. Failover, for some reason,
does not work and sentinels do not talk to the master node but to one of the
slaves. Sentinels elect this slave for master promotion, but they do not
perform the failover.

I posted all details in the above topic. If needed, I can post them here.

Regards,

Daniel Andrzejewski



--
Sent from: http://redis-db.2338650.n4.nabble.com/