HAProxy with redis master-slave replication and Sentinel

[Adi Chiru]

Hi guys,

I've been testing the above scenario for quite a while now and, although haproxy v1.5-dev20 and newer added support for redis (option redis-check to go further to check redis protocol not only tcp) there is still a problem with the setup.

Sentinel is working as expected and, although it take about 8 seconds, it will eventually fail-over properly, according to my tests.

HAProxy configured with:

.....

backend b_redis

    balance leastconn

    option tcplog

    option tcp-check

#  tcp-check connect port 26379

    tcp-check send PING\r\n

    tcp-check expect string +PONG

#  tcp-check expect string PONG

#  option redis-check

    tcp-check send info\ replication\r\n

#  tcp-check send sentinel\ masters\r\n

    tcp-check expect string role:master

#  tcp-check expect string "redis-master"

    tcp-check send QUIT\r\n

    tcp-check expect string +OK

    server redis1 10.0.0.1:6379 maxconn 1024 check inter 1s rise 30

    server redis2 10.0.0.2:6379 maxconn 1024 check inter 1s rise 30

...............

will detect the new master and everything works beautifully.

If the failed redis-master would die in any way I was able to test, except for a net partitioning, everything looks good.

However, if I just isolate the redis-master from a network perspective (like an iptables rule to drop all packages to it) the process remains up and running (same pid, same run_id) and unaware that sentinel promoted another slave to replace it. I do run sentinel on each redis node including the initial master and I do not block sentinel communication.

Once I remove the iptables rule and make the old/initial master available again to the haproxy checks, haproxy will see it as a valid master and start sending requests to it. Sentinel will detect eventually that the old master re-became available and will reconfigure it as slave but a little bit too late, definitely later than haproxy will detect and start using it with the above configuration.

As far as I know now, there are 2 solutions:

1. increasing the rise in haproxy config to about 30; this combined with inter 1s makes haproxy consider the newly detected master as a valid backend server after 30 seconds which gives sentinel enough time to make reconfigure this master to be slave and so the problem would be solved. However, the 30 seconds delay will always happen, in all other failure scenarios so now you have HA but the fail-over is very slow (to be honest I would have expected for the sentinel to fail-over in less than a second...) and also, you can never be 100% sure that the 30 seconds delay will be long enough...

2. make a script (ran as a daemon) that constantly queries a sentinel instance (which one? or maybe all?), finds the current master, changes the haproxy configuration and soft-reloads haproxy - not a good idea at all in my opinion - many things can go wrong here and it is not really elegant either

I tried making HAProxy health-check sentinels and not redis masters and it can be done but all sentinel instances will have exactly the same data so all of them will output a sting that will show "they" have the master role. HAProxy at this moment can't reconfigure itself based on info (like the IP of the current redis master) that sentinel can give.

I agree that this is more a HAProxy issues/opportunity to improve it's redis support but maybe people here more familiar with redis came up with a better idea for this.

What do you think?

Thanks,

Adi

[Josiah Carlson]

I was going to advise using twemproxy, but the existence of https://github.com/twitter/twemproxy/issues/67 from a year ago shows that they still don't have Sentinel support.

Out of curiosity, why are you using HAProxy?

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Adi Chiru]

It's just a technology already in use in the company. We also have
Nginx which I rather prefer but since HAProxy will support sentinel
natively I thought it's a good idea to use it.

What other options I have?

I could go with some python or shell scripts too but it does not seem
like a better approach except for the fact that I will have more
control.

Any other ideas on how to make a reverse proxy work with sentinel?

[Josiah Carlson]

I'm sorry, I should have asked a better question that was clearer regarding what I actually wanted to know (which may help me or others answer the question better): Why do you need a reverse proxy?

As for actually making it work, sentinel has built-in support for handling configuration changes via script. That script can alter whatever reverse proxy configuration you have/need, or can push configuration changes to another configuration management system that actually handles updating your configs.

 - Josiah

[Adi Chiru]

Thanks Josiah,

1. I need a reverse proxy to hide the master change (during a
failover, when a slave is promoted to master) from my application
layer - I do not believe that manually or automaticatlly updating DNS
entries is a good idea for this. Basically all my clients need to know
one single fqdn/IP and port that represents a redis service,
regardless of what redis instance is behind it as master.

2. Yes, that is correct - I can use the reconfiguration script with
the parameters received from sentinel to update a config on my reverse
proxies. However there are these drowbacks/limitations/risks:
- I will have at least 2 reverse proxy servers so that they are not a
single point of failure so now the script must make sure that it
updates the info into haproxy.cfg (or whatever reverse proxy I use)
consistently - implementing atomic change sets in scripts is not that
trivial
- I will need to do these updates on remote hosts - meaning ssh with
keys and so one - lots of things to go wrong
- using the script to update a config file implies a restart of
haproxy which means cutting all connections (I just found out that
HAProxy has a soft restart / reload feature too so this may not be a
major issue)

Thanks,
Adi

[Matt Stancliff]

On May 8, 2014, at 8:07 PM, Adi Chiru <adic...@gmail.com> wrote:

> 1. I need a reverse proxy to hide the master change (during a
> failover, when a slave is promoted to master) from my application
> layer - I do not believe that manually or automaticatlly updating DNS
> entries is a good idea for this. Basically all my clients need to know
> one single fqdn/IP and port that represents a redis service,
> regardless of what redis instance is behind it as master.

Sentinel can notify clients of a failover. The auto-notification allows clients to switch to the new IP without any external actions. That’s the most efficient way to deal with locating the current Redis master.

For figuring out which master to use on startup, see the "SENTINEL get-master-addr-by-name <name>” command.
If your client supports Sentinel, it should support subscribing to the "switch-master” channel. That channel gets a message each time a new master is promoted. The published message contains all the new connection details you need to use the newly promoted master instance.


-Matt

[Josiah Carlson]

Ahh, but if you ran Redis sentinel on your HAProxy boxes (which are the only machines that seem to *need* to know what the masters are, in addition to the slaves), then you don't need SSH, public keys, etc. All you need is a template HAProxy configuration, and a script to inject those lines into the template and overwrite the old config, followed by a SIGHUP to reload the config in HAProxy.

In the past, reloading the config reset the stats. IIRC, modern HAProxy no longer resets stats after a SIGHUP config reload.

And regarding DNS; aside from the fact that existing connections don't die, where's the drawback? You *may* need to reload configuration after a DNS switch, but that can be accomplished with a < 20 line Python daemon that polls DNS every second for changes. You then need to write your DNS update script to complete the circle.

But if we take a little hint from Matt, you could trivially run a couple 20-line Python daemons on the HAProxy servers that listens to the master update channel on the sentinel for the updated master address. You can then perform your config rewrite + SIGHUP. But maybe the first option with Sentinel + a script is better.

 - Josiah

[Adi Chiru]

What Matt said is new to me but I am not sure that all our clients are
properly implemented to be able to work in that way.

As for your idea, it was one of the scenarios but the smallest setup
we have is 1 redis master, 1 redis slave and 2 haproxy. That would
mean that I will only have 2 sentinels - which is not a good idea due
to the way quorum works and even if it would be possible with the 2
sentinels, one on each haproxy system, I still need to allow one
sentinel (the one that becomes the leader) to change the config of the
other haproxy - so via ssh.
It is possible now, that I think of this, to make a python daemon that
would take care of this and update haproxy config only locally....

Anyway, except for Matt's hint, all the other solutions involving
scripts and daemons shows that there is a bit of code missing
somewhere. I guess what Matt said, combined with the reconfiguration
script sentinel can run make redis server + redis sentinel a pretty
complete solution....


Thanks guys!