Application HAProxy issue with Redis sentinels

[Amit Bondwal]

Hi Everyone,

I setup three node redis sentinels with one master and two slave with quorum of 2.

Set up haproxy in front of redis. I used a lot of setting for haproxy like below link.

https://jambr.co.uk/2015/07/23/redis-sentinel-behind-haproxy/
http://blog.haproxy.com/2014/01/02/haproxy-advanced-redis-health-check/

Now issue is that to test this setup when I manually shut down master server in redis
it automatically promote the one of slave server to master after some time. but
my application gives the error  not connected to any server.

"ERROR:1110221 Not connected to any server at /usr/local/share/perl/5.20.2/Redis.pm line 890"

To run it again I have to restart my app again refresh of page don't work.

My application is in perl catalyst framework.
haproxy version:- 1.5.8-3+deb8u2 
redis-server version :- 2.8.17-1+deb8u3
OS is debian jessie, and I installed package from repo.

How can I solve this. Where is the issue, is it with haproxy or with my app?

--
Thanks

Amit Bondwal

[Greg Andrews]

Using Haproxy and Sentinel together has come up a couple of times in this mailing list in the past couple of months.  Here are links to the threads:

https://groups.google.com/d/topic/redis-db/p98BE3FyJy8/discussion  (29 posts in the thread, but the last 5-6 highlight some issues)
https://groups.google.com/d/topic/redis-db/RcI57a-Z9sk/discussion  (3 posts in the thread)

I believe there are two important problems with mixing Haproxy and Sentinel:

• Haproxy's master/slave detection mechanism knows nothing about Sentinel's monitoring and control of the Redis configurations
• Sentinel knows nothing about Haproxy choosing to send connections to Redis instances

In essence, Haproxy was not written to work cooperatively with Sentinel, and Sentinel was not written to work cooperatively with Haproxy.

I know there is a blogger on the Internet wrote a post saying this will work.  However, as I said in the 3-post thread above, I don't see how they will.  As both of the discussion threads describe, trouble arises as soon as a master fails or starts back up again.

In my view it's not best to use both Haproxy and Sentinel.  Use one or the other.

  -Greg

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[The Real Bill]

As someone with first hand expeeienx with over a thousand setups of haproxy with sentinel I will have to disagree entirely. Again the mistake lies in thinking haproxy can automatically interact with sentinel on a reliable basis. It can not. However there is a myriad of ways to make it work ranging from a simple shell script which the sentinel leader executes to reconfigure haproxy, or something as elegant as and consul-template managing haproxy. To say it can't be done is demonstrably false. While not designed to work specifically together each system has a design that allows the two to work together. The rub is it requires some sort of third party tool. But those tools are easy to write and already exist.

Sure the people saying to have haproxy do a role check are being shortsighted and promoting a broken configuration. Sure the people promoting haproxy checking in with a sentinel aren't doing any better. And sure the people saying to have haproxy do a check against all sentinels still get it wrong given haproxy' behavior on backend failures. But to say that simply because haproxy isn't suitable to manage itself you can't use it is to throw the baby out with the bath water.

As to the original problem you'll experience this if the client library can not reliably reconnect and it has nothing to do with haproxy or sentinel. From the information presented the perl client library does not appear to properly handle reconnection so you will need to catch the "not connected" error and have your code reconnect on that error.

Of course you'll need to implement a circuit breaker and back off mechanism so as to not flood reconnection attempts or be stuck in a loop trying to reconnect. But fundamentally that is what you need to do: detect connection failure and reconnect. You should also see this same behavior if you connected directly to a Redis server and after establishing and using the connection restart the Redis instance. This will cause a disconnect which ideally the client library catches or at least passes up the specific error so your code can reconnect.

Even some client libraries which retry will only retry once and will do so immediately. Of course if the service isn't back immediately it propagates a disconnected error and can look like it doesn't retry. As such you should've prepared to implement a mechanism to manage that reconnection attempt to fit your specific requirements.

I haven't touched perl in about two decades, much to my coding nirvana, so other than the basic principles of what needs to be done, I am unable to give specific aid in Doing so in that language. Hopefully knowing what is happening and what to do to catch and handle the condition is enough to point you in the right code direction.

Cheers and good luck,
Bill

[Bill Anderson]

Wow thanks autocorrect. ;) experience not whatever that was, and mucking not kicking

> --
> You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/31tpxKaCLh4/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.

[Greg Andrews]

The Real Bill <ucn...@gmail.com> wrote:

As someone with first hand experience with over a thousand setups of haproxy with sentinel I will have to disagree entirely. Again the mistake lies in thinking haproxy can automatically interact with sentinel on a reliable basis. It can not.

My exposure to the combination of Haproxy and Sentinel has been strictly from the threads posted to this list and the one or two blogs cited in the threads.  And you describe them perfectly:  Those threads and blogs say it can, when the reality is that it can not.  Not with the superficial configurations described by the blogs and the e-mail threads.

However there is a myriad of ways to make it work ranging from a simple shell script which the sentinel leader executes to reconfigure haproxy, or something as elegant as and consul-template managing haproxy. To say it can't be done is demonstrably false. While not designed to work specifically together each system has a design that allows the two to work together. The rub is it requires some sort of third party tool. But those tools are easy to write and already exist.

I'm glad to hear that.  I'll point out that I've never said, "This can't be done."  I've cited the Haproxy/Sentinel configs described in the threads - which lack the additional script/tools you mention - and said, "I can't see how it will work."

Thank you Bill, for describing how it actually can work.

  -Greg

[Amit Bondwal]

Thanks you very much bill, I enable reconnect in perl redis library and it works like a charm.
Thanks again.