How to specify redis connectionstring for master-slave servers ,which can work while one server is down and one is up

[shyamal....@gmail.com]

Hi,

I have  2 redis server installd on 2 different machines. and also both are master-slave configured.i also configure sentinel which make one slave server  as master if previous master down.

my connection string is like below

<sessionState mode="Custom" customProvider="RedisSessionStateStore"

        timeout="5">

        <providers>

          <add connectionstring="192.168.1.7:6379,192.168.1.4:6379" connectionTimeoutInMilliseconds="5000"

            accessKey="" ssl="false" throwOnError="true" abortConnect="false"

            name="RedisSessionStateStore" type="Microsoft.Web.Redis.RedisSessionStateProvider,Microsoft.Web.RedisSessionStateProvider" />

        </providers>

      </sessionState>

when my local machine's(192.168.1.4:6379) redis server is down then sentinel makes 192.168.1.7:6379 as master .but the problem is my application only works when

my local machine's redis server is running . it is not working while my local redis server is down and other is up inspite specifying in connection string

Thanks

Shyamal

[shyamal....@gmail.com]

   quick response will be appreciated....!

Thanks

Shyamal

[The Real Bill]

You don't, as this isn't how Redis is designed. For having failover (with is functionally what you're talking about here), you'll need to either set up Sentinel, or set up a TCP proxy which you can configure directly. I can not, however, help with the latter as I'm not a Windows person.

If that seems to complicated for what you are trying to do (which I suspect is likely), that is because what you are doing isn't the way Redis is designed.

Cheers,
Bill

[Shaun Davis]

A good way to handle this is to use an Haproxy and its advanced redis health checks.  Here is a good example.  

http://blog.haproxy.com/2014/01/02/haproxy-advanced-redis-health-check/

[Bill Anderson]

No it isn't. The health check written there won't work properly right when you need it to. It exposes you to split brain syndrome. When more than one node reports itself as master, which will happen during normal failover and recovery operations, haproxy will not be able to tell the difference. 

As a result you will have data in both places and no way to merge them at best, and data loss otherwise. As a result no one should ever recommend that setup if having correct data matters.

At best you could leverage sentinel's client reconfigure script to rewrite a hosts file and use the name specified in the hosts file in the connection config. Assuming that all works on Windows. I know windows can use a hosts file, so it is a matter of the reconfigure script working and the app doing the proper lookups when it needs to. 

That said the overall use case is not a suitable one for sentinel anyway. You have two nodes and apparently need them both to be master when disconnected from each other. Which means you can't merge data correctly when the laptop returns to the network. This isn't a situation Redis or sentinel are designed for. 

Cheers,

Bill

On Jul 11, 2016, at 11:01, Shaun Davis <davi...@gmail.com> wrote:

A good way to handle this is to use an Haproxy and its advanced redis health checks.  Here is a good example.  

http://blog.haproxy.com/2014/01/02/haproxy-advanced-redis-health-check/

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/VUCztDRodC0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[Shaun Davis]

If the sentinels are properly configured, split brain shouldn't happen eh?  The master_link_status on the slave would go DOWN, but it wouldn't be considered to take over master until the "down-after-milliseconds" threshold is hit and the election process happens.  Basically I don't think this should be an issue unless there's a misconfiguration.  Seems like the worst case you may see some connection timeouts from the haproxy while the sentinels are doing their thing, but that would depend on the sentinel configuration.  

Am I missing something?  

[Shaun Davis]

Sorry, looks like I may have misinterpreted the question.  Carry on.  

[Bill Anderson]

It is impossible for Sentinel to be configured to prevent split-brain.

There are several scenarios where it can happen, one of which is pretty much guaranteed to occur so I'll go into it first: Recovery.

Scenario: Master dies, slave promoted, master comes back.

When the old master comes back it will start up as a master. It does this because last it knew, it was the master. it has to be up and running before Sentinel can inform it of its demotion. During this time you will have both systems reporting as master and HAProxy will proxy to each based on its load balancing choices.  

That is just the general case of every single recover of the master short of a complete rebuild. However, that isn't all. For this case it is even worse. The overall setup is terrible for Sentinel. You have a case where one instance is expected to be available offline. Where do you run sentinel and what settings do you give it? When the laptop goes offline, how is the master on the network to be informed, and what is it to do? If you tell it to be master, how will you do it?

You could do it before removing the laptop. But with sentinel you'll need to be either running a single one on the laptop or you will immediately create split brain. You'll have two masters because if sentinel runs on the network it will always promote the network master. Yet  the app needs to talk to a master offline - ie. the local one. So now you'd have two masters. Ok, so what if we only run a single sentinel on the laptop, and issue a manual failover request before disconnecting? 

You've just created split brain in HAProxy. This is because during a manually issued failover, there will be a time where both instances will report master. Again, w/the "advanced checks" the HAProxy people put out there you will now have requests "load balanced" across two masters.  So what if you run the sentinels on the network? The reverse happens. 

You can't let sentinel just handle it because if the only sentinel is on the laptop the network one will never get promoted, meaning you are not meeting the requirement. So the HAProxy checks won't solve the problem presented, and are in general a terrible idea if you care about data integrity.

This is why this scenario is not a good fit. Redis is not designed for disconnected use, and neither is Sentinel. HAProxy can't change that, and shouldn't be configured the way that link says if data integrity is important anyway.

Cheers,

Bill

[Bill Anderson]

Hehe no worries, Shaun. :)

[Shaun Davis]

I appreciate the response!  

[Shyamal Gajjar]

Thank you all for your responses.....!