Redis Sentinel: let's improve clients detection of failover events

[Salvatore Sanfilippo]

Dear list, this is an important topic I want to share with you about
the evolution of Sentinel.

Currently we have a draft document that describes how to write a
Sentinel-aware client:

http://redis.io/topics/sentinel-clients

Basically advanced clients may take a Pub/Sub channel open on
Sentienls to trap the event of the switch to the new master, while
normal blocking clients with minimal Sentinel logic should just ask
Senitnels at every reconnection to the master.

We can't just rely on the fact every client will use Pub/Sub. Most
clients will use simpler systems, like check on reconneciton, and if
the clients are configured to have a reasonable timeout, most of the
times they'll disconnect from the failing master, and ask the
Sentinels at every reconnection attempt, to finally be redirected to
the new master.

I've the feeling that while this may work, we need to improve things
in this simple case, to make the system more reliable, because it is
surely possible that the client for some reason does not sense the
failover happening (will not disconnect) even Sentinels reach quorum
about the master being down.

There are a couple of things we can do, both in the side of the
guidelines we provide to write a Sentinel-aware client, and in the
implementation of Sentinel.

PROPOSAL
===

My proposal is to make the following changes in the guideline, so that
clients do these two additioanl checks:

1) When redis replies with -READONLY, a Sentinel-aware client should
reconnect, asking Sentinels again what is the master.
2) Every Sentinel-aware client should periodically verify with
Sentinels that the master is yet what it believes to be. So for
instance if the latest check is older than N seconds, without
disconnecting the link the client contacts a Sentinel, verifies that
the master is still what we believe it is, if so everything will
continue as expected, otherwise we disconnect the link explicitly.

Moreover it is possible to modify the CLIENT command with a new
subcommand so that it can kill all the clients, but the current one
sending the command, CLIENT KILLALL or alike basically.

This way when the old master is detected to be back by Sentinels, we
may not just reconfigure it as a slave as already happens, but also
force a CLIENT KILLALL operation, so that clients will likely notice.

Your feedback is appreciated,
Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Andy McCurdy]

Hey Salvatore,

Making a client Sentinel-aware is quickly becoming non-trivial. As you detail below, we already have multiple ways for clients to talk to Sentinel (commands vs. pub/sub). Clients then also need to pick a strategy on how to run the Sentinel logic. For example, some clients might choose to create a separate thread that listens for published messages or that periodically polls Sentinel and updates the client connections in some way. Other clients may try to inject a Sentinel "ping" every N seconds just before another command is executed.

It seems to me that we could eliminate a lot of this complexity if Sentinel became a proxy for Redis hosts. Sentinel is already aware of which host is the master and which host(s) are slaves. Sentinel already speaks the Redis protocol. We're already maintaining active connections to Sentinel in our environments. Why complicate every client library with this redundant logic when Sentinel already has all the information it needs to do the right thing for us?

A proxy strategy also seems easier to explain to end users who are setting up Redis in their infrastructure. These folks are likely already used to other systems using the same strategy, e.g reverse proxies in front of HTTP servers, HA Proxy or similar in front various TCP services, PGBouncer/etc. in front of relational databases, etc.

-andy

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Salvatore Sanfilippo]

Hello Andy,

I understand your point of view, but I don't agree (as this basically
is a design thing, so just a matter of personal taste ultimately) for
the following reasons:

1) While a sentinel aware client is not as trivial as to write just a
few lines of code, it looks like all we need to do is to check with
Senitnel from time to time, without requiring another thread or
complex systems. Basically: storing the time at which we contacted
Sentinel the last time, and requesting it again. Once we disconnect
clients when an old resurrecting master is demoted, even clients
without this step will be a lot safer.

It's worth to note that there are additional strategies, not requiring
a thread, but still using Pub/Sub, that is to just listen to the
switch master event via Pub/Sub, but actually check if there is
something to read from that socket only when we need to send commadns
to our master.

2) Given "1", that still, this is not going to be a really hard part
of implementing a client (as there are many other non Sentinel related
issues in a good client library that are not triival at all), the win
is huge of not having an intermediate layer. Proxies take away a
significant amount of requests per second and latency, even the *best*
proxies.

3) Reason three is modularity. We already have a powerful Redis proxy
with a BSD licensed, that's Twitter's twemproxy. All there is to do is
to take it and add Sentinel-awareness to it. This will serve the users
that want a proxy solution well with a vertical solution designed to
do the proxy with very good performances, and also supporting other
stuff like sharding if needed.

So basically it's a tradeoff, I still believe that from my point of
view the weight is much more pending in the side of talkign directly
with instances without intermediate layers. We just need to provide a
set of simple rules, with optional parts to implement to make the
client even more safe but not mandatory, that client library users can
follow in order to end with a robst enough client. At the same point
nobody prevents us from implement strategies inside Sentinel and/or
Redis like the "disconnect all" thing, in order to make even the
simplest of the clients robust enough in most cases.

Cheers,
Salvatore

[Andy McCurdy]

Hey Salvatore. Some replies below.

On May 3, 2013, at 4:19 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

> Hello Andy,
>
> I understand your point of view, but I don't agree (as this basically
> is a design thing, so just a matter of personal taste ultimately) for
> the following reasons:
>
> 1) While a sentinel aware client is not as trivial as to write just a
> few lines of code, it looks like all we need to do is to check with
> Senitnel from time to time, without requiring another thread or
> complex systems. Basically: storing the time at which we contacted
> Sentinel the last time, and requesting it again. Once we disconnect
> clients when an old resurrecting master is demoted, even clients
> without this step will be a lot safer.
>
> It's worth to note that there are additional strategies, not requiring
> a thread, but still using Pub/Sub, that is to just listen to the
> switch master event via Pub/Sub, but actually check if there is
> something to read from that socket only when we need to send commadns
> to our master.
>

The occasional check to Sentinel is just part of what a client will have to do. Clients will also have to accept additional configuration options (at least a list/array of Sentinel ip:port values) and use that information to look up the real Redis servers. A sentinel aware client constructor might look something like this:

>>> Redis(hostname, port, ..., sentinel_servers=[])

This sounds trivial at first, but consider all the libraries (not client libs, but libs that are using a client lib to talk to Redis) out there that have code like this:

>>> redis = Redis(REDIS_HOSTNAME, REDIS_PORT)
>>> redis.get(...)

where REDIS_HOSTNAME and REDIS_PORT come from environment variables, config files, etc. Even if the underlying client supports Sentinel, libs that use this common pattern won't work in Sentinel-based deployments until those libs also add more config options for Sentinel. This really sucks.

> 2) Given "1", that still, this is not going to be a really hard part
> of implementing a client (as there are many other non Sentinel related
> issues in a good client library that are not triival at all), the win
> is huge of not having an intermediate layer. Proxies take away a
> significant amount of requests per second and latency, even the *best*
> proxies.
>

A single connection might see a slight drop in throughput, but assuming multiple Sentinels would be used to proxy connections to the Redis server, the system as a whole will still perform at the same level. In fact, in relational database communities, adding reverse proxies is often a good way to *increase* performance because they pool connections to the database server. Some clients like redis-py already do this, but it would be nice for everyone to have access to this functionality, regardless of client lib.

> 3) Reason three is modularity. We already have a powerful Redis proxy
> with a BSD licensed, that's Twitter's twemproxy. All there is to do is
> to take it and add Sentinel-awareness to it. This will serve the users
> that want a proxy solution well with a vertical solution designed to
> do the proxy with very good performances, and also supporting other
> stuff like sharding if needed.
>

I haven't used it yet, but twemproxy looks great. Unfortunately, it doesn't seem to know anything about master/slave relationships. If twemproxy becomes Sentinel aware, that's great. But then I would need to run 3 different daemons (redis-server, sentinel, and twemproxy) to get high availability.

[Andy McCurdy]

On May 3, 2013, at 5:45 PM, Andy McCurdy <sed...@gmail.com> wrote:

>
> The occasional check to Sentinel is just part of what a client will have to do. Clients will also have to accept additional configuration options (at least a list/array of Sentinel ip:port values) and use that information to look up the real Redis servers. A sentinel aware client constructor might look something like this:
>
>>>> Redis(hostname, port, ..., sentinel_servers=[])
>
> This sounds trivial at first, but consider all the libraries (not client libs, but libs that are using a client lib to talk to Redis) out there that have code like this:
>
>>>> redis = Redis(REDIS_HOSTNAME, REDIS_PORT)
>>>> redis.get(...)
>
> where REDIS_HOSTNAME and REDIS_PORT come from environment variables, config files, etc. Even if the underlying client supports Sentinel, libs that use this common pattern won't work in Sentinel-based deployments until those libs also add more config options for Sentinel. This really sucks.

I should have also noted that no matter what direction is taken, it's likely that clients will have to adopt a way to at least specify a list/array of hosts to connect to in order to gain high availability. Nevertheless, libraries that use a Redis client shouldn't need to know anything about Sentinel. Sentinel should be considered an ops thing, not something that developers need to be concerned with. An ops person should be able to turn deploy Sentinel without making any changes to apps apart from replacing Redis connection strings with Sentinel connection strings in a config file somewhere.

[Josiah Carlson]

If we're going to be talking about how to deploy this operationally, we may as well talk a brief moment as to how one of the largest services providers handles failover of their hosted database servers for some thousands (perhaps tens of thousands) of highly-available synchronously-replicated database clusters.

Of course I'm talking about Amazon's RDS hosted database servers in Multi-AZ configuration. For those of you who aren't familiar with their tech, they host MySQL with some unknown number of synchronous replicas of the master (at least one, though their pricing is only ~2.5x what a single server would run). On failover, DNS is updated. Clients existing connections fail (because the old master is forcibly killed), and will get the new master IP on reconnect.

What can we learn from this?

1. Make sure you aren't caching DNS requests, or if you are caching, that you can clear out specific hostnames during failover

2. Your failover script should be updating your DNS (and you should be using a DNS provider with a viable API)

That's actually a reasonable method of handling this to determine the master. But what about read slaves (AWS RDS instances don't offer direct access to read slaves, you have to add additional read slaves manually)? Well, *that* is where round-robin DNS comes in. On failover, your script should also update DNS for the 'slaves.clusterX.domain' dns entry to contain the list of slaves for clusterX. Then, your client can either use this information as part of its internal round-robin efforts, or it can always just make a request to slaves.clusterX.domain any time it needs a new connection.

With these two pieces, the simple answer to handling failover is:

1. update DNS for your master and your slaves

2. disconnect all clients and clear your DNS cache (you can wait until commands complete before closing connections)

3. reconnect as necessary

Now, if you are careful, you can disconnect only those client connections to slaves that are no longer slaves (according to DNS), as well as any connections to the old master. But I'd call those "advanced" clients, as I would consider any client that explicitly communicated with the sentinel, or which hung out on pubsub channels. But even if you did it the "dumb" way that I list above as 1-2-3, you've still got a pretty solid platform.

Want to turn failover up to 11 and bypass DNS server APIs? Implement a DNS server that polls Redis sentinels for the most up-to-date information, watches pubsub, etc., and set that to be a "subzone" in your DNS. Then, all requests to master.clusterX.domain and slave.clusterX.domain will hit clusterX.domain for DNS, which will always be up-to-date, according to the sentinels it knows about.

Regards,

 - Josiah

[Salvatore Sanfilippo]

On Sat, May 4, 2013 at 2:45 AM, Andy McCurdy <sed...@gmail.com> wrote:
> Hey Salvatore. Some replies below.

Thank you Andy,

> The occasional check to Sentinel is just part of what a client will have to do. Clients will also have to accept additional configuration options (at least a list/array of Sentinel ip:port values) and use that information to look up the real Redis servers. A sentinel aware client constructor might look something like this:
>
>>>> Redis(hostname, port, ..., sentinel_servers=[])
>
> This sounds trivial at first, but consider all the libraries (not client libs, but libs that are using a client lib to talk to Redis) out there that have code like this:
>
>>>> redis = Redis(REDIS_HOSTNAME, REDIS_PORT)
>>>> redis.get(...)
>
> where REDIS_HOSTNAME and REDIS_PORT come from environment variables, config files, etc. Even if the underlying client supports Sentinel, libs that use this common pattern won't work in Sentinel-based deployments until those libs also add more config options for Sentinel. This really sucks.

In environments where HA is needed, the only thing you can do is to
make things at least a bit more complex. There is no escape from that.

Even if we would use proxies, you still need to provide the list of
Sentinels, because if the client can't connect to the first, it is not
ok to fail, it needs to connect to the next.
Basically every distributed system that is reliable requires that
every client know about multiple access points to the system, there is
no escape from that whatever solution you use, otherwise the proxy
turns into a single point of failure.

That said, fortunately there are many things that can be done in
interesting ways.

1) Redis, as specified in the Sentinel doc (I don't remember if it's
in the spec or standard doc), will feature a command that will list
all the sentinels ip:port pairs that is monitoring a given instance.
This is trivial as Sentinels are required to send Pub/Sub messages for
auto-discovery. This means that clients, in case the user did not
specified the list of a few sentinels (they should as it is safer for
a cold start), can do a best-effort thing and at least fetch the list
of Sentinels once the object is created, assuming the master is up at
the moment the Redis object is created.

2) Users don't need to list all the Sentinels, it is enough to list a
few Sentinels in different conceptual availability zones, so that the
client wil be likely able to contact at least one. So at least the
list does not *need* to be esaustive.

> A single connection might see a slight drop in throughput, but assuming multiple Sentinels would be used to proxy connections to the Redis server, the system as a whole will still perform at the same level. In fact, in relational database communities, adding reverse proxies is often a good way to *increase* performance because they pool connections to the database server. Some clients like redis-py already do this, but it would be nice for everyone to have access to this functionality, regardless of client lib.

Well here I don't want to argue much, as this should be backed by
tests, but I can't agree that a middle layer can increase the
performance with the Redis architecture, unless you do sub-optimal
things in the client that a proxy can fix in some way. Proxies are
also another component that can fail, and another component where you
need to implement subtle things like TCP keep-alive, client timeouts,
and so forth. They will mask informations about client peer address
and port. I think that's really a shame to lost a lot of things just
for something that is definitely implementable client side easily. I
can understand using a proxy because of sharding concerns, if the
alternative is to implement sharding client side: this is a fair trade
off, but for HA, I don't see the win at all.

> I haven't used it yet, but twemproxy looks great. Unfortunately, it doesn't seem to know anything about master/slave relationships. If twemproxy becomes Sentinel aware, that's great. But then I would need to run 3 different daemons (redis-server, sentinel, and twemproxy) to get high availability.

It's three daemons but it would allow you to run with a setup that is
a different pick on trade offs, one that I don't like as stated
already but that many other very skilled people does. So I think there
is really a great value about having a Sentinel aware Twemproxy.

About the three daemons, that adds sysop complexity and is a good
point indeed, but to use this example to show my point of view, I'm a
lot more concenred with *three conceptual roles* added by the proxy
even if in the same daemon of Sentinel, as this raises the conceptual
complexity of the system.

Long story short, while I'll not go for the proxy, I think the
arguments you used for the proxy are the ones we could use to improve
the current system in different ways.

Cheers,

[Salvatore Sanfilippo]

I think this is a good point Josiah,

with its ability to execute scripts, Sentinel provides a very easy to
mount system-level solution to failovers where there is no need at all
to have intelligent clients. This was a design decision since the
start that there would be a way to use unmodified clients and client
reconfiguration using either tricks at level-2, DNS, and alike.

Cheers,
Salvatore

[Andy McCurdy]

On May 6, 2013, at 2:17 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

> On Sat, May 4, 2013 at 2:45 AM, Andy McCurdy <sed...@gmail.com> wrote:
>> Hey Salvatore. Some replies below.
>
> Thank you Andy,
>
>> The occasional check to Sentinel is just part of what a client will have to do. Clients will also have to accept additional configuration options (at least a list/array of Sentinel ip:port values) and use that information to look up the real Redis servers. A sentinel aware client constructor might look something like this:
>>
>>>>> Redis(hostname, port, ..., sentinel_servers=[])
>>
>> This sounds trivial at first, but consider all the libraries (not client libs, but libs that are using a client lib to talk to Redis) out there that have code like this:
>>
>>>>> redis = Redis(REDIS_HOSTNAME, REDIS_PORT)
>>>>> redis.get(...)
>>
>> where REDIS_HOSTNAME and REDIS_PORT come from environment variables, config files, etc. Even if the underlying client supports Sentinel, libs that use this common pattern won't work in Sentinel-based deployments until those libs also add more config options for Sentinel. This really sucks.
>
> In environments where HA is needed, the only thing you can do is to
> make things at least a bit more complex. There is no escape from that.
>
> Even if we would use proxies, you still need to provide the list of
> Sentinels, because if the client can't connect to the first, it is not
> ok to fail, it needs to connect to the next.
> Basically every distributed system that is reliable requires that
> every client know about multiple access points to the system, there is
> no escape from that whatever solution you use, otherwise the proxy
> turns into a single point of failure.
>

You're right -- at the very least, clients will need to to provide a list of servers to connect to for HA. I think that can be done in a backwards compatible way for most client libs such that we don't break existing apps.

However, I'd really like to avoid making the hundreds of Redis backed reusable applications out there write code and documentation about how to use their app with a traditional Redis setup vs. with a Sentinel setup. Right now, those apps have to write code that looks something like this:

if "<am-i-using-sentinel>":
client = Redis(sentinel_servers=['sentinel1.example.com', 'sentinel2.example.com'])
else:
client = Redis(redis_servers=['redis1.example.com', 'redis2.example.com'])

If I'm a Sentinel user and an app that uses Redis doesn't do this, then I can't use that app in my architecture. At least not without jumping through hoops.

I do think there's a way to get around this though. I'd like to propose a new command that all services that speak the Redis protocol respond to: SERVICETYPE

Redis servers would respond to the SERVICETYPE command with "redis".
Sentinel servers would respond to the SERVICETYPE command with "sentinel".

A client lib connection routine could then look something like this (I'm heavily paraphrasing, but I hope it makes sense)

def Redis(servers=[], ...):
for server in servers:
connection = get_connection(server)
if connection.servicetype() == 'sentinel':
# query sentinel to get the real redis server configs.
# create connections to the real redis server, etc.


This then allows apps to connect to Redis the same way, whether they're using Sentinel or not:

client = Redis(servers=['redis1.example.com', 'redis2.example.com'])
OR
client = Redis(servers['sentinel1.example.com', 'sentinel2.example.com'])

[Andy McCurdy]

Hey Salvatore, just floating the SERVICETYPE command idea (see below) now that you're back working on Sentinel. Would love to get your thoughts.

[Andy McCurdy]

I've been thinking more about this. We're seeing a rise in outsourced operations. Services like Heroku abstract away the details of operations and infrastructure. As a user of such a service, I shouldn't care whether they're using Sentinel or plain Redis. I pay them explicitly so I don't have to know the details. I just want to connect to Redis with the hostname(s) they provide to me.

Also, consider the scenario that thousands of apps are already deployed on services like Heroku. If one of these services wants to start using Sentinel, we should find a way to allow that without forcing their users to update app code.

Really what I'm getting at is that the existence (or not) of Sentinel in an infrastructure shouldn't bleed past the Redis client libraries. And that includes the method signature(s) those libraries expose to make connections to Redis.

Obviously we still need to expose the Sentinel specific commands for operations folks that explicitly want to talk to Sentinel (i.e. writing monitoring code, etc.). However, we shouldn't complicate everyone else's code in doing so.

-andy

[Josiah Carlson]

Andy,

Do you have a specific plan for making clients connect to and use Redis sentinel-enabled clusters (not to be confused with Redis cluser) without them having to worry about all of those extra details?

In order to get that level of "it just works" with failover, it would seem that you need some infrastructure somewhere. Either that is embedded in the client (with extra threads, connections, etc.), or it must be somewhere else. A twmemproxy or similar that knows how to handle sentinel events would work well, as would the DNS method I described in my earlier post.

 - Josiah

[Andy McCurdy]

The original proposal that Salvatore started this thread with suggests that clients should be able to handle this. My concerns to this have mostly been about keeping a separation between app and infrastructure code. Specifically, I believe that an app developer shouldn't have to change his/her code based on the presence of Sentinel in their environment.

I think clients can provide this abstraction provided they can identify the type of service they are connecting to. At a minimum, if the host name(s) provided to a client connect to a sentinel instance, behind the scenes the client should be able to do the right thing -- that is ask for the master's info and connect to it. There's also some management that the client would need to take care of detailed in Salvatore's proposal.

In the DNS example you provided, having a sentinel aware client is completely unnecessary. But not everyone has the luxury to control DNS, for example an app that runs on heroku or other similar services.

-andy

[Josiah Carlson]

It's not unreasonable to expect that a client that connects to a sentinel should be able to do the right thing, though it may make sense to require providing an argument to the client to say "you're connecting to a sentinel, figure out who to connect to in cluster X".

Maybe I'm thinking that Heroku is better than it actually is, but if someone is going to be providing master/slave clusters in Heroku, the provider of said services should also be handling all of the failover, DNS, proxies, etc., under the covers (like you say, if you're in Heroku, you're going to be connecting to the hostname that is provided). But maybe my expectations are high for paid-for services.

 - Josiah

[Andy McCurdy]

On May 17, 2013, at 5:09 PM, Josiah Carlson <josiah....@gmail.com> wrote:

It's not unreasonable to expect that a client that connects to a sentinel should be able to do the right thing, though it may make sense to require providing an argument to the client to say "you're connecting to a sentinel, figure out who to connect to in cluster X".

That's exactly why I'm proposing the addition of a command (SERVICETYPE) that would indicate the type of server connected to. A client could issue this command immediately on connection and decide what to do based on the result. This way the user doesn't need to specify anything. I wrote some pseudo code earlier in this thread detailing how it might be used. 

[Mike Peters]

Hi Salvatore,

We absolutely love Redis and it's an integral part of everything we do!  Keep up the great work

Over the last two weeks, we have been struggling with integrating Redis Sentinel (trunk version 2.6.13) into our cross data center, production environment, with no luck.

We finally decided to give up on Sentinel and revert to a home-grown solution.  I wanted to share some of our frustrations and thought-process, with the hopes this can help future development.

Sentinel got our attention with the promise of high-availability and automatic failover.

Diving into the implementation process, we had three goals - 

1. Avoid changes to client code

2. Continually monitor redis instances and if an instance goes down, either prevent clients from connecting to it again, or send a notification so that the global map can be updated

3. If master node goes down, automatically promote slave to be a master.  When master comes back, demote slave and resurrect master.

-

It was clear from the get-go that with Sentinel, we have to throw #1 out the window.

Sentinel requires clients to be aware of its existence, querying Sentinel to find the current active master and monitoring ongoing notifications.  Fine, at least we're left with #2 and #3.

Unfortunately, notifications didn't work either.  We enabled this line in the log:

  sentinel notification-script mymaster /home/sentinel_notification.php 

With no real load, about one of every three times we would take the master down, the notification never made it to the client.

The automatic failover also proved buggy in our environment.  

Failing over from master => slave worked flawlessly, but when the master came back online, Sentinel failed to properly resurrect the master.

Full log here: http://pastebin.com/b4hKgTU2

Obviously Redis Sentinel is still work-in-progress and the automatic master resurrection is a brand new feature.  We know the code will improve with time, but this experience made us dig deeper into the Sentinel architecture, as well as evaluate other solutions, like twemproxy and using zookeeper to coordinate node changes.

Each of the solutions we looked at, came with a hefty price:

sentinel: Requires changes to client code and (at least in our environment) both notifications and master resurrection didn't work

twemproxy: No changes to client code, but 30% performance penalty (in our tests), no automatic failover

zookeeper: Requires changes to client code, problematic in cross data center configurations, no automatic failover 

The redis cluster sounds very promising at this point.  However, considering the state of sentinel and the amount of time redis cluster has been pending, it is probably safe to say we are at least six months away from redis cluster being production ready.  

We need a solid high-availability + automatic failover solution TODAY.

Studying the approach others use to achieve these goals, we looked at Aerospike, Cassandra and Amazon RDS.  

Here's what they all have in common - 

* No changes to client code (clients don't have to be aware of the underlying HA infrastructure)

* Continually monitor all nodes and update the global ring (Cassandra), client hash-table (Aerospike) or DNS (Amazon) when a node goes down

* Automatic failover and resurrection 

-

After ruling out available solutions, we decided to build a home-grown architecture to achieve our goals.

We are big users of HAProxy.  Clients already connect to all redis nodes via HAProxy.  The idea is to create two groups in HAProxy per each node, of [redis write-master] and [redis read].  

Client reads are routed through the HAProxy [redis read] group, where they are distributed among all live read replicas.

Client writes are always sent to the HAProxy [redis write-master] group, where we have two servers, one defined as a backup.  The [redis write-master] "is alive" check, consists of an inetd script, that checks if the server is "master".  If it's not master, it is considered down.

This way, the HAProxy [redis write-master] will always see a single server as live.  A local script monitors the HAProxy log and handles failover / resurrection, sending "slaveof" commands to the participants of the "redis write-master" cluster, as needed.

I'll write up a separate post, outlining this implementation in greater detail.

The benefits of this approach are: No changes to client code, Automatic monitoring, Automatic failover and resurrection, Scales well across data centers (no paxos voting, all decisions are local to the current master node).

-

Would love to hear your thoughts.

All the best,

Mike Peters, General Manager

Software Projects Inc.

http://www.softwareprojects.com