Redis HA agents, half of an idea, at best.

[Salvatore Sanfilippo]

Hi all,

Redis is used every day in more places, so many people need an HA
setup. Different users end doing it in different ways with an home
brew solution given that we don't provide any.

While it is true that Redis Cluster will also offer fault tolerance, I
don't think it will solve our HA issues, since many people don't
really need a distributed version of Redis: they just have a master
instance that has plenty of room for all the data, and a slave that is
used just for fault tolerance. It is time to provide a solution.

One interesting observation to do is that in setups with a master and
a slave there are only two nodes, so the "cluster way" of detecting
failing instances via some kind of consensus algorithm with external
nodes checking if you are still alive is not going to work, so just
using Redis Cluster with a single master and a single salve is not a
good idea. If there is a netsplit between the two, both will think the
other is not working, and it is not exactly a very interesting
information.

In this setups what you really care about is the availability from the
point of view of the clients (that are often in the web servers).

My current idea is that the best way to solve this problem is writing
a small specialized client, that is part of the Reds distribution, is
implemented in C, is rock solid, and uses almost no memory.
This is called an HA agent, or /usr/local/bin/rha-agent for the
friends (Redis HA agent).

It takes a very simple configuration via command line or alike, where
we tell the HA what the IP address of the master and all its slaves.
What the HA does is the following: it performs a small query every
second against the master. If the master is up nothing is done.
If the master is down, the HA agent writes that the master is not
working as expected into one hey in the redis slave instance.

Note that if the redis-slave instance is also not available the HA
agent will simply think it is the problem, and will discard its state
and try again the next second.

The user should run many HA agents in different places of its network,
especially in all the places where there are clients. For instance
every web server should have an HA agent running.

In the HA agent configuration we tell a few things, especially the
failover triggers. For instance the failover can be triggered if N
consecutive errors as signaled by at least M HA agents.
It's like this:

ha-agent --master 1.2.3.4:6379 --slave 3.4.5.6:6380 --slave
3.4.5.6:8888 --failover-agents 2 --failover-errors = 30

Now we can have three of those agents running in our three web
servers. If the master goes down the agents will start writing the
errors in the slaves. This is obtained using the following commands
(more or less):

MULTI
LPUSH master-errors <ha-agent-unique-id>:<unix time>
TTL master-errors 5
EXEC

So every error that is separated by the previous one by less than 5
seconds will end queued into the list.
(perhaps this commands should be executed against every slave, still
not sure but probably it is a good idea).

If there are errors this list will finally reach an interesting length.
ha-agents don't simply ping the master, they also fetch this list from
time to time. So when enough errors by enough ha-agents are reported,
the failover is performed.

Big open question: how to perform the failover? What we can do is to
elect the slave to master and then execute an arbitrary command passed
by the sysadmin.
How will clients know that something is wrong with the master and will
switch to the slave instance? Is the executed command by ha-agents
that is able to switch IP addresses or alike?

Basically I don't have still a good solution, but just a draft, but I
really want to address this problem.
Maybe we can put together the knowledge of the ad-hoc implementations
of HA for Redis that is already in different production environments
of people following Redis development?

Any help is appreciated,
Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Salvatore Sanfilippo]

p.s. an alternative is to create a protocol that can be implemented
directly in the client side.
All the redis clients implementing it will be HA-ready.

Salvatore

[Josiah Carlson]

My opinion: Implement it on top of Redis Cluster. Redis Cluster
already has the "sorry, your key isn't here, check this other machine"
failover, redirection, etc. The only additional functionality that
Redis needs is the ability to say "disable sharding for this cluster"
and an optional agent to help arbitrate the election of the master.

Your simple agent is easier to implement once than to implement in the
dozen or so client libraries. Checking health every second is
reasonable, but it may make sense for it to be configurable.

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Didier Spezia]

While I would love to see a simple solution to implement a basic master/slave HA setup,

I don't think it can be implemented in a simple way. IMO there are only two paths to HA:

- physical clustering, based on an extra private network, an optional quorum device, and

a clusterware such as Pacemaker, Heartbeat, VCS etc ... If you do not want to add the

complexity of a resource manager, then you need to embed HA features in Redis itself,

for instance by linking to the corosync library ( http://www.corosync.org )

- logical clustering, based on fancy distributed algorithms. You can find this kind of

strategies in MySQL Cluster, Membase, Riak, Zookeeper, Doozer, etc ...

Both paths are complex in their own way.

Implementing reliable distributed algorithms is hard and error prone. Others have done

it already, so why not reuse their work? Building a solution over Zookeeper or Doozer

may be simpler and more robust.

Regards,

Didier.

[Jeremy Zawodny]

Or put redis behind an haproxy installation?

Jeremy

On Tue, Nov 15, 2011 at 9:02 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

[Josiah Carlson]

That just handles availability of slaves, which is only a small part
of the problem.

- Josiah

[Jeremy Zawodny]

Is it?  How is this different from the ways people deploy haproxy pairs in front of their mysql masters?

Jeremy

[Josiah Carlson]

Redis doesn't do multi-master.

- Josiah

[Pau Freixes]

HI Salvatore, use a external program sounds good solution, because the
user know how is the architecture and know how he can change "redis
client" environment...

For example, you could use a dns name strategy like this.

At first time there are a three servers with this name :

1.2.3.4 > master.mynetwork.com
3.4.5.6 > slave-a.mynetwork.com
4.5.6.7 > slave-b.mynetwork.com

And then execute a N instance of ha-redis with this arguments

ha-agent --master master.mynetwork.com:6379 --slave
slave-a.mynetwork.com:6380 --slave slave-b:8888 --failover-agents 2
--failover-errors = 30 --failover-cmd /path/file

With --failover-cmd you can address the file will be executed, in this
environment file script will be change dns rewriting
master.mynetwork.com CNAME register to another slave node.

Another interesting argument could be --failover-end, to say if
ha-agent will be died after reach failover situation. File script
could run again ha-agent with other parameters.

On Tue, Nov 15, 2011 at 5:55 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
--pau

[Jeremy Zawodny]

Multi-master is rather orthogonal to this discussion.  MySQL doesn't do mutli-master either (really) unless you're incredibly careful--and most people are not.

Once the decision has been made to takeover the IP of the failed node, you can also issue the 'salveof no one' against the new master to ensure that it doesn't think it's a slave anymore.

I guess I'm not seeing what makes redis unique enough to require engineering a new, redis-specific way of failing over to a secondary/slave/standby node...

Jeremy

[Javier Guerra Giraldez]

On Tue, Nov 15, 2011 at 4:59 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:
> MySQL doesn't do mutli-master either (really) unless you're incredibly
> careful--and most people are not.

i hope those people aren't using haproxy for that

--
Javier

[Jeremy Zawodny]

I'm not saying it's the best solution for MySQL, but people are doing it.

http://www.alexwilliams.ca/blog/2009/08/10/using-haproxy-for-mysql-failover-and-redundancy/

That's one example of several I found when researching haproxy for MySQL.  We're in the midst of deploying it for our read (slaves) traffic and it works wonderfully in that role.

Jeremy

[Dvir Volk]

one important thing to note regarding failover, no matter how you manage it:

once you tell all the slaves to go to a new master, they all sync at once, rendering them useless until the data is loaded, possibly bringing the whole cluster to a grind.

we've talked in the past about pacing this or limiting the number of concurrent slave syncs.

probably an HA manager can also solve this one too, but maybe part of the solution is doing this limitation in the master.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

Dvir Volk

System Architect, DoAT, http://doat.com

[Javier Guerra Giraldez]

On Tue, Nov 15, 2011 at 5:16 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:
> We're in the midst of deploying it for our read (slaves) traffic and it
> works wonderfully in that role.

oh, yes. if you're not distributing writes, simply adding haproxy does
work (and very well, i've seen).

and, as Josiah said, that's just part of the problem. if you want to
use haproxy for the full scenario, you need master-master.

--
Javier

[Jeremy Zawodny]

This is the source of my confusion.  This thread began discussing the common 1 master, 1 slave scenario.  Not clustering or many machines with sharding.  What's the "full scenario" you're referring to?  Is it something more than Salvatore's simple 2 node setup that we began this with?

Jeremy

--
Javier

[Josiah Carlson]

Right now the solution for even the most simple 1 master 1 slave
failover to the slave becoming the master is essentially a class of
client-side hacks that dozens (if not hundreds) of developers have put
together individually, sometimes discussed here. That's not a viable
path going forward if we are to consider Redis as a potential
High-Availability service (for each individual definition of "High
Availability", and in this discussion, that is master failover).

So, for this situation, HAProxy doesn't help. Sure, you can read from
the master or the slave, and HAProxy will pass reads to whichever is
still up... but it doesn't solve the data write problem.

Once the data write problem is solved for this simple 2-node scenario,
going with 1 master k slaves is the next obvious candidate (and should
be straightforward, given the election procedures determined as part
of the 2-node scenario). Subsequently, j masters with k slaves should
also be informed by the earlier decisions.

Regards,
- Josiah

[Javier Guerra Giraldez]

On Tue, Nov 15, 2011 at 5:43 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:
> This is the source of my confusion.  This thread began discussing the common
> 1 master, 1 slave scenario.  Not clustering or many machines with sharding.
>  What's the "full scenario" you're referring to?  Is it something more than
> Salvatore's simple 2 node setup that we began this with?

as I see it:

Redis HA proposal: 1 master, 1+ slaves, failure detection, automatic
slave->master promotion

you ask about just putting haproxy in front of Redis. nice points:
haproxy distributes load and if a sever fails, it simply stops sending
traffic there.

but haproxy (in TCP mode) needs the servers behind it to be all equal.
in MySQL world, it can be done with master-master, or (your setup) if
put in front of all the slaves and handle only reads.

so, 'full-haproxy' doesn't work (because there's no master-master in
redis) and 'slave-only haproxy' works but doesn't help (because
read-only slave balancing is a solved issue).

and, in any case, it doesn't begin to help in slave->master promotion

--
Javier

[Salvatore Sanfilippo]

On Tue, Nov 15, 2011 at 6:21 PM, Josiah Carlson
<josiah....@gmail.com> wrote:
> My opinion: Implement it on top of Redis Cluster. Redis Cluster
> already has the "sorry, your key isn't here, check this other machine"
> failover, redirection, etc. The only additional functionality that
> Redis needs is the ability to say "disable sharding for this cluster"
> and an optional agent to help arbitrate the election of the master.

Not sure how this could possibly work in a master/slave setup. If the
master is down it can't redirect.
If the master returns back alive, it does not know it should redirect.

[Salvatore Sanfilippo]

On Tue, Nov 15, 2011 at 7:27 PM, Didier Spezia <didi...@gmail.com> wrote:

> - physical clustering, based on an extra private network, an optional quorum
> device, and
> a clusterware such as Pacemaker, Heartbeat, VCS etc ... If you do not want
> to add the
> complexity of a resource manager, then you need to embed HA features in
> Redis itself,
> for instance by linking to the corosync library ( http://www.corosync.org )

Yes this is the stuff I was trying to 'fake' basically.

I don't want logical clustering as this is what Redis Cluster is going to be.
Just a simpler than 'made at home' stuff ensuring HA.

The solutions you mention received much attention and care for sure,
but still it is probably worth exploring if we can in some way provide
something that works for 90% of users that don't want to add too much
complexity.

There are many solutions after all that we can use and our goal is
reasonably simple. We just want:

1) Automatic failover when a master instance fails.
2) No single point of failure.

We don't want load balancing, or ability to restore automatically the
condition after failover (this will require sysadmin intervention
possibly).

So conceptually what we need is:

1) A device that is able to sense failure in a "from the point of view
fo the client" fashion.
2) A safe slave election when a failure is detected.
3) A system to switch the client configuration to the new host
permanently before sysadmin intervention.

And the set of solutions is really big.
For instance both 1,2,3 can be put inside the client itself.

Or we can have a simple agent for "1", that also performs "2" and is
able to communicate with "3" that is a simple proxy, and you can have
multiple agents and proxies.

Or in a simpler scenario the proxy is also able to do the check (that
is pretty obvious since it already handles connections to the
servers). But is also able to stay connected to the other proxies
instances so that they can agree if something is wrong with the
master. Here we don't need to handle fancy net splits: proxies can't
communicate? No deal, no failover, nothing. It is enough to simply
make it working when *just* the master instance no longer works.

In the past I worked a bit at this kind of systems and what I learned
is: if something is odd don't do the failover, even if you can cover a
smaller number of failures, because trying to be too much smart almost
always results in failovers triggered by the system itself even if
what we want to make redundant is still working well :)

[Salvatore Sanfilippo]

On Tue, Nov 15, 2011 at 9:11 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:

> Or put redis behind an haproxy installation?

This is definitely a way to solve the problem: if there is a simple
system that works well for Redis we can just put the configuration
instructions in our web site. Done.

But I don't know HA enough to comment and I wonder if it also allows
us to avoid a single point of failure.

Salvatore

[Salvatore Sanfilippo]

On Tue, Nov 15, 2011 at 11:43 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:
> This is the source of my confusion.  This thread began discussing the common
> 1 master, 1 slave scenario.  Not clustering or many machines with sharding.
>  What's the "full scenario" you're referring to?  Is it something more than
> Salvatore's simple 2 node setup that we began this with?

For everything over master + slave with failover I think Redis Cluster
is the way to go indeed...
I just want a solution that is easy to deploy for the base case of a
single instance and a slave, that is a very common setup.

Of course this setup does not mean you should have a single instance.
You can have even 100, but your application can still treat every
instance as a single logical unit, so you can still HA using a master
<-> slave setup.
If you do your own sharding at application-level I think everything
still needs a simple HA without any need to mix details about how the
application uses the different logical units: they can be used to
store 100 different kind of data, or your app can implement PAXOS, or
everything in the middle :)

Salvatore

--

[Jak Sprats]

Hi All,

I think Jeremy's suggestion of using haproxy makes the most sense.

Haproxy can be used for TCP (not just HTTP)

Haproxy can distribute reads to either master or (master & slave)

Haproxy can regularly ping the master and has settings to redirect
writes to the slave if their are N(configurable) failures (i.e.
automatic promotion)

Haproxy can be located on the same machine as a client, given it the
client's view of the world.

Haproxy is generic & battle-tested.

That said Didier is also correct :) Netsplits can happen with clients
on Switch1 still seeing only the old master and clients on Switch2
seeing only the was-slave-is-newly-promoted-master ... solving that
one requires the most-masters protocols or something of that sort.

- jak

[Josiah Carlson]

On Wed, Nov 16, 2011 at 2:24 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> On Tue, Nov 15, 2011 at 6:21 PM, Josiah Carlson
> <josiah....@gmail.com> wrote:
>> My opinion: Implement it on top of Redis Cluster. Redis Cluster
>> already has the "sorry, your key isn't here, check this other machine"
>> failover, redirection, etc. The only additional functionality that
>> Redis needs is the ability to say "disable sharding for this cluster"
>> and an optional agent to help arbitrate the election of the master.
>
> Not sure how this could possibly work in a master/slave setup. If the
> master is down it can't redirect.
> If the master returns back alive, it does not know it should redirect.

The client should be caching information about other members of the
cluster, correct? If it fails to connect to the master, it just
retries on one of the other known cluster members. Those other members
will be able to either serve the (read) request, or redirect to the
(newly elected) master. If the client fails on repeated attempts
(fails to hit the master, and a new master hasn't been elected yet),
surely the ha-agent will also discover this, force an election, etc.

Regards,
- Josiah

[Josiah Carlson]

HAProxy is great for distributing TCP requests across a cluster. It
solves the load balancing and failover handling for Redis read slaves.
It doesn't solve the master failover situation.

Regards,
- Josiah

[Daniel Mezzatto]

I have implemented a daemon called "rcloudd" and is in our production
environment since May/2011.

Our redis "cluster" have 64 master and 64 slave instances, spread
around 18 machines. Every master has only one slave. Each machine has
a single rcloudd process.

The daemon checks the master every 60 seconds sending a PING command.
If the ping fails, the daemon asks 2 machines from the neighborhood to
ping the master as well. If the 2 machines also fail to ping the
master instance, the daemon issues the "SLAVEOF NO ONE" command to the
slave instance. Meanwhile, the daemon still tries to ping the original
master. When the original master returns, the daemon makes it the
master again and sets the original slave to be a slave again (this is
due to performance problems that we were having since a single machine
cannot have too many master because it will be flooded with writes).
Its a kind of consensus algorithm.

The slaves are not "backups", we read data from them. Here we aimed
for availability (with eventual consistency). My client implementation
randomically chooses if it will read from the master or the slave
(rand() & 1) and if the request fails it sends the request to the
other instance.

It is not a bulletproof solution, but it is good enough while we wait
for the Redis Cluster :D

Cheers,

Daniel Mezzatto

[Greg Richardson]

Salvatore - thanks for opening up this round of discussions -
appreciated!

I like the point of re-using common LB technologies, most of us
already have these in spades (HAproxy/appliances). To Josiah and
others point, the problem is ultimately with the writes.

Personally, I always thought when a redis node(s) becomes a slave it
should prohibit the ability for client to write directly to it. I
think this catches many first timers off-guard and we usually find out
the hard way :)

If the ability could added have slave's could enforce the notion of
'no-writes in slave mode', then we could leverage logic from the
upcoming cluster rev where the slave replies back to the client with
the direct address:port to the master (not the LB address). For
clients that can cache this; great! For those that can't, some
overhead will be met for write attempts through the LB address.

Keeping in line with having backend HA-agent's, maybe these could be
spawned into a separate thread via each master/slave(s) when HA is
enabled. HA nodes should ultimately be responsible for managing the
'SLAVE NO ONE' to the next slave elected to assume master. HA nodes
should also update remaining slaves to the newly elected master.

On the sync side, maybe before doing complete re-sync's, sometime of
checksum could be performed to identify if a sync needs to occur? For
low traffic environments that still want sanity, this would be ideal
and prevent the env from being at a standstill while sync's occur.
Probably not feasible for high-traffic environments.

On Nov 17, 7:50 am, Josiah Carlson <josiah.carl...@gmail.com> wrote:
> HAProxy is great for distributing TCP requests across a cluster. It
> solves the load balancing and failover handling for Redis read slaves.
> It doesn't solve the master failover situation.
>
> Regards,
>  - Josiah
>
>
>
>
>
>
>
> On Wed, Nov 16, 2011 at 2:37 AM, Salvatore Sanfilippo <anti...@gmail.com> wrote:
> > On Tue, Nov 15, 2011 at 9:11 PM, Jeremy Zawodny <Jer...@zawodny.com> wrote:
>
> >> Or put redis behind an haproxy installation?
>
> > This is definitely a way to solve the problem: if there is a simple
> > system that works well for Redis we can just put the configuration
> > instructions in our web site. Done.
>
> > But I don't know HA enough to comment and I wonder if it also allows
> > us to avoid a single point of failure.
>
> > Salvatore
>
> >> Jeremy
>

> >> On Tue, Nov 15, 2011 at 9:02 AM, Salvatore Sanfilippo <anti...@gmail.com>

> >> wrote:
>
> >>> p.s. an alternative is to create a protocol that can be implemented
> >>> directly in the client side.
> >>> All the redis clients implementing it will be HA-ready.
>
> >>> Salvatore
>

> >>> On Tue, Nov 15, 2011 at 5:55 PM, Salvatore Sanfilippo <anti...@gmail.com>

[Josiah Carlson]

I agree with almost all of what you said; there are scenarios where
writing to a slave is a reasonable use-case (I have on specifically in
mind, I'm sure others have others). But that situation could be
handled with a configuration option in non-HA/non-Clustered situations
(plain master/slave). In clustered/HA scenarios, disabled slave
writing is not unreasonable.

Regards,
- Josiah

[Nathan Davis]

Salvatore,
I like this idea of a protocol for implementing auto-failover in the
client. Putting a separate process in between has the unfortunate
effect of creating a new SPOF. Implementing the fail-over and health
checking in the client lib will keep it simple, avoid adding a new
SPOF, and won't add any additional latency. Also, you could make
hiredis the reference implementation.

-Nathan

[Javier Guerra Giraldez]

On Wed, Nov 23, 2011 at 2:43 PM, Nathan Davis <nathan...@gmail.com> wrote:
> Putting a separate process in between has the unfortunate
> effect of creating a new SPOF.

not if it's designed to be several copies of that extra process (which
isn't "in between", if i understood correctly). having a copy of that
monitor process at each client seems reasonable, thought, and it's
cleaner if it's part of the client itself.

--
Javier

[Josiah Carlson]

If you embed it in the client, you now have as many implementations of
the monitoring method as you have client libraries.

Regards,
- Josiah

[Jonathan Shook]

A study of Cassandra's "gossip" between nodes may be helpful here. Also, sharding will probably become a necessary concern at some point, so a protocol/scheme which works with multiple peers from the beginning may get more mileage down the road. 

[Javier Guerra Giraldez]

On Thu, Nov 24, 2011 at 7:15 PM, Josiah Carlson
<josiah....@gmail.com> wrote:
> If you embed it in the client, you now have as many implementations of
> the monitoring method as you have client libraries.

minus the number of client libraries that wrap hiredis

--
Javier