*automatic* failover

[Brian Hammond]

It would be great to not have to manually set a former slave to be a
master but have it happen automatically.

Consider 3 redis instances: A, B, C

A is the current master
B and C are slaves of A

A goes down (for whatever reason)

Currently, you have to manage this scenario yourself (AFAIK) by

1. fixing the issue with A, bring it back online, and continue with A
as master
2. pick one of B or C (say B) and make it a "SLAVEOF" the other (this
extends to N other slaves)

When A is brought back online, it can be made a SLAVEOF the new master
(say B)

This "sucks". I don't want to have to babysit my servers. I just
want them to be available as much as possible.

I know there are monitoring tools out there that watch a server and if
it goes offline, *does something* like alerting someone or ARPing the
IP on another host (e.g. Wackamole). That's fine and good but there's
nothing that I know of that will send a command to several other hosts
(SLAVEOF). If there is, please let me know.

Here's what I am thinking would work. You have to have a free server
to run this on which of course raises the question of "who is going
monitor your monitor?" - oy! I'm not going there.

I'm planning on creating "redis-cluster-monitor", a simple script
(Python say) that is configured with a list of redis instances, one of
which can be designated as the master (or the script could pick one at
random), the rest being slaves of the master.

Every N seconds the monitor connects to the master and runs an INFO
command or the like. If it fails more than F times (say default F=1)
then it picks an online slave from the list and promotes it to
master. Then it connects to the other servers and updates them to be
slaves of the new master. The old master is marked as "former
master". If the former master comes back online, it is updated to be
a slave of the new master.

Pretty simple.

Am I missing anything?

[Sergey Shepelev]

On Thu, Oct 29, 2009 at 8:36 PM, Brian Hammond <br...@fictorial.com> wrote:
>
> It would be great to not have to manually set a former slave to be a
> master but have it happen automatically.
>
> Consider 3 redis instances: A, B, C
>
> A is the current master
> B and C are slaves of A
>
> A goes down (for whatever reason)
>
> Currently, you have to manage this scenario yourself (AFAIK) by
>
> 1. fixing the issue with A, bring it back online, and continue with A
> as master
> 2. pick one of B or C (say B) and make it a "SLAVEOF" the other (this
> extends to N other slaves)
>
> When A is brought back online, it can be made a SLAVEOF the new master
> (say B)
>
> This "sucks".  I don't want to have to babysit my servers.  I just
> want them to be available as much as possible.
>
> I know there are monitoring tools out there that watch a server and if
> it goes offline, *does something* like alerting someone or ARPing the
> IP on another host (e.g. Wackamole).  That's fine and good but there's
> nothing that I know of that will send a command to several other hosts
> (SLAVEOF).  If there is, please let me know.
>

Yup, there are tons of tools like monit that "poll for some event"
(e.g. run redis-cli with particular and test its return code) and "do
something" when test fails (e.g. run redis-cli with other command).
You can run redis-cli with N commands to send SLAVEOF to N other hosts
in one "do something".

> Here's what I am thinking would work.  You have to have a free server
> to run this on which of course raises the question of "who is going
> monitor your monitor?" - oy!  I'm not going there.
>

Let everyone monitor everyone.

[Aníbal Rojas]

Sergey,

There is Monit, and God (Ruby) I agree; I suppose there is bunch
more of monitoring tools out there. But something I find very
interesting is MongoDB's Replica Pairs http://bit.ly/2Vf7rE

This requires a little bit of intelligence at client side, but not
too much. The replica pairs coordinate which one is the Master and
which one is the Slave, you setup you client with the connection info
for both.

When the client first connects it will notified which one is the
Master and default to it, if the Master fails the Slave takes over.
After a number of failures to connect to the original Master, the
client library asks the Slave what is going on and resumes operation.

If/When the original Master comes back the roles are negotiated
again.

It doesn't have the complexity of a Multimaster setup, and I think
it is pretty simple to implement both in redis-server and client
libraries.

Best regards,

--
Aníbal Rojas
Ruby on Rails Web Developer
http://www.google.com/profiles/anibalrojas

[Adam Michaels]

This should go in the proposed redis-cluster daemon.

On Oct 29, 11:52 am, Aníbal Rojas <anibalro...@gmail.com> wrote:
> Sergey,
>
>     There is Monit, and God (Ruby) I agree; I suppose there is bunch
> more of monitoring tools out there. But something I find very

> interesting is MongoDB's Replica Pairshttp://bit.ly/2Vf7rE

>
>     This requires a little bit of intelligence at client side, but not
> too much. The replica pairs coordinate which one is the Master and
> which one is the Slave, you setup you client with the connection info
> for both.
>
>     When the client first connects it will notified which one is the
> Master and default to it, if the Master fails the Slave takes over.
> After a number of failures to connect to the original Master, the
> client library asks the Slave what is going on and resumes operation.
>
>     If/When the original Master comes back the roles are negotiated
> again.
>
>     It doesn't have the complexity of a Multimaster setup, and I think
> it is pretty simple to implement both in redis-server and client
> libraries.
>
>     Best regards,
>
> --
> Aníbal Rojas

> Ruby on Rails Web Developerhttp://www.google.com/profiles/anibalrojas

[Brian Hammond]

That's funny, I forgot about redis-cli for a moment.

Oh well, I wrote the thing anyway. Only took a hour :)

http://github.com/fictorial/redis-cluster-monitor

Brian