Re: Master returns writable after restart

[Salvatore Sanfilippo]

On Thu, Mar 28, 2013 at 4:04 PM, Sheldon Hearn <shel...@starjuice.net> wrote:
> And then the fun begins. When I start the failed master up again, it is
> writable. I thought maybe this was just a tiny window for data loss until
> sentinel could slave it, but it stays writable indefinitely.
>
> So now I'm left wondering how a master/slave cluster could be configured to
> survive a reboot of a bootstrapped master.
>
> I've been testing with the configs below.
>
> Any guidance would be greatly appreciated.


Hello, this problem was still not addressed in Sentinel, and I was
wondering about two possible options:

1) Kill the master once it is back again.
2) Setup it as a slave for the new master.

The problem with "2" is that it's much better in terms of how the
system behaves but is also a bit more risky, since if there is
something wrong the the slave's copy of data, the master will also
duplicate the slave problem and data is gone.

However I recently implemented solution "2" in Redis Cluster, and it
works very very well, so I'm more inclined to do this for Sentinel as
well.
In Redis Cluster it's simpler because when the master node reconnects
it notices one of its slave is now a master (and this can happen only
if the node was authorized by the majority of masters to fail over),
so it's a non-brainer to turn itself into a slave of the new master.

With Redis Sentinel we'll have to remember that the given master at a
given address was failed over, and if we see it connecting again, we
need to address it with the SLAVE OF command. However there must be a
time limit for this I guess. Probably configurable.

Makes sense from your POV?

Cheers,
Salvatore

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware
http://invece.org

Beauty is more important in computing than anywhere else in technology
because software is so complicated. Beauty is the ultimate defence
against complexity.
— David Gelernter

[Sheldon Hearn]

On Thursday, March 28, 2013 5:36:50 PM UTC+2, Salvatore Sanfilippo wrote:

With Redis Sentinel we'll have to remember that the given master at a
given address was failed over, and if we see it connecting again, we
need to address it with the SLAVE OF command. However there must be a
time limit for this I guess. Probably configurable.

 

I've come up with something that works for 2 nodes. I configure both nodes as slaves to the other. The worst that can happen is that neither one comes up.

To get the pair going, I do "slaveof no one" on one of them. From then on, as long as they don't both go down at the same time, sentinel does the right thing.

I need to explore this idea with more than 2 nodes, and I need to solve the "no active master" problem. I suspect "no active master" will be easy.

I'm ducking for the week-end, but I wanted to share the idea and thank you for your quick reply.

Ciao,
Sheldon.

[Josiah Carlson]

No. The worst-case scenario is that your two Redis servers max out their processor, disk, and network bandwidth, and when Sentinel choosing a master, you lose data. Let me explain.

When a machine that acts as a master connects to its own master and sends SYNC, it disconnects all of its slave connections to force them to reconnect and resync. If you have A and B set up as slaves of each other, they will alternate disconnecting, reconnecting, and resyncing data. This is very nasty in practice. DON'T DO IT, unless you like to be in pain and waste your time. Sentinel basically adds a bit of mystery, as it will arbitrarily choose one machine as the new master, potentially setting a machine as master before it even *starts* syncing data.

My advice: set up two slaves, use sentinel to automatically choose a new master/slave on failure, and manually reconfigure the old master before letting it back into the pool.

Regards,

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sheldon Hearn]

On Thursday, March 28, 2013 5:36:50 PM UTC+2, Salvatore Sanfilippo wrote:

Hello, this problem was still not addressed in Sentinel, and I was
wondering about two possible options:

1) Kill the master once it is back again.
2) Setup it as a slave for the new master.

On one level, Sentinel is designed to be "just another client" of Redis. This is great, because it gets to lean on Redis (e.g. not having to reinvent its own pub/sub protocol).

However, I think this approach demands that any method of failed instance reintroduction requires the cooperation of clients.

In other words, I think it will be more valuable to think about how clients identify the active master and slave instances, than to think about how to reduce the window during which their naive instance selection leads to pain. Then it's no train smash for an instance to come up as master momentarily, because clients aren't using it.

If Sentinel operated "out of band" (not as "just another client"), then Redis could support a startup mode in which it was neither a master nor a slave, and in fact was not even accepting client connections. Redis in this state could then wait for Sentinel to tell it what to do, including an "okay, start listening" instruction.

At first, this seemed attractive to me. But it still requires some intelligence from clients, because they must cope with instances that are not listening.

And if there must be intelligence in the client anyway, then I think it's better to preserve the elegant design of Sentinel. Clients need to acquire the active master and slave instance information from sentinel. I don't think we can have "Sentinel is just a client" and "clients don't have to know about Sentinel" at the same time.

So I go for your option 2, leaving risk mitigation to the client, assisted by Sentinel.

Ciao,
Sheldon.

[Sheldon Hearn]

On Thursday, March 28, 2013 6:13:23 PM UTC+2, Josiah Carlson wrote:

No. The worst-case scenario is that your two Redis servers max out their processor, disk, and network bandwidth, and when Sentinel choosing a master, you lose data. Let me explain.

That is definitely a worse scenario! Thank you for taking the time to explain.
 

My advice: set up two slaves, use sentinel to automatically choose a new master/slave on failure, and manually reconfigure the old master before letting it back into the pool.

Thanks, it's good to have a fallback position.

I am trying very hard to avoid the operational burden of manual reintroduction of failed instances. Our teams are used to services behind which you can reboot all nodes in some safe sequence without taking down the service. We can teach them new tricks, but we're talking about meat, documentation and tools. If I can put in a little extra elbow grease to avoid that, great! :-)

If this were a high priority to you, would you consider redis_failover[1] or redis-proxy[2] or something else?

I hope this doesn't sound like I'm criticizing your advice. I know very close to nothing about Redis. I'm just wanting to hear whether your advice changes now that I've explained my operational sensitivities, which I didn't take the time to do initially.

Thanks,
Sheldon.

[1] https://github.com/ryanlecompte/redis_failover
[2] https://github.com/sreeix/redis-proxy

[Josiah Carlson]

My replies are inline.

On Tue, Apr 2, 2013 at 6:42 AM, Sheldon Hearn <shel...@starjuice.net> wrote:

On Thursday, March 28, 2013 6:13:23 PM UTC+2, Josiah Carlson wrote:

No. The worst-case scenario is that your two Redis servers max out their processor, disk, and network bandwidth, and when Sentinel choosing a master, you lose data. Let me explain.

That is definitely a worse scenario! Thank you for taking the time to explain.
 

My advice: set up two slaves, use sentinel to automatically choose a new master/slave on failure, and manually reconfigure the old master before letting it back into the pool.

Thanks, it's good to have a fallback position.

I am trying very hard to avoid the operational burden of manual reintroduction of failed instances. Our teams are used to services behind which you can reboot all nodes in some safe sequence without taking down the service. We can teach them new tricks, but we're talking about meat, documentation and tools. If I can put in a little extra elbow grease to avoid that, great! :-)

I understand completely. None of us wants to get a 3AM phone call from our ops guy saying, "Redis 23 went down, can you help us get a replacement box back online?"

If this were a high priority to you, would you consider redis_failover[1] or redis-proxy[2] or something else?

I don't have personal experience with any automated failure and recovery method, so I don't have practical experience on the situation. I will point out that redis-proxy *does* slow down your available throughput, so if you're running at the limits of Redis (which is why you are setting up slaves, etc.), it's probably not a good answer.

One thing to consider is that you *could* initially configure your Redis instances to come up listening on a different port, configured with no on-disk storage. Then you can run a configuration script at the tail end of your init.d that connects to that Redis server, queries the existing configuration (available via Redis, Zookeeper, or whatever), updates the local Redis configuration, then changes the listening socket* just before telling it to become a slave of the current master.

* I haven't tested whether this works, but if not, that's a patch that I suspect would be very appealing to a lot of people.

That gets you "starts up braindead outside the pool", and with maybe 20 lines of scripts, you can bring it back into the pool. From there, it only needs sentinel-aware clients, and for sentinel to write configuration information to a configuration management server. You will definitely have to be careful with your data backups in this scenario, which will vary depending on your operational expectations on data integrity and persistence.

Regards,

 - Josiah

I hope this doesn't sound like I'm criticizing your advice. I know very close to nothing about Redis. I'm just wanting to hear whether your advice changes now that I've explained my operational sensitivities, which I didn't take the time to do initially.

Thanks,
Sheldon.

[1] https://github.com/ryanlecompte/redis_failover
[2] https://github.com/sreeix/redis-proxy

[Sheldon Hearn]

I have discovered haredis[1].

Wow. Once you've let go of the idea that your clients have to be able to pretend they're only interacting with one process over one TCP socket, the world is your oyster!

Redis's builtin pub/sub is a stroke of genius. It lets clients manage the cluster. And once you get over the inintial crawling skin sensation behind that idea, it's brilliant.

If there are no clients connected, do you care who the master is? If there are clients connected, who better to establish quorum and elect a master?

Now I can have high availability without any of the operational complexity that I usually introduce to achieve it. No CARP, no floating virtual address, no backlinks for heartbeat, no split brain (unless you decide you want to risk it).

I don't mean to detract from the effort going into Sentinel. Sentinel will be great. But it's wonderful to have something else to lean on in the meantime.

[Salvatore Sanfilippo]

On Mon, Apr 8, 2013 at 5:54 PM, Sheldon Hearn <shel...@starjuice.net> wrote:
> Wow. Once you've let go of the idea that your clients have to be able to
> pretend they're only interacting with one process over one TCP socket, the
> world is your oyster!

Hello Sheldon,

this is true, however what Sentinel is trying to do is to provide an
HA system where there are two characteristics that are not otherwise
easy to achieve:

1) Failure is much more objective than, client XYZ can't talk with the
master. You can dislocate sentinels across your network, they need
agreement. In failover systems nothing is worse than a false positive
of failure IMHO: you risk problems without that actual problems are
happening.

2) Allow any kind of client to directly interact with the Redis
master, without intermediate layers. So you can have different kind of
clients: a PHP application, a Ruby script doing cron stuff, a fast C
server serving over HTTP, and so forth. As long they support the
minimal semantics of "before reconnecting a socket, ask Sentinel" they
can live together well in an HA system.

That said Sentinel IS FLAWED :-) Because it is not acceptable that
masters return back and are writable again. This was a design
consideration since the start, but I did the mistake of saying, let's
add it later to ship ASAP. Instead this is a core feature that needs
to be inside ASAP.

For this reason I'm pausing other work, studying a plausible design,
and implementing it in the next days.

Moreover documentation of Sentinel will be improved so that there is a
go-to resource to understand how the system really behaves, should be
configured, and so forth.

[Sheldon Hearn]

On Thu, Apr 18, 2013 at 7:11 PM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> this is true, however what Sentinel is trying to do is to provide an
> HA system where there are two characteristics that are not otherwise
> easy to achieve:

I do hope you didn't receive my previous email as a dismissal of the
future value of Sentinel. I very much look forward to it. :-)

> 1) Failure is much more objective than, client XYZ can't talk with the
> master.

The great thing about Redis pub/sub is that it provides a fast,
reliable, in-band transport for sentinels and/or HA clients to
establish quorum.

In my mind, means that HA clients (like haredis) can achieve as
objective a perspective of the cluster as sentinel can. What do you
think?

> 2) Allow any kind of client to directly interact with the Redis
> master, without intermediate layers.

I think this is why it's important to have a Sentinel *client*.

And for HA clients, the Sentinel *protocolo* will be great. And if all
your clients are HA clients that use the Sentinel protocol... well you
don't need dedicated Sentinel clients. You get to choose, and I think
that will be wonderful.

Ciao,
Sheldon.

[Salvatore Sanfilippo]

On Fri, Apr 26, 2013 at 9:01 AM, Sheldon Hearn <shel...@starjuice.net> wrote:

> I do hope you didn't receive my previous email as a dismissal of the
> future value of Sentinel. I very much look forward to it. :-)

No prob, I would love if Sentinel was not needed at all and Redis HA
would be trivial, less problems!

Btw in recent commits the initial problem you had was now addressed
(reconfiguration of returning master).

>> 1) Failure is much more objective than, client XYZ can't talk with the
>> master.
>
> The great thing about Redis pub/sub is that it provides a fast,
> reliable, in-band transport for sentinels and/or HA clients to
> establish quorum.

Implementing this client side is possible but not a design I like.

1) Quorum is bound to number of participants you have, and with
clients it is not easy to guess, they change over time.
2) Persistent connections are probably needed, and some way to wake-up
idle connections, otherwise if there is just one active client
everything will stop working.
3) There are complex failure scenarios like the server responding but
very slowly, clients need to have all explicit timeouts to handle
that.
4) You need to take a lot of state client side or need zookeeper or
alike as a coherent repository of the current configuration.

And so forth. You can solve everything and do this client side, but my
subjective consideration is that Sentinel is a better solution to
client-triggered failover.

>> 2) Allow any kind of client to directly interact with the Redis
>> master, without intermediate layers.
>
> I think this is why it's important to have a Sentinel *client*.
>
> And for HA clients, the Sentinel *protocolo* will be great. And if all
> your clients are HA clients that use the Sentinel protocol... well you
> don't need dedicated Sentinel clients. You get to choose, and I think
> that will be wonderful.

This does not change the overall design, at the end of the day you
have different possibilities:

1) A proxy that handles everything and is transparent.
2) Client-driven HA with something like zookeeper to avoid split-brain issues.
3) A completely distributed system like Sentinel.

This is a oversimplification, you have many components:

A) Detection of failure.
B) Point of view of the detection: where the controller is placed on
the network?
C) Action: who will perform the failover.
D) Collaboration: will another device continue or restart the work of
a device that failed, for instance, during the failover?
E) Configuration propagation: how will be clients be informed?

There are infinite solutions, I picked what I think is a good trade
off in Sentinel because while it's a "ready to run" system once you
have minimal client support, you can hack a lot of the above if you
want, by placing sentinels in different places, executing scripts, and
configuring a given quorum.

Thanks for arguing, it's an interesting topic.

Cheers,
Salvatore

>
> Ciao,
> Sheldon.

>
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Sheldon Hearn]

On Fri, Apr 26, 2013 at 10:39 AM, Salvatore Sanfilippo
<ant...@gmail.com> wrote:
> > And for HA clients, the Sentinel *protocolo* will be great. [...]

>
> This does not change the overall design, at the end of the day you
> have different possibilities:
>
> 1) A proxy that handles everything and is transparent.
> 2) Client-driven HA with something like zookeeper to avoid split-brain issues.
> 3) A completely distributed system like Sentinel.

Interesting. In my head, the HA clients of the future would basically
be Sentinels that also happen to do actual db client work. Or
Sentinels would be HA clients that only take care of cluster.

Are you seeing an aspect of the Sentinel role that would be obstructed
by actual db client work?

Hmmm. I think I get it. You're seeing a lot of the Sentinel protocol
being logic that lives in Sentinel, beyond the pub/sub. Getting that
logic right in every client would be a waste of effort and error
prone.

Am I on the right track?

I will be away from civilization for 10 days, but will take Sentinel
from redis-unstable for a spin when I get back. Thanks!

Ciao,
Sheldon.

[Salvatore Sanfilippo]

Hello Sheldon,

the current design is what we'll use in the future, basically :-)
Experiment with other options if you like.

Cheers,
Salvatore