Synchronization in HA configuration

[Didier Spezia]

Hi,

here is a question for people using an HA Redis configuration.

I consider in any HA Redis solution, we need:

- some kind of resource management: starting/stopping Redis instances,

  relaunching failed instances, etc ... It can be covered by the OS init process

  (initd, launchd, systemd, upstart), or an ad-hoc process

  (supervisor,god,monit, etc ...)

- some kind of service/role management, deciding who the masters/slaves are.

  It can be covered by Sentinel, redis_failover/Zookeeper or other similar

  software.

- a mechanism so the clients can reconnect on alternative Redis instances.

  It can be a router (like haproxy), or some tricks regarding the IP address

  (like with LVS), or just the master/slave topology distributed to the (smart)

  clients.

Synchronization is also required between the various mechanisms.

For instance, let's take a simple master/slave configuration, with no

persistency option. Instance A is the master, instance B is the slave.

If A fails, the service/role manager will try to promote B as a new master,

but the resource manager will also try to launch A again.

If A is relaunched before the SLAVEOF NO ONE command is executed on B,

B will reconnect to a fresh A instance which happens to be empty

(no persistency). The data on B will be wiped out, and once B is

promoted to master, the data are lost on both instances. So it is really

critical the service/role manager takes precedence over the resource manager.

Now, I'm just curious: what are your solutions to synchronize the activities

of the management processes?

Thanks.

Regards,

Didier.

[Matthew Palmer]

On Thu, Aug 09, 2012 at 09:36:10AM -0700, Didier Spezia wrote:
> here is a question for people using an HA Redis configuration.
> I consider in any HA Redis solution, we need:
>
> - some kind of resource management: starting/stopping Redis instances,
> relaunching failed instances, etc ... It can be covered by the OS init
> process
> (initd, launchd, systemd, upstart), or an ad-hoc process
> (supervisor,god,monit, etc ...)

Pacemaker.

> - some kind of service/role management, deciding who the masters/slaves are.
> It can be covered by Sentinel, redis_failover/Zookeeper or other similar
> software.

Pacemaker.

> - a mechanism so the clients can reconnect on alternative Redis instances.
> It can be a router (like haproxy), or some tricks regarding the IP address
> (like with LVS), or just the master/slave topology distributed to the
> (smart)
> clients.

https://groups.google.com/group/redis-db/attach/7d6240fe06dc8cf9/live-rebind.patch?part=2&authuser=0

> For instance, let's take a simple master/slave configuration, with no
> persistency option. Instance A is the master, instance B is the slave.
> If A fails, the service/role manager will try to promote B as a new master,
> but the resource manager will also try to launch A again.

You'd really, really hope not. That's the whole point of the cluster
manager -- it takes care of ordering of promotion/demotion and resource
takeover to ensure those sorts of shenanigans don't occur.

> If A is relaunched before the SLAVEOF NO ONE command is executed on B,
> B will reconnect to a fresh A instance which happens to be empty
> (no persistency). The data on B will be wiped out, and once B is
> promoted to master, the data are lost on both instances. So it is really
> critical the service/role manager takes precedence over the resource
> manager.

Well, if you run your cluster without persistence, the data should be
something you can afford to lose. Just like RAID isn't a substitute for
backups, neither is replication a substitute for AOF (or at least RDB).

> Now, I'm just curious: what are your solutions to synchronize the activities
> of the management processes?

As above -- Pacemaker and IP address takeover.

- Matt

[Didier Spezia]

>> As above -- Pacemaker and IP address takeover.

Thanks for the answer.

Yes, physical clustering can be a solution.
I've found the following recipe for MySQL replication which I think can be adapted to Redis:

http://www.percona.com/files/presentations/percona-live/dc-2012/PLDC2012-high-availability-with-percona-prm.pdf

https://github.com/jayjanssen/Percona-Pacemaker-Resource-Agents/blob/master/doc/PRM-setup-guide.rst

Now to make a physical cluster reliable, we need a private network, an optional SAN/DAS to host a quorum device,

a fencing device to implement STONITH, and a sysadmin to setup everything. My feeling is not that many people

actually run Redis on such configuration (but perhaps I'm wrong?).

That's why I was more interested in automated HA solutions involving logical clustering only

(even if they are probably less reliable - it is debatable). 

So, any people out there planning to run something like Sentinel on production setup?

Or using things like Zookeeper or Doozer as a role manager?

What do you use (or plan to use) for your automated HA Redis configuration?

Thanks.

Regards,

Didier.

[Matthew Palmer]

[Please don't Cc me; I read the list]

On Sat, Aug 11, 2012 at 04:12:25AM -0700, Didier Spezia wrote:
> >> As above -- Pacemaker and IP address takeover.
>

> Yes, physical clustering can be a solution.

"Physical clustering"? What is that supposed to be?

> Now to make a physical cluster reliable, we need a private network,

You don't need a private network, but it does help.

> an optional SAN/DAS to host a quorum device,

Yes, "optional". I've never used a separate quorum device in any cluster
I've setup.

> a fencing device to implement STONITH,

Well, yes, you do need a way to avoid split-brain, but you need that
regardless of the clustering technology you use.

> and a sysadmin to setup everything.

Well, yes, you do need someone who knows what they're doing, but you need
that regardless of the clustering technology you use.

> My feeling is not that many people
> actually run Redis on such configuration (but perhaps I'm wrong?).

Probably not. Doesn't mean it isn't the right way to do it, though.

> That's why I was more interested in automated HA solutions involving
> logical clustering only
> (even if they are probably less reliable - it is debatable).

Can you please define the difference between "logical" and "physical"
clustering? I don't understand what you're getting at.

- Matt

--
A committee is a cul-de-sac down which ideas are lured and then quietly
strangled.
-- Sir Barnett Cocks (1907-1989) (QOTD 20 Feb 2003)

[Didier Spezia]

>> "Physical clustering"?  What is that supposed to be? 

>> Can you please define the difference between "logical" and "physical" 
>> clustering?  I don't understand what you're getting at. 

Well, it is just a terminology we have here.

What I call physical clustering is a cluster relying on the OS kernel

and hardware to implement HA mechanisms (VIPs, quorum devices,

private network, fencing, etc ...) Physical clusters can be used to

implement share-everything (Oracle RAC) or share-nothing stores

(MySQL master/slave). In CAP terminology, they are typically used to

support CP systems. In case of failure, you get reliability and

consistency. However, when the cluster reconfigure itself to evict

nodes, VIPs are moving (i.e. at a point the addresses are not anymore

on the network), filesystems may have to be unmounted/mounted,

and it takes time. So you only have "eventual" availability.

Logical clusters are implemented without any support from specific OS

level services or hardware, just by using fancy distributed algorithms

like Paxos ( http://en.wikipedia.org/wiki/Paxos_(computer_science) ).

They are typically used to implement AP share-nothing stores (still

using CAP terminology). Examples of well-known AP logical clusters are

Cassandra and Riak. Couchbase, or MySQL Cluster are other examples,

even if I'm not sure they are strictly AP systems.

One of the benefit of logical clusters is availability may be guaranteed while

the cluster is reconfiguring itself (this is due to the fact no VIP is used).

With this terminology, Redis Cluster is clearly a logical cluster

(somewhat similar to Couchbase).

Redis Sentinel is trying to provide a simple HA solution by emulating a logical cluster,

using unconventional, simpler, minimalist algorithms. I guess the reliability of

this approach remains to be proven, but it is a very interesting project.

Regards,

Didier.