Running Redis in a multi data center environment

[Mike Peters]

Hi,

We're running Redis across two data centers (east coast / west coast).

The web servers in each data center connect to the local Redis, but
are designed to fail-over to the remote Redis if the local one fails
to respond.

The data saved into Redis by the web-servers in each location, is
global and need to persist in both locations.

To meet our needs, right now we are using a jobqueue on both locations
which keeps a transaction-log of all local Redis writes and transmits
them to the remote Redis (multi-threaded). While this works well, I
wonder if this home-grown master-master replication, is what others
use too, or if there's a more elegant solution.

We want data to persist in both locations and we want to avoid web-
servers connecting to a remote Redis unless the local one is down. So
a master-slave setup across the two data centers is ruled out.

Another setup we considered is:

= East coast:
* node1: Master_East1
* node2: Slave_of_West1

= West coast:
* node1: Master_West1
* node2: Slave_of_East1

This requires x2 the number of machines and forces the web servers to
hit 2 nodes (master+slave) and merge the data locally. We don't like
this approach.

-

Would really appreciate any feedback from the community about the
recommended architecture for this use case.

Yes, I know it's not supported and is tricky. But one way or another
we need to support high availability across two data centers and
partition tolerance.

Thanks,
Mike Peters

[Jak Sprats]

Hi Mike,

is security an issue? are you just sending replication streams over
the internet? if you need to encrypt it: stunnels are pretty simple to
slide in.

At a high level your architecture seems a little bit odd in its
failover decisions. If EastWebserver notices EastRedisMaster is down,
then failing to WestRedisMaster means your webserver and your redis-
server are communicating OVER the internet which is both slow and not-
secure.

This might be annoying, but I am going to not answer your question,
but rather ask a question :) How partitionable is your data? can you
partition it to user? If so, then what you are saying is doable, if
not (i.e. your data is highly inter-related and non partitionable)
then having master<->master replication in a geo distributed fashion
has loads of problems (mainly due to replicating the data over the
slowest link in the chain {the internet}).

Can you give more information on your data schema, and its
partitionableness.

- jak

[Burt]

Hi Mike,

Can you give more detail on how did you implemented this master-master
replication?

I'm willing to do the same but only for local HA.

tks

[Mike Peters]

Hi Jak,

Thank you for your response - much appreciated!

We're doing this for redundancy. Our application provides the e-
commerce infrastructure for 3,000 businesses. We simply cannot afford
to go down and need to stay up and running, even in the case of losing
an entire data center (it happened before).

We cannot afford to be down for a single minute and need automatic
failover, with no disruption in service.

Under normal operating conditions, our global load balancer connects
users in the East Coast with the East Coast web servers/redis/mysql
etc. and users in the West Coast with the West Coast web servers/redis/
mysql etc.

Consider this simplified scenario -

A customer begins his transaction on the East coast. A few seconds
later, the entire East coast data center goes down. That same
customer is now redirected to the West coast. All data should be
there for the customer to seamlessly continue transacting with the
site.

Our redundancy protocol calls for supporting three types of disasters:
#1. We lose one local redis node (no problem, because we have a few in
each data center and they're all in-sync)
#2. We lose all redis nodes in the local data center due to a software/
hardware problem => switch to using the remote redis
#3. We lose the entire data center => customers are redirected to the
remote data center

Under #2 - obviously things will get a bit slower. Not too bad and
it's generally only going to last a few minutes at most.
Under #3 - we simply need to ensure that data persists from the local
redis to the remote one, just like MySQL master/master replication
works.

Security is not a concern for us because both data centers are
connected via a private network and all information streams across
private ip addresses, outside of the public Internet.

As I said, right now the solution we're using is our home-grown master/
master redis replication, by running a jobqueue on each side which
keeps a transaction log of all writes and then transmits them to the
remote redis asynchronously.

Thanks again for any insight anyone can offer.

[Josiah Carlson]

Job queues are probably about your best option right now. If you could
partition based on where your customers are connecting to, that may
help at least from a locality perspective (the delayed writes to the
other DCs might not need to be read over there soon). But ultimately
you're going to need to pay the price to replicate your data out to
multiple DCs. Sorry to be a buzz-kill :(

As an aside, how are you dealing with updates to the same data in
multiple DCs simultaneously overwriting each other (thus leading to
data inconsistency)? Are you versioning your data, or are you hoping
it is too rare for it to matter?

Regards,
- Josiah

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

[Jak Sprats]

>
> As an aside, how are you dealing with updates to the same data in
> multiple DCs simultaneously overwriting each other (thus leading to
> data inconsistency)? Are you versioning your data, or are you hoping
> it is too rare for it to matter?
>

This is what I was getting at w/ how shardable is the data. If you
dont shard data and serve it out of 2 locations and then blindly merge
the data, you get inconsistent data (most redis ops are not
commutative). Even if you are doing simple key-value GET/SETs only,
these are idempotent, but not commutative, so you get inconsistent
data ... other seemingly harmless ops like APPEND,LPOP,,,, can lead to
truly inconsistent data.

If you can shard the data to say customer_id%data_center_id there are
loads of replication strategies (namely blind master-master) that will
work fine (to fail a datacenter) so long as the decision to switch
from 2->1 datacenters (or vice versa) is more or less atomic.

Since you are talking about an entire web architecture, and not just
redis failover, then you also need to make sure that your app fails
over (e.g. from 2->1 datacenters w/ redirects to only 1 datacenter)
CONSISTENTLY/ATOMICALLY w/ your redis setup.

It is notoriously easy to miss corner cases doing geographically
redundant setups.