Redis replication across data centers.

[Sairam M P]

Hi,

We are using redis-2.8.17 in our production servers. For each redis server(master) we maintain a slave for HA in the same data center(dc). So in case of any issues in the master redis server we have slave as back up and once the slave takes over(as master) either the old master will be rejoined as slave or a new server is provisioned as slave. This way we are ensuring the HA of redis servers inside the dc.

In our infrastructure we have a stand by DC for unforeseen disasters in the primary DC. In case of any disasters or issues in the primary DC, we need to open up the secondary DC without any delay. So our requirement is to maintain a upto date copy of the redis data at the secondary DC from primary DC. Once the secondary DC takes over(as main DC), the primary DC should be upto date with the secondary DC. 

*********************It will be helpful if someone can through some lights on the flawless possible ways to achieve our above requirement and how other redis users are doing the inter-dc replication *********************************

We were able to configure replication between primary DC and secondary DC. But this replication comes with its own cost. As the psync is used by redis internally for only a limited cases, for all other cases a full sync has to be performed and there by lot of data transfer across the network. We also increased "repl-backlog-size, repl-backlog-ttl, repl-timeout, client-output-buffer-limit" redis parameters to reduce the full sync cases. But still it doesn't looks like a fool proof solution.

Also after making the secondary DC as main DC, we are forced to configure a full sync(as there is no binlog position kind of thing in redis as like mysql) from secondary DC to primary DC(again a lot of data transfer across the network, which might saturate the network bandwidth). This is because we have 'n' number of redis servers with ~20GB of data in each. 

To avoid the challenges in redis full sync, we had an idea of capturing the modified/new keys from the client and an asynchronous thread fetching the values of these keys from one DC and writing it to the other DC. But we are not sure of the practical difficulties in implementing this solution and we would be grateful if someone can share the ideas on same. 

In short : Both the data centers(whichever happens to be the main/stand by DC) should be in sync all the time, kindly help us on achieving this with a better solution.

-Sairam.

[Dvir Volk]

I've ran a cross DC redis database for about a year using PSYNC. a few insights:

1. the replication and timeout factors indeed needed a lot of tuning, but some times you will have to do a full resync, no matter what. I don't remember what we did exactly but there were a few non trivial settings we had to do.

2. In out case there was only one main DC, and if it was down the secondary one was read only. I know it's not usually possible but this approach simplified a lot of things - the serving DC never writes anything, it might just enqueue writes to be performed on the main DC, but that's all. And it could work in read only mode flawlessly.

3. You shuold not have N slaves in a remote DC reading from the master. Use chained replication. i.e. in DC 2 have a local master, who's a slave of DC 1's master. That way, if there is disconnection between them, only one instance needs to resync.

4. This is indeed problematic with very large datasets. Working with smaller datasets (<10G) is much easier.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.