Migrating a Redis Cluster

[David Thomson]

Hi,

I've recently joined a project that switched to a Redis Cluster (an actual Redis Cluster, currently running on 3.0.0). The cluster consists of 4 masters, 4 slaves and 1 floating slave, and is being used as a cache. Each node has around 1.25m keys, and about 6.25Gb of data.

Each node in the cluster runs on an AWS EC2 instance, but for various reasons we'll need to migrate the nodes and data to brand new instances in the near future. 

Our current thoughts on how to approach this is as follows:

- Bring up the new cluster in the same configuration (4 masters, 4 slaves, 1 floating slave)

- Set the new master nodes to be slaves of their equivalents in the existing cluster (with a slave-priority of 0)

- Once replication is complete, alter the configuration of our app to point to the new cluster

- Change the slave-priority of the new master nodes

- Takedown the old cluster once we’re happy things are OK

Does this sound reasonable? Would it be better to move things over one master at a time? My assumption here is that we only need to replicate the new master nodes from the old master nodes, and the slave set up on the new cluster will act as normal.

If anyone has any experience of doing such a migration, or there’s a suggested method for handling this, please let me know.

Thanks in advance,

-David

[Josiah Carlson]

Aside from the missing details of your configuration change, this sounds fairly reasonable.

I would personally try to do a single master at a time, and maybe try to pause writes for a few moments when swapping individual masters out.

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[David Thomson]

Hi Josiah,

There's no real configuration change - we just have to move to a new set of machines in AWS.

Unfortunately, my investigations yesterday revealed that you can't set a cluster node to be a 'slaveof' anything else, so my original theory would seem to be a non-starter unless there's something I'm missing.

I can use the redis-trib tool (with minor modifications) to migrate data from an old master node to a new master node (using import command with copy and replace enabled), and that seems to get all the data across. If I run that for each master node (redis-trib doesn't want the source to be a cluster) then that gets all the data across while keeping all nodes active. That seems to work well enough (at least well enough for our purposes).

An alternative approach I thought of last night was to add the two clusters together and reshard to eventually take the older nodes out of the cluster all together - I've not done any experiments with that as yet, however. 

Thanks,

-David

[Josiah Carlson]

Your description of your results lead me to believe that I misunderstood your original intent. Yes, adding the new machines to the cluster, then using redis-trib to migrate shards over should work.

 - Josiah