on Cluster backup-restore rdb getting re-written.

[bennet samuel]

I have a 5 node cluster say "A" with 2 slaves per node[ Version of redis is 4.08 ]. I have another cluster say "B" with 5 nodes and 2 slaves per node[ Version of redis is 3.2.11 ].

Both the clusters have rdb  enabled and AOF  disabled.

My intention is to load the data from B to A.

I did the following steps:

1. stoped one of the masters say "M1" in A using service redis stop

2. copied a dump from one of the master in B to one of master(M1)  in A.

3. restarted the master(M1) where the dump is copied with the service redis start command

I observed that the dump file which I copied is re-written and the master is overtaken by slave. So M  became slave to one of its slave.

I tried to make the master the "SLAVEOF NO ONE" but this is giving "ERR SAVEOF  not allowed in cluster mode"

I am stuck and not sure how to restore the data with rdb file. Please help.

[bennet samuel]

@salvatore

Is this a product limitation.? or these things only works if I provision my redis with redislab.

1. why slaveof doesnot work in cluster mode ?

2. how can I restore an rdb to a new redis cluster ? My above steps not working....

3. is there anywhere we get a proper documentation for these questions...?

[Chris Stanaway]

When it comes to redis clusters, you need to keep in mind that the role of master is not assigned or constant.  In your procedure, when you stopped M1 in cluster A, one the slaves (let's call it node2) was automatically promoted to be the new master in that shard.  When M1 was restarted, it checked in with the other nodes in the cluster and found that node2 is currently the master in that shard and so M1 initiated a replication download from node2 to M1.  Since node2 is master, anything that M1 thought it had is considered old/stale/etc and thus dumped in preference for what node2 said is the current data set.  And since this cluster A had no data to begin with (since it is a new cluster), the result of the replication is that M1 has no data.

The redis cluster dynamically manages the master / slave roles and automatically changes roles as needed.  Thus the SLAVEOF commands make no sense in a cluster mode.

I haven't tried this myself, but if you're really wanting to move data from one cluster to another, I'd suggest having all nodes in the new cluster stopped, then copy over the data files and then start the nodes.  Though I wonder if there is other meta data stored in the rdb files that might present other issues.

If the scenario is simply to upgrade your redis cluster from 3.x to 4.x, there is a section the in the redis cluster tutorial that describes how to go about upgrading a redis cluster and it does not involve standing up a completely new cluster.  See https://redis.io/topics/cluster-tutorial#upgrading-nodes-in-a-redis-cluster.

Chris

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[bennet samuel]

@Chris Thanks a lot for responding. I have 8 nodes with 2 slaves each, that is 30 nodes altogether. So just wondering how I will distribute the rdb from old cluster to new. I understand that I need to copy the rdb file to a group of three servers...

Since I cannot guarantee who will become the master and who will be slave, the above process might be some luck for me.

If you have some solution to this issue, let me know. Any way I will try this and let you know the result.

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.

[bennet samuel]

Please read it as 24 nodes altogether.

[Tuco]

Hi Bennet, 

I think initially you can just configure the new cluster to only have masters. So lets say you only have 8 master nodes in cluster B. Assuming you have 8 masters on cluster A. 

Initially you need to verify and match that the slot nos held by masters are indeed the same, ie. If a OldMaster1 holds slots between 0-2047, the NewMaster1 should hold the slots between 0-2047 only. 

Once you have the mapping between the old masters and the new masters, you will need to copy the rdb on the newMasters and start the redis servers. That should be it.

PS. I think redis will allow you to create a cluster with only the master nodes. If it does not, you can create slave nodes, and set the slave-priority to 0 in the conf file of the slaves only, which will make sure that the slaves don't become masters in your case. But still i prefer the only masters approach. Once your data is transferred, you can create the slaves of each master by using "cluster replicate" command.

[Chris Stanaway]

Yes, redis-trib.rb does allow you to create a cluster with only the master nodes.

1. Create your new cluster with just the masters.  redis-trib.rb allows this:  use with "--replicas 0".

2. Once the cluster is formed, shutdown the nodes.

3. Copy the rdbs as Tuco described.

4. Restart the masters.

5. Add the slaves to the cluster using the --slave and --master-id options in redis-trib.rb.

If you have a mismatch between the number of shards/masters in the old and new clusters, then initially create the new cluster with the same number of shards and slot allocations and then copy the data over.  After step 5, before adding the slave nodes, add the rest of the master nodes into the cluster and then reshard the data (both operations use redis-trib.rb).  And then add in the slave nodes.

Chris

To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+unsubscribe@googlegroups.com.

[bennet samuel]

My sincere thanks to Chris and Tuco for leading me the way to the solution. I tried this approach and its working for me. Thanks a ton.