add-node -> how safe is it?

[Kevin Johnson]

Hi Folks,

I've been testing my Redis Cluster under load, and recently performed an add-node operation as such:

ruby ./bin/redis-trib.rb add-node --slave --master-id MASTER_NODE_ID NEW_NODE_IP:6379 RANDOM_NODE_IP:6379

After executing this command, I noticed that the new node was immediately added to the cluster topology and began serving requests while it was still syncing with the master even though it didn't even have its data set loaded into memory yet.

Of course all of these requests resulted in key space misses causing client failures during the entire sync process, which in my case takes hours to complete.

Is this the expected behavior when adding nodes at runtime? Is there a less intrusive way to scale up the cluster while avoiding what is in effect an outage in my use case?

Thanks!

Kevin

[Tuco]

As per my understanding, it is a good practice to read data from master nodes only. In your case, you are trying to read from slaves.

[Kevin Johnson]

Thanks for the reply.  Our cluster is configured to read from slaves because we can tolerate stale reads and we can scale out more easily to support  our extremely high transaction rates (> 4 million OPS).

Ideally it would be ideal to be able to add a new slave node that doesn't actually come on line until the master SYNC (full) is completed.  Anyway, I'll try setting slave-serve-stale-data to "no" and see how that works.

[Tuco]

Not sure whether adding the param which you mentioned will help. What I meant to say is that the application should try to read data only from masters, so no redis config change will be required.
Not sure about the 4+ versions, but in the earlier cluster versions, slave can try to repeatedly sync whole data from master whenever there is a slight network fluctuation, so that it will be unavailable when it is syncing or loading data in memory.
What I meant is to change the application to read from masters.