Redis Cluster update issue

[Lax kota]

On a mesos provisioned cluster I happened to spawn a 3 node Redis Cluster. After a while of running one of the Redis instance would go down and Mesos would go relaunch the instance.

But right after this happens, I end up being in a state where each node sees (cluster nodes) only one more node instead of 2. As a result when I try adding the 3 node, I would get an error stating 'node is not empty. either the node already knows other nodes or contains some key in database' 

What is the best way to get out of this issue? Is there a cleaner way to form cluster membership in this case? 

[Jan-Erik Rediger]

Hey,

first of all a 3-node Redis Cluster won't get you much HA.
If you need that, you need slaves for each master node.

Does Mesos provide a completely new node?
Or just restart the failed one?
Restarted nodes will simply rejoin the Cluster.

It sounds like you're trying to add a node to the Cluster again, but it
is telling you that it still thinks it is part of some Cluster and thus
has some data around.
Check its nodes.conf (or whatever you set cluster-config-file to).
For nodes that were once part of a Cluster this will contain information
about itself and other nodes. For fresh nodes this file should not even
exist.

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[Lax kota]

Thanks for the response. 

Due to limitation of no of nodes one could launch (until the cap is increased internally), we are launching 3 node cluster(s) though recommended is to use minimum of 6 node cluster. 

Setup got cleaned up now, cant remember if it brought back on same node. But then I was launching Redis a in-memory  mode (so when my container dies and get re-launched it should come back empty). Some how when issue happened I noticed each node having itself and one more node entry.

Right now re-running the tests again with 6node cluster.

Thanks

Lax

[hokie...@gmail.com]

I just started with Redis cluster, but when I see this message, it's because the node where a redis server is failed is the same node that your scheduler (I am using Marathon) is attempting to launch a replacement on that node.

Either get your scheduler to launch the new Redis server somewhere else or I think running cluster forget on dead, disconnected node *may* fix it also.

--John