Why does rs.remove() force election?

[tpneumat]

I would like to remove the replica without forcing an election.  Any way to do that? 

More info:  As part of our scale-up/scale-down process, we add and remove servers a lot.  One annoyance is that when you execute an rs.remove(), it does the following according to the docs:

MongoDB disconnects the shell briefly as the replica set elects a new primary. The shell then automatically reconnects. The shell displays a DBClientCursor::init call() failed error even though the command succeeds.

Does an election need to happen every time I remove a secondary from our replica set?  The annoying part is that if you need to remove one replica at a time, you end up chasing the primary around (and other services which depend on the primary staying where it is need connections reset too).

Thanks,

Jeremy

[Hornet]

Add them via dns and dont remove them from the replicaset. Just assign an unused hostname and it will sync backup once its online. (this is how I do it in AWS)

As far as chasing the primary, run rs.freeze(600) on all the slaves and you wont have to worry about the primary role moving for 10min.

I agree tho, seems unnecessary to stop your DB for a few seconds whilst the re-election is done. I suspect its having to recalculate everything when one is removed. But that still (IMHO) should not make the primary change.

-Erik-

--
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongod...@googlegroups.com
To unsubscribe from this group, send email to
mongodb-user...@googlegroups.com
See also the IRC channel -- freenode.net#mongodb
 
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Asya Kamsky]

Why do you need to remove one replica at a time? There is no such requirement.
You can create a new configuration and do a single rs.reconfig command.

Asya

On Thu, Dec 19, 2013 at 3:32 PM, tpneumat <jer...@dilltree.com> wrote:

[Jeremy Dill]

Hi Asya,

Thanks for your reply. rs.remove() is a supported command that makes removing a replica very easy—1 step vs many for building a new config. The main question here is why does rs.remove() force an election—the primary (which is currently being used to run the command) should remain primary, why would it not? rs.reconfig also forces an election.

Sure, there are ways to deal with removing replicas, but none are as convenient as rs.remove(). In response to Erik, doing a freeze on all other replicas means having to connect to all of them first and running that command. Also, I don’t want to leave offline replicas in my set for various reasons including the fact that when we scale up, we may go from 2+arbiter to 10 servers, then back down to 2+arbiter—can’t have 8 zombies in my config.

But my original question stands.. why does rs.remove() force an election?

Thanks,

Jeremy

> You received this message because you are subscribed to a topic in the Google Groups "mongodb-user" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/mongodb-user/SVqx5MikVfE/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to mongodb-user...@googlegroups.com.

[Asya Kamsky]

You say: "the primary (which is currently being used to run the
command) should remain primary, why would it not?" - but we do *not*
and cannot know that! When you remove a node from the replica set,
you change how many nodes it takes to elect a primary - so in fact it
is possible when you remove or reconfig that the primary now won't be
able to stay as a primary.

Asya

[Jeremy Dill]

Sorry if I am not being clear—all I wanting to do is to be able to remove a secondary without forcing an election. If this is not currently possible, I may just open a JIRA request for that feature. In the mean time, I might be able to mitigate things by playing with the ‘priority’ so, when the election does happen, at least it keeps things the way we want them to be. I think that in oder versions, rs.remove did not force an election.

The reason why I want to do this is best explained via an example: Imagine that during a single day, have to scale up gradually and then scale back down. We start with 3 servers. We gradually add secondaries. NOTE: adding a secondary with rs.add() doesn’t seem to force an election—which is how I would think rs.remove() should work. We get up to, say 7 servers. Then over the next few hours, we want to remove the secondaries one at a time (could do this via rs.remove() or rs.reconfig()). However, upon removal, an election happens every time, and this is disruptive and, in our case, unnecessary. WE (the admins) know what servers we have in place, and we know what server we want to keep as our primary, and we don’t want Mongod to work out a new decision for us automatically every time we remove a secondary (no election please). Even if we were to remove ALL secondaries, we should still be able to run a single server replica set, as you would when starting the very first server in the replica.


Thanks,

Jeremy

[Asya Kamsky]

I'm a little confused about adding and removing secondaries - that
doesn't really have anything to do with scaling (sharding is the way
to scale MongoDB - replica sets are primarily for high availability,
automatic failover, etc).

I think of adding and removing replica nodes are a relatively
infrequent operation, but even then, I would expect that you would use
priority scores if you have a preference that primary is always one of
certain subset of servers.

I understand that you to be able to remove a secondary from
configuration without forcing an election, but I suspect that this is
a lot trickier than additive change (since a freshly added node cannot
be immediately eligible for primary, for example, but removing a node
could change who is eligible to be a primary).

I'm quite certain that rs.remove() always forced an election - but
there is a ticket for the functionality you are asking for - it's
https://jira.mongodb.org/browse/SERVER-5788 you should add your
comments there as well as voting for it, so that it will get
prioritized accordingly.

Asya

[Jeremy Dill]

Thanks Asya, the ticket you directed me to is on target. Adding many secondaries allows us to distribute reads during times of heavy load. Sharding would help us distribute writes too, but we don’t need to do that. Our data size in this case is relatively small.

Jeremy