With rs.stepDown(), server remains secondary

[M. Robot]

Hello,

While doing maintenance on a replica set yesterday, I noticed some unexpected behavior with rs.stepDown(), running Mongo 2.4.5. 

We needed to restart the primary server hardware, so we issued rs.stepDown(900), reasoning that 15 minutes should be more than enough time to shut down our machine, come back up, and then step back up to primary. We can see in the log that this command was issued successfully. Secondary became primary.

However, after restarting the machine, the original primary remained secondary — it never stepped back up. After about 30 minutes, we realized we'd have to switch them back manually. We decided to use the same "bug" we had just observed to our advantage, so on the current primary, we issued rs.stepDown(30). As we expected, the servers swapped role and stayed that way. Over 12 hours now, and things are ok.

Is my understanding of stepDown() incorrect? Does an election not get triggered automatically after the given number of seconds has expired? Or is there something in our configuration that could possibly be awry to cause this?

Thanks in advance!

[Francisco Andrade]

Hi there, I think that the stepDown period avoids the server to become primary, but not necessarily forces the server to get back as primary.

I may be wrong, but i guess the server may become primary only if an event forces an election after the stepDown time.

Abraços,
Francisco Andrade

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/5089eeb7-f222-4a7f-b8a9-1601d3d25df8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Andre]

Are your priorities set in order because I believe if the current primary has a lower priority over the others it will hold on to that state. we've had similar scenarios in our cluster when we were playing around with priorities, after reset it all worked great.

[Asya Kamsky]

Hold on - lower priority means the node cannot be a primary over
another with higher priority.

As far as original post - that is the expected behavior. You had
node1 primary, node2 secondary. You told node1 to step down, node2
became primary.

Why would you expect node1 to take over once it rejoined? All nodes
in a replica set are equivalent and you want to minimize the number of
elections, why not leave them with node2 as primary and node1 as
secondary? Primary is a role, not a node designation...

Asya

> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user"
> group.
>
> For other MongoDB technical support options, see:
> http://www.mongodb.org/about/support/.
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> To post to this group, send email to mongod...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mongodb-user.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mongodb-user/0e2f5cee-0199-4eb5-8318-190ce7c27474%40googlegroups.com.

[M. Robot]

I think you nailed it, Francisco Andrade. I assumed that since you designate a time with rs.stepDown(), the set would wait that long before automatically holding an election (or just returning the node to its primary state). But I should have taken the documentation more literally. The time argument is the time that the node will not attempt to become the primary if there is an election. Perhaps the documentation could be clearer (http://docs.mongodb.org/manual/reference/method/rs.stepDown) — or I may be the only person who this has confused. 

Andre, you make a good point — I should set the priority for the nodes. (The reason I'd like one node to be primary and the other secondary is that these reside on a system that many people access; each server needs to have a consistent role to avoid confusion amongst stakeholders.)

Thanks so much, everyone!

On Thursday, August 14, 2014 11:46:26 AM UTC-4, M. Robot wrote:

[Asya Kamsky]

>  I should set the priority for the nodes. (The reason I'd like one node to be primary and the other secondary is that these reside on a system that many people access; each server needs to have a consistent role to avoid confusion amongst stakeholders.)

I would recommend against this.  Presumably people are accessing these nodes as a replica set - if they use a correct format connection string, then the client (via driver logic) will automatically pick the primary, whichever node that happens to be at that moment.

Asya

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7d90cff7-0182-48cd-802d-c9555885a3ec%40googlegroups.com.

[Glenn Maynard]

On Fri, Aug 22, 2014 at 3:45 AM, Asya Kamsky <as...@mongodb.com> wrote:

>  I should set the priority for the nodes. (The reason I'd like one node to be primary and the other secondary is that these reside on a system that many people access; each server needs to have a consistent role to avoid confusion amongst stakeholders.)

I would recommend against this.  Presumably people are accessing these nodes as a replica set - if they use a correct format connection string, then the client (via driver logic) will automatically pick the primary, whichever node that happens to be at that moment.

And, if you force one node to be primary most of the time instead of allowing it to be handled normally, then you may introduce new confusion.  By configuring it that way, you're allowing users to incorrectly configure their clients to only know about the usually-primary node, and not ensure that it really does work as a replica set.  If that server goes down and a different node becomes primary, those clients may not actually work.  If you configure the server correctly, this problem is more likely to be noticed sooner.

In fact, the description above sounds like he's encouraging this error and telling people to connect to a single node, instead of configuring the replica set correctly.  It may be simpler to never expose the actual Mongo servers at all to clients.  Instead, run mongos (even if you only have one shard) and have people connect to that instead, so the replica configuration is hidden to the user.

--
Glenn Maynard

[M. Robot]

Glenn and Asya,

That makes sense. Your comments led me to check our connection strings and make sure they were configured correctly for a replica set, and they were not optimized for the situation.

I agree that the strategy of preferring one over the other for primary encourages faulty practice. The obstacle that must be overcome in this case may be a political one, rather than a technological one — convincing stakeholders that having servers with fluid roles is best. A battle I'm willing to have.

Thanks again!