3 node replica set - 2 nodes are down - how to make the only remaining node PRIMARY ?

[leck...@gmail.com]

Hi,

I am running quite usual "by the book" 3 node replica set configuration on 3 differnet nodes.
It is running well, now I am running some failure scenarios.

I noticed that if I stop all 3 mongos, then I try to run just mongo on primary.

It issues many error like this

2016-08-17T13:39:46.992+0200 W NETWORK  [ReplicationExecutor] Failed to connect to 10.139.123.7:27017, reason: errno:111 Connection refused
2016-08-17T13:39:46.993+0200 W NETWORK  [ReplicationExecutor] Failed to connect to 10.139.123.5:27017, reason: errno:111 Connection refused
2016-08-17T13:39:46.993+0200 I REPL     [ReplicationExecutor] transition to STARTUP2
2016-08-17T13:39:46.993+0200 I REPL     [ReplicationExecutor] Starting replication applier threads
2016-08-17T13:39:46.994+0200 I REPL     [ReplicationExecutor] transition to RECOVERING
2016-08-17T13:39:46.996+0200 I REPL     [ReplicationExecutor] Error in heartbeat request to 10.139.123.5:27017; HostUnreachable: Connection refused
2016-08-17T13:39:46.997+0200 I REPL     [ReplicationExecutor] transition to SECONDARY
2016-08-17T13:40:19.126+0200 I REPL     [ReplicationExecutor] Not starting an election, since we are not electable

I would expect that primary would at least start as PRimary, but no, it just remains as SECONDARY.

Is this behavior OK ?
I am thinking that following the election logic this is ok. mongo process sess it is alone, other 12 are not reachable so there is no election, no PRIMARY.

I am thinking what should I do if it happens that my other 2 nodes go down for longer time.

I am thinking that in such scenario the best solutio is to use FORCE reconfigure and config just the ione node into new replica ?

Here is a copy/paste of another thread:

"You can force reconfigure the replica set to drop Server 2/secondary and Server 2/arbiter from the rs.config().  Essentially creating a 1 node replica set.

https://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

As per the documentation:

"Use this procedure only to recover from catastrophic interruptions."

"

Thanks!

[William Hagan]

Hello,

Basically, in a replica set, a primary node can only come about if a majority of the eligible voting members (nodes) are available.

If a majority of the voting members are not available, then the available nodes will become secondary nodes.

In a 3-node replica set (like you have), where all nodes are eligible to vote, you will require at least 2 of the nodes up before you can have a primary node.

All said, as you have noted, we can with MongoDB, force the available members that do not form a majority, to continue normal operations pending when the other members of the replica set are available.

Cheers

[leck...@gmail.com]

Thank you @William Hagan for the answer. You confirmed that the procedure that I suggested is the right one.