Question about MongoDB-replication-guide

[Mike Fisher]

I have read in the MongoDB-replication-guide from the MongoDB website (http://docs.mongodb.org/master/MongoDB-replication-guide.pdf) the following statement on page 28, under section "2.3.1 Replica Set Elections":

 

"If a majority of the replica set is inaccessible or unavailable to the current primary, the primary will step down and become a secondary. The replica set cannot accept writes after this occurs, but remaining members can continue to serve read queries if such queries are configured to run on secondaries."

 

Is this statement true?  Unless I misunderstand the above statement, this scenario seems very "BAD" to me.  To me this statement is telling me that if I have a Replica set with three members that spans three data centers; i.e. data center 1 contains the primary member, data center 2 contains a secondary member, and data center 3 contains a secondary member; and if data centers 2 and 3 go down, then data center 1 will have the primary member revert to being a secondary member.  This would be very bad for a production system running on data center 1.  In this scenario, the primary member would become a secondary member and no more write operations would be allowed by the production system running on data center 1.  Is this true?  Why would anyone want this behavior?  I can see a situation where one would want to take down data centers 2 and 3 for maintenance, or they might go down for some kind of failure, but that should not affect the production system running on data center 1.  Can anyone please explain this statement in more detail to me?

[Ashish Kaushik]

This is to confirm that the statement you pointed out is absolutely true and we have recently faces it in our production system. Looking for a method where I can solve this happening from on my production system at least.

Any insights from the community is helpful.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
To post to this group, send email to mongod...@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/d52cba00-903c-4e88-b61c-2c8e00a4c4ab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Reno]

This is by design. Consider a replica set with one member in each of 3 data centers, DC1, DC2, and DC3. Mongo1 is primary and mongo2 and 3 are secondary. Now a network partition occurs which isolates DC1 but mongo1 stays up. The Mongos in DC2 and DC3 are unaffected, constitute a majority and vote Mongo2 as primary. If Mongo1 does not step down you will have a replica set with 2 primaries.... Very bad. Some writes may go to Mongo1 from local users and other writes will go to Mongo2 from users on the other side of the partition. When the partition clears, it will be very difficult or impossible to reconcile the two primaries.

The solution is to add data centers and mongo processes until you feel comfortable that you will always have a majority up :). Or accept that if you lose two data centers you will have to manually break the replica set at the surviving data center to allow writes again.

[Mike Fisher]

John,

    Thanks for the reply.  Rather than adding additional data centers, could I achieve the same result by adding additional secondary members to the replica set at data center 1 to assure that data center 1 will maintain a majority and a primary member in the case that both DC 2 and DC3 go down?

[John Reno]

Yes, if a majority of the replset instances are in DC1 than a loss of DC2 and DC3 will not affect your primary in DC1 as it will still be able to contact a majority. However if DC1 goes down you will still have to do a manual reconfiguration to move the primary to DC2 or DC3.

In short, there is no way to guarantee a majority of the instances will always be available if enough of your nodes go down; all you can do is add diversity (multiple servers, racks, power supplies, routers, ISPs, data centers,...) to reduce the possibility of losing contact with a majority of the replicas. But one node in 3 geographically diverse data centers gets you 95% of the way there.

I am currently building a sharded cluster made up of 5 member replica sets, two members at the main data center, two at our alternate location and an arbiter in a third location. Both members at the main site have priority 9 and the members at the alternate site have priority 1. This causes the main site members to be preferred primary; I do that because our analytic batch jobs run at the main site so I want the primary to be there if possible. If you have a good enough network (we don't) it shouldn't matter where the primary is, but you will find out it usually does.

With this setup, I can take down either of the members at the main site and be assured the primary will remain local. If both nodes or the main site go down, the replicas at the alternate site consult with the arbiter to see if one of them should become primary. If I lose a node during maintenance the replica set remains writeable. It's as much availability as I can afford.

You are correct to worry about a failure during maintenance. Our operations group tends to do all kinds of updates in short outage windows so a firewall reconfiguration at the same time as an OS patch/reboot at the same time as I'm upgrading a database is very possible.

Hope this helps

[Mike Fisher]

John,

    Thanks for the great insight!  I think that this answers my question.

[Darshan Shah]

John,

What do you think of using replicaset member priority to solve this?

For example. primary at DC1 has a priority of 5 and secondaries at DC2 & DC3 have a priority of 2.

So in the event of a network partition or DC2 / DC3 going down, primary at DC1 would continue to be primary.

But in the event of the primary going down, one of the secondary would step up, but as soon as the DC1 primary is available again and caught up, it automatically becomes the primary

I did a small test of this scenario and saw it work - but want more inputs and to confirm my assumptions

Thanks!

[Wan Bachtiar]

What do you think of using replicaset member priority to solve this?
For example. primary at DC1 has a priority of 5 and secondaries at DC2 & DC3 have a priority of 2.
So in the event of a network partition or DC2 / DC3 going down, primary at DC1 would continue to be primary.

Hi Darshan,

By ‘solving this’, I assumed that you are referring to the original question on this thread. Which is, if the majority of the replica set is inaccessible to the current primary, will the primary step down and become secondary ?

To clarify, the priority settings of replica set members is to affect the outcomes of elections for primary, and not to affect the fault-tolerance of the replica set. Using your example, if the two secondaries on DC2 and DC3 (majority) are not available to the primary on DC1, the primary will still step down to secondary regardless of priority values. See also Consider Fault Tolerance.

Setting the priority value is suitable if you prefer that the member(s) in one data center (i.e. DC1) be elected primary before the members in the other data centers. As you mentioned, in the event of the primary loss, one of the secondaries would step up temporarily until the member on DC1 is available again and caught up. See also member priority.

You may also be interested in:

• Replica Sets Distributed Across Two or More Data Centers
• Replica Set Deployment Architectures
• Replica Set: Factors and Conditions that affect elections

Regards,

Wan.

​

[Darshan Shah]

Thanks Wan for the details.

In case the member on DC3 is a hidden member, would this still hold good?

That is if DC1 cannot see DC2 & DC3, would it still step down to become secondary?

Thanks