MongoDB automatic fail over between 2 data centers

[Virendra Agarwal]

Hi,

If I deploy a MongoDb replica set across only 2 data centers, if one entire data center fails, can there be automatic failover to the other data center?

There is a scenario where we have 5 member replica set across two data centers.

Data center 1 : 3 Members

Data center 2 : 2 Members

Situation is if DC 1 goes down DC 2 becomes read only but we need to keep it ready to accept writes also.

Is there any possibility for same using one extra member like hidden node in DC 2 which can be nmanually updated to secondary in acse of DC 1 failure.

Will this solution be able to elect new primary.

We do not have third data center.

Adding a third data center to house the arbiter is not an option.

Thanks

[Rhys Campbell]

I think you'd need to add two new hosts to DC2 in this event so you would then have a replset with 7 members. This would then give you 4/7 votes to be able to elect a primary. This would obviously take a bit of time if there's a large amound of data to pull on the initial sync. It would also be possibe to reconfig one of the DC2 mongo hosts to accept writes in standalone node https://docs.mongodb.com/manual/reference/method/rs.reconfig/ (Probably also possible master/slave type setup).

Here's a thread about Master/Slave setup https://groups.google.com/forum/m/#!msg/mongodb-user/YOMjysriMzM/2e1920lRBgAJ

You should test this in your own environment to ensure it works for you.

As you already know you really need DC3.

Cheers,

Rhys

[Kevin Adistambha]

Hi,

If I deploy a MongoDb replica set across only 2 data centers, if one entire data center fails, can there be automatic failover to the other data center?

As pointed out by Rhys, automatic failover will happen with the set still accepting writes (i.e. with an active primary) if the failing data center contains fewer members vs. the operational data center.

Using your example, if datacenter 2 went offline, the remaining 3 members of the replica set still forms a majority (3 out of 5) and can elect a primary between themselves.

However, should datacenter 1 went offline, there is not enough members in the remaining datacenter to elect a primary (2 out of 5), thus it cannot elect a primary, and the set will become read-only.

This design is intentional, since you can never be sure if datacenter 1 is really offline, or is there a network issue preventing communications between the two datacenters. If you allow members in datacenter 2 to elect a new primary, you could potentially have two primaries in the replica set operating simultaneously. In this “split brain” scenario, it’s difficult to reconcile the writes that happened between the two primaries. Hence a replica set requires a majority vote to be able to elect a primary.

Adding a third data center to house the arbiter is not an option.

Unfortunately you cannot have it both ways and have automatic failover. You cannot have high availability with a guarantee that no split brain scenario can occur, while at the same time require that no arbitration could take place. Without an arbiter, manual intervention will always be required, since there is no way for the members of a replica set to know whether a datacenter is truly offline or it’s just a temporary network issue that could last only a short moment.

One possible intervention method would require you to manually reconfigure the replica set in datacenter 2, by removing two of the members in datacenter 1. After this reconfiguration, it will appear that only one member is offline, and the remaining members in datacenter 2 can thus elect a new primary (since it will be 2 votes out of 3).

Having said that, adding/removing members manually involve a certain amount of risk, not to mention that MongoDB replica set was never designed to work like this.

Please note that it’s generally not recommended to perform writes to a member of a replica set while being started as a standalone node. On another note, the old master/slave model was deprecated and is not recommended anymore for new deployments.

For more information, please see Replica Set Deployment Architectures and Replica Sets Distributed Across Two or More Data Centers.

Best regards,
Kevin

​

[Virendra Agarwal]

I think in that case to avoid manual intervention third DC is the solution with one arbiter.

It should work in split brain scenario too. Pleas elet me kwno if tehre might be some more issues.

Thanks

Virendra Agarwal