High availability with 2 datacenters

142 views
Skip to first unread message

Alessandro Gherardi

unread,
Feb 13, 2014, 3:23:37 PM2/13/14
to mongod...@googlegroups.com
Consider the following configuration:
- 2 datacenters - say DC1 and DC2.
- Replica set: 1 mongod (primary) in DC1, 1 mongod (secondary) in DC2, 1 arbiter in a 3rd location.
- No sharding.
- 1 app server in DC1, 1 app server in DC2.
- 1 load balancer in a 3rd location.

First scenario: DC1 goes down.
mongod in DC2 cannot talk to mongod in DC1. An election is held, mongod in DC2 becomes primary. All client traffic goes to app server in DC2, which talks to mongod in DC2 which can process both reads and writes. All is well.

Second scenario: Network partition between DC1 and DC2.
mongod in DC1 cannot talk to mongod in DC2 and vice-versa, so an election is held.

If mongod in DC1 wins the election, any client requests going to the app server in DC2 will fail (fail to write and also fail to read if slaveOK is false). So I want the app server in DC2 to stop taking requests. I can accomplish this by having the app server in DC2 fail the load balancer health check - e.g., I can have the health check try to update a well-know dummy document.

Likewise, if mongod in DC2 wins the election I can force all client traffic to go to DC2.

Is this the recommended way of handling this situation? Or is there a better way to do this?


To complicate things further, let's assume that I need to shard the data across 2 replica sets, and that there is a network partition between DC1 and DC2.

Now I need to ensure that the mongod's that win the election on each replica set are located in the same datacenter. Otherwise, if the primary for the first replica set is in DC1 but the primary for the second replicaset is in DC2, the app server in DC1 cannot access data in the second replicaset and vice-versa.

What's the best way to ensure that both replica sets elect mongod's in the same datacenter? Should I assign a different priority to the mongod's in that datacenter?

Thanks in advance.


Andrew Ryder

unread,
Feb 14, 2014, 2:50:38 AM2/14/14
to mongod...@googlegroups.com
Hi Alessandro!

That physical layout is fantastic, that is exactly what we recommend for high availability. Note however that your layout is effectively 3 DCs, with data bearing at only 2 of them.

To your specific question around detecting network partitions via a MongoDB driver:
Certainly writing to a document will confirm if the driver is able to contact a primary but the ability to write is effectively a side-effect of the replica-set status, so I think it would be better to just query the real thing. You can use rs.isMaster() (and related rs.status()) to query the current status of the replica-set. The output of those commands are well documented and I'm sure you'll find what you need there.

I hope this helps.

Kind regards,
Andrew

Alessandro Gherardi

unread,
Feb 14, 2014, 1:50:53 PM2/14/14
to mongod...@googlegroups.com
Thank you, Andrew. Your reply does help a lot.

A couple of follow up questions:
- If a replicaset primary goes down, my understanding is that all writes and reads with slaveOK=false fail until a new primary is elected. How long does an election typically take?
- Is there anything that can be done to speed up the election process? If yes, what are the con's of doing that?

Asya Kamsky

unread,
Feb 19, 2014, 7:57:05 PM2/19/14
to mongodb-user
Alessandro:

How long an election takes depends on a lot of things, most of them
having to do with network latency, the nature of failure (refuse
connection or hangs/times out) etc. The fastest you can count on is
on the order of a couple of seconds, the slowest should be no more
than maybe 30 seconds (I think the docs say less than one minute, to
be on the safe side).

The downside of a very fast election/failover would be a false
failover - in other words, the faster you make your failover, the
more likely you are to elect a new primary when the old primary isn't
actually down but just had a network hiccup, or some other tiny delay
in response.

Over long time in production systems, we've seen more problems with
false failover (and the ensuing flip-flop that it tends to cause).
This is why I would caution you to not over-engineer your system for
fastest possible failover at the cost of unnecessary failover
scenarios.

Asya
> --
> --
> You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> To post to this group, send email to mongod...@googlegroups.com
> To unsubscribe from this group, send email to
> mongodb-user...@googlegroups.com
> See also the IRC channel -- freenode.net#mongodb
>
> ---
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to mongodb-user...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

Rob Pecor

unread,
May 30, 2014, 2:33:28 PM5/30/14
to mongod...@googlegroups.com
In the 2nd scenario, why would there be an election in DC1, and how can the Primary change?  If DC1 has the Primary and it can see the Arbiter, it still has a Majority and will remain Primary with no election.  DC2 will not see the Primary, will try to elect itself, but the Arbiter will veto it because it can see the primary, so DC2 will remain a secondary.

Asya Kamsky

unread,
Jun 1, 2014, 11:32:38 PM6/1/14
to mongodb-user
Alessandro:

Rob is correct, in the second scenario, node in DC2 will try to initiate an election which the arbiter will veto.
But node in DC1 will be the primary throughout.  

So your second scenario assumes some things that cannot happen given the rules of elections in replica sets.

Asya



--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
 
For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.

---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages