Scenario in case Node gets disconnected

[Georgii Shagov]

Hello Team,

The original question is here: https://groups.google.com/g/etcd-dev/c/Pl-OLlWJCTc

Let me phrase it it that way. Let we Image our application is set in two Data Center (DC), DC1 (like Munich) and DC2 (Frankfurt, for instance). Let me say that we are using etcd (that is Raft-based) cluster and we have three nodes in DC1 and three in DC2, let me say the leading node is in DC1. Now let me say that our applications, that are using etcd, in DC1 are connected only to the nodes of etcd in DC1 and know nothing about DC2, the same for DC2, mirrored.  Now let me say due to technical outage the connectivity between two DCs (DC1 and DC2) are lost for some period of time.

The question: What would be the behaviour for etcd nodes during the outage and after.

IMU the nodes are connected to leading one  (say DC1) are to be in operational state, yet the nodes in DC2 are to be in non-operational state, though they are up and running. Is this Correct? So, when the connectivity between DC1 and DC2 will be restored the nodes from DC2 should inherit the data from DC1. Does it make sense?

thx in advace

[Diego Ongaro]

Hi Georgii,

If you want a Raft cluster to be available within a datacenter even when that datacenter is disconnected from the rest of the world, the majority of nodes in that cluster (a quorum) must be contained within that datacenter. That doesn't work the moment you want this Raft cluster to be available in more than one datacenter. You probably want to run independent Raft clusters in each datacenter and then deal with how to merge state across them, or to not use Raft at all. The Raft algorithm isn't designed to do that part.

-Diego

--
You received this message because you are subscribed to the Google Groups "raft-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to raft-dev+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/raft-dev/6da0cdef-3aa0-4ed7-97b9-9db1729cee1fn%40googlegroups.com.

[Georgii Shagov]

Hello Diego

I really appreciate your reply. This is very to the purpose. If I understand you right in case the Leading node goes away the rest/connected nodes makes quorum and new Leader is elected. This practically makes two clusters, if they get connected again this undoubtedly makes merging between them Insane or impossible.

It makes sense absolutely. Thanks again.