What's the practical consul architecture of multi datacenter deployment in production

[Zim]

Hi guys

I've read some docs about the consul multi dc deployment, now I want to know if my understanding is correct.

The picture is a architecture of multi dc deployment which is use to do service discovery

Components in this pic:

• Agent Server: as[1-3] in dc1, bs[1-3] in dc2, each in a single server
• Agent Client: ac[1-6] in dc1, bc[1-6] in dc2, each deploy with the a provider/consumer in the same server
• RPC Provider: arp[1-3] in dc1, brp[1-3] in dc2
• RPC Consumer: arc[1-3] in dc1, brc[1-3] in dc2

I got some conclusions about the deployment, and I'd like to make sure they are correct. (and also some questions)

1. Is the picture above correct?
2. What we benefit from the multi datacenter deployment is: once the RPC Provider arp[1-3] fail, we could use GEO FAILOVER to let arc[1-3] invoke brp[1-3] in dc2 (cross dc invoke).
3. The consul agent client should be deployed with a provider or a consumer.
4. When a provider or a consumer does a service register or service discovery, it communicates with the local consul agent client, and the agent client forward the register/discovery request to the consul agent server
5. The consul agent client is not a part of Raft protocol
6. Is it possible or practical to deploy the agent client in a single server to receive all the provider/consumer request?
7. If the consul agent server in DC1 all fail, is it still possible for the RPC consumers in DC1 to discovery the service providers in dc2 (Would the agent client in dc1 forward the service discovery request to the agent server in DC2)?  
8. What's the scenario to use consul-replicate in production.
9. consul-replicate could only support a unidirectional data replication either DC1 -> DC2 or DC2 -> DC1 (What will happen if I set two consul-replicate to do a bidirectional data replication?)
10. consul-replicate would overwrite a value if a key exists both in DC1 and DC2, such as 'key1' -> 'v1' in dc1, and 'key2' -> 'v2' in dc2, when replicate from dc1 -> dc2. the value in dc2 would change to 'v1'
11. when you use consul-replicate, will it replicate the service registry data to another dc? Will it overwrite the service provider data or just append the provider in dc1 to dc2? 

[Zim]

Another question.

12. Can we take the client agents away from the application server and make them a cluster (maybe 10 or 20 servers to deploy 20 client agents)? 

[pba...@hashicorp.com]

Hey, some great questions!

> 1. Is the picture above correct?

Yep seems right.

 > 2. What we benefit from the multi datacenter deployment is: once the RPC Provider arp[1-3] fail, we could use GEO FAILOVER to let arc[1-3] invoke brp[1-3] in dc2 (cross dc invoke).

 

That's a possible setup yes. The benefit of multi-DC is fault isolation - one entire DC being unavailable should not affect other DCs. (More detail on this late).

> 3. The consul agent client should be deployed with a provider or a consumer.

Not sure what you mean by provider and consumer - in general the client agents run on every host in your datacenter that provides any sort of service.

> 4. When a provider or a consumer does a service register or service discovery, it communicates with the local consul agent client, and the agent client forward the register/discovery request to the consul agent server

More or less. Register and discover are actually more different than that in practice (the agent is treated as the source-of-truth for registrations so it continually syncs it's state with servers, vs discovery which is a point-in-time request that gets forwarded up to the servers)

> 5. The consul agent client is not a part of Raft protocol

Correct

> 6. Is it possible or practical to deploy the agent client in a single server to receive all the provider/consumer request?

No. The agent is designed to be colocated on same node since it's failure is taken to imply failure of all the services registered on it - if you had a central client and it fail all services in the cluster would become "unhealthy". If you need to monitor or discover external services there is a separate tool for that that luckily got blogged about this week: https://www.hashicorp.com/blog/consul-and-external-services

> 7. If the consul agent server in DC1 all fail, is it still possible for the RPC consumers in DC1 to discovery the service providers in dc2 (Would the agent client in dc1 forward the service discovery request to the agent server in DC2)?  

No, all servers failing in a DC is equivalent to that DC being offline in terms of service discovery - both inside same DC and outside.

> 8. What's the scenario to use consul-replicate in production. (same answer for 9,10,11)

I'd suggest checking docs for that project and coming back with more specific question. In general consul-replicate only replicates KV, only one way and with no staleness/consistency gaurantees. If you have config you want to be "global" that can work under those assumptions then it can be used to have a single "authoritative" DC which is the only one you write those shared KV entries in, and then use consul-replicate to copy those out to others. Keep in mind other DCs must not write same keys and must not rely on them being strongly consistent (e.g. using them as locks).

> Can we take the client agents away from the application server and make them a cluster (maybe 10 or 20 servers to deploy 20 client agents)? 

This doesn't really make sense. See answer to 6.

Hope this helps!

[Zim]

@pba...@hashicorp.com

thanks so much, you solve my doubts about consul!