CP vs AP for service discovery

[Paul Lorenz]

Hi all,

   We're about to make a decision on a service discovery mechanism, where consul is front-runner. We have some concerns around the CP nature of consul, and hope that you can share your perspectives on this.

We're aware than read consistency is configurable.

The specific scenario we're concerned about is scaling up during a network partition. It seem likely that a network partition might require scaling up, since your services might be split and could be processing more slowly. If you're on the non-quorum side of a split and you're trying to scale up, you won't be able to register your services. This means your scaled up instances won't be able to contribute to making forward progress.

Some thoughts we had on this:

1. This is not an issue for all services
1. Several common services work around this. A new Cassandra node, for example, can join the C* cluster by looking up existing nodes in service discovery. Clients will also be to use the new node as long as they can reach the old nodes, since the C* client handles getting a full client list using gossip.
2. If your application doesn't need to publish itself, if for example it is just a queue processor, then this issue doesn't affect it either.
2. CP for services discovery seems common (ZK, etcd, consul), though not universal (Eureka).

Questions:

1. Are there problems with an AP system that we haven't thought of?
2. Why are so many service discovery solutions CP? It seems this must have been a conscious choice for consul, since the underlying implementation (Serf) is AP. 
3. Are there workarounds (that don't involve writing a backup AP service discovery) that we haven't considered?

Thank you,

Paul

[James Phillips]

Hi Paul,

Sorry for the late reply on this - things got delayed due to some travel. These are great questions!

> 1. Are there problems with an AP system that we haven't thought of?

> 2. Why are so many service discovery solutions CP? It seems this must have been a conscious choice for consul, since the underlying implementation (Serf) is AP. 

The main driver for us going CP with Consul is that it's much easier to reason about during operations. There aren't as many dynamic behaviors that are possible as a change ripples across a cluster in an AP system, and the behavior in the face of failures is also easier to understand (what happens to the minority state when the AP system gets un-partitioned, for example). Adding Consul's KV store with coordination primitives like sessions and locks into the mix makes the AP solution much more challenging to implement, scale (how do you efficiently manage KV updates with thousands of nodes), and reason about. Raft is a distributed log that is serialized, sequential, and agreed upon - these are very convenient properties to rely on and simplify nearly everything else above it.

You are right that some services can deal with these issues on their own, but in general it makes them more complicated internally. Consul tries to hide these details and allow for zero-touch integration with almost anything via DNS, and other things with simple HTTP API calls, so it favors much lighter weight clients for the common case while exposing deeper functionality for clients who need it (locking for leader elections, etc.). In the extreme you could run something like Serf alongside Consul if you needed a truly AP system for a subset of your services, but that's relatively rare to do.

> 3. Are there workarounds (that don't involve writing a backup AP service discovery) that we haven't considered?

For your specific use case of scaling the cluster during an extended partition, you'd want to separate segments of your cluster that are susceptible to extended partitions into different Consul datacenters (running a set of Consul servers on each side of the link). This would give you the independence you need to manipulate each side during a partition, and the ability to query the other side as needed when things are working properly, because Consul will forward requests between the datacenters.

Hopefully this helps! If you think that you'd still need a separate AP backup system we'd be glad to discuss it here, or if you'd like to contact us with any concerns you can't share on the list feel free to send a note to sup...@hashicorp.com.

-- James

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/CAP7Q6Yb-UF72aJVEKV00FG-fnsBiTPkmHpdSLPfJbE8iJ0mn5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.