Questions regarding consul bootstrap flow

[Shimshon Zimmerman]

Hi guys,

I went over the bootstrap guide in the documentation, as well as some of the bootstrap-related discussions here, but still have a few questions I hope you can help with.

1. You mention in the documentation that bootstrapping consul with a single node is not recommended and error-prone. How is bootstrap-expect=N different, and why is it better? The reason I'm asking is that in the old bootstrap flow, we could start using the key/value store even when only one node was up, before more nodes started joining the cluster. With the new flow, we basically have to wait for more nodes to join before being able to use the key/value store.

2. According to the documentation, once the N servers started consul with bootstrap-expect=N, one of them must run consul join with the IP addresses of all N nodes. Is there any way for the servers to discover each other and join? Alternatively - is it possible that each server will run "consul join" with an IP of only 1 other server? (i.e. if servers B and C both have A's IP address, can each of them run "consul join server_A"?).

3. If more than 3 nodes join consul with bootstrap-expect=3 and -server - what will be the behavior? Will we only have the first 3 as servers, or will all of them become servers?

Many thanks in advance,

Shimshon.

[Michael Fischer]

On Thu, Apr 16, 2015 at 3:01 AM, Shimshon Zimmerman <shim...@stratoscale.com> wrote:

Hi guys,

I went over the bootstrap guide in the documentation, as well as some of the bootstrap-related discussions here, but still have a few questions I hope you can help with.

1. You mention in the documentation that bootstrapping consul with a single node is not recommended and error-prone. How is bootstrap-expect=N different, and why is it better? The reason I'm asking is that in the old bootstrap flow, we could start using the key/value store even when only one node was up, before more nodes started joining the cluster. With the new flow, we basically have to wait for more nodes to join before being able to use the key/value store.

It's recommended to configure at least 3 server nodes and bootstrap them before adding any non-server agents to the cluster.   If you set bootstrap-expect to 3, then the cluster will bootstrap on its own.  This is to ensure service availability in a real-world configuration where equipment fails from time to time.

 

2. According to the documentation, once the N servers started consul with bootstrap-expect=N, one of them must run consul join with the IP addresses of all N nodes. Is there any way for the servers to discover each other and join? Alternatively - is it possible that each server will run "consul join" with an IP of only 1 other server? (i.e. if servers B and C both have A's IP address, can each of them run "consul join server_A"?).

I don't think the documentation says this, but it is incorrect if it does.  Cluster membership information will propagate automatically, provided the server that is joined to is already a cluster member.  An agent need not to specify all servers when joining; one is sufficient.  However it is often good to list more than one in case one of them is unavailable.

 

3. If more than 3 nodes join consul with bootstrap-expect=3 and -server - what will be the behavior? Will we only have the first 3 as servers, or will all of them become servers?

Only agents started with the '-server' argument will become servers.  They don't promote themselves automatically.

--Michael

[Shimshon Zimmerman]

Hi Michael, 

Thanks for the prompt reply, please see some clarification questions inline.

Thanks,

Shimshon.

On Thu, Apr 16, 2015 at 3:55 AM, Michael Fischer <mfis...@zendesk.com> wrote:

On Thu, Apr 16, 2015 at 3:01 AM, Shimshon Zimmerman <shim...@stratoscale.com> wrote:

Hi guys,

I went over the bootstrap guide in the documentation, as well as some of the bootstrap-related discussions here, but still have a few questions I hope you can help with.

1. You mention in the documentation that bootstrapping consul with a single node is not recommended and error-prone. How is bootstrap-expect=N different, and why is it better? The reason I'm asking is that in the old bootstrap flow, we could start using the key/value store even when only one node was up, before more nodes started joining the cluster. With the new flow, we basically have to wait for more nodes to join before being able to use the key/value store.

It's recommended to configure at least 3 server nodes and bootstrap them before adding any non-server agents to the cluster.   If you set bootstrap-expect to 3, then the cluster will bootstrap on its own.  This is to ensure service availability in a real-world configuration where equipment fails from time to time.

Why is this better than bootstrapping the cluster with one server, shortly thereafter joining 2-3 more servers to the bootstrapped server, and then joining more "regular" agents? 

 

2. According to the documentation, once the N servers started consul with bootstrap-expect=N, one of them must run consul join with the IP addresses of all N nodes. Is there any way for the servers to discover each other and join? Alternatively - is it possible that each server will run "consul join" with an IP of only 1 other server? (i.e. if servers B and C both have A's IP address, can each of them run "consul join server_A"?).

I don't think the documentation says this, but it is incorrect if it does.  Cluster membership information will propagate automatically, provided the server that is joined to is already a cluster member.  An agent need not to specify all servers when joining; one is sufficient.  However it is often good to list more than one in case one of them is unavailable.

In that case - will the following flow work?

- node A starts with -bootstrap-expect=3, -server

- node B starts with -bootstrap-expect=3, -server, -join=<node A IP address>

- node C starts with -bootstrap-expect=3, -server, -join=<node A IP address>

<cluster bootstraps at this point I guess...>

- node D and all the rest of them start consul with -join=<node A IP address>  (without -server, as they will be regular agents).

 

3. If more than 3 nodes join consul with bootstrap-expect=3 and -server - what will be the behavior? Will we only have the first 3 as servers, or will all of them become servers?

Only agents started with the '-server' argument will become servers.  They don't promote themselves automatically.

--Michael

--

Shimshon Zimmerman

Web | Blog | Twitter | Google+ | Linkedin

[Michael Fischer]

On Thu, Apr 16, 2015 at 7:34 PM, Shimshon Zimmerman <shim...@stratoscale.com> wrote:

It's recommended to configure at least 3 server nodes and bootstrap them before adding any non-server agents to the cluster.   If you set bootstrap-expect to 3, then the cluster will bootstrap on its own.  This is to ensure service availability in a real-world configuration where equipment fails from time to time.

Why is this better than bootstrapping the cluster with one server, shortly thereafter joining 2-3 more servers to the bootstrapped server, and then joining more "regular" agents? 

The reason you bootstrap a minimum of 3 nodes is to survive a partition (assuming there are at least 2 nodes that can form a majority after the partition occurs).  See https://www.consul.io/docs/internals/consensus.html. 

In that case - will the following flow work?

- node A starts with -bootstrap-expect=3, -server

- node B starts with -bootstrap-expect=3, -server, -join=<node A IP address>

- node C starts with -bootstrap-expect=3, -server, -join=<node A IP address>

<cluster bootstraps at this point I guess...>

- node D and all the rest of them start consul with -join=<node A IP address>  (without -server, as they will be regular agents)

This should work.

Best regards,

--Michael 

[Shimshon Zimmerman]

Many thanks. 

[Brian Spacecraft]

I'm still confused about the expected behavior for Shimshon's question 3.

I understand that an agent can not promote itself to a server node, but I'm interested in knowing the expected behavior in the following scenario:

1. Bootstrap cluster with 3 agents, each in server mode and with -bootstrap-expect 3
2. Join agents 2 and 3 to 1, leadership election succeeds and cluster is online
3. I decide I want to increase the fault tolerance of my cluster, so I add two new hosts running consul agent in server mode

Should the two agents in step 3 have -bootstrap-expect 3, or -bootstrap-expect 5? If 5, should I restart agents 1, 2, and 3 with the new -bootstrap-expect value?

[Armon Dadgar]

Hey Brian,

To clarify, the `-bootstrap-expect` flag only operates a single time in the history of the cluster.

It is only used for that initial bootstrap, e.g. the first time a leader is elected. At that point forward,

scaling the cluster up/down is done dynamically through the quorum. The existing members replicate

among themselves commands to add or remove peers.

So in this example, you would start with `-bootstrap-expect 3`, but then the new agents do not even

need the bootstrap flag. They are joining an existing cluster which has a leader, so they are not

performing a bootstrapping of the leader.

Best Regards,

Armon Dadgar

--
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Spacecraft]

Hi Armon,

Thanks for clarifying.

Why expose this option at all?

Since quorum can't be reached with less than 3 nodes, then couldn't consul always just use 3 as the -bootstrap-expect value?

On Wednesday, April 15, 2015 at 2:01:44 PM UTC-5, Shimshon Zimmerman wrote:

[Armon Dadgar]

Hey Brian,

There are other possible choices. You may have a single node cluster (testing / stage),

and we’ve seen deploys of 3, 5 and 7 servers. We couldn’t just default to 3 for all cases.

Best Regards,

Armon Dadgar

From: Brian Spacecraft <br...@gospacecraft.com>
Reply: Brian Spacecraft <br...@gospacecraft.com>>
Date: May 15, 2015 at 2:01:33 PM
To: consu...@googlegroups.com <consu...@googlegroups.com>>
Cc: yae...@stratoscale.com <yae...@stratoscale.com>>, mo...@stratoscale.com <mo...@stratoscale.com>>
Subject:  Re: Questions regarding consul bootstrap flow

--