Bootstrapping Consul + configuration management

[Diego Zamboni]

Hi,

As I'm exploring Consul, I'm trying to figure out what the best way to bootstrap new machines in which both Consul and a configuration management mechanism (I'm partial to CFEngine, but the same should apply to any other) are running. The way I see it, this could be done in any of the following ways:

1- Start Consul on the new machines, and configure it with a service/check/watch for CFEngine, which starts it if it's not running.
2- Start CFEngine on the new machines, with appropriate policy for downloading/installing/configuring Consul if needed.

I think I'll do (2) and then have CFEngine do (1) so they are both keeping an eye on each other, but I'm curious as to what others are doing wrt bootstrapping, and what your experiences have been.

Thanks,
--Diego

[Carl Johnson]

We use Icinga to watch for our config management not running, Salt in
our case. We've considered wrapping the Salt minion (agent) process
with something like supervisord or runit to just make sure it stays
running, though. I'm not sure where Consul is a great benefit here.

And yes, I'm writing Salt states right now to install Consul and
restart it as needed when configuration changes or the package is
upgraded.

The tricky part is the initial cluster join. Luckily our home-grown
inventory system writes data into Salt concerning what machines are
part of what role. So it's a simple matter of iterating through the
Consul server hostnames and running "consul join". Would love to use
Consul for this, but that's a chicken/egg problem, eh? We could also
probably accomplish this if we set up consistent DNS names for our
Consul servers.

Carl

> --
> You received this message because you are subscribed to the Google Groups "Consul" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Diego Zamboni]

Hi Carl,

Thanks for your comments, they match many of my own thoughts.

WRT the initial cluster join, I was thinking that if your DNS is configured to forward .consul queries to Consul (e.g. http://www.consul.io/docs/guides/forwarding.html) it shouldn be sufficient to do a DNS lookup for "consul.service.dc.consul" to find where to join.

--Diego

[Dan McKean]

I made a sample demo script that bootstraps Consul with Ansible as the provisioner.  It builds a small server cluster inside of Vagrant; the code is here: https://github.com/danlmarmot/demo-consul-vagrant.  It also works on EC2, but I haven't added that to this repo yet.  It uses Supervisor for process management and to keep Consul running, and dnsmasq to forward DNS requests that look like *.consul to the Consul DNS service.

My experiences: 

1.  The new bootstrap-expect in Consul 0.4.0 really works well, though you do need to gather the IP addresses.  You don't need to run it on one machine, it can be run on any or all of the machines you want to turn into a cluster.

2.  Boolean values in the consul.conf file don't have quotes, it is JSON.   server: true is correct.  server: "true" does not work and Consul won't run.  That took awhile to debug.  Grrrr JSON.

3.  I don't particularly like JSON as a config file format; just adding a config line to enable the Consul UI is somewhat ugly from a templating perspective and it can't be easily commented for testing out because it's JSON and then there's the trailing comma with JSON that always bites me.  Just a personal preference.

4.  Consul with multiple network interfaces is a bit odd--the bind address is important to specify.

5.  On EC2 security groups, I had to enable several ports, both UDP and TCP, for Consul to work. This seems like a bit of excessive sprawl.

6.  The Consul UI really seems to only work with 127.0.0.1 (or the Consul HTTP binding IP address).  This threw me for awhile, but then I just remembered to "ssh <box> 8500:localhost:8500" and I could browse from another machine.

To join an existing Consul cluster, well... that's interesting, as you need to somehow discover another node's IP address.  On EC2 I look for other instances with the tag "consul-server" running in my AWS region and account, and the value of my environment (such as "prod" or "dev" or "staging"), get all of their private IP addresses, and then do a consul join on those IP addresses.  this works fine but seems hackish.  For Vagrant I already know their IPs so I just join them directly.  

Dan

[Diego Zamboni]

Hi Dan,

Thanks for your notes, they are very useful. I know the solution to one of your problems:

6.  The Consul UI really seems to only work with 127.0.0.1 (or the Consul HTTP binding IP address).  This threw me for awhile, but then I just remembered to "ssh <box> 8500:localhost:8500" and I could browse from another machine.

You need to start consul with "-client 0.0.0.0" (or specify the "client_addr" option in the config file) to make the consul UI bind to all the local interfaces. This was mentioned here by Armon a few days ago.

--Diego

[Laurent Raufaste]

Hey Diego ;)

FWIW, we bootstrap cfengine, which bootstrap consul. And a health-check is checking CFE, which is checking Consul.

We use CFE for system stuff, and consul for app config.

Pretty straight forward frontiers.

[Diego Zamboni]

Hey Laurent, thanks for your comments! It's nice to have another CFEngine wonk around playing with Consul ;)

Do you manage the Consul config files (e.g. service definitions and checks) using CFEngine as well?

--Diego

[Laurent Raufaste]

Yes I do.

--
You received this message because you are subscribed to a topic in the Google Groups "Consul" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/consul-tool/lyJ5jBDw1A8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to consul-tool...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Laurent Raufaste
<http://www.glop.org/>