What is the best way to update a consul cluster without downtime

[George Chilumbu]

What is the best way to update a consul cluster without downtime? I am running my consul cluster on Ubuntu, and has three servers and several clients.

[Chris Stevens]

There is a dedicated docs section for upgrading your cluster here:

https://www.consul.io/docs/upgrading.html

I've found the process outlined there to work very well when upgrading in-place.

You can also add new servers/instances to the cluster already running the new version and then shut down the old.

[David Adams]

For context, we run Consul in AWS, and we package the consul binary into a .deb package on our local APT repo. So when we upgrade Consul on the servers, I've written an Ansible playbook that prompts to ask for the datacenter to upgrade, then matches the hosts based on AWS tags, then cycles through them one at a time, stopping Consul, updating the APT package, starting Consul, waiting a few seconds, running some sanity checks that the cluster is alive and some expected minimum number of WAN and LAN members are present, then moves on to the next server in the cluster. Obviously you'd do this slightly differently if you're using Docker or immutable cloud servers or some other architecture, but the main point is that it works pretty smoothly to stop a server, upgrade/replace it, start Consul back up, check everything's working, then move on to the next. I've gone through this process twice now since we've polished it off, and it works very smoothly.

As for upgrading the agents, the process is basically the same, but I do some more sanity checking, and I allow more than one machine to be upgrading at a time. We don't yet have any hard production dependencies on Consul, so taking the local agent down for a few seconds can be done without much thinking about the apps, but if your apps rely on the agent to be 100% available, you'll want to take more care with the agent upgrades.

Our plans in the future are to move to launch all server types that we can using Amazon's Auto Scaling Groups (without the load-scaling bits turned on--just the maintain-cluster pieces) and immutable AMI images that we bake when updates need to be made and cycle into place. In that pattern, I expect we would add an extra server with the new version and cycle out the old one at a time as described by Chris Stevens. Going the plus-one route is safer, of course.

On Tue, May 3, 2016 at 5:47 AM, George Chilumbu <chilumb...@gmail.com> wrote:

What is the best way to update a consul cluster without downtime? I am running my consul cluster on Ubuntu, and has three servers and several clients.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
IRC: #consul on Freenode
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/3d524832-7c80-415a-845a-e0b376ee30f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.