Service Discovery - Consul or Saltstack?

[aos...@tibco.com]

Hi Everyone,

I've been looking into service discovery with Consul and am interested in making use of it for stateless resources. However I'm curious if someone can point out why a service discovery framework would be of better use than an automation framework.

One example is that I can just use Salt with the reactor to execute some things in the highstate when a minion comes online with a new service node. How can I best make use of both Consul and Salt in an environment? Doesn't have to be Salt (could be Puppet/Chef/Ansible/etc).

Thanks everyone,

Ahmed

[William Huba]

We use Consul along with Chef running in local mode. Although some of the service discovery can be replicated with a centralized conf management server, Consul has some really nice features.

For example: we use Docker heavily, and each container is registered in consul with the ports that it has opened and a health check. HAProxy on a frontend node is setup with consul-template to route traffic to the correct server/port combos and only on healthy nodes. Changes are generally reflected in under a second.

There's actually a page in the docs for consul vs CM as well which might help expand on my anecdote: http://www.consul.io/intro/vs/chef-puppet.html

Cheers,
William

--
You received this message because you are subscribed to the Google Groups "devops-toolchain" group.
To unsubscribe from this group and stop receiving emails from it, send an email to devops-toolcha...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ryan Thomas-Martin Miller, O.P.]

I worked in a Zookeeper/Puppet world rather than Consul/Salt, but for us the demarcation was about whether stale caches were bad.  For DNS servers and other network resources (think of resolv.conf as a service discovery cache, from a config management perspective), failover is often built in, so doing service discovery in config management works nicely.  For database handles, stale data is really bad, so you need something more realtime/HA like Zookeeper, if you want to do things on the client (which is to say webserver) side; else you can use something like corosync/pacemaker on the server side. We did the database read handle in zookeeper but pacemaker for the write handle, since we had to use the other pacemaker services to ensure data consistency anyway.  I would be interested in peoples' thoughts about when to handle such things client-side and when server-side, given that both solutions seem scalable but also a bit brittle.   

Ryan

[gareth rushgrove]

On 28 April 2015 at 14:19, Ryan Thomas-Martin Miller, O.P.

<rmm...@gmail.com> wrote:
> I worked in a Zookeeper/Puppet world rather than Consul/Salt, but for us the
> demarcation was about whether stale caches were bad. For DNS servers and
> other network resources (think of resolv.conf as a service discovery cache,
> from a config management perspective), failover is often built in, so doing
> service discovery in config management works nicely. For database handles,
> stale data is really bad, so you need something more realtime/HA like
> Zookeeper, if you want to do things on the client (which is to say
> webserver) side; else you can use something like corosync/pacemaker on the
> server side. We did the database read handle in zookeeper but pacemaker for
> the write handle, since we had to use the other pacemaker services to ensure
> data consistency anyway. I would be interested in peoples' thoughts about
> when to handle such things client-side and when server-side, given that both
> solutions seem scalable but also a bit brittle.
>

I talked a bit about this a few weeks a go at Loadays. The videos
should be available at some point but here are the slides.

https://speakerdeck.com/garethr/service-discovery-and-configuration-management

(The examples are in Puppet and Consul, but I do work at Puppet Labs,
and concepts should be similar in other tools)

For me it comes down to what I've been calling "two speeds of
configuration", namely model driven and emergent. You can use one or
the other, but in most cases you need both. The mix depends on the
organisation constrains I think. I'm hoping to write up a number of
thoughts when I get a moment.

Gareth

Gareth Rushgrove
Web Geek

morethanseven.net
garethrushgrove.com

[Jason Edgecombe]

Hi Ahmed,

This hour-long podcast helped me to better understand why it's
beneficial to have both a service discovery framework and an automation
framework. It explains why you would want to use a tool called consul
template, which overlaps with puppet:

http://foodfightshow.org/2014/11/consul-template.html

The video is an hour long, but I found it helpful. I hope that you do as
well.

Sincerely,
Jason

[Joaquin Menchaca]

Interesting.  I thought of this idea as well.  Salt + Consul look like really good tools to marry together.  For alterantive CMs, you would use puppet apply and chef localmode.  There's also MCollective, which has some similarities to SaltStack.  

For Ansible, I am not sure, as this is pure synchronous push tech.  Maybe you can query consuls distributed info then use that to centrally manage systems.  It's apples and oranges with this, because consul works better with other asynchronous tech, not pulling from a central server, or pushing from a central server.

At some point you would want to extract some data on the state of the changes, and the current states, slap some front is spiffy web dashboard to show the bosses a snapshot of where things are at...