Consul integration and discovery service for Nomad Servers

[Halil Can Kaşkavalcı]

Hello,

From the documentation I couldn't figure how to configure Consul for service discovery as example client.hcl comments in servers tag

    # For demo assume we are talking to server1. For production,
    # this should be like "nomad.service.consul:4647" and a system
    # like Consul used for service discovery.

So, I need to use Consul for Server IPs and Consul should be a system job within Nomad. Documentation isn't clear how to integrate them nicely and make a cluster. Example client and server configuration as well as consul job files would be nice for a cluster of few machines.

Thanks.

[Mohinish Shaikh]

Yes, I am working on a similar setup towards having 3 Nomad Servers and 5 Nomad clients. We can possibly collaborate. I need to wire them together using Consul.

Documentation says either one can point nomad to a existing consul service or run consul as a system job ... 

[Paulo Pires]

I have it working with 0.2.3-rc1 https://github.com/pires/nomad-vagrant-coreos-cluster.

[Halil Can Kaşkavalcı]

Hi Pires,

Thanks for posting your work. However I could not see how did you register Nomad as a service to Consul? You still have static IP of Nomad Servers in client.hcl file. Am I correct?

Mohinish, that would be great actually. I still haven't figured out how to start consul from Nomad as well as register Nomad as a service so that clients can find via DNS queries. Currently I give the IP addresses of Nomad Servers statically in client.hcl files.

[Pires]

On Monday, December 21, 2015 at 7:56:37 AM UTC, Halil Can Kaşkavalcı wrote:

Hi Pires,

Thanks for posting your work. However I could not see how did you register Nomad as a service to Consul? You still have static IP of Nomad Servers in client.hcl file. Am I correct?

In this particular case, yes.

 

Mohinish, that would be great actually. I still haven't figured out how to start consul from Nomad as well as register Nomad as a service so that clients can find via DNS queries. Currently I give the IP addresses of Nomad Servers statically in client.hcl files.

As far as I can tell from my experiments, Nomad won't register itself as a service in Consul, but only jobs that expose the _service_ block. I'm also not sure how it could be possible for anyone to run Consul as a Nomad job and then discover Consul. You'd still have to rely on something else for this.

In my experiments, I've been using etcd as source of truth for both Nomad and Consul servers. Each time a Nomad node or Consul node boots up, it starts a loop that will register itself into the etcd cluster with a certain TTL. Then, it reads from etcd all current members and joins them. Finally, gossip takes care of the rest.

[Alex Dadgar]

Hey,

The way you would run Consul under nomad is to use reserved ports and not dynamic ports and then have Consul bind to localhost and its default ports. Then any task on that machine could talk to Consul at its default port.

The most common use case is to have setup your servers on machines with known IPs. Then you can use the retry_join mechanism to have them join. 

It is a bit of an impossible bootstrap problem if you want to launch nomad using service discovery and have nomad launch the service discovery layer :)