Register Vault as a service using Consul

[Simone D'Andreta]

Hello,

I am posting here what I already posted in Vault channel.

I am running Vault 0.6.4 on 3 machines and using Consul 0.7.2 as a storage backend. Consul runs as a cluster of 3 nodes, on different boxes - I mean Vault and Consul server are not installed on the same machines. I would like to use Consul service discovery so the developers using Vault can just talk to it using a name (i.e. vault.service.cnqr.io). 

I installed Consul agent on each Vault machine. This is the config.json for the Consul agent:

{

  "datacenter": "mspbat",

  "data_dir": "/opt/consul",

  "log_level": "INFO",

  "node_name": "mspbatvlt001",

  "domain": "cnqr.io",

  "retry_join":["10.24.25.138","10.24.25.139","10.24.25.140"],

  "acl_token": "whatever"

}

and this is the service definition file:

{

 "service":{

 "name": "vault",

 "address": "10.24.26.116",

 "port": 8200,

 "check":  {

    "http": "http://10.24.26.116:8200/v1/sys/health",

    "interval": "10s",

    "timeout": "5s"

 }

}

}

The service is properly registered in Consul and the healthcheck is reporting green when sealed and orange when unsealed. But there are things I am not getting and I tried to read all the docs and groups, but I have difficulties to put all the pieces together. Here my questions:

1) When I connect to the Consul UI I see the Vault service registered under the Vault nodes as I said above. What I see it's that also one of the Consul server nodes has the Vault service registered pointing to one of the Vault nodes.. and it's orange. I don't get it, Vault doesn't run on Consul servers, how a Consul servers can discover Vault? 

2) Looking at the various groups and docs it seems that Vault can automatically register itself in Consul.. is this why I see Vault registered under a Consul server? If yes, why only one node is registered and not all of 3? Automatically registered means I don't need to install a Consul agent on Vault servers?

3) The service is registered as vault, but I'd like to have vault.service.cnqr.io - I still have to double check this with System engineering team, might be the dns forwarder for cnqr.io is not done yet.. apart from that, is there anything else I should do?

4) I have seen advertised address and redirect address variables mentioned almost everywhere.. my Vault config file has redirect_addr set to "http://127.0.0.1:8200" - should I set it differently?

Thanks for all your answers.

Best regards,

Simone

[Simone D'Andreta]

After a lot of digging I installed a Consul agent on the Vault machines and pointed the backend to the local agent. I see the service automatically registered, with its healthcheck and if I run a dig querying the local consul agent I get back the answers for vault.service.cnqr.io.

So far so good. 

Now it comes another problem. We use conditional forwarders in our domain controller DNS so all the queries for the *.service.cnqr.io point to a VIP in our load balancer. The VIP includes our Consul servers.

The bad thing is that no DNS can solve vault.service.cnqr.io (apart from the local consul agent..) so that's why I cannot ping it either. I am a bit worried that our approach with DNS forwarders requires including ALL the Consul cluster members under the VIP pool (which means also the clients). This approach seems unfeasible to me because it would require a server entry in the load balancer for each consul agent (both servers and clients).

Can anyone suggest me and idea here please?

Thank you very much,

Simone

[Michael Corcoro]

I've recently been delving into how to integrate Consul's DNS with our own DNS as well.  I'm no expert, but I did find some useful guides, such as this one on DNS forwarding:  https://www.consul.io/docs/guides/forwarding.html.  Is the forwarding set up to use Consul's DNS port (8600)?  Also maybe look into adding a zone entry specifically for the "vault" sub-domain.  Good luck!

[Simone D'Andreta]

Hi Michael,

we found out that our ACL was denying everything so we need to add the service read policy for the anonymous token. Now we are all good.

And yes we have conditional forwarders setup in our DNS to forward Consul domain requests.

Thanks for your help,

Simone