HA and TLS ... difficult times ..

[Justin DynamicD]

I have a working vault cluster, but I'm starting to do the final hardening steps by introducing TLS and have come to the conclusion that ... well ... vault best practices here are exceedingly vague as to both what dns names a certificate will need as well as what the various "ADDR" addresses do.  it gets especially confusing in regards to consul, which as I read more seems to be used a backend only and should NOT be used for SD of vault itself.

At any rate the more I research, the more I stumble on parameters/variables that are either exceedingly vague in their purpose or flat out undocumented.  Then there's settings that seem like they SHOULD exist but don't?  Oh fun days.

Quick rundown of settings I've stumbled on but that I feel are poorly defined or completely omitted when compared to here:

https://www.vaultproject.io/docs/configuration/index.html

VAULT_ADDR: this is obviously used by clients/agents to find the vault cluster ... but does the cluster need this defined?  it's nowhere on the configuration page, yet shouldn't I need to advertise the main cluster url or no?

VAULT_API_ADDR: this is to advertise a unique URL to other vault members so they know who to redirect clients to.  It also looks like it is MUST be present in the Certificate if you wish to use TLS.

VAULT_CLUSTER_ADDR: another redirect,  but in this case it's a more generic server forwarding that will auto match the API address unless overridden.  Honestly this may be wrong, but if true it's only because I went to a completely unique page to figure that out:  https://www.vaultproject.io/docs/concepts/ha.html.  So this _might_ be additional SAN entries depending.  But we're not done ...

VAULT_REDIRECT_ADDR: here's one that doesn't exist at all in either docs, but it turns out it's critical for HA use, the system will simply attempt to generate a value if it's missing, see the conversation here: https://github.com/hashicorp/vault/issues/1337.  But how is it unique from the other two?  I mean ... at this point api is for clients, cluster is for servers (even though it states they prefer to chat through the back-end storage) and now we have this seemingly important value that is being auto set to ... something.  If you're using Consul you probably need to hardcode this to IP, otherwise Consul will try to use multiple CNAMES for the vault service which will break dns resolution.

Oh and which of these need TLS?  All of them I assume (though I also assume much of this has overlaps).  What of consul address or it's active.vault.service.consul and standby.service.consul entries?  Does the vault service even care to be aware of it's default service name or as long as the name in TLS it takes it?

My _suspicion_ to all the above is as follows:

VAULT_ADDR is a client only thing, I can pull that environment variable and the server wont care.

API/CLUSER_ADDR are for clients vs the internal cluster.  All of these DNS names should be in the certificate (or IP if that's how they are referenced) and if they are unique is completely dependant on how I configure my servers.

REDIRECT_ADDR ... is ... uh ... apparently important but I can't find any definitive info on how it should be set vs. API/CLUSTER.

Do I need to set the generic vault name somewhere?  basically the server equivalent to VAULT_ADDR?  Or does vault not care as long as TLS works?

[Adam Carlin]

I don't really use environment variables but will write them as used in config files. One thing I would note is to be careful with cluster_addr and cluster_address which serve different purposes...

cluster_address in the listener section controls which IP and port Vault binds to for clustering; it is the cluster equivalent of address in the listener section.

cluster_addr in the backend (or ha_backend) section controls the address advertised to other nodes for clustering; this is the clustering equivalent of redirect_addr. This can be used to work around custom request paths (e.g. if the actual bind is on a private address so you need to give the address of a TCP-based load balancer to other nodes). This is a full address, but Vault does enforce HTTPS.

Source: https://github.com/hashicorp/vault/issues/2475

As for TLS in Vault, this is configured in the listener TCP section documented here https://www.vaultproject.io/docs/configuration/listener/tcp.html so my example config is...

{
  "storage": {
    "consul": {
      "address": "X.X.X.X:8500",
      "path": "vault"
    }
  },
  "listener": {
    "tcp": {
      "address": "X.X.X.X:8200",
      "tls_cert_file":"/var/lib/vault/ssl/vault.crt",
      "tls_key_file":"/var/lib/vault/ssl/vault.key"
    }
  },
  "telemetry": {
    "statsd_address": "X.X.X.X:20001"
  },
  "default_lease_ttl": "1h"
}

Hope that helps but yes documentation is lacking and updating maybe a bit too much (info here https://github.com/hashicorp/vault/tree/master/website/source/docs updates information on the site).

[Jeff Mitchell]

Hi Justin,

On Mon, Dec 4, 2017 at 3:17 PM, Justin DynamicD <justin....@gmail.com> wrote:

At any rate the more I research, the more I stumble on parameters/variables that are either exceedingly vague in their purpose or flat out undocumented.  Then there's settings that seem like they SHOULD exist but don't?  Oh fun days.

Sorry you're having trouble here. All current parameters should be documented, but the website currently displays only documentation for the latest released version (I hope that at some point we'll let you switch between versions, but we're not there yet). You may be getting confused if looking at others' posts or configs, if they are using now-deprecated parameters.

 

Quick rundown of settings I've stumbled on but that I feel are poorly defined or completely omitted when compared to here:

https://www.vaultproject.io/docs/configuration/index.html

VAULT_ADDR isn't defined on that page because it's not a configuration parameter. You can see client env vars at https://www.vaultproject.io/docs/commands/environment.html

 

VAULT_ADDR: this is obviously used by clients/agents to find the vault cluster ... but does the cluster need this defined?  it's nowhere on the configuration page, yet shouldn't I need to advertise the main cluster url or no?

This is not needed for the cluster.

 

VAULT_API_ADDR: this is to advertise a unique URL to other vault members so they know who to redirect clients to.  It also looks like it is MUST be present in the Certificate if you wish to use TLS.

If a Vault server redirects a client from a.example.com to the active node at b.example.com, b.example.com will need its name on its certificate (standard TLS behavior).

 

VAULT_CLUSTER_ADDR: another redirect,  but in this case it's a more generic server forwarding that will auto match the API address unless overridden.  Honestly this may be wrong, but if true it's only because I went to a completely unique page to figure that out:  https://www.vaultproject.io/docs/concepts/ha.html.  So this _might_ be additional SAN entries depending.  But we're not done ...

This is not a redirect, and has nothing to do with Vault's TLS certificates. This address is used to specify/override the address that Vault nodes give to each other for clustering. It's here because sometimes people want cluster traffic to go over LBs or don't have direct access via the same IP or DNS name used for the API address.

 

VAULT_REDIRECT_ADDR: here's one that doesn't exist at all in either docs, but it turns out it's critical for HA use

This is just an old name for API_ADDR. We renamed it to API_ADDR because it was being used for more than simply client redirection and it was confusing. We still accept it if it's there, but you should use API_ADDR instead.

 

Best,

Jeff