Setting up Vault for production and configuring TLS

[Anthony Powles]

Hi,

I'm trying to set up Vault for production, I've read quite a few posts here about setting vault with TLS, and tried a few of the suggestions I found, without much success.

It's running in AWS, backed by a consul cluster, running 3 vault servers.

I've configured an ELB on top of the vault servers with the following health check HTTPS:8200/v1/sys/health

And it's working fine in detecting the sealed/unsealed vaults. (Only have 1 unsealed vault server, so only 1 shows as InService)

I had it working fine without TLS but now I need to set that up, and hitting a few walls.

Vaults instances are on private networks, with the following configuration file

backend "consul" {
  address = "<internal_consul_elb_address>:8500"
  path = "vault"
  advertise_addr = "https://<public_elb_address>:8200"
  tls_skip_verify = 1
}

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}

listener "tcp" {
  address = "10.0.1.49:8200"
  tls_cert_file = "/etc/vault.d/server.pem"
  tls_key_file = "/etc/vault.d/server.key"
}

the <public_elb_address> looks something like a.b.c.com

The TLS certificate is a wildcard certificate from GlobalSign, matching *.c.com and c.com

After I unseal vault, and check the status using the localhost ip (127.0.0.1), the status looks like this:

Sealed: false
Key Shares: 5
Key Threshold: 3
Unseal Progress: 0

High-Availability Enabled: true
 Mode: active
 Leader: https://a.b.c.com:8200

Now, if I try the following command from outside the network:

VAULT_ADDR=https://a.b.c.com:8200 vault status

I get this error on the client:

Error checking seal status: Get https://a.b.c.com:8200/v1/sys/seal-status: x509: certificate is valid for *.c.com, c.com, not a.b.c.com

And in vault log I can see:

2015/11/20 01:03:53 http: TLS handshake error from <elb_internal_ip>:17187: remote error: bad certificate

The ELB is configured to listen on TCP:8200 and forward to the instance on TCP:8200

I also see the following logs, which I believe are generated when the ELB runs the health checks, but it can still correctly detect sealed/unsealed vaults from that somehow.:

2015/11/20 01:18:30 http: TLS handshake error from 10.0.0.252:13400: tls: client offered an unsupported, maximum protocol version of 301
2015/11/20 01:18:50 http: TLS handshake error from 10.0.2.245:17596: tls: client offered an unsupported, maximum protocol version of 301

Initially I had tried configuring the ELB to handle SSL, and failed, and read somewhere that vault needed to do this, so now it's just passing through the ELB.

I've tried using both the server.pem and chain.pem files for the certificates, without any success.

I've also tried creating self signed certs, but I get other errors doing so.

Any help would be greatly appreciated

Cheers,

Anthony

[Jeff Mitchell]

Hi Anthony,

On Thu, Nov 19, 2015 at 8:25 PM, Anthony Powles <ant...@anyperk.com> wrote:
> And it's working fine in detecting the sealed/unsealed vaults. (Only have 1
> unsealed vault server, so only 1 shows as InService)

Any reason only one is unsealed? If only the active node is unsealed,
you basically have no HA capability, because if that node goes down no
standby node is able to take over.

> the <public_elb_address> looks something like a.b.c.com
>
> The TLS certificate is a wildcard certificate from GlobalSign, matching
> *.c.com and c.com

This is going to be a problem (and as we'll see, it is), because your
certificate isn't valid for *.*.c.com. So it would match b.c.com
(*.c.com), but not a.b.c.com (which would require *.*.c.com).

> Now, if I try the following command from outside the network:
>
> VAULT_ADDR=https://a.b.c.com:8200 vault status
>
> I get this error on the client:
>
> Error checking seal status: Get https://a.b.c.com:8200/v1/sys/seal-status:
> x509: certificate is valid for *.c.com, c.com, not a.b.c.com

This is Vault telling you exactly what I wrote above -- your wildcard
certificate is only valid one level down from c.com, but you're trying
to use it two levels down.

> I also see the following logs, which I believe are generated when the ELB
> runs the health checks, but it can still correctly detect sealed/unsealed
> vaults from that somehow.:
>
> 2015/11/20 01:18:30 http: TLS handshake error from 10.0.0.252:13400: tls:
> client offered an unsupported, maximum protocol version of 301
> 2015/11/20 01:18:50 http: TLS handshake error from 10.0.2.245:17596: tls:
> client offered an unsupported, maximum protocol version of 301

This is the ELB health check trying to use TLS 1.1 instead of TLS 1.2.
We see this constantly in our own Vault instance (where the health
checks also work.) You can configure Vault to support TLS 1.1, but so
long as the health checks are working, I don't see a compelling reason
to decrease the connection's security. I'm not sure why ELB does this,
but it's irritating.

> Initially I had tried configuring the ELB to handle SSL, and failed, and
> read somewhere that vault needed to do this, so now it's just passing
> through the ELB.

You can have ELB perform the TLS connection if you like (although you
may have failed for the same reason, if you were using the same domain
and certificates). The problem there is that if the connection from
the ELB to Vault is then unencrypted, you have lost a good chunk of
security. You could certainly issue certificates for the ELB to use to
encrypt the connection to Vault on the backend, and these could be
different certificates than what you have on the ELB's front end; even
in this case, however, you have a transitive trust issue, because
somewhere in the ELB the request and response are handled in plain
text and could therefore be snooped or modified. This transitive trust
problem may or may not be an issue for you depending on your needs
(but it is for a lot of organizations, especially if they have
specific industry standards they must be in compliance with).

Hope that helps...let us know if you have further issues.

Thanks,
Jeff

[Brian Lalor]

On Nov 20, 2015, at 10:27 AM, Jeff Mitchell <je...@hashicorp.com> wrote:
>
> You can have ELB perform the TLS connection if you like (although you
> may have failed for the same reason, if you were using the same domain
> and certificates). The problem there is that if the connection from
> the ELB to Vault is then unencrypted, you have lost a good chunk of
> security. You could certainly issue certificates for the ELB to use to
> encrypt the connection to Vault on the backend, and these could be
> different certificates than what you have on the ELB's front end; even
> in this case, however, you have a transitive trust issue, because
> somewhere in the ELB the request and response are handled in plain
> text and could therefore be snooped or modified. This transitive trust
> problem may or may not be an issue for you depending on your needs
> (but it is for a lot of organizations, especially if they have
> specific industry standards they must be in compliance with).

I did a little research on this myself. ELBs are PCI compliant, but only if you do *not* terminate TLS at the ELB. I confirmed this 2nd-hand by way of a friend who asked this question of Amazon on my behalf. I haven’t verified this myself, but I believe that if you use your ELB in TCP mode, your vault server will not see the real IP address of the client. That makes auditing by client IP basically impossible. For the token-granting service I’m building based on the cubbyhole post on the Hashicorp blog (that I think Jeff wrote), I want to use the client’s IP address as a component of validating that the client is allowed to request a token.

I guess Vault isn’t too different from other TLS-secured services, but there are several hostnames (or wildcards) that Vault needs to be reachable via that make procuring the server certificate difficult: vault.myco.com, vault.service.consul, vault-instance.internal.myco.com, for example.

—
Brian Lalor
bla...@bravo5.org

[Jeff Mitchell]

On Fri, Nov 20, 2015 at 10:45 AM, Brian Lalor <bla...@bravo5.org> wrote:
> I did a little research on this myself. ELBs are PCI compliant, but only if you do *not* terminate TLS at the ELB. I confirmed this 2nd-hand by way of a friend who asked this question of Amazon on my behalf. I haven’t verified this myself, but I believe that if you use your ELB in TCP mode, your vault server will not see the real IP address of the client.

I also believe this to be true, although I haven't checked if there's
a way to make ELB do something crazy to try to prevent this. A
combination of a load balancer that spoofs the outgoing IP and
specific route settings on backend servers could get around this, but
I highly doubt ELB supports something like that.

> That makes auditing by client IP basically impossible. For the token-granting service I’m building based on the cubbyhole post on the Hashicorp blog (that I think Jeff wrote), I want to use the client’s IP address as a component of validating that the client is allowed to request a token.

Note that cubbyhole is targeted at the secret introduction problem,
where there isn't any assumption of an initial secure channel. Because
of the various other security properties it offers, there may be a way
to be compliant even if you do not have a TLS-protected connection
between your service and the client. TLS is designed to protect the
secret in transit, but in cubbyhole the real secret isn't the one that
you're sending over the wire. Any MITM attack is either useless later
(because the actual secret is no longer retrievable due to the temp
token being used or the timeout being hit), or is detectable (because
if they MITM and then fetch the perm token value, your service will be
unable to).

> I guess Vault isn’t too different from other TLS-secured services, but there are several hostnames (or wildcards) that Vault needs to be reachable via that make procuring the server certificate difficult: vault.myco.com, vault.service.consul, vault-instance.internal.myco.com, for example.

That's very setup-specific to what you're doing. Most setups simply
use a canonical hostname and access path.

--Jeff

[Brian Lalor]

On Nov 20, 2015, at 10:59 AM, Jeff Mitchell <je...@hashicorp.com> wrote:
>
>> I guess Vault isn’t too different from other TLS-secured services, but there are several hostnames (or wildcards) that Vault needs to be reachable via that make procuring the server certificate difficult: vault.myco.com, vault.service.consul, vault-instance.internal.myco.com, for example.
>
> That's very setup-specific to what you're doing. Most setups simply
> use a canonical hostname and access path.

Not if you want to use TLS in the request forwarding from standby to active Vault servers. Then the Vault servers need to advertise an address that’s on the cert being used.

—
Brian Lalor
bla...@bravo5.org

[Jeff Mitchell]

Sure, so you do something like 1.vault.example.com and
2.vault.example.com with a cert for *.vault.example.com. Or
vault1.example.com/vault2.example.com and use *.example.com or a cert
with DNS SANs for both host names.

What I was meaning by a canonical host name and access path was that
most setups aren't referring to each Vault instance by three separate
names spanning two TLDs (service.consul and myco.com, in your example)
-- although DNS SANs can handle that case just fine, too. Most setups
refer to each particular Vault instance by a canonical hostname. And
if you're using an ELB, most also refer to each particular Vault
instance by a canonical access path, with the advertise_addr of each
node set to the ELB's address.

--Jeff

[Anthony Powles]

Hi Jeff, thanks for your help

Any reason only one is unsealed? If only the active node is unsealed,
you basically have no HA capability, because if that node goes down no
standby node is able to take over.

It was mostly just for testing purposes for now, definitely intend to have all the vaults unsealed

 

> the <public_elb_address> looks something like a.b.c.com
>
> The TLS certificate is a wildcard certificate from GlobalSign, matching
> *.c.com and c.com

This is going to be a problem (and as we'll see, it is), because your
certificate isn't valid for *.*.c.com. So it would match b.c.com
(*.c.com), but not a.b.c.com (which would require *.*.c.com).

Wow, well, I guess I never knew that, for some reason I thought wildcard certs *.a.com would cover sub, subsub, subsubsub... domains

I guess I was wrong! I did change the domain to a simple subdomain and it now works fine!

 

You can have ELB perform the TLS connection if you like (although you
may have failed for the same reason, if you were using the same domain
and certificates). The problem there is that if the connection from
the ELB to Vault is then unencrypted, you have lost a good chunk of
security.

The main reason I wanted to terminate at the ELB was that AWS already has my certs, so it's easy to setup (I'm using Terraform by the way)

I just specify the ID of the key, but now I actually have to copy the cert keys to the servers, which I find a bit annoying, but it's not like I'm going

to have to do this very often either, so, better security, little hassle, all is good

 

Hope that helps...let us know if you have further issues.

It's all running now, thank you! 

[Michael Fischer]

On Fri, Nov 20, 2015 at 7:45 AM, Brian Lalor <bla...@bravo5.org> wrote:

I did a little research on this myself. ELBs are PCI compliant, but only if you do *not* terminate TLS at the ELB. I confirmed this 2nd-hand by way of a friend who asked this question of Amazon on my behalf. I haven’t verified this myself, but I believe that if you use your ELB in TCP mode, your vault server will not see the real IP address of the client. That makes auditing by client IP basically impossible. For the token-granting service I’m building based on the cubbyhole post on the Hashicorp blog (that I think Jeff wrote), I want to use the client’s IP address as a component of validating that the client is allowed to request a token.]

There's a little-known feature available in ELB configuration that directs it to pass the client IP address to the backend via the PROXY protocol. 

Instructions are located here: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/enable-proxy-protocol.html

The next step would be for Vault to support it; I'd recommend filing a GitHub issue to track it.  (I've noticed Armon Dadgar has already written some Go code to support it - https://github.com/armon/go-proxyproto)

Best regards,

--Michael

[Jeff Mitchell]

Oh, interesting -- thanks for the pointer, Michael!

Although I think for Brian's case it's not actually Vault needing the
IP address, but rather his token service (which I'm guessing is also
behind ELB). Unless I misunderstood?

--Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/CABHxtY6myLG-twguGONxY6_63ha%2BKp5d%3Do%3DT8a92B7Y4N4qgMQ%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Brian Lalor]

> On Nov 23, 2015, at 9:38 AM, Jeff Mitchell <je...@hashicorp.com> wrote:
>
> Oh, interesting -- thanks for the pointer, Michael!
>
> Although I think for Brian's case it's not actually Vault needing the
> IP address, but rather his token service (which I'm guessing is also
> behind ELB). Unless I misunderstood?

That’s correct, Jeff.

I hadn’t seen that reference for the proxy protocol before. That looks really interesting!

—
Brian Lalor
bla...@bravo5.org

[Michael Fischer]

Vault could probably use the support as well, to help ensure the accuracy of its audit logs when placed behind an ELB. I'll file an issue.  

--Michael

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/CAORe8GECBLmUZAjUMNE0jNZa0xYXOYPd_RGYTPZup8sNktrqVQ%40mail.gmail.com.

[Jonathan Sokolowski]

I created a PR (https://github.com/hashicorp/vault/pull/291) a while back to add X-Forwarded-For support, in an attempt to determine the client's remote address properly on the logical request struct. 

[Michael Fischer]

https://github.com/hashicorp/vault/issues/799

[Anthony Powles]

Since I got it working, I unsealed all 3 Vault instances and today I'm noticing that they key going in and out of service on the ELB.

Running vault status against individual instances, they key loosing leadership and their mode keep changing between standby and active.

Also in the logs I see the following:

2015/11/24 20:41:26 [INFO] core: acquired lock, enabling active operation
2015/11/24 20:41:26 [INFO] core: post-unseal setup starting
2015/11/24 20:41:26 [INFO] core: mounted backend of type generic at logical/e7c6fc9e-cd19-e20f-0077-22a904e3d3ef/
2015/11/24 20:41:26 [INFO] core: mounted backend of type cubbyhole at logical/62f458ab-16f8-dbf2-b8da-72e08c9b58e5/
2015/11/24 20:41:26 [INFO] core: mounted backend of type system at sys/
2015/11/24 20:41:26 [INFO] rollback: starting rollback manager
2015/11/24 20:41:26 [INFO] expire: restored 5 leases
2015/11/24 20:41:26 [INFO] core: post-unseal setup complete
2015/11/24 20:42:25 [WARN] core: leadership lost, stopping active operation
2015/11/24 20:42:25 [INFO] core: pre-seal teardown starting
2015/11/24 20:42:25 [INFO] rollback: stopping rollback manager
2015/11/24 20:42:25 [INFO] core: pre-seal teardown complete
2015/11/24 20:42:25 [INFO] core: pre-seal teardown starting
2015/11/24 20:42:25 [INFO] core: pre-seal teardown complete
2015/11/24 20:43:25 [INFO] core: acquired lock, enabling active

All 3 of them show similar logs.

This is the config I have on all 3 instances (already posted above):

backend "consul" {
  address = "<internal_consul_elb_address>:8500"
  path = "vault"
  advertise_addr = "https://<public_elb_address>:8200"
  tls_skip_verify = 1
}

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}

listener "tcp" {

  address = "10.0.3.137:8200"

  tls_cert_file = "/etc/vault.d/server.pem"
  tls_key_file = "/etc/vault.d/server.key"
}

What would make a vault loose leadership?

[Armon Dadgar]

Hey Anthony,

Given the lock acquisition to lock failure is almost exactly 60 seconds apart and you are

going through ELB, it’s likely you are hitting the ELB timeout and it is terminating the 

connection of Vault to Consul.

Vault is typically configured to talk to a local Consul agent, and that agent will automatically

forward traffic to servers as appropriate, and that would avoid the ELB timeout issue.

Hope that helps!

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/694f0b50-188b-4d11-81ff-66b19f7bae7f%40googlegroups.com.

[Anthony Powles]

I have my consul server agents running on the same hosts as vault, so from your suggestion, I've updated Vault's config to point to their local instance of consul instead of the ELB address. Seems ok so far.

However, something strange is happening, doesn't seem to be related, as I've tried with the consul elb address and the local instance, but 2 out of the 3 vault instances are returning a strange http code during the ELB health check:

$ curl -i http://127.0.0.1:8200/v1/sys/health
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Date: Wed, 25 Nov 2015 01:44:13 GMT
Content-Length: 51

{"initialized":true,"sealed":false,"standby":true}

Not sure why it's returning that status code, since nothing is hitting that vault instance.

The 2 instances that are returning this status are therefore OutOfService in the ELB.

I've tried sealing/unsealing and restarting vault, but nothing changes.

Right after restarting vault, I get the 500 because the vault is sealed, so far so good

$ curl -i http://127.0.0.1:8200/v1/sys/health
HTTP/1.1 500 Internal Server Error
Content-Type: application/json
Date: Wed, 25 Nov 2015 01:52:29 GMT
Content-Length: 50

{"initialized":true,"sealed":true,"standby":true}

As soon as I unseal it, I get the 429

$ curl -i http://127.0.0.1:8200/v1/sys/health
HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Date: Wed, 25 Nov 2015 01:52:59 GMT
Content-Length: 51

{"initialized":true,"sealed":false,"standby":true}

All 3 instances have the exact same configuration, but only 2 are affected by this

[Brian Lalor]

I find this weird, but it is documented, I believe. The 429 is because that instance is a standby. The ELB will only send requests to instances that return 200. Consul sees 429 as passing or warning. Since standby instances will forward requests to the active server, I think this behavior is confusing, but it is by design. I wrote my own health check for consul for this reason. 

-- 
Brian Lalor
bla...@bravo5.org

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/332d1f8f-442c-4820-9fa1-36e13c8e48e7%40googlegroups.com.

[Jeff Mitchell]

As Brian pointed out, 429 is the status code that Consul expects for a
"warning" status from health checks. Vault uses this for standby nodes
to make it easy to set up a Consul health check. This keeps Consul
from returning standby instances in a query for Vault.

--Jeff

> https://groups.google.com/d/msgid/vault-tool/59EAA282-3BC0-4822-8ABD-4BCF9CA662DA%40bravo5.org.

[Brian Lalor]

On Nov 25, 2015, at 11:14 AM, Jeff Mitchell <je...@hashicorp.com> wrote:
>
> As Brian pointed out, 429 is the status code that Consul expects for a
> "warning" status from health checks. Vault uses this for standby nodes
> to make it easy to set up a Consul health check. This keeps Consul
> from returning standby instances in a query for Vault.

But isn’t it ok if a standby instance is returned by Consul, since Vault will forward the request?

—
Brian Lalor
bla...@bravo5.org

[Jeff Mitchell]

It's OK, yes, so long as your advertise_addrs are correctly set up (or
detected). But why send the client through a redirect when they can
simply connect straight to the leader?

--Jeff

[Brian Lalor]

On Nov 25, 2015, at 11:23 AM, Jeff Mitchell <je...@hashicorp.com> wrote:
>
>> But isn’t it ok if a standby instance is returned by Consul, since Vault will forward the request?
>
> It's OK, yes, so long as your advertise_addrs are correctly set up (or
> detected). But why send the client through a redirect when they can
> simply connect straight to the leader?

The bigger issue is that I don’t want checks to be in a non-passing state unless they’re actually not passing. Persistent warnings lead to alert fatigue.

—
Brian Lalor
bla...@bravo5.org

[junen...@gmail.com]

Hi Anthony,
I am trying to set up similar configuration for Vault with Consul in AWS VPC.  Understand the public ELB for Vault and setup the health check for Vault, but not quite understand the purpose of the <internal_consul_elb_address>:8500 you specified in your Vault config file. Could you please explain?  I am new to Vault and Consul.  Thanks a lot!

June


[Anthony Powles]

Hi June,

So initially, Vault's configuration pointed to the load balancer we put in front of all the consul server instances, which is the internal_consul_elb_address you mention

However, I was getting strange behaviors in Vault, the vault cluster kept changing leaders because the consul ELB was timing out somehow. See the last post from Armon in this thread.

So the way I solved it was to point vault directly to a consul instance, instead of the ELB.

Luckily, I have 3 servers each running consul and vault, so vault communicates directly the local instance of consul.

This is what my current Vault configuration file looks like:

backend "consul" {
  address = "consul0:8500"
  path = "vault"
  advertise_addr = "https://my.domain.com:8200"

  tls_skip_verify = 1
}

listener "tcp" {
  address = "127.0.0.1:8200"
  tls_disable = 1
}

listener "tcp" {

  address = "10.0.1.154:8200"

  tls_cert_file = "/etc/vault.d/server.pem"
  tls_key_file = "/etc/vault.d/server.key"
}

So, on a server named consul0, which is running both an instance of consul and vault, the address on the consul backend is now consul0, which is just the local machine

Let me know if you have more questions, happy to help.

Cheers,

Anthony

[junen...@gmail.com]

Thanks Anthony.  I got the cluster setup working with the local address for the consul backend. 

[junen...@gmail.com]

Hi Anthony,
I'm trying to set up the TLS with Vault.  When I concatenate the domain pem with CA (digicert) pem into one pem file, I got "NSS error -12263 (SSL_ERROR_RX_RECORD_TOO_LONG)" error. Without concatenation, I got bad certificate and unknown authority error. Also do you need add the CA root pem file? Could you please shed some light on how to config these pem files under /etc/vault.d ?

Thanks,
June

[Anthony Powles]

I just put the certs files I received from the CA (GlobalSign in my case) I didn't have to do anything to them.

I'm guessing the unknown authority error could happen if you're using a self-signed cert?

I'm afraid I'm not familiar enough on this subject to be of much help, I'm sorry