vault unable to establish MTLS connection with consul as a storage backend

177 views

Skip to first unread message

Joseph Lorenzini

unread,

Aug 26, 2017, 8:40:02 PM8/26/17

to Vault

Hi all,

We have configured vault to use consul as its backend storage and have encountered a problem with setting up TLS between the two endpoints. I am curious to know if anyone has gotten this work.

Both consul and vault have been configured to communicate over TLS and consul has been configured to perform TLS client authentication on vault's certificate. When vault starts up, the following warning is printed out.

2017/08/27 00:27:29.849871 [WARN ] physical/consul: reconcile unable to talk with Consul backend: error=service registration failed: Put https://localhost:8500/v1/agent/service/register: remote error: tls: bad certificate

We have set up a self-signed internal CA that has issued two leaf certificates, one for vault and the other for consul. Note that vault is only talking to consul agent in client mode running on the same system as vault. I performed tcpdump of the consul port and verified that the SSL handshake progresses up until the server (consul) requests the client's (vault's) SSL certificate. At that point, the connection drops. It seems like the vault (as a SSL client) is simply refusing to send its SSL certificate to consul (the server).

In consul, if we set verify_incoming_https to false, then vault is able to connect to consul without a problem. We can successfully connect to the consul server using openssl s_client and curl, where in both cases we are using the same certificate and key that vault has been provisioned to use.

Here's the vault version.

Vault v0.8.1 ('8d76a41854608c547a233f2e6292ae5355154695')

We can provide additional information including configurations and x509 certificates if anyone would find this helpful for debugging the problem.

Thanks,

Joe

Joseph Lorenzini

unread,

Aug 30, 2017, 10:33:10 AM8/30/17

to Vault

Hi all,

Based on our analysis, we believe this is in fact a bug in vault. We've opened an issue accordingly with steps to reproduce. For anyone interested, here's the ticket.