Vault Cert Authentication behind NGINX

[gandre...@gmail.com]

Hello Team,

We are just exploring Vault as a viable option to store secrets. We are in initial setup where our initial stack comprises of:

Nginx -> Vault -> Zookeeper

We plan to rely on Cert based authentication for services accessing their secrets from Vault. Where we are stuck for now is that

Nginx is unable to pass the Client cert to the Vault backend. We have tried multiple options.

So when we connect to the VIP:

$curl -vk --request POST --cert .client_cert.pem --key ./client_key.pem https://nginx-vip/v1/auth/cert/login"

we get a response:  {"errors":["client certificate must be supplied"]}

But when we try directly with vault:

$curl -vk --request POST --cert .client_cert.pem --key ./client_key.pem https://vault-server:8200/v1/auth/cert/login"

we Get the correct response with a Vault token.

Our Nginx config has proxy pass set as:

location / {

        proxy_pass https://vault-server:8200/;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

proxy_set_header X-SSL-Client-Cert $ssl_client_escaped_cert;

proxy_set_header X-SSL-CERT $ssl_client_escaped_cert;

proxy_set_header Host $host;

expires -1;

}

and for the Vault tcp listener, we have this config:

listener "tcp" {

  address     = "vault-server:8200"

  tls_cert_file = "../chain.server.pem"

  tls_key_file = "../chain.server.key.pem"

  proxy_protocol_behavior = "use_always"

  x_forwarded_for_reject_not_present = "false"

  x_forwarded_for_reject_not_authorized = "false"

  x_forwarded_for_authorized_addrs = "<redacted>"  

}

Do you see if we are missing something?

Thanks in advance!

- Andrew

[Jeff Mitchell]

Hi Andrew,

Generally the only way to perform client authentication is by using TCP proxying. If you are terminating the TLS connection by nginx, nginx doesn't have the private key to use to then connect to Vault. For obvious reasons, you can't configure Vault to simply pretend that one TLS connection is another, including proxy headers. (The proxy headers allow the proxy to specify the client IP, and that's also potentially spoofed, but unless you very specifically turn off some other features it is never the only method of authentication.)

If TCP proxying isn't possible for you, you'll need to use a different authentication mechanism.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/bc0d3115-6ee4-4dd3-b838-afeebfe7179a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gandre...@gmail.com]

Thank you for a prompt reply Jeff. Appreciate it!

We thought as much but we were hoping that Vault might have a way to read a client certificate from the Proxy header. 

Since it already provides "proxy_protocol_authorized_addrs" which mean we can trust only the VIP IP and parse the cert that came in the header.

But anyways, we will look into another solution for TCP proxy.

We also had a query about the Zookeeper backend. We are setting the ACL's on the node but we could not find a configuration that could allows 

TLS configuration like other backends. Is there a way to enable TLS for Zookeeper?

We understand that data sent to zookeeper is encrypted, but without TLS the authinfo can be read and data be corrupted.

We are trying to follow your best practices stated here for end to end TLS: https://www.vaultproject.io/guides/operations/production.html

Thanks!

- Andrew

[Jeff Mitchell]

I'm honestly not sure. If the docs don't mention any tls options, try searching through GitHub issues to see if anyone has talked about it.

Best,

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/8b325640-da77-446f-8a1f-a1f48b8de858%40googlegroups.com.

[gandre...@gmail.com]

Thank you for the pointer.

We have gone through the documentation but we couldn't find any config flag to enable Zookeeper TLS. We found this configuration 

available for Cassandra and other backends though. We are looking into ZK since we already have a setup for it.

I just did a search on the Vault repository and found two issues on the same thing referring each other. So it looks like this is not supported yet :(

https://github.com/hashicorp/vault/issues/1652

https://github.com/hashicorp/vault/issues/4907

There is a PR(https://github.com/hashicorp/vault/pull/4856) referred in the issue for this change but it doesn't look like it is available in Vault source yet.

Any chance Vault will support this ?

Thanks!

- Andy

[Jeff Mitchell]

Hi Andy,

We don't use or test the ZK storage engine so a good way to get it supported in Vault would be to test out the PR, make sure it's working well for you, and leave a comment to that effect (or describing what's going wrong).

Best,

Jeff

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/a6cee6d6-366b-40c1-851b-6c1dd9ae8b48%40googlegroups.com.

[Andrew Goldman]

Thanks Jeff. 

After going through the PR It does not look like that the Vault CD actually produces a binary for each PR build.

So I am assuming the only way to test is to build from source and try.. as mentioned on the installation page. https://www.vaultproject.io/docs/install/index.html#compiling-from-source

Since we need Zookeeper at least for HA for now we will test this and provide our result.

Thanks!

- Andrew

[Andrew Goldman]

Hi Jeff,

We have tested this PR - building Vault from source. As per our requirement we were able to establish a mutual TLS connection with

our Zookeeper cluster without issues. We have been running this code for about a day now. 

I have provided my comments on the PR.

Thanks!

-Andrew