Can't manage to get TLS auth to work

[Brian Rodgers]

I've been fighting TLS auth now for awhile.  Here's what I'm trying to do.  First and foremost, I'm trying to totally automate this process.  We are setting up multiple VPCs in AWS that will all talk to a vault instance.  I want to generate a self signed cert, register it as trusted in vault (using the user auth of who is starting the VPC), and then transfer that cert securely to the EC2 instances in that VPC so they can authenticate to Vault using the cert.

First, I generate a self signed cert.  I generate a CN in a dynamically created openssl conf.  I've tried a few other approaches to generate the cert, all with the same results.  

cat >openssl.conf <<EOF
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
CN = $STAX_VAULT_CERT_CN

[v3_req]
basicConstraints = CA:TRUE
EOF

    openssl req -x509 -nodes -days 3650 \
        -newkey rsa:2048 -keyout $STAX_VAULT_CERT_CN.key \
        -out $STAX_VAULT_CERT_CN.crt -config openssl.conf

Next up I create a policy.  I don't think that part matters for my issue.  Then I register the cert as follows:

vault write auth/cert/certs/$STAX_VAULT_CERT_CN \
    display_name=aws-stax-$STAX_NAME \
    policies=aws-stax-$STAX_NAME \
    certificate=@${STAX_VAULT_CERT_CN}.crt

I can then read the cert out from that path in vault.

However, when I try to login using either the CLI or curl, I get the same response: "invalid certificate or no client certificate supplied"

I can see in a curl -v req that it is sending a cert:  "Client certificate: aws_stax_vpc_stax_24302_scytheless"

A couple of things that might be relevant:

1. Being a self signed cert, I'm not passing in cacerts to either curl or vault CLI.  I tried passing in the cert itself as the cacert just to see if that might work on a self signed cert, and (not surprisingly) it didn't.
1. Ideally I'm looking to stay with self signed certs and not set up a CA.  
2. The client cert I'm generating isn't in any way signed by the TLS cert that the server is using for its own TLS listener.  I wouldn't think it would need to.  I was expecting that registering the cert by writing it to auth/cert/certs that would mean it's "trusted."  
1. I'm hoping there's no such requirement that it be signed by the TLS cert the server is using.  I dynamically generate that cert as a self signed cert because it's only used to provide encryption between the load balancer I have in front of vault.  The "real" cert that the vault client sees terminates on the load balancer, so Vault server doesn't have it.  I don't want to have to change that for a number of reasons.
3. Initially I wasn't worrying about whether the CN in the cert matched anything in particular.  Does it matter?  For example, I initially wasn't using the CN as the path under auth/cert/certs/ that I registered the cert as. I tried to make sure I'm consistent now and that didn't solve anything. 

Thanks for any help you can provide!

[Clay Bowen]

Hey Brian, I use TLS certs a lot (you can see my posts on this forum) so I may be able to help.  Can you provide the code for how you're logging into Vault?  Also, is your Vault server using a self-signed certificate for access? (do you use https for access to the vault server?)  If the server certificate is self-signed you'll need to modify your command a little.  I wrote up an article in this group on authenticating using x509 certs - you may find something in it.

1. I do the same - create self-signed certs for all my users and they work just fine

2. There's no requirement for the x509 user certs to be verified

3. The CN shouldn't matter

Thanks,

Clay

[Brian Rodgers]

Thanks Clay!  I did try following your guide for the most part.  I am using a self signed cert on the server.  What actually happens is that I have a Thawte cert that terminates on the load balancer, and each of the nodes behind the load balancer (I run 2 more in standby) generate a self signed cert on startup.  The only purpose of that is to enable encryption on traffic between the load balancer and the Vault server.  Clients never see that self signed cert.  Is there any relationship necessary between the self signed cert the server is using and the client certs?  It didn't look like it in your guide, but I might have missed something.

To login, I've been using this for the cli:

vault auth -method=cert -client-cert=aws_stax_vpc_stax_24302_scytheless.crt -client-key=aws_stax_vpc_stax_24302_scytheless.key

For curl, I'm on a mac which apparently doesn't support pem files in curl.  So i convert the cert and key to a pkcs file and call it like this:

curl  --cert certificate.pfx  $VAULT_ADDR/v1/auth/cert/login -XPOST -v

Both ways I get an invalid cert or no cert found error.  

I didn't try disabling audit logging yet as that discussion and the associated fix look like they should be in .2.  

[Armon Dadgar]

Hey Brian,

For mutual TLS authentication to work correctly, Vault must terminate the TLS connection directly.

Otherwise, the client is negotiating with the load balancer, and the load balancer separately with Vault.

In that arrangement, there is no way for Vault to verify the client identity since that information is not

passed through the load balancer.

Hope that helps!

Best Regards,

Armon Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/d9d0ebce-4a3f-483c-b54f-41bf11245a04%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brian Rodgers]

Ahh ok, that's starting to make sense.  Even though it worked for one way, what you're saying is that now that I'm trying to pass a client TLS cert in, the load balancer doesn't pass it along to vault at all since it's terminating and recencrypting it?  I had been assuming (honestly without really thinking) that it'd still pass the client cert on to vault in the connection between the ELB and vault.  You're saying it's just getting dropped at the ELB, right?  I suppose that makes sense.  

So no way around this I take it, other than moving the cert to the Vault server?  I can do that, it's just a bit more difficult to manage compared with keeping all our certs on the ELB.  

[Clay Bowen]

Thanks Armon, that makes a lot of sense.  Brian, if you want to keep your "real" cert off of the Vault server you can use a self-signed cert on it and either use the "insecure" flag (so the client won't verify the certificate with a CA) or add the Vault cert to the CA file on the clients.  Some security groups won't allow the "real" cert to be placed somewhere outside of their direct control.

Thanks,

Clay

On Tuesday, August 4, 2015 at 11:56:07 PM UTC-5, Brian Rodgers wrote:

[Brian Rodgers]

Yeah, for now I changed my ELB to pass TCP 443 rather than terminate, and that solved the problem (with the insecure flag for now).  So it definitely was the ELB TLS termination.  

The main reason why we like terminating on the ELB is that we use a wildcard cert typically so we don't have to buy a cert for our subdomains.  So terminating on the ELBs means that key lives only in AWS IAM.  Makes rotation easier and it has fewer places it could get compromised.  They're OK with installing a cert within an application if it's not a wildcard cert.  So I can get through this without having to go with insecure certs or install the cert on clients by just buying a specific cert for this subdomain.  I've got that in progress.

Thanks to both of you for your quick help!  I appreciate it!

[Clay Bowen]

Glad to be of help, Brian.  I've found Armon (and HashiCorps in general) to be very easy to work with and very responsive.

Clay

On Tuesday, August 4, 2015 at 11:56:07 PM UTC-5, Brian Rodgers wrote:

[Brian Rodgers]

OK, I'm now having a new issue.  I've got a Thawte issued cert installed into Vault and I think it's setup right.  The cert shows up as green in chrome if I pull up the vault server there, and curl is happy.  The command below gives me no complaints about TLS and it returns a token.

curl -XPUT https://<vault-server>/v1/auth/cert/login --cert /vault/<client-cert>.crt --key /vault/<client-cert>.key

However, if I try to use the CLI, I get this:  vault auth -method=cert -client-cert=/vault/<client-cert>.crt -client-key=/vault/<client-cert>.key

Error making API request.

URL: PUT https://<vault-server>/v1/auth/cert/login

Code: 400. Errors:

* invalid certificate or no client certificate supplied

But if I add -tls-skip-verify to the CLI call, it works fine.

On the vault server, I simply appended the intermediate cert from Thawte to the end of the vault server cert file, which I thought was all I needed to do.  It seems to have made Chrome and curl happy.  Is there something else I need to do for the CLI to work?

On Tuesday, August 4, 2015 at 11:56:07 PM UTC-5, Brian Rodgers wrote:

[Armon Dadgar]

Brian,

The error is indicating that no client certificate is being presented to Vault. Is there an intermediary load balancer or SSL terminator that might be stripping the client certificate?

Also which version of Vault are you running as there was a bug related to this for a while.

Best Regards,

Armon Dadgar

Sent from my iPhone

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/af26d6ad-c475-4b10-9a3d-cd0db91263ba%40googlegroups.com.

[Brian Rodgers]

I don't think so, and if that were true, wouldn't curl have the same problem?  That's the part I don't get.  Unless the Go https library is somehow more picky about something, they should behave the same, right?

I'm running 0.2.0.  Was the bug fixed in that version, or do I need to do my own build to get that fix?

[Brian Rodgers]

Also, the client cert does appear to be getting passed through even with the CLI, since it works if I do -tls-skip-verify.  There's something about the CLI that isn't liking the cert as it's being presented, while curl is fine.

[Clay Bowen]

It may be worthwhile to check your environment variable VAULT_ADDR to ensure it's pointing at the right location (mine, for example, is set to VAULT_ADDR=https://10.16.19.21:8000). 

If that's set OK, I'd suggest to enable auditing (you can enable this through curl using the /sys/audit/file path) and see what comes up in the file when making the same request through curl and the CLI.

Thanks,

Clay

[Armon Dadgar]

Clay’s suggestions are good, there maybe an environment variable affecting the CLI behavior

and the audit logs are a great way to debug this further. The error you are getting from the

login endpoint is indicating that either no client TLS certificate was presented or that it is not

matching any of the trusted certificates.

Does the client cert use intermediary certificates? There may be an issue with presenting

a chain of certificates via the CLI that curl handles properly.

Best Regards,

Armon Dadgar

From: Clay Bowen <on.rode...@gmail.com>
Reply: vault...@googlegroups.com <vault...@googlegroups.com>>
Date: August 14, 2015 at 7:24:09 AM
To: Vault <vault...@googlegroups.com>>

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/4eb9bbc8-acdb-4fe0-84f7-b5cf84177396%40googlegroups.com.

[Brian Rodgers]

I do have the VAULT_ADDR variable set correctly, and I can see the correct URL in the error message the CLI spits out.

The client cert is self signed, so no intermediate cert there.  The server cert is Thawte issued, so there is an intermediate cert.  My understanding was that all I need to do is concatenate that intermediate cert onto the end of the server cert and it'll present them both to the client so it has the full chain.  Is that correct, or do I need to do something else with the intermediate cert?  That seems to be working when I hit the Vault server via curl or even if I just pull up the vault server in chrome.  But the CLI doesn't like it.  Additionally, it only seems to not like it on the client cert login process.  I get no complaints from vault CLI -- with tls verify enabled -- when doing read/writes with the vault token that I obtained with cert auth.  There's just something in the mutual TLS auth that the CLI doesn't like.

When the CLI sends a req, the audit log gets this:

{"type":"request","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.1.250"},"error":""}

{"type":"response","error":"","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.1.250"},"response":{"secret":null,"data":{"error":"sha1:d765da7404d3839d712da268cfaa93dad52cece0"},"redirect":""}}

{"type":"request","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.3.71"},"error":""}

{"type":"response","error":"","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.3.71"},"response":{"secret":null,"data":{"error":"sha1:d765da7404d3839d712da268cfaa93dad52cece0"},"redirect":""}}

When I do it with curl, I get:

{"type":"request","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.1.250"},"error":""}

{"type":"response","error":"","auth":{"display_name":"","policies":["aws-stax-vpc-stax-28007-unovercome"],"metadata":{"cert_name":"vpc_stax_28007_unovercome","common_name":"vpc-stax-28007-unovercome"}},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.1.250"},"response":{"auth":{"client_token":"sha1:20b551fe0becfab79e73dcd631dcfcf58d0107bd","display_name":"cert-aws-stax-vpc-stax-28007-unovercome","policies":["aws-stax-vpc-stax-28007-unovercome"],"metadata":{"cert_name":"vpc_stax_28007_unovercome","common_name":"vpc-stax-28007-unovercome"}},"secret":null,"data":null,"redirect":""}}

When I do it with the CLI with tls verify turned off, I get:

{"type":"request","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.3.71"},"error":""}

{"type":"response","error":"","auth":{"display_name":"","policies":["aws-stax-vpc-stax-28007-unovercome"],"metadata":{"cert_name":"vpc_stax_28007_unovercome","common_name":"vpc-stax-28007-unovercome"}},"request":{"operation":"write","path":"auth/cert/login","data":null,"remote_address":"10.183.3.71"},"response":{"auth":{"client_token":"sha1:c32eee8449b10a84f9cc5773501c8971d0a57c24","display_name":"cert-aws-stax-vpc-stax-28007-unovercome","policies":["aws-stax-vpc-stax-28007-unovercome"],"metadata":{"cert_name":"vpc_stax_28007_unovercome","common_name":"vpc-stax-28007-unovercome"}},"secret":null,"data":null,"redirect":""}}

{"type":"request","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"read","path":"auth/token/lookup-self","data":null,"remote_address":"10.183.1.250"},"error":""}

{"type":"response","error":"","auth":{"display_name":"","policies":null,"metadata":null},"request":{"operation":"read","path":"auth/token/lookup-self","data":null,"remote_address":"10.183.1.250"},"response":{"secret":null,"data":{"display_name":"sha1:572029b667182883b9735f2c2fa5695c5790ae10","id":"sha1:c32eee8449b10a84f9cc5773501c8971d0a57c24","meta":{"cert_name":"sha1:845db70a72e86a885b3bfb488be9713f452e534d","common_name":"sha1:1fbc59d7c135bf60da30ff64ab22d7a4cf818c3a"},"num_uses":0,"path":"sha1:1beff447d620b21ed7c2a5951e828de08d6197d5","policies":["sha1:7939847343b407411df78e85cca55fa880da3988"]},"redirect":""}}

[Armon Dadgar]

Brian,

Could you please provide the CLI commands you are using when these audits are being generated?

It is possible that there is a bug in how we are initializing the TLS client in the CLI.

Best Regards,

Armon Dadgar

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/15113e58-def3-4fe9-b216-cf66905d9980%40googlegroups.com.

[Brian Rodgers]

Armon,

The first audit log (unsuccessful CLI) command was (with VAULT_ADDR set to the vault server as https://<url>/):

vault auth -method=cert -client-cert=/root/vault/vpc-stax-28007-unovercome.crt -client-key=/root/vault/vpc-stax-28007-unovercome.key

The third log (successful CLI) one is the same, but with tls-skip-verify:

vault auth -method=cert -client-cert=/root/vault/vpc-stax-28007-unovercome.crt -client-key=/root/vault/vpc-stax-28007-unovercome.key -tls-skip-verify

The middle one (successful CURL) is:

curl -XPUT https://<vault-url>/v1/auth/cert/login   --cert /root/vault/vpc-stax-28007-unovercome.crt   --key /root/vault/vpc-stax-28007-unovercome.key

Is there any client side debugging I can turn on?

On Friday, August 14, 2015 at 7:36:02 PM UTC-5, Armon Dadgar wrote:

Brian,

...

[Armon Dadgar]

Hey Brian,

Looks like it was a bug in our client setup for the CLI. Setting either a CA or the insecure flag would

cause the client certificates to be loaded, but otherwise they were skipped.

Fixed in master: https://github.com/hashicorp/vault/commit/e659d931638808ed47eeb3b65eb308acee614c20

Thanks for reporting!

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/91693d49-df88-4b44-94ca-f768ac8b7ab3%40googlegroups.com.

[Brian Rodgers]

Thanks Armon!  I'll do a build when I get a chance and report back if it solved it for me.  Are we close to a .3 release or should I plan to deploy my own build for awhile?

-Brian

[Armon Dadgar]

Hey Brian,

We are probably a few weeks out from a 0.3, so I’d plan on building it for now.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/cdb3e096-1810-4f3e-8364-0d535ec79d67%40googlegroups.com.