Vault CLI not using OS X System Keychain for Certificates

[Jeromy Labit]

Greetings,

  I'm wondering if anyone else has seen this issue, and hopefully resolved it. It appears that the Vault CLI is not using the certificate keychains properly on OS X. I've seen some discussion of this in different golfing places, but it's always been an issue with cross-compiling. I compiled Vault natively, and am still running into the issue.

  For this test, I created a ROOT and Intermediate CA, and created a TLS certificate for localhost. I imported the CA chain into my OS X system keychain and set it to always trust the ROOT.

$ vault version

Vault v0.2.1-dev  <--- I've tried with 0.2.0 as well

$ vault status

Error checking seal status: Get https://localhost:8200/v1/sys/seal-status: x509: certificate signed by unknown authority

$ curl -XGET https://localhost:8200/v1/sys/seal-status

{"sealed":true,"t":1,"n":1,"progress":0}

  For my purposes, I have to verify the certificate, so I can't "work around" this by skipping verification. I would prefer to use the CLI, but will use curl if I must.

  Any ideas?

Jeromy

[Clay Bowen]

For testing, you can append the Vault Server's self-signed certificate to your client's certificate file (/etc/ssl/certs/ca-bundle.crt).  That will allow your testing machine to "validate" the certificate.

Thanks,

Clay

[Jeromy Labit]

Clay,

  I think that would work fine for Linux, but I don't think that's the right location for OS X. I think have the bundle in the right place, because curl, Safari, and Chrome all verify the certificate properly when I connect to Vault. It's the Vault CLI that fails :(

  Can you confirm this has worked for you on OS X?

/jeromy

[Clay Bowen]

The location will be different on OSX (which is based on BSD) but the concept is the same.  Since OSX has a decent GUI you can do it that way -- here's an article on adding your cert to OSX:

http://www.techrepublic.com/blog/apple-in-the-enterprise/managing-ssl-certificate-authorities-on-os-x/

To get the CLI to work you'll need to add the self-signed cert to the Vault Server's certificate chain.  I had the same issue, but I was doing it on Linux.

Thanks,

Clay

[Jeromy Labit]

Clay,

  Thanks for your help, but I think you might have missed that I already have imported the chain. My statement is that the CLI does not use the System Keychain, and I'm trying to see if anyone else has experienced this problem before I open an issue.

  Everything else on my system (i.e. curl, Safari, Chrome, etc) works properly when connecting to the Vault server. It's only the Vault CLI that does not work.

/jeromy

[Clay Bowen]

Yeah, sorry Jeromy I did miss that.  

Clay

[Jeromy Labit]

All,

  Just to pass along, I resolved this. Through research, I saw a lot of references to Go being very picky about how a server certificate is presented. I must have had something out of order in how I created the bundle, because I created a new localhost certificate, and added my intermediate and root certificates... and now it is working with the Vault CLI.

  When all else fails, start from scratch... It worked in this case.

  If anyone runs into an issue like this again, I would recommend taking a look at the server certificate... Hopefully this thread will prevent someone else from wasting hours of frustration. :)

  Thanks Clay for all of your responses.

[Clay Bowen]

No problem Jeromy - I know exactly what you're dealing with (see my posts on getting curl to work with Vault on base linux 6.6 systems).  :-)

Clay

[pavan kumar s]

Hi Jeremy,

Can you help me with those commands you used to create the certificates  and keys ? 

I have a vault server on ubuntu where I generated the server certificate and key.

I installed that certificate on my MAC OS(through KeyChain) which i am using as client to the Vault Server. I get the error 'certificate verify failed' on client and then on vault server I get remote error: unknown certificate authority.

I am not sure what is wrong. Any idea ?

Thanks,

Pavan

[Jeromy Labit]

If it is a self-signed certificate, then you'll need to make sure the certificate is trusted. When you import through keychain, it is not automatically trusted. Double click the certificate in Keychain, and expand the Trust section. You will have an option to "Always Trust" the certificate.

If you created your own ROOT CA, or intermediate CAs, on the client side you will have to trust the ROOT CA in the same fashion, and make sure you include the entire chain on the server side (i.e. cat ica.crt >> client.crt; cat root.crt >> client.crt)

Hope this helps.

[Fatima M]

@jeromy - people who end up here are here for the same reason you opened this post: 1) they have successfully added their self signed cert to the system keychain and set it to always trust, their web browsers are working 2) they cannot get vault cli to use the OS X system keychain... hence the name of your post.

it is not clear how you fixed your problem.... this is rather frustrating.

[Jeff Mitchell]

Hi there,

It depends on whether it's compiled dynamically or statically, but
based on https://golang.org/src/crypto/x509/root_darwin.go I think it
only looks in one specific keychain:

cmd := exec.Command("/usr/bin/security", "find-certificate", "-a",
"-p", "/System/Library/Keychains/SystemRootCertificates.keychain")

Can you make sure that you're adding it to that keychain?

Thanks,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/b066607f-ac5b-454f-8092-dde5fd7cf1cd%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Fatima M]

Hi Jeff, was also referred to your comments here: https://groups.google.com/forum/#!msg/vault-tool/u7l0jl8q4V0/qFRn2_cVDQAJ

Using the --ca-cert=/path/to/cert worked.

[Mitchell Hashimoto]

I want to note that we just tackled this in Terraform as well.

There is a Go issue for OS X to support the Login keychain as well as the System keychain. 

https://github.com/golang/go/issues/14514

Hopefully that gets merged as well since it is very common for users to have certs in their login keychain.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/37040756-9b76-460f-bc95-11f8a394c1b0%40googlegroups.com.