Vault-Consul over HTTPS with a self signed cert

[gran...@distilnetworks.com]

I am having problems getting Vault working with Consul using TLS.

The error I am getting is:

[ERR] core: failed to read seal configuration: Get https://0.0.0.0:8501/v1/kv/newvault/core/seal-config: remote error: bad certificate

I am generating my certs using the following commands:

openssl req -x509 -newkey rsa:2048 -days 3650 -nodes -out ca.cert
openssl req -newkey rsa:2048 -nodes -out consul.csr -keyout consul.key
openssl ca -batch -config vault_openssl.cnf -notext -in consul.csr -out consul.cert

My consul config looks like this:

{
  "bootstrap_expect": 2,
  "ca_file": "/etc/consul/ssl/CA/ca.crt",
  "cert_file": "/etc/consul/ssl/certs/consul.crt",
  "client_addr": "0.0.0.0",
  "data_dir": "/var/lib/consul",
  "key_file": "/etc/consul/ssl/private/consul.key",
  "ports": {
    "dns": 8600,
    "http": 8500,
    "rpc": 8400,
    "serf_lan": 8301,
    "serf_wan": 8302,
    "server": 8300,
    "https": 8501
  },
  "server": true,
  "ui": true,
  "ui_dir": "/srv/consul-ui/current",
  "verify_incoming": true,
  "verify_outgoing": true
}

And my vault config looks like this:

{
  "disable_mlock": false,
  "backend": {
    "consul": {
      "address": "0.0.0.0:8501",
      "path": "vault",
      "advertise_addr": "https://my.dns.here.com:8501",
      "scheme": "https"
    }
  },
  "listener": {
    "tcp": {
      "address": "0.0.0.0:8200",
      "tls_cert_file": "/etc/vault/ssl/certs/vault.crt",
      "tls_key_file": "/etc/vault/ssl/private/vault.key"
    }
  }
}

I am using Ubuntu 14.04 following these directions.

I have tried many different configurations without luck and would appreciate any suggestions. If more information is needed I will happily provide it. Am I not adding the certs to server correctly?

[Hridyesh Pant]

Not sure , just putting my idea.

i will try with below changes

In consul setting

"consul": {
      "address": "IP:8500",

in vault :

 "listener": { 
    "tcp": { 
      "address": "127.0.0.1:8200"

now i will configure nginx with https which will redirect all request to 127.0.0.1:8200.

access vault via nginx end point.

--Thanks

Hridyesh

[Jeff Mitchell]

Hi Grant,

Without having the certificates to look at, I can't really say too
much, but from looking at your openssl commands I don't think you're
actually generating a self-signed CA certificate. At a minimum I'd
expect a -signkey argument. I suggest either using Vault's PKI
backend, or looking for openssl instructions on generating a
self-signed CA (for instance, here
http://www.akadia.com/services/ssh_test_certificate.html), or looking
into easy-rsa.

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/vault-tool/192f799a-6e01-4ad7-ac83-d684ee927667%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[gran...@distilnetworks.com]

Editing those addresses doesn't change the result. I don't think bringing in nginx should be necessary here, as the issue is with the Vault server running on the same machine that the Vault client is attempting to run from.

Bringing in nginx would also not fix the problem of needing a HTTPS endpoint for consul and that would still be a problem.

[gran...@distilnetworks.com]

Hi Jeff,

I don't think Vault's PKI system would work here, as I can't even get "vault status" to return a valid response, so no backends are in play.

I will give the other suggestions that you have for generating keys a shot today and come back with the results. Regarding the "signkey" argument, it is part of my "vault_openssl.cnf" config.

I should note that I am able to run run "openssl verify" on the cert and it says "OK".

Thanks!
Grant

Also, here is my vault_openssl.cnf file:

[ ca ]
default_ca = myca
 
[ myca ]
unique_subject = no
new_certs_dir = .
certificate = ca.cert
database = certindex
private_key = privkey.pem
serial = serial
default_days = 3650
default_md = sha1
policy = myca_policy
x509_extensions = myca_extensions

[ myca_policy ]
commonName = supplied
stateOrProvinceName = supplied
countryName = supplied
emailAddress = optional
organizationName = supplied
organizationalUnitName = optional

 
[ myca_extensions ]
basicConstraints = CA:false
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always
keyUsage = digitalSignature,keyEncipherment
extendedKeyUsage = serverAuth,clientAuth

[Jeff Mitchell]

On Fri, Mar 4, 2016 at 12:33 PM, <gran...@distilnetworks.com> wrote:
> I don't think Vault's PKI system would work here, as I can't even get "vault
> status" to return a valid response, so no backends are in play.

Well, it would if you did initial setup without TLS. But that depends
on your setup and your comfort with performing bootstrapping
unencrypted :-)

> I will give the other suggestions that you have for generating keys a shot
> today and come back with the results. Regarding the "signkey" argument, it
> is part of my "vault_openssl.cnf" config.
>
> I should note that I am able to run run "openssl verify" on the cert and it
> says "OK".

Any chance you can paste (or privately email me) the consul.crt and
CA.crt certificates (not the private keys)?

Thanks,
Jeff

[gran...@distilnetworks.com]

Ahhh, interesting idea about the bootstrapping... but I really would rather find a better solution than that.

Regarding the certs, I don't mind pasting them here as I can just generate more; but I think this problem may be related to me generating the certs on my local machine (OS X) and then moving them to the Ubuntu 14.04 machine. I am currently attempting to generate on the Ubuntu machine to see if that changes my results. If I still have problems, I will paste the newly generated certs here.

One interesting thing I have noticed is running this command returns "OK"

openssl s_client -connect 127.0.0.1:8200 -CAfile /etc/ssl/certs/ca-certificates.crt

But if I leave off the -CAfile and path, it fails. I get "Verify return code: 21 (unable to verify the first certificate)"

Thanks again for the help, I will be back after I generate the certs on Ubuntu.

[Jeff Mitchell]

Sounds good -- I believe the error message you're seeing is coming
from the Consul API that Vault is using (rather than Vault itself) but
I figured that if I check the certs over and make sure that the CA
looks like an appropriate CA and the client looks like it was properly
issued by that CA, etc., that maybe I'd spot something that might be
causing issues. Let me know how it goes.

--Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/d6f48f1d-fa28-498a-8e54-fe161cd61b58%40googlegroups.com.

[mlap...@newrelic.com]

FWIW, I used CertStrap by Square (https://github.com/square/certstrap) to generate all my Vault/Consul certs without issue. It has a very simple CLI for creating a CA and generating/signing certificates. It abstracts away all the headache of using OpenSSL manually and uses the same SSL/TLS code as Vault/Consul (Go). 

HTH,

Matthew

[gran...@distilnetworks.com]

Thank you very much, I will give this a shot right now. I did read about some issues between Go and openssl, so this seems like a good way to go.

I will be back with my results.

[gran...@distilnetworks.com]

This got me back to the same situation that I was already in, unfortunately. Are you using Ubuntu?

I get "x509: certificate signed by unknown authority" when I start Consul and Vault, so I add the cert to "/usr/share/ca-certificates" and run "sudo dpkg-reconfigure ca-certificates", and I get the same "remote error: bad certificate".

Am I missing a step?

[gran...@distilnetworks.com]

This seems strange to me. When I post to "seal-status", the cert verifies but returns "Method Not Allowed." This is the not the case when I post to "auth", it accurately tells me that the Vault is sealed.

$ curl -v -X POST  "https://127.0.0.1:8200/v1/sys/seal-status"

* Hostname was NOT found in DNS cache

*   Trying 127.0.0.1...

* Connected to 127.0.0.1 (127.0.0.1) port 8200 (#0)

* successfully set certificate verify locations:

*   CAfile: none

 CApath: /etc/ssl/certs

* SSLv3, TLS handshake, Client hello (1):

* SSLv3, TLS handshake, Server hello (2):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS handshake, Server key exchange (12):

* SSLv3, TLS handshake, Request CERT (13):

* SSLv3, TLS handshake, Server finished (14):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS handshake, Client key exchange (16):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSL connection using ECDHE-RSA-AES256-GCM-SHA384

* Server certificate:

*        subject: CN=Vault

*        start date: 2016-03-04 21:22:11 GMT

*        expire date: 2018-03-04 21:22:11 GMT

*        subjectAltName: 127.0.0.1 matched

*        issuer: CN=VaultCA

*        SSL certificate verify ok.

> POST /v1/sys/seal-status HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 127.0.0.1:8200

> Accept: */*

>

< HTTP/1.1 405 Method Not Allowed

< Content-Type: application/json

< Date: Fri, 04 Mar 2016 22:28:31 GMT

< Content-Length: 14

<

{"errors":[]}

* Connection #0 to host 127.0.0.1 left intact

$ curl -v -X POST  "https://127.0.0.1:8200/v1/auth"

* Hostname was NOT found in DNS cache

*   Trying 127.0.0.1...

* Connected to 127.0.0.1 (127.0.0.1) port 8200 (#0)

* successfully set certificate verify locations:

*   CAfile: none

 CApath: /etc/ssl/certs

* SSLv3, TLS handshake, Client hello (1):

* SSLv3, TLS handshake, Server hello (2):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS handshake, Server key exchange (12):

* SSLv3, TLS handshake, Request CERT (13):

* SSLv3, TLS handshake, Server finished (14):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS handshake, Client key exchange (16):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSL connection using ECDHE-RSA-AES256-GCM-SHA384

* Server certificate:

*        subject: CN=Vault

*        start date: 2016-03-04 21:22:11 GMT

*        expire date: 2018-03-04 21:22:11 GMT

*        subjectAltName: 127.0.0.1 matched

*        issuer: CN=VaultCA

*        SSL certificate verify ok.

> POST /v1/auth HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 127.0.0.1:8200

> Accept: */*

>

< HTTP/1.1 503 Service Unavailable

< Content-Type: application/json

< Date: Fri, 04 Mar 2016 22:28:43 GMT

< Content-Length: 31

<

{"errors":["Vault is sealed"]}

* Connection #0 to host 127.0.0.1 left intact

[Jeff Mitchell]

Hi Grant,

/sys/seal-status takes a GET. The appropriate HTTP verbs for each
operation are listed in the documentation -- Vault tries to be CRUDdy
whenever possible, with the caveat that for security it foregoes query
parameters in favor of using POST/PUT with JSON body data.

See https://www.vaultproject.io/docs/http/sys-seal-status.html for example.

Best,
Jeff

> --
> This mailing list is governed under the HashiCorp Community Guidelines -
> https://www.hashicorp.com/community-guidelines.html. Behavior in violation
> of those guidelines may result in your removal from this mailing list.
>
> GitHub Issues: https://github.com/hashicorp/vault/issues
> IRC: #vault-tool on Freenode
> ---
> You received this message because you are subscribed to the Google Groups
> "Vault" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to vault-tool+...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/vault-tool/24f41101-106f-4692-8cf9-c0ded512e606%40googlegroups.com.

[gran...@distilnetworks.com]

Ah, thanks Jeff.

So, that still gives the same result. The cert is verified, and the connection goes through, but I think you were right that the problem is in the backend connection from Vault to Consul that the error occurs.

Thanks again.

$ curl -v -X GET "https://127.0.0.1:8200/v1/sys/seal-status"

> GET /v1/sys/seal-status HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 127.0.0.1:8200

> Accept: */*

> 

< HTTP/1.1 500 Internal Server Error

< Content-Type: application/json

< Date: Fri, 04 Mar 2016 22:37:47 GMT

< Content-Length: 136

< 

{"errors":["failed to check seal configuration: Get https://0.0.0.0:8501/v1/kv/vault/core/seal-config: remote error: bad certificate"]}

* Connection #0 to host 127.0.0.1 left intact

[Jeff Mitchell]

On Fri, Mar 4, 2016 at 4:43 PM, <gran...@distilnetworks.com> wrote:
> I get "x509: certificate signed by unknown authority" when I start Consul
> and Vault, so I add the cert to "/usr/share/ca-certificates" and run "sudo
> dpkg-reconfigure ca-certificates", and I get the same "remote error: bad
> certificate".

Just to check, you added the CA cert, not the issued server cert, right?

I have found in the past that update-ca-certificates is very specific
about what it wants and also often doesn't give good errors. You need
to put local CA certificates in /usr/local/share/ca-certificates, not
/usr/share/ca-certificates; also, when you run the reconfigure on
ca-certificates, you have to make sure to select to trust all new
ones. If in doubt, select "ask" and verify that the local one you've
added is in the list.

Also, if you manually configure 'tls_ca_file' in the Consul backend
settings (see https://www.vaultproject.io/docs/config/index.html),
does it work? If it does, but it doesn't otherwise, it's a good
indication that your CA cert isn't properly installed on the system.

Thanks,
Jeff

[Jeff Mitchell]

Oh, along the lines of what I said before -- one such pain point is
that certificates must end in .crt. 'man update-ca-certificates',
which is what is eventually run by reconfiguring ca-certificates, has
some info.

--Jeff

[gran...@distilnetworks.com]

THIS FIXED MY ISSUE!!! Thank you so much!

I was missing the tls_cert_file and tls_key_file in my backend options config for Vault. If you look at my original post, you can see that I had these specified in my listener, but that is (obviously) not enough. I didn't even have to specify the CA, the cert and key for the backend are enough, because I actually did have my CA working correctly after all.

I can't tell you how happy this makes me. I owe you one, Jeff.
Grant

[Jeff Mitchell]

Huh...was it the same exact cert as Consul was using? If they were separate certs it shouldn't have helped, you would need the CA instead. But, glad it's working!

--

This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/3087fe48-30ff-46c7-9373-ecf025d61d03%40googlegroups.com.

[gran...@distilnetworks.com]

Yes, they were the exact same cert, though different files, Consul files were owned by a consul user and Vault files were owned by a vault user.

I am using the vault-cluster cookbook from the chef supermarket to handle the creation and that is how that was written. It has pretty poor support for this use case (Consul+Vault with TLS and self signed certs) so if this is a problem for anyone else or if someone needs help with their wrapper cookbook, I would be happy to help.

One upside to this mess is that I have learned a whole lot more about Vault and Consul as well as TLS in general as I debugged this. :)

[Jeffrey Kwan]

Hi:

I could really use some help on how to use the vault-cluster cookbook.  I'm running into a number of issues and have a few questions:

1. Which version of the hashicorp-vault cookbook are you using?  It looks like the vault-cluster cookbooks needs to rely on 2.1.0 to work.  The newer version of hashicorp-vault cookbook fails with the following error:

undefined method `bag_item' for VaultCookbook::Resource::VaultConfig

2. If I force the cookbook to rely on the 2.1.0 cookbook, I get farther, but now see the following error:

================================================================================
Error executing action `create` on resource 'ssl_certificate[consul]'
 ================================================================================

RuntimeError
 ------------
Cannot read SSL key from chef-vault: secrets.consul[private_key]

Any idea on what the problem may be?