Vault CLI: x509: certificate signed by unknown authority

[Chris Hill]

I have followed multiple guides but I just can't seem to get Vault working correctly. This is quite aggravating. You may have seen I've posted 2 other issues, hopefully this one is easier to fix (and maybe the root of the problem).

I'm just trying to run a simple vault status command, but I keep getting:

$ vault status
Error checking seal status: Get https://vault.our.internal.domain.io/v1/sys/seal-status: x509: certificate signed by unknown authority

I've followed these guides:

https://testdriven.io/blog/running-vault-and-consul-on-kubernetes/

https://github.com/sethvargo/vault-kubernetes-workshop

https://github.com/kelseyhightower/vault-on-google-kubernetes-engine

That is, I've run exactly these commands (pulled from my history):

cfssl gencert -initca config/ca-csr.json | cfssljson -bare ca
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=config/ca-config.json -profile=default config/consul-csr.json | cfssljson -bare consul
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=config/ca-config.json -profile=default config/vault-csr.json | cfssljson -bare vault

I've also created a "combined" or "full chain"(???) file by running:

cat vault.pem ca.pem > vault-combined.pem

(This was in Kelsey Hightowers guide)

Additionally, I've used openssl to verify my vault certificates:

$ openssl verify -verbose -CAfile certs/ca.pem certs/vault.pem
certs/vault.pem: OK
$ openssl verify -verbose -CAfile certs/ca.pem certs/vault-combined.pem
certs/vault-combined.pem: OK

For my vault configuration, the listener looks like so:

      "listener": [{
        "tcp":{
          "address":"[::]:8200",
          "cluster_address":"[::]:8201",
          "tls_cert_file": "/etc/vault/tls/vault-combined.pem",
          "tls_key_file": "/etc/vault/tls/vault-key.pem",
        },
      }]

Note that for tls_cert_file I've tried both vault.pem and vault-combined.pem.  Both of them cause the "signed by unknown authority" error.

For my local environment, I've set:

export VAULT_ADDR="https://vault.our.internal.domain.io"
export VAULT_CACERT="/path/to/certs/ca.pem"
#export VAULT_CLIENT_CERT="/path/to/certs/vault.pem"
#export VAULT_CLIENT_KEY="/path/to/certs/vault-key.pem"
export VAULT_TOKEN=<redacted>

Note that I've tried commenting/uncommenting those VAULT_CLIENT_CERT/KEY values and both fail.

I'm at a loss here. I've followed multiple guides exactly, I've regenerated certificates, I've tried every combination of settings I can think of. Can anybody help point me in the right direction?

[Michel Vocks]

Hi Chris,

I'm really sorry to hear that you run into some issues setting up a Vault cluster inside Kubernetes. If you have any pointers for us to improve our documentation/learnings please let us know!

You already pointed out that the problem is the following line:

x509: certificate signed by unknown authority

This always indicates that the TLS handshake was not successful and in this case the client certificate verification failed.

From your information above, I noticed that your Vault instance is running inside a Kubernetes Cluster and you try to access the Vault API from your local computer.

Usually, if you want to access a service which is running inside of a Kubernetes cluster, you have to expose it via the Kubernetes ingress.

This Kubernetes ingress is mostly a L7 load balancer which means the ingress needs to decrypt every request (in case HTTPS is used), lookup and decide to which POD this request should be routed to and encrypt the request again.

In your case, I assume that the TLS handshake fails already at the Kubernetes ingress since the Vault CLI expects a different certificate which has been replied (the certificate from Vault and not from the ingress).

A possible workaround is to use skip the Vault client verification by setting "VAULT_SKIP_VERIFY" as an ENV variable in your local computer session. The better solution is to add the Kubernetes ingress certificate to your local os tls truststore.

Cheers,

Michel

[Chris Hill]

Michel, thanks again for your input.

Usually, if you want to access a service which is running inside of a Kubernetes cluster, you have to expose it via the Kubernetes ingress.

I thought it was a Vault configuration issue this whole time, but it was actually my service definition. In our case I was able to fix it by tweaking the Kubernetes service config. Here is the working service.yaml file:

apiVersion: v1
kind: Service
metadata:
  name: vault
  labels:
    app: vault
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: ssl
    external-dns.alpha.kubernetes.io/hostname: vault.our.internal.domain.io
spec:
  type: LoadBalancer
  ports:
    - name: api
      port: 443
      targetPort: api
    - name: cluster
      port: 8201
      targetPort: cluster
  selector:
    app: vault

I bolded and underlined the key line in there.  Before I was using "HTTPS" think that it meant "continue passing through HTTPS traffic" (i.e. don't handshake and terminate the TLS).  However, after reviewing the docs (https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/#load-balancers) "If http (default) or https, an HTTPS listener that terminates the connection and parses headers is created. If set to ssl or tcp, a “raw” SSL listener is used".  My assumption was wrong, and I changed it to "ssl".  Now I think the load-balancer is just passing through the TLS session to Vault and allowing it to terminate?  Everything is working now without certificate errors.

Thanks again Michel!