Unable to read secrets on an HA vault with TLS enabled

[Mrinal Mukherjee]

I was trying to setup an HA vault ecosystem which is TLS-enabled. However, I am using self-signed certs as of now. Once things start working, I intend to use proper CA signed certificates. So, please don't judge me now for the use of self-signed certs :)

While on the standby vault, when I try to read a secret which was written on the active vault, I get this error -

/ # vault read secret/hello
Error reading secret/hello: Get https://10.228.18.20/v1/secret/hello: http: server gave HTTP response to HTTPS client

Here are the details of my setup. Please note that the entire setup is done using Docker containers.

My setup involves -

- A cluster of 3 Consul servers.  Each consul server runs as a docker container. The servers talk/gossip using an encryption key. The servers are TLS-enabled with self-signed certs. A sample consul output from one of the servers looks like this -

==> Consul agent running!
           Version: 'v1.0.6'
           Node ID: '2c4b2160-7063-41af-fbba-xxxx'
         Node name: 'xxx4042xxx'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: x.x.18.42 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true

- 1 Active Vault. The server has 1 vault container and 1 consul-client container. The consul-client container is started with the following docker-run command -

docker run --net=host -v /opt/consul:/opt/consul -v /etc/pki:/etc/pki -v consul-data:/consul/data -e 'CONSUL_LOCAL_CONFIG={"leave_on_terminate": true, "encrypt":"xxxx","ca_file": "/opt/consul/ssl/demo-root.cer", "cert_file": "/opt/consul/ssl/server.cer", "key_file": "/opt/consul/ssl/server.key", "verify_outgoing":true, "verify_incoming":true, "verify_server_hostname": true, "ports":{"https":8501}}' consul:1.0.6 agent -bind=x.x.18.20 -retry-join=x.x.18.42

In this command, the value - 'x.x.18.20' corresponds to the IP of the local server where the client is running. The value - 'x.x.18.42' refers to one of the IPs of the Consul server cluster. This results in the consul-client starting as -

==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.6'
           Node ID: 'b9205cc7-209f-b224-f389-xxxx'
         Node name: 'xxxx4020xxx'
        Datacenter: 'dc1' (Segment: '')
            Server: false (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: 8501, DNS: 8600)
      Cluster Addr: x.x.18.20 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true

Now, on the same server hosting the consul-client, I am starting my (active) vault using the following docker-run command

docker run --net=host -v /opt/consul:/opt/consul -v /etc/pki:/etc/pki --cap-add IPC_LOCK -e 'VAULT_LOCAL_CONFIG={"backend": {"consul": {"tls_skip_verify":"true","address":"127.0.0.1:8501","advertise_addr":"https://x.x.18.20","path":"vault/","scheme":"https","tls_cert_file":"/opt/consul/ssl/server.cer","tls_key_file":"/opt/consul/ssl/server.key","tls_ca_file":"/opt/consul/ssl/demo-root.cer"}}, "default_lease_ttl": "168h", "max_lease_ttl": "720h", "listener": {"tcp":{"address":"0.0.0.0:8200", "tls_disable": "0","tls_cert_file":"/opt/consul/ssl/server.cer","tls_key_file":"/opt/consul/ssl/server.key"}}}' --hostname vault --name vault vault:0.9.5 server

With this, the vault starts up -

==> Vault server configuration:


                     Cgo: disabled
         Cluster Address: https://x.x.18.20:444
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "enabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
        Redirect Address: https://x.x.18.20
                 Storage: consul (HA available)
                 Version: Vault v0.9.5
             Version Sha: 36edb4d42380d89a897e7f633046423240xxxx


==> Vault server started! Log data will stream in below:

I used the docker-exec command to log into the vault container and set the following environment variables. I had to do the 'VAULT_SKIP_VERIFY' because I was using self-signed certificates.

/ # export VAULT_ADDR=https://127.0.0.1:8200
/ # export VAULT_SKIP_VERIFY=true

With this in place, I was able to initialise the vault, unseal it, write secrets and read from it. The vault status was as follows -

Key             Value
---             -----
Seal Type       shamir
Sealed          false
Total Shares    5
Threshold       3
Version         0.9.5
Cluster Name    vault-cluster-4994xxxx
Cluster ID      73034975-d34f-ea09-dd9f-8e88cdd94xxx
HA Enabled      true
HA Cluster      https://x.x.18.20:444
HA Mode         active

With this done, I started setting up the stand-by vault in a similar manner. This time on server - x.x.18.41. The corresponding consul status was -

==> Consul agent running!
           Version: 'v1.0.6'
           Node ID: 'b9205cc7-209f-b224-f389-6dfda11bf340'
         Node name: 'nzakdot4041szrw'
        Datacenter: 'dc1' (Segment: '')
            Server: false (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: 8501, DNS: 8600)
      Cluster Addr: x.x.18.41 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: true, TLS-Outgoing: true, TLS-Incoming: true


==> Log data will now stream in as it occurs:

The corresponding vault status was 

==> Vault server configuration:


                     Cgo: disabled
         Cluster Address: https://x.x.18.41:444
              Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", tls: "enabled")
               Log Level: info
                   Mlock: supported: true, enabled: true
        Redirect Address: https://x.x.18.41
                 Storage: consul (HA available)
                 Version: Vault v0.9.5
             Version Sha: 36edb4d42380d89a897e7f633046423240b710d9


==> Vault server started! Log data will stream in below:

As I did in the case of the active vault, I logged into the vault container and set the following environment variables

/ # export VAULT_ADDR=https://127.0.0.1:8200
/ # export VAULT_SKIP_VERIFY=true

Post this, I issued the unseal command and was able to 'unseal' the vault.

Now, when I execute a vault status command, I see this - which is what I expected

/ # vault status
Key                     Value
---                     -----
Seal Type               shamir
Sealed                  false
Total Shares            5
Threshold               3
Version                 0.9.5
Cluster Name            vault-cluster-49946axx
Cluster ID              73034975-d34f-ea09-dd9f-8e88cdd943xx
HA Enabled              true
HA Cluster              https://x.x.18.20:444
HA Mode                 standby
Active Node Address:    https://x.x.18.20

Now, while I am on this standby vault, when I try to read the secret which was written earlier on the active vault, I get this error

/ # vault read secret/hello
Error reading secret/hello: Get https://10.228.18.20/v1/secret/hello: http: server gave HTTP response to HTTPS client

Where am I going wrong?

Also, if there are configurations which I am missing or could be optimised, please let me know!

Cheers!

Mrinal M.

[Jeff Mitchell]

Hi Mrinal,

It seems from your output like everything is working fine with the standby when you exec commands on the standby Vault from within the container. To me this then suggests that https://10.228.18.20 is not actually pointing to your standby Vault server, but rather to some other service.

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/4baa499d-6675-475d-935f-f9dc05eaebca%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.