Vault Deployment - TLS Handshake Error

[Srikanth Chandika]

Hi,

I have 3 node vault HA and 3 node consul cluster in test environment. Consul cluster is used as vault's back end storage.

all 3 vault nodes are behind load balancer vault.domain.com. 10.10.10.2 and 10.10.10.3 are the F5 ip address sending heartbeats to vault nodes from load balancer.

I am facing problems when I enabled TLS encryption in vault and consul.

Error:

vault[20101]: 2019-01-25T04:56:31.379-0600 [INFO]  http: TLS handshake error from 10.10.10.2:54109: tls: first record does not look like a TLS handshake

vault[20101]: 2019-01-25T04:56:31.415-0600 [INFO]  http: TLS handshake error from 10.10.10.3:1997: tls: first record does not look like a TLS handshake

vault[20101]: 2019-01-25T05:23:57.511-0600 [INFO]  http: TLS handshake error from 10.10.10.20:55069: tls: first record does not look like a TLS handshake

vault[20101]: 2019-01-25T05:24:01.377-0600 [INFO]  http: TLS handshake error from 10.10.10.3:35070: tls: first record does not look like a TLS handshake

vault[20101]: 2019-01-25T05:24:01.397-0600 [INFO]  http: TLS handshake error from 10.10.10.2:55177: tls: first record does not look like a TLS handshake

vault[20101]: 2019-01-25T05:24:01.887-0600 [INFO]  http: TLS handshake error from 10.10.10.20:55072: tls: first record does not look like a TLS handshake

10.10.10.2 and 10.10.10.3 are the F5 addresses.

10.10.10.20 is the IP when I tried to access one of the vault node directly from UI for testing.

My configurations are attached.

Kindly, help me what I did wrong in the configurations.

[Vasilev Vjacheslav]

Do you have the same gossip key across consul instances?

[eedwa...@gmail.com]

this looks vaguely familiar -- are you sure you're providing the client certificate when accessing vault? e.g. from the F5 heartbeats and your client machine (.20)

I use PKI / TLS with vault and consul, as well. Here's an example of querying a local consul agent's health with curl using client certificates:

curl \

 --cacert /opt/snapkitchen/pki/consul-client/ca/ca-chain.pem \

 --cert /opt/snapkitchen/pki/consul-client/client.pem \

 --key /opt/snapkitchen/pki/consul-client/client-key.pem \

 https://127.0.0.1:8501/v1/status/leader

[Srikanth Chandika]

yes I am using same gossip key in all 6 consul instances (3 agents in server mode, 3 agents in client mode)

[Srikanth Chandika]

To be honest I am not familiar with certificates usage. I only know the purpose of it. In terms of implementation I am novice.

Please help me if anything wrong in the configurations.

Below are my config's and cluster status commands:

I have total 6 instances: 3 consul server instances and 3 vault + consul client instances.

In all consul configuration, I am using ca.crt(Issuing CA1 + Root CA + Intermediate CA), individual host certificate and host key issued by CA.

Consul Servers:

hostname: consul_node01.domain.com
openssl x509 -noout -text -in consul_node01.crt | grep DNS
DNS:consul_node01, DNS:consul_node01.domain.com

hostname: consul_node02.domain.com
openssl x509 -noout -text -in consul_node02.crt | grep DNS
DNS:consul_node02, DNS:consul_node02.domain.com

hostname: consul_node01.domain.com
openssl x509 -noout -text -in consul_node03.crt | grep DNS
DNS:consul_node03, DNS:consul_node03.domain.com

Vault Servers:

hostname: vault_node01.domain.com
openssl x509 -noout -text -in vault_node01.crt | grep DNS
DNS:vault_node01, DNS:vault_node01.domain.com

hostname: vault_node02.domain.com
openssl x509 -noout -text -in vault_node02.crt | grep DNS
DNS:vault_node02, DNS:vault_node02.domain.com

hostname: vault_node03.domain.com
openssl x509 -noout -text -in vault_node03.crt | grep DNS
DNS:vault_node03, DNS:vault_node03.domain.com

Consul Server Instances config.json

consul_node01:

"ca_file": "/etc/consul/ssl/ca.crt",
"cert_file": "/etc/consul/ssl/consul_node01.crt",
"key_file": "/etc/consul/ssl/consul_node01.key",

consul_node02:

"ca_file": "/etc/consul/ssl/ca.crt",
"cert_file": "/etc/consul/ssl/consul_node02.crt",
"key_file": "/etc/consul/ssl/consul_node02.key",

consul_node03:

"ca_file": "/etc/consul/ssl/ca.crt",
"cert_file": "/etc/consul/ssl/consul_node03.crt",
"key_file": "/etc/consul/ssl/consul_node03.key",

Vault + Consul Client Instances configuration

vault_node01 consul_config.json

"ca_file": "/etc/consul/ssl/ca.crt",
"cert_file": "/etc/consul/ssl/vault_node01.crt",
"key_file": "/etc/consul/ssl/vault_node01.key",

vault_node02 consul_config.json

"ca_file": "/etc/consul/ssl/ca.crt",
"cert_file": "/etc/consul/ssl/vault_node02.crt",
"key_file": "/etc/consul/ssl/vault_node02.key",

vault_node03 consul_config.json

"ca_file": "/etc/consul/ssl/ca.crt",
"cert_file": "/etc/consul/ssl/vault_node03.crt",
"key_file": "/etc/consul/ssl/vault_node03.key",

vault_node01 vault.hcl

api_addr = "https://vault_node01:8200"
cluster_address = "https://vault_node01:8201"
ui = true
listener "tcp" {
  address = "vault_node01:8200"
  cluster_address = "vault_node01:8201"
  tls_disable = "False"
  tls_cert_file = "/etc/consul/ssl/vault_node01.crt"
  tls_key_file = "/etc/consul/ssl/vault_node01.key"
  tls_min_version = "tls12"
}
storage "consul" {
  path = "vault/"
  address = "vault_node01:8500"
  scheme = "https"
  datacenter = "DC1"
  tls_ca_file = "/etc/consul/ssl/ca.crt"
  tls_cert_file = "/etc/consul/ssl/vault_node01.crt"
  tls_key_file = "/etc/consul/ssl/vault_node01.key"
  service = "vault"
}

vault_node02 vault.hcl

api_addr = "https://vault_node02:8200"
cluster_address = "https://vault_node02:8201"
ui = true
listener "tcp" {
  address = "vault_node02:8200"
  cluster_address = "vault_node02:8201"
  tls_disable = "False"
  tls_cert_file = "/etc/consul/ssl/vault_node02.crt"
  tls_key_file = "/etc/consul/ssl/vault_node02.key"
  tls_min_version = "tls12"
}

storage "consul" {
  path = "vault/"
  address = "vault_node02:8500"
  scheme = "https"
  datacenter = "DC1"
  tls_ca_file = "/etc/consul/ssl/ca.crt"
  tls_cert_file = "/etc/consul/ssl/vault_node02.crt"
  tls_key_file = "/etc/consul/ssl/vault_node02.key"
  service = "vault"
}

vault_node03 vault.hcl

api_addr = "https://vault_node03:8200"
cluster_address = "https://vault_node03:8201"
ui = true

listener "tcp" {
  address = "vault_node03:8200"
  cluster_address = "vault_node03:8201"
  tls_disable = "False"
  tls_cert_file = "/etc/consul/ssl/vault_node03.crt"
  tls_key_file = "/etc/consul/ssl/vault_node03.key"
  tls_min_version = "tls12"
}
storage "consul" {
  path = "vault/"
  address = "vault_node03:8500"
  scheme = "https"
  datacenter = "DC1"
  tls_ca_file = "/etc/consul/ssl/ca.crt"
  tls_cert_file = "/etc/consul/ssl/vault_node03.crt"
  tls_key_file = "/etc/consul/ssl/vault_node03.key"
  service = "vault"
}

> consul members

Node              Address             Status  Type    Build  Protocol  DC  Segment
consul_node01     10.10.10.101:8301   alive   server  1.4.1  2         dc1  <all>
consul_node02     10.10.10.102:8301   alive   server  1.4.1  2         dc1  <all>
consul_node03     10.10.10.103:8301   alive   server  1.4.1  2         dc1  <all>
vault_node01      10.10.10.11:8301    alive   client  1.4.1  2         dc1  <default>
vault_node02      10.10.10.12:8301    alive   client  1.4.1  2         dc1  <default>
vault_node03      10.10.10.13:8301    alive   client  1.4.1  2         dc1  <default>

vault status

vault_node01

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.0.2
Cluster Name    vault.domain.com
Cluster ID      2b98cc8c-d208-9fba-cddb-ffc2ebca84cf
HA Enabled      true
HA Cluster      https://vault_node01:8201
HA Mode         active

vault_node02

Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           5
Threshold              3
Version                1.0.2
Cluster Name           vault.domain.com
Cluster ID             2b98cc8c-d208-9fba-cddb-ffc2ebca84cf
HA Enabled             true
HA Cluster             https://vault_node01:8201
HA Mode                standby
Active Node Address    https://vault_node01:8200

vault_node3

Key                    Value
---                    -----
Seal Type              shamir
Initialized            true
Sealed                 false
Total Shares           5
Threshold              3
Version                1.0.2
Cluster Name           vault.domain.com
Cluster ID             2b98cc8c-d208-9fba-cddb-ffc2ebca84cf
HA Enabled             true
HA Cluster             https://vault_node01:8201
HA Mode                standby
Active Node Address    https://vault_node01:8200