Error "first record does not look like a TLS handshake"

[Chris Hill]

I'm quite sure this is just a TLS configuration problem, but I don't know what is wrong or how to fix it.  I've setup Vault in HA (backed by Consul).  I've been following Seth Vargo's guide (yaml manifest here: https://github.com/sethvargo/vault-kubernetes-workshop/blob/master/k8s/vault.yaml) with some tweaks for our environment (AWS and consul storage).

Here is my configuration (via ENV):

            api_addr      = "https://vault.our.internal.domain.io:8200"
            cluster_addr  = "https://$(POD_IP):8201"
            ui = true
            log_level = "debug"
            seal "awskms" {
              region      = "us-west-2",
              kms_key_id  = "12345678-1234-1234-1234-123456789012"
            }
            storage "consul" {
              address     = "localhost:8500",
              path        = "vault/",
              disable_registration  = "true",
              ha_enabled  = "true"
            }
            listener "tcp" {
              address     = "127.0.0.1:8200"
              tls_disable = "true"
            }
            listener "tcp" {
              address       = "$(POD_IP):8200"
              tls_cert_file = "/etc/vault/tls/vault.pem"
              tls_key_file  = "/etc/vault/tls/vault-key.pem"
              tls_disable_client_certs = true
            }

And here is the entire Pod configuration:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vault
  labels:
    app: vault
spec:
  replicas: 3
  serviceName: vault
  selector:
    matchLabels:
      app: vault
  template:
    metadata:
      labels:
        app: vault
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values: ["vault"]
            topologyKey: kubernetes.io/hostname
      terminationGracePeriodSeconds: 10
      containers:
      - name: consul-vault-agent
        image: "consul:1.5.3"
        env:
          - name: GOSSIP_ENCRYPTION_KEY
            valueFrom:
              secretKeyRef:
                name: consul-gossip-encryption-key
                key: gossip-encryption-key
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
        args:
          - "agent"
          - "-encrypt=$(GOSSIP_ENCRYPTION_KEY)"
          - "-config-file=/etc/consul/config/config.json"
          - "-node=$(POD_NAME)"
        volumeMounts:
          - name: vault-consul-agent-config
            mountPath: /etc/consul/config
          - name: consul-tls
            mountPath: /etc/tls
      - name: vault
        image: vault:1.2.2
        args:
          - "server"
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: VAULT_ADDR
          value: "http://127.0.0.1:8200"
        - name: VAULT_LOCAL_CONFIG
          value: |
            api_addr      = "https://vault.our.internal.domain.io:8200"
            cluster_addr  = "https://$(POD_IP):8201"
            ui = true
            log_level = "debug"
            seal "awskms" {
              region      = "us-west-2",
              kms_key_id  = "12345678-1234-1234-1234-123456789012"
            }
            storage "consul" {
              address     = "localhost:8500",
              path        = "vault/",
              disable_registration  = "true",
              ha_enabled  = "true"
            }
            listener "tcp" {
              address     = "127.0.0.1:8200"
              tls_disable = "true"
            }
            listener "tcp" {
              address       = "$(POD_IP):8200"
              tls_cert_file = "/etc/vault/tls/vault.pem"
              tls_key_file  = "/etc/vault/tls/vault-key.pem"
              tls_disable_client_certs = true
            }
        securityContext:
          capabilities:
            add: ["IPC_LOCK"]
        ports:
        - containerPort: 8200
          name: http
          protocol: TCP
        - containerPort: 8201
          name: cluster
          protocol: TCP
        resources:
          requests:
            cpu: "500m"
            memory: "256Mi"
        readinessProbe:
          httpGet:
            path: /v1/sys/health?standbyok=true
            port: 8200
            scheme: HTTPS
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
          - name: vault-tls
            mountPath: /etc/vault/tls
      volumes:
        - name: vault-config
          configMap:
            name: vault-config
        - name: vault-consul-agent-config
          configMap:
            name: vault-consul-agent-config
        - name: vault-tls
          secret:
            secretName: vault-certs
        - name: consul-tls
          secret:
            secretName: consul-certs
  volumeClaimTemplates: []

Vault comes up and runs, but I think the servers fail to see each other.  They are all dumping errors like this to the logs:

2019-08-29T03:30:31.473Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:31.726Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:31.726Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:39.503Z [INFO]  http: TLS handshake error from 100.100.0.0:43486: tls: first record does not look like a TLS handshake
2019-08-29T03:30:39.503Z [INFO]  http: TLS handshake error from 100.113.128.0:61702: tls: first record does not look like a TLS handshake
2019-08-29T03:30:39.974Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:39.974Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:39.974Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:39.974Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:40.383Z [INFO]  http: TLS handshake error from 100.116.0.0:45864: tls: first record does not look like a TLS handshake
2019-08-29T03:30:47.309Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:47.309Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:48.472Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:48.472Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:48.472Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:48.472Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:48.726Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:48.726Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:49.503Z [INFO]  http: TLS handshake error from 100.104.0.0:23794: tls: first record does not look like a TLS handshake
2019-08-29T03:30:50.247Z [INFO]  http: TLS handshake error from 100.99.0.0:43514: tls: first record does not look like a TLS handshake
2019-08-29T03:30:50.383Z [INFO]  http: TLS handshake error from 100.114.0.0:18798: tls: first record does not look like a TLS handshake
2019-08-29T03:30:56.973Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:56.973Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"
2019-08-29T03:30:57.226Z [WARN]  core.cluster-listener: no TLS config found for ALPN: ALPN=[]
2019-08-29T03:30:57.226Z [DEBUG] core.cluster-listener: error handshaking cluster connection: error="unsupported protocol"

From what I can tell its a HTTP/HTTPS error, but I've set the cluster_addr to https. I can provide any more details if needed. Any help would be appreciated.