HELP: x509: certificate signed by unknown authority when vault try to talk with consul client

[Alex Wang]

PROBLEM:
I have stuck to this problem for some time. The log showed:
vault.server1 | [WARN ] storage.consul: reconcile unable to talk with Consul backend: error="service registration failed: Put https://consul-c1:8080/v1/agent/service/register: x509: certificate signed by unknown authority"
all the time.
If I use http scheme connecting to 8500, it just works fine. But if I use https scheme connecting to 8500, it reports an error of connect http to https.

What I've done:
I am trying to build an HA vault cluster with consul as storage backend according to this instruction: https://www.vaultproject.io/guides/operations/vault-ha-consul.html, and this one: https://www.hashicorp.com/resources/hashicorp-vault-administrative-guide

Below is my docker compose yml file:

version: '3.7'

services:
  consul_s1:
    container_name: consul.server1
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server1:/consul/config
      - ./consul/data/data_s1:/consul/data
    restart: unless-stopped
    ports:
      - "8090:8080"
      - "8300:8300"
      - "8500:8500"
      - "8600:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.53

  consul_s2:
    container_name: consul.server2
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server2:/consul/config
      - ./consul/data/data_s2:/consul/data
    restart: unless-stopped
    ports:
      - "8091:8080"
      - "8310:8300"
      - "8510:8500"
      - "8610:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.54

  consul_s3:
    container_name: consul.server3
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/server_agent.json"]
    volumes:
      - ./consul/config/server3:/consul/config
      - ./consul/data/data_s3:/consul/data
    restart: unless-stopped
    ports:
      - "8092:8080"
      - "8320:8300"
      - "8520:8500"
      - "8620:8600/udp"
    networks:
      internal_net:
        ipv4_address: 172.16.238.55

  consul_c1:
    container_name: consul.client1
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/client_agent.json"]
    volumes:
      - ./consul/config/client1:/consul/config
      - ./consul/data/data_c1:/consul/data
    restart: unless-stopped
    ports:
      - "8093:8080"
      - "8330:8300"
      - "8530:8500"
      - "8630:8600/udp"
    depends_on:
      - consul_s1
    networks:
      internal_net:
        ipv4_address: 172.16.238.201
        aliases:
          - consul-c1

  consul_c2:
    container_name: consul.client2
    image: consul:latest
    command: ["agent", "-config-file=/consul/config/client_agent.json"]
    volumes:
      - ./consul/config/client2:/consul/config
      - ./consul/data/data_c2:/consul/data
    restart: unless-stopped
    ports:
      - "8094:8080"
      - "8340:8300"
      - "8540:8500"
      - "8640:8600/udp"
    depends_on:
      - consul_s2
    networks:
      internal_net:
        ipv4_address: 172.16.238.202
        aliases:
          - consul-c2

  vault_s1:
    container_name: vault.server1
    image: vault:latest
    ports:
      - "9200:8200"
    expose:
      - "8500"
    volumes:
      - ./vault/config/server1:/vault/config
      - ./vault/logs_s1:/vault/logs
    cap_add:
      - IPC_LOCK
    command: ["server", "-log-level=info"]
    environment:
      - VAULT_LOCAL_CONFIG={"backend":{"consul":{"address":"consul-c1:8080", "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/tls/certs/root.pem", "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_skip_verify":0}}, "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201", "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}}, "api_addr":"https://s1.vault:8200", "cluster_addr":"https://s1.vault:8201"}
    depends_on:
      - consul_c1
    networks:
      internal_net:
        ipv4_address: 172.16.238.151
        aliases:
          - s1.vault

  vault_s2:
    container_name: vault.server2
    image: vault:latest
    ports:
      - "9210:9200"
    expose:
      - "8500"
    volumes:
      - ./vault/config/server2:/vault/config
      - ./vault/logs_s2:/vault/logs
    cap_add:
      - IPC_LOCK
    command: ["server", "-log-level=info"]
    environment:
      - VAULT_LOCAL_CONFIG={"backend":{"consul":{"address":"consul-c2:8080", "path":"vault", "scheme":"https", "tls_ca_file":"/vault/config/tls/certs/root.pem", "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_skip_verify":0}}, "listener":{"tcp":{"address":"0.0.0.0:8200", "cluster_address":"0.0.0.0:8201", "tls_disable":0, "tls_cert_file":"/vault/config/tls/certs/server.pem", "tls_key_file":"/vault/config/tls/private/server-key.pem", "tls_min_version":"tls12"}}, "api_addr":"https://s2.vault:8200", "cluster_addr":"https://s2.vault:8201"}
    depends_on:
      - consul_c2
    networks:
      internal_net:
        ipv4_address: 172.16.238.152
        aliases:
          - s2.vault

networks:
  internal_net:
    driver: bridge
    ipam:
      driver: default
      config:
        - subnet: 172.16.238.0/24

and below is one of the consul client config json:

{
  "server": false,
  "node_name": "consul-c1",
  "datacenter": "dc1",
  "data_dir": "/consul/data",
  "bind_addr": "172.16.238.201",
  "client_addr": "172.16.238.201",
  "retry_join": ["172.16.238.53", "172.16.238.54", "172.16.238.55"],
  "log_level": "INFO",
  "enable_syslog": false,
  "acl_enforce_version_8": false,
  "ports":{
    "https": 8080
  },
  "ca_file": "/consul/config/certs/root.pem",
  "key_file": "/consul/config/private/client-key.pem",
  "cert_file": "/consul/config/certs/client.pem",
  "verify_incoming":true,
  "verify_outgoing":true
}

The connections between consul servers and clients just work fine.

I will be very very appreciated if anyone could point out what's wrong with my configuration.
Thanks in advance.

[Nathan Basanese]

  // , It will be very very appreciated if a question about x509 troubleshooting included at least _some_ content about the certificates themselves, even if it has to be redacted to remove sensitive data. 

[Alex Wang]

Sorry. The certs are generated by cfssl tools.

First I generated a root.key and root.cert.

Second, I used this root.key and cert generating keys and certs for the three consul servers and two clients respectively.

The host names designated to each are: server1.consul, server2.consul, server3.consul, consul-c1, consul-c2 with localhost and 127.0.0.1 appended.

Third, I used the same root.key and cert generating keys and certs for the two vault servers with host names: server1.vault, server2.vault and localhost, 127.0.0.1 appended.

Please let me know if any piece of critical helpful information missed. 

[Alex Wang]

Could anyone kindly help me for this?

Nearly a month passed. T_T

[Jeff Mitchell]

Hi Alex,

See https://www.vaultproject.io/docs/configuration/storage/consul.html#tls_ca_file

Best,

Jeff

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/vault/issues
IRC: #vault-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Vault" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vault-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/vault-tool/96b135f4-8024-47d1-bc46-9d89c51ee8c3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.