Can't Connect to Consul from itself, or from Vault. Loopback issues?

[Corey Fonseca]

Hello all, 

I'm running into a weird Consul connectivity issue that seems to be loopback-related. Here's all of the relevant info I could think of.

I'm running three Consul containers in a cluster. They will be used as a backend for Vault and Terraform. I also have two other servers, with each one running both a Vault container and a Consul container in client mode. All of the containers are using --net=host in Docker, and are on the same subnet with no firewall between them.

My Vault containers are configured to use Consul as a backend, and are configured to hit the local Consul agent on 127.0.0.1:8500 (because each Vault server also runs a Consul container bound to the same IP). The Consul agents are configured to bind to eth0.

All of my Consul agents are able to connect to each other and join a cluster, but my Vault servers can't communicate with Consul for some reason. Additionally, if I'm directly on a Consul node and try running "Consul Info", I get an error about not being able to communicate with Consul, even though I'm on the node itself.

If I'm on any of the servers with a Consul agent running, I can run "telnet <actual ip address> 8500" and get a successful connection, but if I run "telnet 127.0.0.1 8500", I get connection refused.

Here's an example of what I see from Vault (docker logs vault):

[WARN] storage.consul: reconcile unable to talk with Consul backend: error="service registration failed: Put http://127.0.0.1:8500/v1/agent/service/register: dial tcp 127.0.0.1:8500: connect: connection refused"
[WARN] storage.consul: check unable to talk with Consul backend: error="Put http://127.0.0.1:8500/v1/agent/check/fail/vault:vault01.domain.local:8200:vault-sealed-check?note=Vault+Sealed: dial tcp 127.0.0.1:8500: connect: connection refused"

Here's what I see from Consul (consul info):

Error querying agent: Get http://127.0.0.1:8500/v1/agent/self: dial tcp 127.0.0.1:8500: connect: connection refused

Here are the full "docker run" commands I'm using to fire up the containers:

docker container run -d --name=consul --net=host -v /etc/consul/data:/consul/data -v /etc/consul/config:/consul/config -e CONSUL_CLIENT_INTERFACE='eth0' -e CONSUL_BIND_INTERFACE='eth0' -p 8300:8300 -p 8301:8301 -p 8302:8302 -p 8500:8500 -p 8501:8501 -p 8600:8600 consul agent -ui -node {{ inventory_hostname }}

docker container run -d --name=vault --cap-add=IPC_LOCK --net=host -p 8200:8200 -p 8201:8201 -v /etc/vault/data:/vault/data -v /etc/vault/config:/vault/config -v /etc/vault/logs:/vault/logs -e VAULT_API_ADDR='https://{{ inventory_hostname }}:8200' -e VAULT_CLUSTER_ADDR='https://{{ inventory_hostname }}:8201' vault server

Here's my Vault config:

storage "consul" {

    address = "127.0.0.1:8500"

    path = "vault/"

    scheme = "http"

    service = "vault"

    tls_ca_file = "/vault/config/consul-ca.pem"

    tls_cert_file = "/vault/config/client.pem"

    tls_key_file = "/vault/config/client-key.pem"

    tls_min_version = "tls12"

}

listener "tcp" {

    address = "0.0.0.0:8200"

    cluster_address = "0.0.0.0:8201"

    tls_cert_file = "/vault/config/cert.crt"

    tls_key_file = "/vault/config/key.key"

    tls_min_version = "tls12"

}

Here's my Consul Client config. My server config is similar, except for server being set to true, and a bootstrap_expect of 3.

{

    "datacenter": "companyname"

    "data_dir": "/consul/data",

    "encrypt": "jfieowjfieowjfieow",

    "retry_join": [

        "10.100.204.9:8301",

        "10.100.204.10:8301",

        "10.100.204.11:8301"

    ],

    "server": false,

    "key_file": "/consul/config/client-key.pem",

    "cert_file": "/consul/config/client.pem",

    "ca_file": "/consul/config/consul-ca.pem",

    "verify_incoming": true,

    "verify_incoming_rpc": true,

    "verify_outgoing": true

}

I'm running this on Ubuntu 18 and the latest version of Docker, using the official Vault and Consul containers. Anyone have any ideas on why I can't connect? I'm stuck right now so I'd really appreciate any pointers. Thanks in advance!

[Justin DynamicD]

So immediately I see you're running both systems in a container, PATing addresses (-p), and in Vault you're trying to connect to 127.0.0.1.  This will never work, as in a PAT scenario, each container effectively has it's own 127.0.0.1 lookup that only functions from within the container.

As I'm assuming you're testing thigns out,  try using --network host to tell the containers to actually use hte host network.  You will not longer need to -p ports because it will bind right to the host IP.  This should allow 127.0.0.1 to work for your testing.  Otheriwse, you need your containers in the same network.


In order for both containers to 

[Corey Fonseca]

Hi Justin,

Thank you for the reply! I'm using --network=host already. Based on your response, it sounds like I can't or shouldn't use the PAT while I'm using the host network, is that correct? 

I didn't realize that the host network also gave the container access to the host ports, but I suppose that does make sense! I'll give this a try shortly. 

Thank you for your help!

Corey

Sent from my iPhone

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/consul/issues
Community chat: https://gitter.im/hashicorp-consul/Lobby
---
You received this message because you are subscribed to the Google Groups "Consul" group.
To unsubscribe from this group and stop receiving emails from it, send an email to consul-tool...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/consul-tool/78ea6745-b27a-42d8-b39e-e76a8d4ce64d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Justin DynamicD]

I *think* (I forget now) that if the network is in host mode the -p is simply ignored.  Easiest test though is to start your container then from your host simply type:

netstat -plnt

this should list all the bound tcp listeners, including Consul.  If that doesn't happen then yeah ... we have a problem :)  

[Pankaj Bhagwani]

hey, guys if you are running consul cluster(bootstrap) in containers. to get the information please use this command "consul catalog services   -http-addr=172.16.0.1:8500" 
 
use this "-http-addr=172.16.0.1:8500" with consul command.