nomad's health checks in Consul are failing due to incorrect IP address

[Zdenek Styblik]

Hello,

I'm back with rather non-standard testing setup and I'm having issues with nomad health checks, but not the service health checks.

Setup is as follows:

1. nomad agent v0.7.1 is running at the Host itself. Configuration is:

```
log_level = "INFO"
bind_addr = "0.0.0.0" # the default

data_dir  = "/some/persistent/state/dir"

advertise {
  # Defaults to the node's hostname. If the hostname resolves to a loopback
  # address you must manually configure advertise addresses.
  http = "172.17.0.1:4646"
  rpc  = "172.17.0.1:4647"
  serf = "172.17.0.1:4648"
}

server {
  enabled          = true
  bootstrap_expect = 1
}

client {
  enabled       = true
  network_speed = 1000
  options {
    "driver.docker.enable" = "1"
    "docker.cleanup.image" = "false"
  }
}

consul {
  address = "172.17.0.1:8500"
  auto_advertise = true
  ssl = false
}

```

2. Consul is running in docker on the Host like so:

```
docker run -d --name=dev-consul -e CONSUL_BIND_INTERFACE=eth0 -p 8500:8500 -p 8600:8600 -p 8300:8300 -p 8301:8301 consul
```

Now, nomad agent health checks "nomad" and "nomad-client" are failing because of address '0.0.0.0'. Here is some example:

```
Get http://0.0.0.0:4646/v1/agent/health?type=client: dial tcp 0.0.0.0:4646: getsockopt: connection refused

dial tcp 0.0.0.0:4647: getsockopt: connection refused
```

I'm a bit confused why is it so. Shouldn't these checks be conducted against advertised IP address rather than bind IP address? Of course, when consul is running with `--net=host` everything works, but that makes sense. Also, service health checks do work as expected, although these are conducted against the IP address of docker container.

If it does sound like a bug, I'll open issue at GitHub. I just wanted to check first since it's rather non-standard setup and it's likely I'm doing it wrong :) Perhaps I shouldn't forward those consul ports. Perhaps I should point nomad agent to the IP address assigned to Consul Docker container? I mean, could it be that nomad agent is confused by the fact that IP address of Consul is the same as advertised IP addresses, therefore "yeah, those are the same, let's use bind address"?

Thank you.

Best regards,
Zdenek (zstyblik@github)

[Zdenek Styblik]

On Tuesday, January 23, 2018 at 10:20:54 AM UTC+1, Zdenek Styblik wrote:

Perhaps I should point nomad agent to the IP address assigned to Consul Docker container? I mean, could it be that nomad agent is confused by the fact that IP address of Consul is the same as advertised IP addresses, therefore "yeah, those are the same, let's use bind address"?

Nope, this doesn't seem to be it and nomad agent's health checks are still registered with '0.0.0.0' even if Consul's IP address is different from advertised IP addresses.

Zdenek

[Zdenek Styblik]

Please, dear me, just RTFM next time.

My apologies for the noise.

Zdenek

On Tuesday, January 23, 2018 at 10:20:54 AM UTC+1, Zdenek Styblik wrote: