Advertising the "right" IP address for containers registered in Consul

[Jonathan Ballet]

Hi,

I'm using Nomad 0.5.0 for scheduling Docker containers.

My instances running Docker have Nomad running as a client, Consul
listening on localhost and a dummy interface configured as 169.254.1.1,
and well, the Docker Engine.

I'm trying to schedule a simple job like this:

job "nginx" {
region = "eu-central"
datacenters = ["lausanne"]
type = "service"

group "load-balancer" {
count = 1

task "nginx" {
driver = "docker"

config {
image = "nginx:1.11.5-alpine"
port_map { http = 80 }
}

resources {
cpu = 500 # 500 MHz
memory = 256 # 256MB
network { mbits = 10 port "http" {} }
}

service {
name = "test-nginx"
port = "http"
}
}
}
}

which goes fine until the service gets registered into Consul with the
following information:

$ http localhost:8500/v1/catalog/service/test-nginx
HTTP/1.1 200 OK
Content-Length: 315
Content-Type: application/json
Date: Thu, 17 Nov 2016 18:56:27 GMT
X-Consul-Index: 12631547
X-Consul-Knownleader: true
X-Consul-Lastcontact: 0

[
{
"Address": "10.40.0.45",
"CreateIndex": 12631378,
"ModifyIndex": 12631378,
"Node": "docker6.lan.my.company",
"ServiceAddress": "172.17.0.1",
"ServiceEnableTagOverride": false,
"ServiceID":
"_nomad-executor-3d5738a2-3275-ee61-e843-e5c382768b1b-nginx-test-nginx",
"ServiceName": "test-nginx",
"ServicePort": 55345,
"ServiceTags": []
}
]

The ServiceAddress field is not right as it is the IP address of the
docker0 interface running on the instance, and not the eth0 as I would
have expected.
So, for some reason, my eth0 is not the "first one" on these instances:

$ ip link | grep -e eth0: -e docker0:
2: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
state UP mode DEFAULT group default
753: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
pfifo_fast state UP mode DEFAULT group default qlen 1000

... and I guess Nomad takes the first one which is not "lo" and expose
this as the ServiceAddress in Consul.

I couldn't find the way to correctly configure this in the documentation
- Nomad client configuration settings on this particular host are like
this:

{
"advertise": {
"http": "10.40.0.45:4646",
"rpc": "10.40.0.45:4647",
"serf": "10.40.0.45:4648"
},
"bind_addr": "10.40.0.45",
"client": {
"enabled": true,
"options": {}
},
"consul": {
"address": "localhost:8500"
},
"data_dir": "/srv/nomad/data",
"datacenter": "lausanne",
"http_api_response_headers": {
"Access-Control-Allow-Method": "GET, POST, OPTIONS",
"Access-Control-Allow-Origin": "*",
"Access-Control-Expose-Headers": "x-nomad-index"
},
"leave_on_terminate": true,
"log_level": "debug",
"region": "eu-central",
"server": {
"enabled": false
},
"telemetry": {
"statsd_address": "statsd.lan.my.company:8125"
}
}

And when the job starts, the client logs are showing this:

[DEBUG] client: updated allocations at index 72574 (pulled 1)
(filtered 1)
[DEBUG] client: RPC failed to server 10.40.2.2:4647: rpc error:
unknown alloc id "3d5738a2-3275-ee61-e843-e5c382768b1b"
[DEBUG] client: allocs: (added 1) (removed 0) (updated 0) (ignore 1)
[DEBUG] client: starting task runners for alloc
'3d5738a2-3275-ee61-e843-e5c382768b1b'
[DEBUG] client: starting task context for 'nginx' (alloc
'3d5738a2-3275-ee61-e843-e5c382768b1b')
[DEBUG] client: updated allocations at index 72576 (pulled 0)
(filtered 2)
[DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 2)
[DEBUG] driver.docker: docker pull nginx:1.11.5-alpine succeeded
[DEBUG] driver.docker: identified image nginx:1.11.5-alpine as
sha256:2f3c6710d8f2091223a9e43d2dc306ab1d6064faf90a482041cbf900fe0fdf15
[DEBUG] plugin: starting plugin: /opt/nomad/0.5.0/nomad
[]string{"/opt/nomad/0.5.0/nomad", "executor",
"/srv/nomad/data/alloc/3d5738a2-3275-ee61-e843-e5c382768b1b/nginx/nginx-executor.out"}
[DEBUG] plugin: waiting for RPC address for: /opt/nomad/0.5.0/nomad
[DEBUG] plugin: nomad: 2016/11/17 19:53:57 [DEBUG] plugin: plugin
address: unix /tmp/plugin706652199
[DEBUG] driver.docker: Setting default logging options to syslog and
unix:///tmp/plugin881426266
[DEBUG] driver.docker: Using config for logging: {Type:syslog
ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin881426266]}
[DEBUG] driver.docker: using 268435456 bytes memory for nginx
[DEBUG] driver.docker: using 500 cpu shares for nginx
[DEBUG] driver.docker: binding directories []string{"..."}
[DEBUG] driver.docker: networking mode not specified; defaulting to
bridge
[DEBUG] driver.docker: allocated port 172.17.0.1:55345 -> 80
(mapped)
[DEBUG] driver.docker: exposed port 80
[DEBUG] driver.docker: setting container name to:
nginx-3d5738a2-3275-ee61-e843-e5c382768b1b
[INFO] driver.docker: created container
a96bdb1453982e2521b37cce5591fa3759fa8e8657cc824b751f69294f95dc1e
[INFO] driver.docker: started container
a96bdb1453982e2521b37cce5591fa3759fa8e8657cc824b751f69294f95dc1e
[WARN] client: error fetching stats of task nginx: stats collection
hasn't started yet
[DEBUG] client: updated allocations at index 72577 (pulled 0)
(filtered 2)
[DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 2)


Am I missing something? Thanks!

[Alex Dadgar]

Hey,

Can you try specifying the network interface you would like Nomad to use on the client by specifying `network_interface` in the client config: https://www.nomadproject.io/docs/agent/configuration/client.html#network_interface

Thanks,

Alex

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.

GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/ecff0d9a273de7d16fcf00bbe35a9a0b%40multani.info.
For more options, visit https://groups.google.com/d/optout.

[Jonathan Ballet]

Hey Alex,

I tried to set up this and it looks like it's working, thanks!

Can I suggest some documentation improvement? I stumbled upon this setting when I was trying to find how to specify the interface and thought it would do the job, but reading the description, it looks like it doesn't really do what I was looking for - although it apparently do the job! I can open a ticket for that if you'd like.

Thanks,

Jonathan

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/737b9668-d908-4f85-aa0b-124cc2edf115%40Alexs-MacBook.local.

[Alex Dadgar]

Hey Jonathan,

Feel free to open up a PR with the docs improvement :)

- Alex Dadgar

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/ecff0d9a273de7d16fcf00bbe35a9a0b%40multani.info.
For more options, visit https://groups.google.com/d/optout.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+unsubscribe@googlegroups.com.