Macvlan and IP pinning with nomad

[johannes amorosa]

Dear Community,

we are trying to host a proprietary service on our on-premise nomad/consul cluster. These dockerized services need to be in the same network as the cluster hosts due to broadcasting and sending magic packets. We are using the docker macvlan bridge for doing this:

docker network create --driver macvlan --subnet=192.168.0.0/20 --gateway=192.168.1.254 -o host_iface=eno1 exposed_nw

and

...
config {
   network_mode = "exposed_nw"
}
...

Unfortunately with macvlan dhcp from "outside" is not yet supported. I can start several instances with nomad of our service but they all get the ip 192.168.0.1 (First IP to hand out). When I start the container by hand I can pin the ip with --ip x.x.x.x:

docker run --name='container3' --hostname='container3' --net=exposed_nw --ip=192.168.9.229 --detach=true ourimage:0.1

We could wrap the docker run in a script that fetches a "free"-IP and use a raw/exec to execute the container start, but this seems brutal - does someone on this list had similar problems to solve and can point me to an elegant solution how to pin IP's with nomad and maybe even register these in consul.

Thanks for your time.

Johannes

[johannes amorosa]

Talking to myself: There seems to be a development for a transparent dhcp proxy like macvlan IPAM driver feature that would

solve my problem.

https://gist.github.com/nerdalert/3d2b891d41e0fa8d688c

Until this has a official pull request I will ping the ip host to a different ip range. With the option --iprange. Not elegant I know.

https://docs.docker.com/engine/userguide/networking/get-started-macvlan/#macvlan-bridge-mode-example-usage

[Alex Dadgar]

Hmm,

Yeah it seems like DHCP is really what you need from docker in this instance.

Thanks,
Alex Dadgar

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/e2c9a7ab-b67c-48df-b90f-a2cfe7c04165%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[johannes amorosa]

Hi Alex,
I do get a weird behavior with my workaround. Nomad is not picking up the assigned IP number.
This is the output from one of my nodes. Im forcing the IP address into the "pseudo range" of one IP.
Network is created like this:



docker network create -d macvlan --subnet=192.168.1.0/20 --gateway=192.168.1.254  --ip-range=192.168.9.201/32 -o parent=eno1 pulse_nw




docker network inspect 5a118f030160
[
    {
        "Name": "pulse_nw",
        "Id": "5a118f030160cd63930a5d5430913e39cb818c08d6b0a770ac4bfafdbc310f52",
        "Created": "2017-05-03T11:19:53.126572684+02:00",
        "Scope": "local",
        "Driver": "macvlan",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "192.168.1.0/20",
                    "IPRange": "192.168.9.201/32",
                    "Gateway": "192.168.1.254"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "41032752750c9a59ed4cf3090be6f04730ca59c5d1ae416efa465ab715023168": {
                "Name": "pulse-b718b186-8f33-0457-7e91-339942c3b3e9",
                "EndpointID": "11546be65d202bb8047aa3a21dd3cbd303b157b67860782785c748f024f7dcef",
                "MacAddress": "02:42:c0:a8:09:c9",
                "IPv4Address": "192.168.9.201/20",
                "IPv6Address": ""
            }
        },
        "Options": {
            "parent": "eno1"
        },
        "Labels": {}
    }
]

But consul/nomad won't see the IP, so of course all service checks fail. Is this a bug?

cell-ls-24 nomad alloc-status b718b186
ID                  = b718b186
Eval ID             = cba2f495
Name                = mypulse.service[0]
Node ID             = 1c48fde8
Job ID              = mypulse
Client Status       = running
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created At          = 05/09/17 17:46:04 CEST


Task "pulse" is "running"
Task Resources
CPU          Memory           Disk     IOPS  Addresses
172/500 MHz  230 MiB/1.0 GiB  300 MiB  0     pulse_listening: 192.168.8.1:5046
                                             pulse_command: 192.168.8.1:5047
                                             alive: 192.168.8.1:8080

Port forwarding is of course useless in an vlan setup.

Johannes

[Alex Dadgar]

Hey Johannes,

Yeah currently Nomad will only register the IP it has assigned with Consul and does not inspect the container. I believe there is already an issue open for this. For now you will have to register your own checks if running non-host based networks.

Thanks,
Alex Dadgar

To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/dec7305b-9b0a-4977-850d-b6bdc0afd7c8%40googlegroups.com.

[johannes amorosa]

Hey Alex,

I created a service in consul by hand and now I have the monitoring I need.

Thanks

Johannes.