Sidecar networking configuration

[Michael Fischer]

Hi everyone,

We've discussed Docker sidecars a little bit (they can be provisioned as members of a task group), but we haven't really discussed the networking and addressability of them.

In particular I'd like to ensure that we can employ the Ambassador pattern.  In this pattern, a containerized application that needs to communicate with the outside world does so through proxy containers which share the same network stack as the primary's.  The proxy containers decouple service discovery, secret injection, sharding, etc. from the application.  The application just talks to "db" or "cache" or "api", which are set to 127.0.0.1 in the application container's /etc/hosts file.

With Docker, this means we need to configure a few custom runtime options when launching the tasks with "docker run":

• the application task needs "--add-host=PROXY_CONTAINER_NAME:127.0.0.1" arguments for each proxy container
  
• the proxy tasks need the "--net=container:DEPENDENT_CONTAINER_NAME" argument

To make this work properly, the application task must be started first; otherwise the container networking can't be properly set up.  So there's a dependency that needs to be established and enforced.

Are we going to be able to attain this with Nomad?  If not, which issues should I file for tracking?

Thanks,

--Michael

[Diptanu Choudhury]

Hi Michael,

We are working on the service discovery part of Nomad right now. And we are trying to see if we can do this as uniformly as we can across all our drivers so that a task running in a docker container can find out a task running inside a plain cgroup (using Nomad's exec or java driver).

Users would be able to label ports in a task and they would be able to define a service block in their task which would specify which ports they want to register as a service with a local Consul agent, and we could potentially expose all the service definitions in a task group to all tasks via environment variables or via a local metadata service. Basically, we think discovering tasks within a task group or across task groups could be achieved cleanly via a good service discovery solution.

Would that satisfy your needs?

With regards to tasks in a TaskGroup sharing the same network namespace we should be able to do that in the future since with the introduction of libnetwork it is easy to instruct the docker engine to use a specific network interface in a namespace.

--
This mailing list is governed under the HashiCorp Community Guidelines - https://www.hashicorp.com/community-guidelines.html. Behavior in violation of those guidelines may result in your removal from this mailing list.
 
GitHub Issues: https://github.com/hashicorp/nomad/issues
IRC: #nomad-tool on Freenode
---
You received this message because you are subscribed to the Google Groups "Nomad" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nomad-tool+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nomad-tool/CABHxtY4LAws3XKYDY5-tXYSJp_4rQYjTGZ-04Fc%3D2UiRKR0F-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--

Thanks,
Diptanu Choudhury

Web - www.linkedin.com/in/diptanu

Twitter - @diptanu

[Michael Fischer]

On Sat, Nov 14, 2015 at 5:13 AM, Diptanu Choudhury <dipt...@gmail.com> wrote:

Hi Michael,

We are working on the service discovery part of Nomad right now. And we are trying to see if we can do this as uniformly as we can across all our drivers so that a task running in a docker container can find out a task running inside a plain cgroup (using Nomad's exec or java driver).

Users would be able to label ports in a task and they would be able to define a service block in their task which would specify which ports they want to register as a service with a local Consul agent, and we could potentially expose all the service definitions in a task group to all tasks via environment variables or via a local metadata service. Basically, we think discovering tasks within a task group or across task groups could be achieved cleanly via a good service discovery solution.

Would that satisfy your needs?

Personally, I view task scheduling and service discovery as two separate things.  Both are needed in a modern cloud infrastructure, but I don't feel both duties should necessarily be solved by the same tool.  This is what Kubernetes is doing; and, while for a lot of users a solution where you get everything you need "out of the box" would work well, it imposes some limitations on folks like us who would prefer a more bespoke solution.

Service discovery is more or less a solved problem now, thanks to Consul.  (Tips hat.)  Task scheduling is a much more difficult problem (i.e. constraint satisfaction), and so far all the solutions I've seen don't actually focus on the problem as well as Nomad seems to; they seem distracted by the other pieces.  So I'd hate for Nomad's development to focus on the auxiliary issues like service discovery.

Also, keep in mind that this is not just about service discovery: we want smart proxies that do more than simply keep track of where services are: we want to do secret injection, sharding, etc.   We actually want to delegate service discovery to another component for this reason.

With regards to tasks in a TaskGroup sharing the same network namespace we should be able to do that in the future since with the introduction of libnetwork it is easy to instruct the docker engine to use a specific network interface in a namespace.

It's absolutely critical that we have that ASAP.

Thanks,

--Michael

 

[Diptanu Choudhury]

Hi Michael,

We are not trying to re-invent service discovery inside Nomad. Like you said, Consul does a good job with Service Discovery and we will make it easy for tasks launched by Nomad to register with Consul. To discover services running on Nomad, applications can talk to Consul using the standard Consul APIs. 

[Michael Fischer]

Thanks! https://github.com/hashicorp/nomad/issues/419

On Sat, Nov 14, 2015 at 7:37 PM, Diptanu Choudhury <dipt...@gmail.com> wrote:

We will absolutely support more networking options, sharing networking namespaces across multiple tasks in the same task group, etc in the future. 

We have not started discussing ordering or startup dependency of containers yet. Please create issues on GitHub, so that we can track your use cases and think through some ways of solving them.

On Sat, Nov 14, 2015 at 3:27 AM, Michael Fischer <mfis...@zendesk.com> wrote:

That's good to hear -- but I'm not sure you've answered my question. Is the intent that Nomad support the kind of container arrangement, ordering and networking options I discussed?  This is quite important to us. 

Thanks,

Michael