embedded dns server

[patrick...@gmail.com]

So I wanted to play around with implementing the docker embedded DNS server idea in https://github.com/docker/docker/issues/17195, and was looking for some feedback.

I'm not sure if I'll submit my implementation as a PR though. So don't take this as a PR review :-).

The implementation is not complete, but the core is functional. When using the bridge driver, containers launched will be pointed at a DNS server which responds to queries for all the same records as in `/etc/hosts`, as well as forwarding unknown requests upstream.

https://github.com/phemmer/libnetwork/compare/master...dns

The change is entirely within libnetwork. No changes at all to docker aside from vendoring. I also haven't updated godeps (for github.com/miekg/dns) just to keep the diff clean.

The main area I'd like feedback in is not the DNS server itself, but in the integration with the rest of libnetwork.

In a nutshell, the way it works is that libnetwork starts up a DNS handler. This handler uses the sandbox paradigm, and maintains a set of DNS records for each sandbox (`libnetwork.sandbox`). The DNS handler is shared among all containers/sandboxes though, so when a query comes in, the handler looks up the DNS sandbox it needs to use by the source address. The same entry points into the libnetwork.sandbox that adjust /etc/hosts now adjust the DNS sandbox as well.

One of the core ideas I was trying to go with was that providing the server (the thing that actually listens for UDP/TCP traffic) would be left up to the libnetwork driver. Thus for things like host networking, we don't listen for traffic. This also prevents us from having to listen on `0.0.0.0`.

To implement this:

We first need to know whether a driver provides the DNS server, so I added an attribute to `libnetwork/DriverAPI.Capability`.

The driver then needs to send the queries into the shared DNS handler, so `libnetwork/DriverAPI.DriverCallback` got a function to expose the DNS handler.

However a problem is that when the sandbox's `resolv.conf` is first created, it's not joined to the driver network yet, so we don't know what DNS server it should be using. To solve this I added a call to rebuild the resolv.conf when it joins the network.

As mentioned, I'm mostly just interested in feedback on the integration, and if that seems like the right approach. The DNS handler itself still needs a work before I'd be ready for review on it.

The only other approach I can think of would be to have the driver maintain everything (DNS server, handler, sandboxing), and not use a shared handler. But at first thought that feels like it'd make things too complex.

Thanks

[Solomon Hykes]

Before looking into the specifics I just want to say that switching to DNS for standard service discovery is an excellent idea, I fully support it, and THANK YOU for contributing this!

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[santho...@docker.com]

Thanks for the proposal and prototype implementation. Apologies for the delay in getting back. 

After evaluating the options we also agree that embedded DNS based service discovery is the right approach to address the current limitations.

This has been under discussion for a while and we feel the following are some additional libnetwork design goals..

- The cut-over to the DNS based implementation should be as seamless as possible.

- libnetwork core already has all the endpoint and IP address information. It shouldn't be duplicated elsewhere to provide DNS based name resolution.

For these reasons we think the right design is to have the embedded DNS server completely within the core libnetwork without any driver side changes. 

This will also avoid repeating some of the DNS specific handling in every driver and provides the exact same user experience for service discovery as it exists today.

We will soon send out a detailed proposal with the state goals (and implementation aspects) for a wider discussion. We would love to have you participate in the

proposal and move it forward for the 1.10 release.

thanks,

Santhosh.

[Madhu Venugopal]

Including Patrick & Solomon.

> --
> You received this message because you are subscribed to the Google Groups
> "docker-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to docker-dev+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--

- Madhu
Sr. Director, Networking
Docker Inc.

[Patrick Hemmer]

> For these reasons we think the right design is to have the embedded DNS server completely within the core libnetwork without any driver side changes. 

The biggest issue with that is the DNS listener. For it to be driver-agnostic, the listener has to be global. If the docker host already has something listening on port 53, docker won't be able to start its own DNS server. Aside from the obvious of the host running a name server such as BIND, this is rather common with things like dnsmasq.

You could put the listener on a non-standard port, but then you have to redirect it somehow, and now you're back into the realm of a driver.

The only other option I can see is to use a driver to start up a DNS server it its own network namespace. Then the listener would get its own IP address which we can be sure won't conflict with the host. But now the DNS server can't be within the same process as the rest of docker, which makes things much more complicated.

Off the wall idea: It might be possible to start a process in the new namespace, open the socket listening for connections, and then pass the file descriptor over a unix domain socket back to the main docker process.

Yeah, this is getting a little ahead into the implementation, but implementation can affect what is even possible, and thus the goals.

-Patrick

[Patrick Hemmer]

> Off the wall idea: It might be possible to start a process in the new namespace, open the socket listening for connections, and then pass the file descriptor over a unix domain socket back to the main docker process.

I put together a test to see if this was possible, and it is. I was able to start a listener inside a docker container, send the file descriptor for the listener to a process outside the container via a unix domain socket (and exit after it was sent), have the host process listen for connections on the FD it received, and connect to the host process using the container's IP address (from outside the container).

I really do love linux, and the crazy shit it lets you do :-P

-Patrick

[Tim Hockin]

But why?  I have a hard time imagining this as anything other than insanity.

--

[Patrick Hemmer]

See the earlier emails. Listening on the host on port 53 is a bad idea as the port might already be in use. This leaves 2 options: Listen on a port other than 53 and implement a redirect (this requires driver support, at least as far as I can see), or listen on port 53 inside a container.

-Patrick

[Santhosh Manohar]

On 11/16/15 8:17 PM, Patrick Hemmer wrote:
> > Off the wall idea: It might be possible to start a process in the new
> namespace, open the socket listening for connections, and then pass the
> file descriptor over a unix domain socket back to the main docker process.
>
> I put together a test to see if this was possible, and it is. I was able
> to start a listener inside a docker container, send the file descriptor
> for the listener to a process outside the container via a unix domain
> socket (and exit after it was sent), have the host process listen for
> connections on the FD it received, and connect to the host process using
> the container's IP address (from outside the container).
>
> I really do love linux, and the crazy shit it lets you do :-P
>

I was thinking of starting a listener and move it to the container nw
namespace. So for the container the local resolver is always at
127.0.0.1 irrespective of the type and number of networks its connected
to. But this does have the problem you mentioned before; docker host
already listening on 53. We can see if there are other option; but what
you tried doesn't really look that crazy to me :)

thanks,
Santhosh.

> <https://github.com/docker/docker/issues/17195>, and was

> looking for some feedback.
> I'm not sure if I'll submit my implementation as a PR
> though. So don't take this as a PR review :-).
> The implementation is not complete, but the core is
> functional. When using the bridge driver, containers
> launched will be pointed at a DNS server which responds to
> queries for all the same records as in `/etc/hosts`, as well
> as forwarding unknown requests upstream.
>
> https://github.com/phemmer/libnetwork/compare/master...dns
> <https://github.com/phemmer/libnetwork/compare/master...dns>
> The change is entirely within libnetwork. No changes at all
> to docker aside from vendoring. I also haven't updated
> godeps (for github.com/miekg/dns

> <http://github.com/miekg/dns>) just to keep the diff clean.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "docker-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/docker-dev/WXkMiPJqh7I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> docker-dev+...@googlegroups.com
> <mailto:docker-dev+...@googlegroups.com>.

[Tim Hockin]

On Mon, Nov 16, 2015 at 9:00 PM, Patrick Hemmer
<patrick...@gmail.com> wrote:
> See the earlier emails. Listening on the host on port 53 is a bad idea as
> the port might already be in use. This leaves 2 options: Listen on a port
> other than 53 and implement a redirect (this requires driver support, at
> least as far as I can see), or listen on port 53 inside a container.

I ack that claiming 53 on every/any machine is likely to cause pain.
Running DNS on anything but 53 is pointless. But I don't understand
the machinations you are trying to go through - why pass an FD around?
Why not just run the DNS server in a container and configure other
containers to resolve from that.

This is EXACTLY what we do in Kubernetes. It works wonderfully and
has for a very long time (proportional to this whole space, of course
:)

[Patrick Hemmer]

>  Why not just run the DNS server in a container and configure other 
containers to resolve from that. 

This was also mentioned in an earlier email :-)

Personally, my main objection to that is now you have to synchronize the DNS records between the main docker daemon, and the containerized DNS server. Granted, this isn't too terrible, but I think it would be cleaner, and simpler, to keep it within the same process.

-Patrick

[Tim Hockin]

I am stunned that you think this ... Rube Goldberg thing is cleaner and simpler than just about anything.

It is pretty trivial to get skydns running with an arbitrary backend.  This is like a 1 week project, max.

[Patrick Hemmer]

All it's doing is passing a file descriptor between processes. This is nothing new or unusual. The only new or unusual part is that the file descriptor is coming from another namespace. Hardly rube goldberg.

SkyDNS won't work for this. SkyDNS is too generic of a server. You wouldn't be able to respond differently based on the container the request is coming from. You'd have to spin up a new SkyDNS server for every single container, or at least every permutation of DNS configuration (`--dns`, `--net`, `--add-host`, etc), and that'd be a huge waste of system resources (and likely overly complex).

-Patrick

[Dreamcat4]

The problem with putting your DNS server on a Docker server is that it no longer works when your Docker host is down.

Wheras the problem with a router or other dedicated network hardware to be running the local DNS is that it has to be manually configured.

In an ideal circumstance, there would be an accepted and established open protocol, that could be adopted. Whereby then Docker can automatically send DNS configuration instructions to local nominated DNS server. Whether that be a dedicated hardware (e.g. router) or simply some popular DNS server running inside another container.

This also requires some strong security / authentication mechanism. Maybe there is already such a protocol, but it's not very good? I don't know. And what about DNSSEC? Does that bring a configuration management protocol with it?

DNS is the only service my Docker host doesn't provide. And it becomes a pain to manually manage with many individual containers. I think this is a gap which hasn't been properly addressed yet. People say 'oh but how about this one or that one'. But they're all too flawed / fragile / unreliable / complicated to setup and/or cannot be used to configure independent dedicated networking hardware. And that's something which simply can't be accepted for such an essential service as DNS. I will be sticking with my dedicated hardware solution for the time being.

[Patrick Hemmer]

> The problem with putting your DNS server on a Docker server is that it no longer works when your Docker host is down.

I don't think docker crashing is every something that can be gracefully handled. However currently, I think as long as you don't need to change the state of the system, you can limp along. So that is a good point.

> In an ideal circumstance, there would be an accepted and established open protocol, that could be adopted. Whereby then Docker can automatically send DNS configuration instructions to local nominated DNS server. Whether that be a dedicated hardware (e.g. router) or simply some popular DNS server running inside another container.

There is, dynamic dns updates: http://docstore.mik.ua/orelly/networking_2ndEd/dns/ch10_02.htm, which fully supports authentication & access control.

But the problem with this is similar to the SkyDNS issue mentioned in the previous email. There's no great way to configure a generic DNS server so that it responds with one answer for one client, and a different answer for a different client, such as if you had 2 containers that were started with `--add-host=foo:1.2.3.4` and `--add-host=foo:4.3.2.1`.

You could give every single container its own DNS zone. And then your records would become `foo.containerA=1.2.3.4` and `foo.containerB=4.3.2.1`, but that seems like it might be a little heavy. Still, the ability to use existing protocols & support support off-the-shelf software is very attractive.

-Patrick

[Santhosh Manohar]

On 11/17/15 9:05 AM, Dreamcat4 wrote:
> The problem with putting your DNS server on a Docker server is that it
> no longer works when your Docker host is down.
>

The embedded DNS server does the name resolution for the containers only
on that host. In this model the DNS server doesn't need any config at
all. When the host (and containers) are up DNS will also be functional.

thanks,
Santhosh.

> <http://github.com/miekg/dns>) just to keep the diff clean.

> <mailto:docker-dev+...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "docker-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to docker-dev+...@googlegroups.com

> <mailto:docker-dev+...@googlegroups.com>.

[Dreamcat4]

On Tue, Nov 17, 2015 at 5:56 PM, Santhosh Manohar <santho...@docker.com> wrote:

On 11/17/15 9:05 AM, Dreamcat4 wrote:

The problem with putting your DNS server on a Docker server is that it
no longer works when your Docker host is down.

The embedded DNS server does the name resolution for the containers only on that host. In this model the DNS server doesn't need any config at all. When the host (and containers) are up DNS will also be functional.

Nope. That would be useless in a mixed environment.

Devices on my LAN rely upon docker servers. They have their DNS server & alt (2nd) DNS server populated by good old DHCP.

There can only be one DNS server provided by DHCP lookup, and it has to handle;

*) Docker containers

*) Other LAN devices configured by the local router.

*) WAN lookups.

So what you're saying would only cover the 1st point, and hence only be any use for docker container <--> other docker container DNS lookups (e.g. inter-dependant docker services).

That's simply not acceptable / good enough.

[Patrick Hemmer]

It sounds like you're arguing that docker should provide a general purpose DNS server you can inject non-docker related records into. If this is not what you're requesting, then my apologies. But if so, this is not a business docker should get into. There are far better utilities for acting as general purpose DNS servers.

If you want to set up a DNS server that can answer with a record from one of many backends servers, there is software that can do this, such as dnsmasq.

-Patrick

thanks,
Santhosh.

    <mailto:docker-dev+unsub...@googlegroups.com>.

    For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google
Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to docker-dev+...@googlegroups.com

<mailto:docker-dev+unsub...@googlegroups.com>.

[Santhosh Manohar]

Please see inline..

On 11/17/15 11:02 AM, Dreamcat4 wrote:
>
>
> On Tue, Nov 17, 2015 at 5:56 PM, Santhosh Manohar
> <santho...@docker.com <mailto:santho...@docker.com>> wrote:
>
> On 11/17/15 9:05 AM, Dreamcat4 wrote:
>
> The problem with putting your DNS server on a Docker server is
> that it
> no longer works when your Docker host is down.
>
>
> The embedded DNS server does the name resolution for the containers
> only on that host. In this model the DNS server doesn't need any
> config at all. When the host (and containers) are up DNS will also
> be functional.
>
>
> Nope. That would be useless in a mixed environment.
>
> Devices on my LAN rely upon docker servers. They have their DNS server &
> alt (2nd) DNS server populated by good old DHCP.
>
> There can only be one DNS server provided by DHCP lookup, and it has to
> handle;
>
> *) Docker containers
> *) Other LAN devices configured by the local router.
> *) WAN lookups.
>
> So what you're saying would only cover the 1st point, and hence only be
> any use for docker container <--> other docker container DNS lookups
> (e.g. inter-dependant docker services).
>

Yes, that is what its meant for. The embedded DNS server in docker is
for inter container name resolution only (ie: its simply replacing the
current /etc/hosts based approach for containers).

You have to continue to run your external DNS servers (populated by
DHCP) for external name resolutions. And you can also pass it to the
containers using --dns option.

thanks,
Santhosh.

> <mailto:patrick...@gmail.com

> <patrick...@gmail.com <mailto:patrick...@gmail.com>>

> wrote:
>
> > Why not just run the DNS server in a container and
> configure other
> containers to resolve from that.
>
> This was also mentioned in an earlier email :-)
> Personally, my main objection to that is now you
> have to
> synchronize the DNS records between the main docker
> daemon,
> and the containerized DNS server. Granted, this
> isn't too
> terrible, but I think it would be cleaner, and
> simpler, to
> keep it within the same process.
>
> -Patrick
>
> On Tuesday, November 17, 2015 at 2:31:51 AM UTC-5, Tim
> Hockin wrote:
>
> On Mon, Nov 16, 2015 at 9:00 PM, Patrick Hemmer
> <patrick...@gmail.com

> santho...@docker.com <mailto:santho...@docker.com>

> <mailto:docker-dev%2B...@googlegroups.com>.

> >>> For more options, visit
> https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are
> subscribed
> to the Google Groups
> > "docker-dev" group.
> > To unsubscribe from this group and stop
> receiving
> emails from it, send an
> > email to docker-dev+...@googlegroups.com

> <mailto:docker-dev%2B...@googlegroups.com>.

> > For more options, visit
> https://groups.google.com/d/optout.
>
> --
> You received this message because you are
> subscribed to the
> Google Groups "docker-dev" group.
> To unsubscribe from this group and stop receiving
> emails
> from it, send an email to
> docker-dev+...@googlegroups.com

> <mailto:docker-dev%2B...@googlegroups.com>.

> For more options, visit
> https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the
> Google
> Groups "docker-dev" group.
> To unsubscribe from this group and stop receiving emails
> from it,
> send an email to docker-dev+...@googlegroups.com

> <mailto:docker-dev%2Bunsu...@googlegroups.com>
> <mailto:docker-dev+...@googlegroups.com
> <mailto:docker-dev%2Bunsu...@googlegroups.com>>.

> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "docker-dev" group.
> To unsubscribe from this group and stop receiving emails from
> it, send
> an email to docker-dev+...@googlegroups.com

> <mailto:docker-dev%2Bunsu...@googlegroups.com>
> <mailto:docker-dev+...@googlegroups.com
> <mailto:docker-dev%2Bunsu...@googlegroups.com>>.

[Dreamcat4]

On Tue, Nov 17, 2015 at 7:20 PM, Patrick Hemmer <patrick...@gmail.com> wrote:

It sounds like you're arguing that docker should provide a general purpose DNS server you can inject non-docker related records into. If this is not what you're requesting, then my apologies. But if so, this is not a business docker should get into. There are far better utilities for acting as general purpose DNS servers.

If you want to set up a DNS server that can answer with a record from one of many backends servers, there is software that can do this, such as dnsmasq.

Well I already have DNSMasq on my Tomato router. So what you're saying is: I should be able to add something my dnsmasq config file to point to a container-based solution? Well that could work.

 

-Patrick

thanks,
Santhosh.

    <mailto:docker-dev+...@googlegroups.com>.

    For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google
Groups "docker-dev" group.
To unsubscribe from this group and stop receiving emails from it, send

an email to docker-dev+...@googlegroups.com
<mailto:docker-dev+...@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "docker-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to docker-dev+...@googlegroups.com.

[Tim Hockin]

On Tue, Nov 17, 2015 at 7:59 AM, Patrick Hemmer
<patrick...@gmail.com> wrote:
> All it's doing is passing a file descriptor between processes. This is
> nothing new or unusual. The only new or unusual part is that the file
> descriptor is coming from another namespace. Hardly rube goldberg.
>
> SkyDNS won't work for this. SkyDNS is too generic of a server. You wouldn't
> be able to respond differently based on the container the request is coming
> from. You'd have to spin up a new SkyDNS server for every single container,
> or at least every permutation of DNS configuration (`--dns`, `--net`,
> `--add-host`, etc), and that'd be a huge waste of system resources (and
> likely overly complex).

SkyDNS has some support for pluggable backends. If your backend is
aware of the client, then it CAN provide different responses. I have
not tried to implement this.

[Tim Hockin]

After much testing and trying, we have come to the realization that
(despite a real spec) DNS 'nameserver' entries should be assumed
fungible. Passing multiple --dns flags to docker where the different
instances resolve different names is going to cause trouble for a lot
of people. We tried it. Don't do it.

Instead, your docker DNS server needs to forward to your other "real"
DNS servers.

> email to docker-dev+...@googlegroups.com.

[Santhosh Manohar]

Hi Patrick,

While discussing this issue with Jana he made a clarification; its
sufficient to open the socket from inside the container's net namespace.
It can be done while populating other network resources in the sandbox.
Its pretty neat. socket will be bound to the container's namespace and
docker process will receive the pkts with that dst port from the container.

I tried this with test code; running multiple containers communicating
with docker on the same port while there is a listener on the host. This
works well.

thanks,
Santhosh.

> <https://github.com/docker/docker/issues/17195>, and was

> looking for some feedback.
> I'm not sure if I'll submit my implementation as a PR
> though. So don't take this as a PR review :-).
> The implementation is not complete, but the core is
> functional. When using the bridge driver, containers
> launched will be pointed at a DNS server which responds to
> queries for all the same records as in `/etc/hosts`, as well
> as forwarding unknown requests upstream.
>
> https://github.com/phemmer/libnetwork/compare/master...dns
> <https://github.com/phemmer/libnetwork/compare/master...dns>
> The change is entirely within libnetwork. No changes at all
> to docker aside from vendoring. I also haven't updated
> godeps (for github.com/miekg/dns

> <http://github.com/miekg/dns>) just to keep the diff clean.

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "docker-dev" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/docker-dev/WXkMiPJqh7I/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> docker-dev+...@googlegroups.com
> <mailto:docker-dev+...@googlegroups.com>.

[Tim Hockin]

Does this mean port 53 is not available for apps?

> You received this message because you are subscribed to the Google Groups
> "docker-dev" group.

> To unsubscribe from this group and stop receiving emails from it, send an
> email to docker-dev+...@googlegroups.com.

[Santhosh Manohar]

On 11/20/15 3:08 PM, Tim Hockin wrote:
> Does this mean port 53 is not available for apps?
>

Yes, apps can't bind to localhost:53

[Tim Hockin]

won't that break everyone who wants to run a DNS server in a container?

On Fri, Nov 20, 2015 at 4:05 PM, Santhosh Manohar

[Santhosh Manohar]

On 11/20/15 4:10 PM, Tim Hockin wrote:
> won't that break everyone who wants to run a DNS server in a container?
>

I don't think so. To run a DNS server in a container it has to be bound
to the physical interface IPs to be reachable from outside.

The embedded docker DNS server will bind only to localhost:53 and not *:53

[Tim Hockin]

Most software binds to 0.0.0.0 by default.  This will not work if localhost is busy.

Additionally people fixate on port mapping.  Native vxlan doesnt use it.  Kubernetes doesn't use it.

You have a namespacing problem here - who owns the netns?  If it is owned by the container then having any conflict - ANY - should not be acceptable.

You could do maybe something like expose a link local address that you trap with iptables or ipvs.  You just can't step on users toes.  I am sort of dismayed to have to point this out.