Resolv.conf override for a pod

[Simone D'Andreta]

Hello,

I wanted to know if there is a way to override DNS settings for pods (not per cluster). I tried to use HostAliases but it only creates an A record for that entry. I basically need a NS record cause I need to point to different Consul clusters and then Consul must be able to do service discovery. So I was thinking to change the resolv.conf for the pod to use Consul for specific requests. Any idea? 

Thank you,

Simone

[Tim Hockin]

There's no supported way to do that, in part because it would give up
all of the Service names that kubernetes provides. I don't know what
would happen if you tried to volumeMount a file over /etc/resolv.conf
- might be worth a shot.

> --
> You received this message because you are subscribed to the Google Groups
> "Kubernetes user discussion and Q&A" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kubernetes-use...@googlegroups.com.
> To post to this group, send email to kubernet...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kubernetes-users.
> For more options, visit https://groups.google.com/d/optout.

[Simone D'Andreta]

Hi Tim,

thanks for your answer. I don't actually need to override all the settings in the resolv.conf, I just need to add another nameserver at the top of the file. How about if I run a command in the pod such as:

command: ['/bin/sh', '-c', 'echo 'nameserver x.y.z.w' | cat - /etc/resolv.conf > temp && mv temp /etc/resolv.conf']

would it work?

Thanks

[Simone D'Andreta]

Bad news, my idea doesn't work. Could you explain me more about the volumeMount? I know how to mount but I don't know how I can effectively add a nameserver on that file.

Thanks

Simone

[Tim Hockin]

You'd have to craft a new file and mount it onto your resolv.conf,
which makes it harder to "just add another line" because you don't
have the base.

But more than that, what you're asking for is really non-standard
behavior. You can't safely add a nameserver record to resolv.conf
that produces different results. The behavior of DNS resolvers varies
widely, and this will cause you pain eventually (kubernetes used to do
this, it was bad).

Consider this - some resolvers ask all DNS servers in parallel, and
take the first response. If one resolver can answer a query and
another can't (NXDOMAIN), your app will sometimes get an address and
will sometimes fail. This actually happens.


On Thu, Sep 21, 2017 at 6:33 AM, Simone D'Andreta

[Simone D'Andreta]

I am trying to mount that volume but my container won't start. I guess I am doing something wrong. This is the yaml

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

  name: {{ template "fullname" . }}

  namespace: code

  labels:

    chart: "{{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}"

  annotations:

    commitSHA: {{ .Chart.AppVersion }}

    isNotifiable : "true"

spec:

  replicas: {{ .Values.replicaCount }}

  template:

    metadata:

      labels:

        app: {{ template "fullname" . }}

  

    spec:

  

      containers:

      - name: {{ .Chart.Name }}

        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"

        imagePullPolicy: {{ .Values.image.pullPolicy }}

        volumeMounts:

        - name: new-resolv

          mountPath: /etc/resolv.conf

        command: ["/bin/sh"]

        args: ["-c", "echo nameserver 10.24.26.102 > /etc/resolv.conf"]

      

        ports:

          - name: frontend

            containerPort: {{ .Values.port}}

        readinessProbe:

          httpGet:

            path: {{ .Values.lifecheck }}

            port: {{ .Values.port}}

          

      volumes:

      - name: new-resolv

        emptyDir: {}

I am using helm to deploy so the variables get expanded via Values.yaml or other template files. 

I think I am just not able to mount that volume properly..

Thanks

[Tim Hockin]

you're trying to mount a directory (emptyDir) onto a file
(/etc/resolv.conf). Without seeing the error that is a wild guess. I
can't stop you from doing this, but I strongly encourage you to
re-read and internalize what I wrote about multiple nameserver
records.

On Fri, Sep 22, 2017 at 6:19 AM, Simone D'Andreta

[Simone D'Andreta]

Hi Tim,

I know what you mean and that's definitely a big issue on our side. This is for us more a proof of concept to understand if we can go this way. Since I am not a big expert with Kubernetes, I wanted to know if there are solutions I haven't considered that might solve my problem.

That said, I thank you a lot for your explanations here. I  found something that can help me, though it sets dns names at node level - not per pod:

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/

If I include the different IPs in the stub domains I definitely should get an answer for the service discovery part. Not ideal, but I think better than mounting and overriding the resolv.conf.

Cheers

Simone

[Rodrigo Campos]

Can you explain what do you what to achieve?

Maybe changing the configmap kube-dns uses it's enough (and is prepared to be changed easily, now).

To unsubscribe from this group and stop receiving emails from it, send an email to kubernetes-users+unsubscribe@googlegroups.com.
To post to this group, send email to kubernetes-users@googlegroups.com.

[Simone D'Andreta]

I need to be able to overwrite the resolv.conf per pods. If I tweak the kube-dns configmap, I will have additional nameservers per node, not per pods.

The scenario is: I have a  kubernetes cluster which is shared between different teams, and each team has his own namespace to deploy pods. Anyway these pods must be able to communicate with different consul clusters which are in different environments. Since they are in different envs, we need to point to different DNS. Since pods get dns resolutions from the nodes, but the nodes are shared, we need a way to overwrite the dns settings per pods. 

I know this is a bad practice - I get what Tim said above in this thread - but this is just a proof of concept and we'd like to know whether there is a way to do this (the cleaner the better :D)

Thanks

Simone

[Tim Hockin]

There isn't a great answer to this.  It is something that has been on my mind we multi-tenant installations, but it is unsolved for now.

[Rodrigo Campos]

Sorry, I must be missing something. But if you want to resolve to different consul clusters in different kubernetes namespaces, can't you just use a service type external on each?

You create a service, named as you want them to contact, and is per ns and returns a CNAME.

Wouldn't that do the trick?

[Simone D'Andreta]

Hi Rodrigo,

ideally we would need this per pod, but I can give it a try with creating a service per namespace.

Thanks for the hint, I will let you know how it goes.

Simone

[Simone D'Andreta]

I am afraid it won't work. I will be able to solve consul.service.domain but I won't be able to solve external services registered within consul, such as mydatabase.service.domain because I need a NS record.. unless I can use that CNAME as a NS record, but I don't think it's the right approach..

[Rodrigo Campos]

Sorry, never used consul and I don't follow what you said.

Does it create records like <service name>.k8s-service that won't work with just a k8s external type service?

Then it might be possible to say to kube dns to use some upstream to some domains, probably? I've not played with it, as I don't need it. But I'd guess something like that might be possible to do. And use different domains on apps that need to query different consul clusters?

[Simone D'Andreta]

It creates records such as myservice.service.domain.io, so your application must be able to contact a dns forwarder which has the zone for that domain.io. 

What I am using at the moment are stubdomains to include the domain.io but it's at cluster level. What I need to do is to solve different consul names per namespace or pod.

I tried to use a service definition as follows:

kind: Service

apiVersion: v1

metadata:

  name: consul-resolution

  namespace: default

spec:

  type: ExternalName

  externalName: consul.service.domain.io

And an endpoint:

kind: Endpoints

apiVersion: v1

metadata:

  name: consulresolution

subsets:

  - addresses:

      - ip: 10.24.7.26

    ports:

      - port: 8300

But I am not even able to ping consul.service.cnqr.io.. any idea? 

Thanks

[Rodrigo Campos]

What I tried to say is using this: http://blog.kubernetes.io/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes.html?m=1

in kube-dns configuration. Not sure how your consul name is and, with all you said in the previous mail, a service type external will help.

As not even able to ping, not sure what you mean. The service has nothing to do with the domain you want to ping, right? Or is that a typo?

[Simone D'Andreta]

Ah yeah that's a typo.. it's setup as consul.service.domain.io but I don't know why I cannot ping it - are the service and the endpoint properly declared? Forgot to mention that I am using minikube to test it (k8s version 1.7.5)

As per the link you provided - I used the stubDomain and gave the domain.io name a list of Consul IPs - all good, but this will overwrite the kube-dns configmap in the kube-system namespace so the new dns resolution will be per node. I need that per namespace or pod.

Is there anyway I can do that?

Hope it's clear and thanks a lot for your help.

Simone

[Rodrigo Campos]

Can you ping the IP? There might not be any route to that subnet? Have you tried/checked that?

And as far as I know, there is no way. Im not 100% sure what problem you are trying to solve, exactly, though. You can have a service type external, configure kube dns stub domains, but having **the same stub domains** resolve to different NS servers according to the kubernetes namespace the query is executed doesn't seem supported. Not sure it's something useful besides your setup, either :-/

You can probably patch kubedns or something, but not sure it's worth the effort.

On Wednesday, September 27, 2017, Simone D'Andreta <simone....@gmail.com> wrote:

Ah yeah that's a typo.. it's setup as consul.service.domain.io but I don't know why I cannot ping it - are the service and the endpoint properly declared?

[Simone D'Andreta]

I can't try today unfortunately but that's definitely the issue, I will have to test it to an environment where I can reach the consul subnets.

As per what I am trying to achieve (and that's perfectly fine if we cannot find a viable solution, it is a proof of concept after all) I need a way to have different nameservers per namespace or pod. It was suggested before that I could do it via volumeMounts, by creating a new resolv.conf and copying it over the resolv.conf of the pod. I tried but I couldn't get it working, but I believe I am doing something wrong when mounting.. do you have any example I can use and adapt to my needs?

Thanks

Simone

[Rodrigo Campos]

Not sure that will work, try step by step.

Start creating the file in some dir in /etc/ and then try to just have the file in /etc/ (probably with subpath or something like that), and then ten try to use it mount it as /etc/resolv.conf

Any example of mounting the configmap as a volume will do the first step. I'd start like that.

Another hack you can try, if this is only for a POC and not will be used in the final solution, is using postStart hook that just overrides/appends to /etc/resolv.conf.

This las option might be more hackish but easier to just try.

[Simone D'Andreta]

Thanks a lot, I will try with the postStart first and then I will retry with the volumeMount. One question on the postStart though. The doc says that there is no guarantee the hook will execute before the entrypoint. Does it mean that if I have i.e. a nodejs app which has an entrypoint 'npm start' defined in the Dockerfile and the hook will execute at the end, my nodeapp won't start?

Cheers

Simone

[Rodrigo Campos]

Why wouldn't the node app start? I don't follow.

And if the node up doesn't start, Kubernetes will restart it.

The doc just says it is not guaranteed (like when mounting a volume) that the hook will be executed and finished before starting the container. It may or it may not happen, no guarantees. And this was a docker limitation some time ago, at least.